Experience first, metrics second: how to get customer experience right with AI

Organizations need to focus on building AI that not only meets customer expectations but exceeds them.

In brief

• Customers demand fast, seamless service and will switch to competitors if their needs aren't met.
• A successful strategy includes defining the desired experience and establishing metrics that reflect customer satisfaction.
• Artificial intelligence should be implemented only after aligning metrics with customer experience goals.

In today’s digital world, customers have more choices – and less patience – than ever. They expect fast, seamless and respectful service, and if they don’t get it, they’ll leave for a competitor. In fact , two-thirds of customers rate speed as equally important as price when dealing with a business, and half of customers won’t wait more than three minutes for help in a store^.1 It’s not just about price or product – it’s about reducing effort and frustration.

To meet these expectations, many organizations are turning to artificial intelligence (AI) as a game-changer for customer service. But AI alone is not a silver bullet. Without the right foundation, even advanced AI can fail to improve customer satisfaction or business outcomes. The key is to start with the experience you want to deliver, then define the metrics that reflect that experience – and only then bring in AI to help move those metrics.

Start with the experience

The foundation of any customer service strategy should be a clear vision of the experience you want to deliver. Is it a fast and effortless resolution? A personalized, empathetic interaction? A sense that the customer is being heard and respected?

Researching what is most important to your customers and what’s expected of your brand is the first step in delivering a lasting experience. The critical factor then becomes how you measure it. Without this clarity, even the most sophisticated AI tools can end up optimizing for the wrong outcomes.

Define the right metrics

Once the target experience is clear, the next step is to define metrics that accurately reflect whether that experience is being delivered. These should be tied to outcomes that matter to customers and to the business – not just internal efficiency.

Many traditional customer service metrics fall short. Net Promoter Score (NPS), First Contact Resolution (FCR) and Average Handle Time (AHT) are widely used, but they’re often gamed or misaligned with real outcomes:

• Net Promoter Score – A high NPS might make a team feel good, but if it doesn’t correlate with repeat purchases or retention, it’s not helping the business. A company may see rising NPS scores even as revenue and retention fall, signaling that NPS alone is an incomplete measure of customer loyalty. Front-line employees can also artificially inflate NPS by coaching customers to give a 9 or 10 on surveys. (Anyone who’s been asked “Will you please give me a 10 on the survey?” at the end of a service interaction has seen this tactic in play.) This same issue applies to other Likert scale questions like Customer Satisfaction (CSAT) and Customer Effort Score (CES).
• First Contact Resolution – Resolving issues on the first contact is great in theory. But if FCR is self-reported by agents without verification, it may hide repeat calls and unresolved problems. An agent might mark an issue “resolved” just to meet the target, while the customer ends up contacting support again later because the problem wasn’t really fixed. In effect, the company is managing to FCR statistics instead of managing to actual resolutions.
• Average Handle Time – Shorter calls or chats can reduce cost per contact, but overly focusing on handle time can cause agents (or bots) to rush customers off the line, leading to a poor experience. It optimizes efficiency at the expense of effectiveness. In fact, while AHT is meant to measure efficiency, it often discourages the best behavior. For example, if a customer has multiple issues, an agent’s AHT target incentivizes them to resolve one issue quickly and end the call, but the customer (and business) would be better served if the agent took the time to address all of the issues. In our experience, VIP and highly engaged customers often willingly spend more time on support calls or chats because of their loyalty and willingness to engage; cutting those interactions short would be counterproductive to the relationship.

AI systems trained solely on these traditional metrics might make customer service faster or cheaper, but not better. For instance, a chatbot could learn to keep interactions short to improve AHT, or to push for high survey ratings to boost NPS – all while failing to actually solve the customer’s underlying issue.

Avoid metric manipulation

Of course, even the best-designed metrics can backfire if they’re implemented poorly or incentivize the wrong behaviors.

In our work with clients, we’ve seen agents hang up quickly to lower their AHT, “selectively” send surveys only to happy customers to boost NPS and find other creative ways to hit targets. At one major retailer, employees even scanned QR codes on receipts themselves to submit fake customer satisfaction surveys. Another company had to create a dedicated team to detect and stop this kind of survey fraud.

This is a classic example of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” If employees feel they can’t control a metric – or that their job depends on an unrealistic target – they will find ways to game it.

To guard against this, companies should:

• Trust but verify: Use telemetry and customer behavior data to validate outcomes rather than relying solely on self-reported metrics. For example, tie your “resolved” status to actual product usage or the absence of repeat calls, not just an agent’s declaration that the issue is fixed.
• Close the loop: Implement a closed-loop resolution process (e.g., follow up with customers or automatically reopen cases if the customer contacts you again about the same issue) to ensure issues are truly fixed and stay fixed.
• Align incentives: Make sure the metrics you set are within employees’ control and directly tied to the customer experience you want to deliver. If you ask agents to provide a great experience, measure things that reflect customer success – and train and empower employees to achieve those metrics in the right way.

Then – and only then – bring in AI

Once you have experience-focused, credible metrics in place, you can bring in AI to help improve those outcomes. In other words, get the metrics right first, then let the machines help.

The most effective customer service AI deployments are targeted, not one-size-fits-all. Instead of trying to automate everything, identify specific high-volume or high-impact scenarios where AI can excel. For example, a well-trained bot might handle password resets, order tracking or simple troubleshooting steps much faster than a person, freeing up human agents to focus on more complex needs.

Customers don’t care whether a bot or a human solves their problem – they care that it’s solved. In fact, 68% of consumers5  say they’re happy to receive AI-driven assistance as long as it effectively addresses their issue. On the other hand, for complex or sensitive matters, 75% of customers 6 still prefer speaking with a human. This underscores the need for a hybrid model: use AI to handle routine, straightforward tasks and to assist agents (e.g., by suggesting answers or pulling up relevant information) while reserving human agents for nuanced or high-emotion interactions.

The goal isn’t to replace humans – it’s to empower them. When AI and human agents work together, customers get the best of both: speed and empathy, automation and understanding.

Experience first, metrics second, AI third

Improving customer service in the age of AI isn’t about choosing between humans and machines. It’s about using machines to enhance the experience – and using the right metrics to get to the experience customers want.

So before you deploy another bot or dashboard, ask:

• What experience do we want our customers to have?
• How will we know if we’re delivering it?
• Are our metrics aligned with that experience, and are they trustworthy?
• Where can we deploy AI to efficiently scale that experience?

If the answer to all of these is “yes,” then you’re ready to innovate boldly. If not, it’s time to pause and realign – starting with the customer experience at the core.

The views reflected in this article are the views of the author and do not necessarily reflect the views of the global EY organization or its member firms.