Evaluation strategy for field teams
The Microsoft Employee Self-Service (ESS) agent had just become publicly available. ESS is an employee experience agent that helps people lookup HR policies and get IT support. Enterprise customers were ready to deploy but they needed to feel confident that their custom LLM experience would actually help employees while also protecting the business from risks. A generative AI agent handling sensitive HR and IT questions for 100,000 employees needs to meet a high standard before it goes live.
I'd been leading evals for the product team and I knew the same thinking could be repackaged for the field teams. I captured my experience in a digestible and sharable format for teams who work directly with customers who are responsible for deploying their company's version of ESS. This guidance is available to all Copilot Studio customers and is a foundation of an ESS development kit that helps create test sets automatically.
Meeting the moment
Most people think LLM evaluations means checking if the system works. Did it respond? Check the box, move on. That mental model made sense for quality assurance testing for traditional software. Write a test flow, run it, go to the next screen. Pass or fail.
But now, every user input is different. Every response is generated at run-time. You're not building a one-to-one experience, you're building a one-to-many experience that has to hold up across infinite variations of human needs, human language, and human context. That's a fundamentally different quality problem. And most of the enterprise customers responsible for deploying this technology weren't sure where to start.
Helping customer-facing teams understand this concept was the first lesson because it's the foundation of what makes evaluations different from quality assurance. Evaluations measure the kind of response you are likely to get most of the time.
Bridging the gap
These customers and field teams were experienced professionals who knew how to talk about and deploy enterprise software. What they needed was a digestible framework for thinking about quality in a probabilistic LLM system.
Their instinct was to treat evals like traditional QA, which is problematic for generative AI. QA doesn't measure whether a specific employee, in a specific situation, actually got help. And it doesn't help us understand if the LLM experience is better than the traditional way of completing the same task, which is crucial for widespread adoption and ROI for our customers.
The second lesson was to establish a mental model and way of talking about quality. "Quality" is too vague to act on. This diagram breaks the concept of response quality into three distinct dimensions — technical performance, UI and interaction design, and output quality — so field teams can facilitate more precise conversations with customers about what's actually falling short.
This LLM response example highlights how the output quality dimensions each play a role in creating the ideal response.
Frameworks are only useful if teams can recognize them in the real world. This annotation is focused on output quality and is an example of the training content I created that shows exactly what each quality dimension looks like inside a real agent response. Field teams who can read a response this way can coach customers on what to prioritize in their eval criteria and how to fix common response quality issues.
The framework that makes it managable
The hardest thing to teach isn't the new technology, but the system thinking and human-centered focus needed to write test sets that will highlight and confirm which kinds of responses are testing ok. There's no single right answer to test against.
The same question asked ten different ways should produce ten semantically consistent responses, but they won't be identical. There are thousands of possible queries to test and not enough time to test all of them. This framework gives field teams a fast, principled way to prioritize so they can focus on protecting what matters most (accuracy on high-stakes questions, hard guardrails on sensitive topics) before investing in the rest. It turns an overwhelming problem into a manageable one.
The framework I use to make that navigable: must be right, nice to have, and must not answer.
-
Must be right covers your highest-value, highest-risk scenarios, the ones where failure causes real harm or erodes trust.
-
Nice to have improves the experience but if the answer is slightly incomplete, the risk of causing harm to the user or company is low.
-
Must not answer defines the guardrails: safety, compliance, and out-of-scope topics the agent should never attempt to answer.
Understanding evaluation aren't a one-time checkbox is the third lesson — evals is a practice that runs throughout the entire customer engagement and the entire life of the agent. This diagram maps each evaluation mode to an engagement stage field teams already work in, so evaluation feels integrated rather than bolted on. It also gives teams clear entry and exit criteria for each phase.
Not all evals use the same kind of test is the fourth lesson. Some tests check for precise outputs like a specific policy number, a date, a link. Others need to evaluate whether the response captures the right meaning, the right tone, the right level of care for someone in a stressful situation.
I built a spectrum of five test types from general behavioral evaluation all the way to exact match so field teams could choose the right tool for each scenario. Knowing which to use is a judgment call that takes practice. Teaching people to develop that judgment is part of the curriculum.
The fifth lesson is showing that the evaluation process isn't a solo exercise. Deploying an employee experience agent is a team effort spread across the organization. The platform team runs the tests, domain owners confirm the answers are actually correct, legal and compliance teams define where the guardrails need to be, and the experience owners decide what "good" looks like in the first place.
Field teams who understand these roles can structure customer conversations around accountability from day one, which is the difference between an evaluation program that sticks and one that gets abandoned after the first sprint.
Semantic vs exact match
After working with the customer-facing folks for about a month, I noticed two concepts needed to most reframing:
1. The first was mentioned earlier in the first lesson and is the sheer number of variables at play in a generative system. Teaching people to evaluate for meaning rather than matching is a genuine conceptual shift.
2. The second hard concept is the difference between semantic evaluation and exact match evaluation. For employee experiences, especially for HR scenarios, there is a relationship between evaluation method and risk.
The judgment call of which tool to use isn't purely technical. It's shaped by how much is at stake if the answer is wrong. A response about vacation balance that's slightly off-tone is a bad experience. A response about FMLA eligibility that's factually incomplete could have legal consequences. Those aren't the same kind of wrong, and they don't get tested the same way.
Like we learned in lesson 4, different queries require fundamentally different evaluation approaches. This diagram helps field teams quickly categorize customer queries and match them to the right evaluation method before writing a single test.
Getting this calibration right requires the people who understand where the risk actually lives as mentioned in the fifth lesson — legal, compliance, HR, benefits, IT. Every business draws that line differently. That's exactly why evaluation strategy can't be built in a silo, and why getting the right stakeholders in the room early is part of the work, not an afterthought.
Putting learning in action
This curriculum was presented to 100 field reps over several workshop sessions over a 6 week period. It was also repurposed into this public-facing guidance which is available to all Copilot Studio customers.
One customer success lead said the curriculum made her "feel like she could actually do this. " Customer adoption of eval tools increased measurably after internal teams saw a live walkthrough of this framework applied to their specific scenarios. The work also surfaced concrete gaps in Copilot Studio's native eval tooling that the platform team is now using to improve the product.
The frameworks are proving durable enough to scale too. The same logic that structures the curriculum is now informing an agentic development kit designed to help customers automate the early decisions involved in building test sets. The goal was always to make evaluation approachable. Tooling that encodes these decisions is the next step in that direction.
What this project is really about
I don't think like an engineer and I don't think like a classic visual designer. I sit somewhere in between — close enough to the technology to understand its behavior, close enough to the human experience to know what it needs to deliver. That position is exactly what this moment in AI development requires.
I genuinely love helping people understand how this technology works and why it matters. Both the engineering side and the human side of it.