Microsoft explains how it tests Copilot with fictional workplaces
Microsoft described an internal Copilot evaluation system on October 30. Its Industry Evaluation Data Generation Platform builds fictional workplaces using public occupational, job-posting and company-filing information. Three environments represent pharmaceutical, capital-markets and software organizations. Microsoft says a composite measure of synthetic-data quality improved 39% against its initial generation approach. That is not a 39% gain in Copilot answer accuracy. The environments are shared through an internal catalog, not announced as a downloadable product. Microsoft says this pipeline uses no customer data. These are vendor-reported results, not independently reproduced findings. Read the Microsoft Research announcement.
Why a believable workplace is not the same as a correct answer
A test collection can look convincing while leaving the important question unanswered: did the assistant find the right evidence and interpret it correctly? A polished email chain is useful test material, but its realism does not establish that a response follows the latest decision. Teams assessing assistants should keep the quality of their test material separate from the quality of the answers produced against it.
Microsoft’s existing Agent Evaluations CLI documentation makes useful distinctions. Retrieval checks examine queries or returned resources. Groundedness checks whether an answer is supported by its context. Relevance asks whether it addresses the question. The citation evaluator counts references. Relevance and coherence are enabled by default; groundedness is not. This documentation is background, not a new release attached to today’s announcement.
Our takeaway: a citation count is not a substitute for checking the cited passage. An answer can include a reference yet point to an outdated proposal. The example below shows how to make that distinction concrete without pretending to have run a benchmark.
A fictional deadline test you can inspect
This is an original, untested illustration, not Microsoft’s dataset or a reported Copilot result. Suppose a fictional project, Cedar, has three records. Monday’s email, C1, proposes delivery on Friday. Tuesday’s approved meeting decision, C2, moves delivery to the following Wednesday. Thursday’s draft memo, C3, mistakenly repeats Friday. A separate project, Cedar Mobile, has its own deadline and records.
For this exercise, explicitly designate C2 as the authoritative decision and C3 as an unapproved draft. Otherwise the evaluator would be guessing which document wins. Ask different questions against the same record set, and define the expected evidence before looking at the assistant’s response.
| Question | Expected evidence | What a failure would expose |
|---|---|---|
| What is Cedar’s agreed deadline now? | Following Wednesday, supported by C2 | Treating the newer draft C3 as the final decision |
| What delivery date had been proposed as of Monday? | Friday, supported by C1 and labeled a proposal | Leaking the later decision into a historical answer |
| What is the deadline for Cedar, not Cedar Mobile? | Cedar records only, including C2 | Mixing similarly named projects |
| Which passage supports the answer? | The actual decision in C2 | Adding a citation that does not support the claim |
How to make the example useful rather than decorative
Keep the document identifiers, question wording and expected answer together. Record whether the right document was retrieved separately from whether the final response used it correctly. If C2 never reached the assistant, changing the answer rubric alone will not tell you why retrieval missed it. If C2 was present but the response still chose Friday, inspect the interpretation of document authority and timing.
Then change one condition at a time. Remove C2 and the current agreed deadline becomes unknown from the remaining records; do not reward an invented Wednesday answer. Rename Cedar Mobile to make the names less similar and compare the specific retrieval behavior, rather than assuming any improvement proves general reliability. These are suggested checks, not findings from testing.
What readers can use today
The immediate value is a way to ask better evaluation questions, not a new tool to install. Start with a small reviewable fixture, an explicit authority rule and answers that include uncertainty when evidence is missing. A realistic fictional organization can create harder cases, but the pass conditions still need to be clear enough for a person to audit.
Featured image: AI-generated illustration by ByteForward, not a product screenshot. Microsoft logo: Used with permission from Microsoft. Microsoft is a trademark of the Microsoft group of companies.
