Evaluating a Skill Like an Engineer: Building a Test Set and Measuring Quality

"It looked good in the demo" is not quality assurance, yet it is how most Now Assist skills get shipped. Generative output is variable by nature, so eyeballing a handful of outputs tells you almost nothing about how the skill behaves across the messy reality of production inputs. If you want skills you can actually trust in front of users, you need an evaluation discipline borrowed from how engineers test anything else: a test set, defined metrics, and regression checks. Here is how to build that without a data-science team.

Why spot-checking fails, specifically. A skill that produces a great answer for your clean example can fail badly on the ugly inputs real users supply: truncated descriptions, multiple issues in one request, jargon, missing context, adversarial phrasing. Because the model is probabilistic, the same input can even produce different outputs across runs. Spot-checking three nice examples gives you false confidence and zero coverage of the cases that actually break. You are testing the happy path and shipping the unhappy ones blind.

Build a representative test set. This is the foundation, and it is unglamorous manual work that pays for itself many times over. Assemble a set of real inputs that span the actual distribution: typical cases, but deliberately weighted toward the edges, short and vague inputs, multi-part requests, inputs with information the skill should not find, and the weird real examples your queue actually contains. For each, record what a correct, acceptable output looks like. Twenty to fifty well-chosen cases beats a thousand random ones. The goal is coverage of the ways the skill can be challenged, not volume.

Define what "good" means before you measure. Generative quality is multi-dimensional, so name the dimensions that matter for this skill. Common ones: factual correctness (is the content accurate), groundedness (is it actually supported by the retrieved or source data, or invented), completeness (did it address the whole request), format adherence (does it match the required output schema), and tone or safety where relevant. A skill that is fluent but ungrounded is worse than useless, so groundedness usually deserves the most weight. Score each test case against these dimensions rather than a vague thumbs up.

Score grounded versus hallucinated explicitly. The single most important measurement for a knowledge skill is whether each answer is supported by real source content or fabricated. Go through your test outputs and label each: grounded, partially grounded, or hallucinated. This one metric tells you more about production risk than any other, and it directly points you back to retrieval and grounding (T2) when the numbers are bad. Track it as a percentage and set a bar you will not ship below.

Regression-test after every change. Here is the discipline that separates a maintained skill from a fragile one. Every time you change the prompt, swap the model, or, critically, change the underlying knowledge base, re-run the full test set. A knowledge-base edit you thought was harmless can quietly degrade answers across many cases, and you will only catch it if you re-measure. Treat the test set as a regression suite that runs on every change, exactly as you would treat unit tests for code.

Automate as far as you reasonably can. You will not manually grade fifty cases on every change for long. Automate what is automatable: format and schema adherence can be checked programmatically, and you can use a model-based evaluation to do a first-pass scoring of correctness and groundedness against your reference answers, with humans spot-reviewing the borderline cases. Even partial automation turns evaluation from a special event into a routine step.

Read the results honestly. The point of measuring is to act on bad numbers, not to manufacture a green dashboard. If groundedness is at 80%, that means one in five answers is unsupported, and you need to decide whether that is shippable for this use case (for a low-stakes draft, maybe; for an autonomous resolution, absolutely not). Let the evaluation set the autonomy level: a skill earns more trust and fewer gates as its measured quality climbs, not as your confidence in it grows.

Evaluation is the difference between "we built an AI feature" and "we operate an AI feature." The first is a demo. The second is engineering.