If you cannot say whether an agent got the answer right, you do not have a system. You have a demo.
Collect cases, not opinions
Sit with the people who do the work and take fifty real examples out of their queue. Messy ones. The ambiguous refund. The email in two languages. The ticket where the customer is wrong about their own account.
For each one, record the input, the correct outcome, and why it is correct. That last column is the valuable one: it captures the business rule that nobody wrote down.
Grade on outcomes, not wording
Scoring an agent on whether its text matches a reference answer produces a system that writes nicely and decides badly. Grade the decision instead: did it route to the right team, apply the right policy, cite the right document, refuse when it should have refused.
We keep roughly a fifth of the set as cases the agent is expected to decline. An agent that never escalates is not confident, it is unsupervised.
Make the set outlive the model
A test set written this way is portable. When a new model appears, we run the set and get a number the same afternoon. That is what makes model choice a measurement rather than a debate, and it is why we never let an agent’s behaviour live only inside a prompt.
Run it on every change. A test set you run once is documentation. A test set you run on every change is a safety net.