You change one sentence in the agent's instructions so it stops offering coupons to people who don't ask for them. It works. And, without anyone noticing, it also stops offering them to people your policy says are entitled to one. You find out two weeks later, from a complaint.
An AI agent doesn't break like a normal program. It doesn't throw an error: it keeps answering, well written, something else. So you can't test it by watching it for a while. You need a set of cases that runs again every time something changes. That's what evals are, and for an agent that acts on your systems they're designed differently than for a chatbot.
Why judging the reply isn't enough
For a chatbot, the output is the text: you check whether the reply is correct, complete and polite. For an agent, the text is the last thing it produces. Before it, the agent looked up systems, decided with your rules and sometimes changed something. A flawless reply on top of the wrong action is the worst possible outcome, because nothing in the text gives it away.
In an agent that acts, the reply is the least interesting part of what needs testing.
What a case contains
At Qopper a case is small on purpose. It has one customer message and four checks, all optional except the message:
- Which actions it must useLooking up the order, for example. They must appear, in any order, and other actions are allowed too: what matters is that it doesn't skip the lookup, not that it follows a script.
- What it must sayPieces of text that must appear in the reply, like the return window your policy states.
- What it must not sayWhat must never come out: a number that isn't in your policy, a promise you can't keep, a coupon that doesn't exist.
- How it must endThe conversation open, resolved or handed to a person. An out-of-policy case the agent “resolves” on its own is a failure even if the customer is delighted.
Why another model doesn't judge it
The usual approach is to ask a second model to score the first one's reply. It's convenient and has a basic problem: the judge is probabilistic too. The same case can pass today and fail tomorrow without the agent changing, and then you no longer know whether the agent broke or the judge did.
We judge each case with fixed rules: whether the expected actions appeared, whether the text includes what it must and leaves out what it mustn't, and whether the final status is the expected one. Text comparisons ignore case. It's cruder than a judge with taste, and we know it. In exchange, the same reply always gets the same verdict: if a case goes from green to red, what changed is what the agent did, not the yardstick.
A test that sometimes fails for no reason isn't a test. It's noise shaped like an alarm.
Testing actions without touching a customer
This is the part that doesn't exist when you evaluate a chatbot. If a case asks for a refund, the agent has to be able to decide to issue it, because that's what's being tested. And no customer can receive a test refund.
- 1Writes are validated, not executed. Cancelling, refunding or creating a return go through every check a real action goes through and stop right before calling the system. The agent gets back “validated, not executed”.
- 2Rules are evaluated the same way. If a refund reaches the cap you set, the test reflects that it would wait for approval. But no request is created and nobody on the team is notified.
- 3Lookups are real. Reading an order or a charge goes to your actual systems. That's deliberate: a case that looks up invented data only proves the agent reads invented data well.
- 4The draft is what's tested. Always the version you haven't published yet. The one serving your customers isn't touched.
Real lookups have a cost worth knowing: if the order a case uses changes status, the case can change result. So point lookup cases at data that doesn't move, like an old order that was already delivered.
The honesty check
There's a failure none of the checks above can see: the agent says “it's done” and did nothing. A model that has read thousands of support conversations knows that's what gets said at the end. So every case gets one extra check nobody has to write.
If the reply claims something already happened (“I've refunded you”, “I've cancelled your order”, and their Spanish equivalents) and no write action ran in that turn, the case fails. Promises in the future tense don't count: “I'll refund you now” is a different thing, and sometimes the right one.
In production the same idea goes further: a write only counts as verified if the system was read again after it and the change was there. Without that second read, the agent can't tell the customer it's done.
What matters is what changed
“38 of 40 cases passed” says little. What you need to know is whether the two failures are the same as last week or two you just broke. So every run is compared with the previous one and every case gets one of four labels:
- BrokeIt used to pass and now it doesn't. The first thing to look at.
- FixedIt used to fail and now it passes.
- SameNothing changed, pass or fail.
- NewIt wasn't in the previous run. It never counts as fixed, even if it passes: a case you just wrote to pass doesn't fix anything.
Publishing goes through the cases
If evals are optional, they get skipped in the week there's a rush, which is exactly the week things break. So publishing an agent first runs all its cases against the draft. If a single one fails, the draft isn't published and the agent serving your customers stays as it was. There's no button to publish anyway.
An agent with no cases can be published, and it's marked as untested. Blocking it would punish whoever is starting out; hiding it would be lying.
Where to start
An empty test suite is the number one reason nobody writes evals. To avoid starting from scratch, Qopper suggests four cases from your business description: a lookup, a request within your policy, one outside it and a knowledge question.
The split of work is deliberate. The model only writes the words: the customer's message and what the reply should and shouldn't say. The code decides the checks. Only the lookup case requires an action, the first read action you have enabled, and none requires a write: what can be done without asking is your decision, not something a model should invent.
What it doesn't do yet
Today each case is a single message: you can't yet test multi-turn conversations, where a good share of failures live. Each case runs once, with no repeats to catch results that change on their own. And text checks look for literal fragments, so a correct reply phrased differently can fail. Those are the next three steps, and we say so here because a test suite is only useful if you know what it doesn't cover.
How to write good cases
- Take them from real conversationsThe ones that went wrong are worth double. Every complaint is a case you don't have yet.
- One per edge of your policyJust under the cap, right at the cap, above it. Failures live at the edges, not in the middle.
- Test the noWhat the agent must not do matters as much as what it must. A case that asks for something forbidden and expects a handoff to a person.
- Test persistenceThe customer asking the same thing for the third time in different words. That's where you see whether the rules live in code or in a sentence in the instructions.
- Few and goodTwenty cases that cover your important decisions beat two hundred that ask for opening hours ten different ways.
An agent that acts on your systems is software, and untested software breaks in production. The difference is that this one doesn't throw an error when it breaks: a customer tells you.
