Skip to content
Cybrails

AI & Automation

How we test AI agents before they reach users

Evaluation sets, red-teaming and staged rollouts: the release checklist we use for agentic workflows.

Cybrails TeamBlog | 3 min readAI & Automation →
Mathematical formulas and diagrams on a blackboard

An AI agent that works in a demo can still fail on a Tuesday afternoon with real data and a real customer. Agents are harder to test than conventional software because they make decisions, call tools and produce different wording every time. This is the release checklist we use before an agentic workflow reaches users. It is a practical routine rather than a guarantee, and it keeps getting longer as we learn.

1. Define the job and the failure modes

Start by writing down what the agent is allowed to do and what a bad outcome looks like. Distinguish failures that are merely annoying, such as an awkward reply, from those that are costly, such as a wrong refund, a leaked record or an action taken twice. The list of costly failures decides how strict every later step needs to be.

2. Build an evaluation set from real cases

Collect real examples of the work, including the awkward ones, and record what a good result looks like for each. Cover normal cases, edge cases and cases where the right answer is to hand over to a person. Version the set, keep it separate from anything used to tune prompts, and score it with a mix of automatic checks and human review. Add every production failure to it so the same mistake is caught next time.

3. Test the tools and the permissions

Most serious agent failures involve actions, not words. List every tool the agent can call and give it the minimum access needed. Test each tool in a dry-run mode first, confirm that destructive actions need explicit approval, and check that the agent behaves sensibly when a tool fails, times out or returns something unexpected.

Treat every prompt, model or tool change like a code change: it does not ship until the evaluation set passes.

4. Red-team before users do

Try to break the agent on purpose. Test prompt injection through documents and web pages, requests that tell it to ignore its instructions, attempts to extract data it should not reveal, and inputs designed to push it into out-of-policy actions. Record what you find, fix it, and keep the cases as permanent tests.

5. Release in stages

Move through stages instead of switching on for everyone. Begin in shadow mode, where the agent proposes and people act. Then let it act with human approval, then with a small group of real users, and only then widen access. Keep a kill switch and agree in advance who can use it and what triggers it.

6. Watch it in production

Log each run with its inputs, the tools it called and the outcome. Track success rate, the share handed to people, latency and cost per task, and review a sample of conversations every week. Production behavior will differ from the test set, and the review is how the test set improves.

None of these steps is complicated on its own. The discipline is in doing them every time, including for small changes.

Want to apply this to your own project?

Talk to our AI & Automation team about where to start.