The Problem with "It Seems to Work"
Traditional software usually fails loudly. A function throws an error, a test goes red, and someone gets an alert. AI agents can fail quietly. An agent can return a confident, well-written answer that is wrong, call the right tool with the wrong input, or finish a task the long way at ten times the cost.
Because nothing crashes, these problems can run for weeks before anyone notices. Evals and observability are the two disciplines that close that gap.
• Evals test whether the agent does the right thing.
• Observability shows you what the agent actually did.
You need both. Evals without observability tell you something failed, but not why. Observability without evals gives you plenty of data, but no agreed definition of good.
Why Agents Are Harder to Test
The same input can produce different results
Models aren't fully deterministic, so a single passing run doesn't prove much. A test that passes once may fail on the next attempt.
There are many steps
An agent may plan, search, call several tools, and revise its work before it answers. A mistake in step two can look like a bad answer in step six.
Small changes can have wide effects
A new model version, a reworded prompt, or a changed tool description can improve one type of task and quietly break another.
Good answers aren't always easy to define
Checking a sum is easy. Judging whether a support reply was accurate, helpful, and on policy takes more work.
Building Your First Eval Set
You don't need a large framework to begin. Start small and use real work.
Collect 20 to 50 real cases
Pull them from actual tickets, documents, or conversations. Include the common cases, the awkward ones, and a few that should make the agent stop and ask for help.
Define what good looks like
For each case, write down what the right outcome is. That could be a correct answer, the right tool being called, or a correct decision to escalate to a person.
Use the simplest check that works
• Code checks suit anything exact, such as a valid format, the correct tool, or a required field.
• Model-graded checks use a second model with a clear rubric for judgment calls, such as tone or completeness.
• Human review is the reference point. Regularly compare a sample against your automated checks to confirm that they agree with your experts.
Run it before every change
Run the set whenever a prompt, model, tool, or data source changes, and run important cases several times to account for variation. Compare the results with the last version, and don't ship a change that breaks something important.
Grow it from real failures
Whenever something goes wrong in production, add it to the set. Over time, your eval set becomes a record of everything your agent has had to learn.
What to Measure
Accuracy alone won't tell you enough. For most agents, we would track:
• Task success: did it complete the job correctly from start to finish?
• Tool accuracy: did it choose the right tools and pass the right inputs?
• Escalation quality: did it hand off to a person when it should have, and only then?
• Cost per completed task: including retries and failed attempts
• Latency: how long a person or a process has to wait
• Business outcome: the measure the project exists to improve
Observability: Seeing Inside a Run
The core tool is the trace, a structured record of a single run. A useful trace captures each model call, tool call, memory read and write, and handoff between agents, along with the inputs, outputs, timing, and cost of each step.
With traces, you can answer questions that are otherwise guesswork:
• What information did the agent have when it made this decision?
• Which tool returned the bad data?
• Why did this task cost five times more than usual?
• Where is the time going?
Add tracing from the first run
It is much easier to build in early than to add later. Tools such as LangSmith and Langfuse are built for this, and OpenTelemetry offers an open standard for carrying traces across systems.
Protect what you record
Traces can contain customer data and other sensitive information. Decide what to mask, who can view traces, and how long to keep them.
Many Failures Are Context Failures
When agents fail, the model often isn't the real problem. It may have been given the wrong information, too much information, or too many tools to choose from. That is why context engineering, deciding exactly what enters the model's limited context at each step, has become such an important skill.
Traces make this visible, because you can see exactly what the model received at the moment it went wrong. The fix is frequently better retrieval or a shorter tool list, not a more expensive model.
Close the Loop
Evals and observability are most useful when they feed each other:
• Monitor production traces for failures, unusual cost, and user complaints.
• Investigate the trace to find the real cause.
• Add the case to your eval set.
• Fix the problem and run the evals to confirm the fix without breaking anything else.
• Release in stages and keep watching.
Each pass through this loop makes the agent more reliable. Agents don't get better simply by running in production. They get better because a team keeps measuring and improving them.
Where to Start
Pick one agent. Add tracing, collect 20 real cases, and agree what good looks like. You could have a basic version of this working quickly, and it will tell you more than any demo.
If you would like help setting up evals and observability for your agents, get in touch through our contact form.