An agent evaluation flywheel can expose the path an AI agent took to reach an answer. A final-answer score can hide wrong tools, needless retries, weak evidence, or a failure to adapt when new information appears. That creates a practical question: what should change next?

This post turns that diagnosis into a five-stage operating model. It captures evidence from real runs, turns repeated failures into testable hypotheses, and improves prompts carefully. Final-answer checks remain useful, but they are no longer the only signal.

Key takeaways

• Final-output quality and trajectory quality answer different questions about an agent.
• A five-stage flywheel can turn evaluation records into concrete prompt-improvement hypotheses.
• Prompt candidates should be tested on held-out and regression cases, not only on the examples that inspired them.
• Human judgement remains necessary for rubrics, safety boundaries, and ambiguous traces.

Why a final answer is not enough to improve an agent

A final-answer evaluation tells you whether a task appears to have been completed. It remains valuable. A support agent that gives an incorrect policy answer has failed, regardless of how elegant its tool sequence looked.

However, the final answer cannot reliably explain why the agent failed. Nor can it show whether a successful run was needlessly expensive or fragile. In Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents, the authors note that answer matching can overlook efficiency, hallucination, and adaptivity.

For example, an agent may get a correct answer only after five unnecessary tool calls. A final-answer score says that run passed. A trace-aware evaluation records a pass with redundant work, which can shape an intervention.

The flywheel below is a proposed framework, not a universal standard. Its job is to make the route from an observed failure to a measured change explicit.

Stage 1: Capture the run, not just the answer

Start by recording enough context to reconstruct a run. The minimum useful record includes the task, relevant inputs, tool calls and results, available intermediate decisions, final output, latency, cost, and an outcome score.

This is not a recommendation for a particular tracing vendor. Instead, it is a design requirement. If evidence disappears after the final response, a later reviewer cannot distinguish an incorrect answer from a poor process that happened to reach a correct one.

Keep the record proportionate to risk. A low-stakes internal helper may need the task, answer, and tool sequence. A customer-facing agent may also need retrieval provenance, policy checks, escalation decisions, and a human review path.

Stage 2: Evaluate outcomes and trajectories separately

Use one set of checks for task success and another for the path taken. Outcome checks might ask whether the response answered the request, used the correct facts, and followed the required format. Trajectory checks should fit the agent’s role.

For a tool-using agent, useful questions include:

• Did it make a claim unsupported by retrieved evidence?
• Did it select the appropriate tool and use its result?
• Did it repeat a call after already obtaining sufficient information?
• Did it adapt when a tool result contradicted its initial assumption?
• Did it take an action that should have required escalation or a safety check?

Consider a fictional internal support agent asked to confirm a subscription status. Two runs return the same correct answer. The first reads the relevant account record once and cites the field it used. By contrast, the second searches unrelated accounts, retries the same lookup twice, then reaches the same record. Final-output scoring rates both as successful, but a trajectory rubric reveals a different reliability and cost profile.

That is diagnosis, not a claim that every extra call is a failure. Some tasks genuinely require exploration. The rubric must explain what counts as sufficient evidence for this agent and task.

Stage 3: Classify failures in the agent evaluation flywheel

Individual traces are anecdotes. A taxonomy turns them into categories that can be counted, reviewed, and acted upon. Avoid a catch-all label such as “bad reasoning.” Instead, use categories tied to observable behaviour: unsupported claim, inappropriate tool choice, redundant retrieval, missed stopping condition, or missing escalation.

Next, separate a symptom from a likely intervention point. “Repeated retrieval” is a symptom. A clearer tool-selection instruction, a sufficiency check, a retrieval rule, or a stopping condition could be the intervention. This separation prevents the team from treating every observed pattern as a prompt problem.

Calibration also matters. Reviewers should label a small shared sample, compare differences, and refine definitions before the labels feed an optimisation loop. As a result, the agent evaluation flywheel can use consistent evidence instead of vague labels.

Stage 4: Build agent evaluation flywheel candidates

For each recurring category, create a small evidence packet: representative test cases, the applicable rubric, the observed pattern, and one concise improvement hypothesis. For example: “When the first account lookup contains the required status field, instruct the agent to cite it and stop rather than issuing another lookup.”

Then split cases before changing the prompt. Development cases help compare candidate prompts. Hold-out cases test whether a promising change generalises. Regression cases preserve behaviours that must not break. This partition is a practical defence against selecting a prompt that merely memorises the examples that produced it.

Trace evaluation takes judgement and time. Therefore, start with failure modes that have the greatest consequence for users or operations. The goal is an evidence set capable of choosing between interventions, not a perfect model of every agent action.

Stage 5: Evolve the agent evaluation flywheel

Evolutionary prompt engineering makes prompt iteration systematic. Keep a population of prompt candidates, generate variations, evaluate them against the evidence set, retain stronger candidates, and repeat. EvoPrompt describes a related research approach that uses evolutionary operators to create prompt variants and improves a population based on a development set.

The important word is balanced. Select candidates using outcome checks and trajectory criteria together. A candidate that raises an answer score by suppressing an escalation, increasing unsupported assertions, or shifting work into unobserved steps is not a genuine improvement.

After selection, run candidates on hold-out and regression cases. Check task success, trajectory labels, cost, latency, and applicable safety constraints. Only then should a team consider a deployment experiment. Subsequently, new failures flow back to Stage 1 as the next round of evidence.

An illustrative flywheel in action

Imagine the internal support agent repeatedly calling the account service after it has already received a complete status record. The outcome is often correct, but traces show extra latency and occasional confusion when later calls return stale data.

1. Capture: retain the request, account-service calls, returned fields, final response, and latency.
2. Evaluate: mark runs that make an additional lookup after the required status field was already present.
3. Classify: group those runs as “missed sufficiency check, ” rather than broadly calling them “bad tool use.”
4. Prepare candidates: add representative cases to the development set and write the hypothesis that the agent should assess whether the required field is already available before calling again.
5. Evolve and verify: generate prompt variants, select only those that preserve answer quality while reducing the labelled pattern, then test winners on hold-out and regression cases.

This example is illustrative. It does not predict a particular percentage improvement or prove that a prompt change will work in every environment. Its value is the chain of evidence: a specific observed pattern produces a specific, testable intervention.

Guardrails that keep the flywheel honest

A trace is evidence, not a complete account of an agent’s internal reasoning. It may omit context, and a reviewer can misinterpret an intermediate step. Treat trace labels as fallible observations that need clear definitions and periodic calibration.

Prompt optimisation can also overfit. A narrow development suite rewards candidates that look good on familiar cases. Hold-out and regression sets counter that risk, but they do not eliminate it. Test changes for safety, reliability, latency, cost, and generalisation before deployment.

Finally, do not turn a composite score into permission to ignore hard constraints. Safety rules, required escalation paths, and critical correctness checks should be gates. A candidate should not compensate for breaking one of them with marginal gains elsewhere.

The practical verdict

This approach suits teams that run agents often enough to see repeated patterns and can capture reliable traces. It is not the first project for a team that lacks basic task-success checks or cannot observe what its agent does between input and answer.

The safest useful first action is small: choose one high-impact failure mode, collect 20 to 30 representative runs, and write a rubric that distinguishes an outcome failure from a trajectory failure. That gives prompt iteration something better than intuition to learn from.

Sources: Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents; EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers.

Categories: AI Agents, Artificial Intelligence

Tags: AI Agents, LLM, System Architecture

The post Build an AI Agent Evaluation Flywheel That Improves Prompts appeared first on Alpesh Kumar.