Evaluating an agent isn't like evaluating a classifier. You can't compute an F1 score and call it a day. Agents take multi-step actions, use tools, maintain state, and produce natural-language outputs that are inherently subjective to evaluate. The interview tests whether you understand this complexity and have practical strategies for dealing with it.

The Three Layers of Agent Evaluation

Layer 1: Final Output Evaluation

Did the agent produce the correct final answer? This is the most basic layer and where most candidates stop.

  • Exact match: For agents with deterministic outputs (data lookups, calculations), compare the output to a gold-standard dataset.
  • LLM-as-a-Judge: For open-ended outputs (summaries, recommendations), use a separate LLM to evaluate the output against criteria: accuracy, completeness, relevance. Use structured rubrics with numerical scores (1-5) and always calibrate against human judgments.
  • Limitation: Final output evaluation tells you what the agent got wrong, but not why. A correct final answer doesn't mean the agent took a good path to get there.

Layer 2: Trajectory Evaluation

Did the agent take the right steps to reach the answer? This is the differentiator in interviews.

  • Step accuracy: For each step in the agent's execution, compare the action taken (tool call, reasoning) to the expected action. Did it call the right tool with the right parameters?
  • Path efficiency: Did the agent reach the answer in the minimum number of steps? An agent that calls 8 tools to answer a question that requires 3 is inefficient, even if the final answer is correct.
  • Error recovery: When the agent encounters a tool failure or ambiguous input, does it recover gracefully or does it spiral into hallucination loops?

Layer 3: Operational Metrics

Is the agent viable in production? This layer is about cost and reliability.

  • Cost-per-task: Total tokens consumed (input + output) × cost per token. An agent that costs $0.50 per query isn't viable for a chatbot that handles 100K queries/day.
  • Latency: Time-to-first-token and total completion time. Users tolerate ~2s for chatbots, ~30s for complex research tasks.
  • Reliability rate: What percentage of tasks does the agent complete successfully without human intervention? Track this over time to catch regressions.

Common Interview Questions

Q1: "How would you set up regression testing for an agent?"

Strong answer: "I'd maintain a golden dataset of 50-100 representative tasks with expected outputs and expected trajectories. On every code or prompt change, I'd run the agent against this dataset and check: (1) Does the final output still match? (2) Is the trajectory still efficient? (3) Did cost-per-task change? I'd set up CI alerts if accuracy drops below a threshold (e.g., 95%) or cost increases by more than 10%. The key insight is that prompt changes are code changes — they need the same regression testing discipline."

Q2: "Your agent's accuracy is 85%. How do you improve it?"

Strong answer: "I wouldn't try to improve the overall number blindly. I'd first segment failures by type: (1) Wrong tool selection — fix schema descriptions or add tool routing. (2) Correct tool, wrong parameters — add validation and few-shot examples. (3) Correct execution, wrong final answer — improve the synthesis prompt or add a self-verification step. (4) Tool failures (API timeouts) — add retries and fallbacks. Each category has a different fix, and investing effort in the largest failure category gives the highest ROI."

Q3: "How do you evaluate an agent that writes and executes code?"

Strong answer: "I'd use a combination of: (1) Unit test pass rate — give the agent a problem with test cases and measure how many pass. (2) Functional correctness — beyond unit tests, does the code handle edge cases? I'd maintain a separate set of hidden test cases. (3) Code quality — use a linter score and check for common antipatterns (hardcoded values, missing error handling). (4) Attempt efficiency — how many iterations did the agent need before all tests passed? Companies like Google increasingly test this in their AI engineering rounds."

The "LLM-as-a-Judge" Deep Dive

This comes up in almost every agentic AI interview. Key points to hit:

  • Use a different model as the judge. If your agent uses GPT-4o, judge with Claude or Gemini to avoid self-serving bias.
  • Structured rubrics > open-ended scoring. Instead of "Rate this output 1-10", break it down: "Accuracy (1-5): Does the output contain factual errors? Completeness (1-5): Does it address all parts of the query? Relevance (1-5): Is it on-topic?"
  • Calibrate against humans. Run 50 examples through both the LLM judge and human evaluators. If agreement (Cohen's kappa) is below 0.7, your rubric needs refinement.
  • Position bias. LLM judges tend to prefer the first option presented. Randomize ordering in pairwise comparisons to mitigate this.

Building an Eval Harness: The Minimum Viable Setup

Interviewers love asking "how would you build this?" Show that you've actually done it:

  1. Dataset: 50-100 tasks with expected outputs. Store as JSONL. Include diverse difficulty levels and edge cases.
  2. Runner: Script that feeds each task to the agent, captures the full trajectory (every LLM call, every tool call, every intermediate state), and stores results.
  3. Evaluator: Separate script that scores results against expected outputs using exact match + LLM-as-a-Judge. Outputs a report with per-task scores and aggregate metrics.
  4. CI Integration: Run on every PR that touches agent code or prompts. Fail the build if accuracy drops or cost increases.

For the foundational evaluation concepts (precision, recall, F1) that underpin agent evaluation, review our Machine Learning question bank.

Continue the Agentic AI Series: