Failure-Focused LLM-as-a-Judge Evaluation for a Transactional Conversational Agent: A 46-Case Repository Study

1. The failure cluster hidden by the headline

Transactional agents are harder to evaluate than single-turn question answering because correctness depends on state. A payment response can be linguistically fluent yet wrong because the cart total was lost; a handoff can sound empathetic yet fail to persist the transfer; a menu answer can be accurate for the catalog but irrelevant to the user's current step. Reference-string metrics are weak for these open-ended responses, while manual review is expensive and difficult to run after each change.

The audited repository implements a separate model judge, a dynamically generated battery, a stateful system-under-test runner, and Langfuse tracing [1,3–5]. A saved run from 8 March 2026 contains case-level outputs, five criterion scores, latency, token counts, and textual reasons in addition to the aggregate [2]. This report reconstructs that experiment and tests the validity of the conclusions one may draw from it.

• Describe the battery, execution path, judge rubric, and aggregation logic from source.
• Report the complete category-level result and analyze the six failures rather than presenting only the 87% headline.
• Audit whether the named test categories actually exercise the properties they claim to measure.
• Document provenance drift between the saved 46-case run and the larger battery in the recorded source revision.
• Propose a calibrated evaluation architecture that separates deterministic properties, open-ended quality, and human judgment.

2. LLM-based evaluation and known judge bias

MT-Bench and Chatbot Arena established LLM-as-a-Judge as a scalable way to approximate human preference on open-ended assistant responses, while explicitly documenting position, verbosity, self-enhancement, and reasoning biases [7]. G-Eval showed that a structured rubric and form-filling output can improve correlation with human evaluation for generation tasks, but also reported potential preference for model-generated text [8]. Prometheus 2 extends the evaluator-model approach to both direct scoring and pairwise ranking with user-defined criteria [9]. These works support using a judge as an instrument; none implies that an uncalibrated judge score is ground truth.

The audited harness uses direct assessment, not pairwise comparison. It supplies the conversation, final answer, selected catalog context, and category to one judge model. The run produces explanations and category summaries. Its main scientific weakness is meta-evaluation: there is no local study of agreement with human raters, repeatability across seeds, or sensitivity to prompt and model changes. It can screen for failures and generate hypotheses about their causes; it cannot establish ground truth.

3. System under test and evaluation architecture

The system under test is a restaurant assistant with catalog-backed answers, ordering, payment handling, persistent conversation state, human handoff, and resilience logic [1]. The test runner first obtains an authenticated catalog snapshot. It then generates cases using available product names and FAQ entries, creates a unique chat identifier per case, sends every user message in sequence to the /message endpoint, and accumulates the observed responses [3,5]. Only the final response is scored, while earlier turns are embedded in the judge's conversation context.

1. Fetch the live catalog using an admin token.
2. Generate a battery whose product references are grounded in that catalog.
3. Create a new conversation identifier for each test to reduce state contamination.
4. Execute all messages of a test sequentially against the system endpoint.
5. Construct context from user turns and non-final bot replies; pass the final reply separately.
6. Invoke the judge, apply the threshold, attach SUT and judge traces, and aggregate by category.

4. Battery design and task coverage

The saved artifact contains 46 cases across eleven categories [2]. Greeting, FAQ, menu, and ordering cases assess ordinary dialogue. Workflow cases carry a transaction across several turns. Payment cases probe exact payment, change, insufficient funds, method selection, and a large bill. Handoff cases ask for a supervisor or express frustration. Security and resilience cases use synthetic messages prefixed with [META:] for behaviors that the chatbot endpoint cannot directly observe.

Dynamic grounding avoids hard-coding a product that is absent from the current catalog. It also reduces repeatability: if product order, availability, prices, or FAQ text changes, the generated cases and judge context change. A publishable benchmark should serialize the resolved battery and catalog fixture before execution. Without that snapshot, the source generator is reproducible in form but not necessarily in content.

The recorded source revision adds two multi-order cases, two active-order edge cases, and an automatic-handoff case relative to the saved output, producing a larger conceptual battery than the 46-case artifact [3]. This is not a problem for ongoing engineering; it is a provenance problem for research reporting. The saved run must be cited as its own dated artifact rather than as a result of every later test definition present in the same repository tree.

5. Judge model, rubric, and pass rule

The judge uses gemma-3-27b-it at temperature 0.3 [4]. Its system prompt requests five 0–100 criteria: relevance, accuracy against supplied context, completeness, tone, and actionability. It adds category-specific reminders for payment, handoff, security, resilience, and meta tests. The model returns JSON containing an overall score, reasoning, and criteria. Values are clamped to [0,100]; malformed output becomes a zero-score failure.

Catalog context is filtered by category. Menu entries are included for order-related cases; selected FAQ entries are ranked by keyword overlap, with extra weight for payment topics [4]. This improves relevance and lowers prompt size, but it can also make the judge more informed than the system under test. Accuracy scores should therefore be interpreted as consistency with supplied catalog context, not independently verified real-world correctness.

6. Saved-run results

The saved console artifact is timestamped 2026-03-08T01:04:26.009Z [2]. It reports 40 passing cases out of 46, an 87% rounded pass rate, a mean overall score of 80, and 671.3 seconds of total duration. Judge usage is 28,010 tokens: 23,038 prompt and 4,972 completion. SUT token usage is recorded as unavailable. The artifact does not contain the run-level p95 later supported by the checked-in report type and generator [5].

The macro-average of the eleven displayed category means is 82.0, while the weighted average reconstructed from rounded category means is approximately 80.5, consistent with the reported case-level mean of 80 after rounding. The weighted mean of displayed category latencies is approximately 12.5 seconds. These derived values are diagnostic only because category means are rounded and the raw JSON report is not checked in.

7. Failure analysis

The payment failures share one pattern: a valid payment intent is routed through product or order-state logic that repeats a previous question. This is stronger evidence for a routing or state-priority defect than for a language-generation defect. Four failed responses leave the current turn unconsumed, which makes tone secondary. PAY-03 reaches 65 because it partially recognizes the constraint, suggesting the category contains more than one failure mode.

F1 is a separate intent-selection failure. The answer states that the restaurant is open but substitutes a menu for the actual hours. The judge assigns accuracy 100 in the stored trace because the listed products match catalog context, while relevance and completeness collapse. This illustrates why an overall score is not enough: a criterion can be locally high even when the response fails the user's task.

• Candidate cause A: payment intent is evaluated after an order-state branch that short-circuits the current message.
• Candidate cause B: payment handling requires fields not yet committed to persistent state.
• Candidate cause C: FAQ and payment routes use overlapping keywords but different precedence.
• Candidate cause D: the judge evaluates only the final turn, so the exact state transition causing the failure remains implicit.

8. Validity audit of security, resilience, and provenance

The security category includes meta descriptions for webhook signature validation, JWT authentication, and rate limiting. In the battery source, comments acknowledge that these properties belong at the HTTP layer [3]. The generic runner nevertheless sends the bracketed meta text to /message and asks the judge whether the conversational reply is safe. A polite fallback cannot establish that an unauthenticated admin request returns 401, that a webhook signature is rejected, or that a rate limiter returns 429. Only the XSS-like and SQL-like text cases exercise user-message handling, and even those do not prove database or browser safety without side-effect assertions.

The same problem affects resilience. A message saying that a circuit breaker or retry should be validated does not induce a provider failure, advance a failure counter, observe an open state, or measure backoff. The reported 5/5 resilience and 5/5 security results are scores for conversational handling of synthetic prompts. They must not be presented as security or fault-tolerance validation.

9. Partial local verification on 4 August 2026

The public repository was cloned, detached at revision 8467633188cd89e4708b8011ee2a0fb39b7f1a24, and installed with npm ci under Node 24.14.0 and npm 11.9.0. The Cuqui repository is not part of this report; all counts below refer to restaurantes. Running the full Vitest suite produced many passing files but did not terminate within the bounded observation window. A narrower command confirmed that all 36 assertions in test-battery.test.ts and all 43 assertions in conversation-assistant.test.ts reported pass. The process again remained alive after test completion and was interrupted.

git checkout --detach 8467633188cd89e4708b8011ee2a0fb39b7f1a24
npm ci
npx vitest run \
+  src/judge/test-battery.test.ts \
+  src/services/conversation-assistant.test.ts

10. Corrected evaluation protocol

A defensible rerun should split the battery by observable property. Layer A contains deterministic integration tests for arithmetic, cart mutations, auth responses, webhook signatures, rate limiting, circuit-breaker state, retry timing, persistence, and handoff side effects. Layer B contains model-judge cases for relevance, completeness, tone, and natural multi-turn behavior. Layer C contains blinded human ratings used to calibrate Layer B and review disagreements.

1. Freeze the catalog fixture and resolved test battery as versioned JSON.
2. Pin SUT revision, judge model, prompt, temperature, dependencies, deployment, and environment.
3. Run deterministic tests once per revision and fix teardown until the command exits zero cleanly.
4. Run the model-judge layer at least three times; report mean, range, and label stability per case.
5. Blind at least two human raters to the automatic score on a stratified subset.
6. Report agreement, adjudicate disagreements, and choose thresholds from calibration rather than convention.
7. Publish raw redacted outputs, test definitions, configuration, and failure taxonomy.

11. Threats to validity and disclosure

• Construct validity: an LLM score approximates conversational quality but cannot observe hidden state or infrastructure controls from text alone.
• Internal validity: catalog changes, provider drift, temperature 0.3, and shared services can change responses across runs.
• External validity: one Spanish-language restaurant fixture does not generalize to other transactional domains or risk levels.
• Conclusion validity: one run provides no estimate of judge variance and no confidence interval for category performance.
• Provenance validity: saved output, sprint summary, and current battery represent different points in an evolving repository.

AI systems contributed to the application code, judge outputs, and editorial preparation of this report. Pablo Cardozo is responsible for the source selection, interpretation, claim boundaries, and corrections. The repository output is quoted through aggregate numbers and short failure descriptions; no client or private user conversation is used.

12. Conclusion

The 46-case artifact is useful because it turns a vague impression of chatbot quality into a concrete failure map. Its most defensible result is not that the assistant is 87% good; it is that five payment cases fail in a coherent way while most ordinary ordering and handoff cases receive high judge scores. That pattern justifies targeted investigation of intent routing and transaction state.

The audit applies the same standard to the harness itself. Meta prompts cannot validate security controls, a moving battery cannot be silently attached to an older run, and an uncalibrated judge cannot substitute for humans. Deterministic control tests, frozen fixtures, repeated judge runs, and blinded calibration would turn the existing harness into a regression instrument whose limits are known.