Sandbox Evaluation Accuracy Report

Date: 2026-04-05 Case: case1 (Qonto FR — commercial dispute / chargeback, ticket #4948053) Scorecard: Stellar Support Scorecard Agents evaluated: Ahmed K M., Hind R M. Effort levels tested: low, medium

Overview

We ran the Modal evaluation sandbox against a human-reviewed QA evaluation to measure check-level agreement. Two runs were completed — one at effort="low" and one at effort="medium" — to test whether deeper reasoning improved alignment with human reviewers.

Top-line results

Metric	Low effort	Medium effort
Checks compared	22	22
Result match (exact)	54.5%	45.5%
Justification match	50.0%	59.1%
Coverage vs human	100%	100%
Cost	~$0.30	$1.23
Duration	~6 min	~14 min

passed and not_applicable are treated as equivalent for result matching.

Medium effort did not improve result-level accuracy. It improved justification selection but made different (not better) interpretive choices on the core pass/fail decisions.

Disagreement breakdown (medium effort run)

8 check-level disagreements out of 22 total checks:

AI too lenient on tone/expression (3 checks)

Agent	Check	Human	AI	Why they diverged
Hind R M.	Tone of Voice — empathy	partial	passed	Human flagged “ravie de prendre en charge votre demande” as tone-deaf for a second agent picking up a dispute. AI saw standard greeting as acceptable.
Ahmed K M.	Tone of Voice — empathy	partial	passed	Human flagged “je suis ravi de vous accompagner” as out of tune with urgency. AI read it as professional empathy.
Ahmed K M.	Tone of Voice — expression	failed	passed	Human flagged “Avec Qonto, vous pouvez facilement contester…” as textbook-sounding with passive phrasing throughout. AI judged overall tone as warm and professional.

Root cause: The AI does not pick up on French-language tone nuance — specifically the mismatch between formulaic cheerfulness and the customer’s distress about an unrecognized transaction. The Qonto Tone of Voice KB document is referenced in the scorecard but the AI may not be reading or applying it with enough specificity.

AI too strict on context carryover (3 checks)

Agent	Check	Human	AI	Why they diverged
Hind R M.	Understand the Request	passed	partial	AI penalized Hind for repeating “contact the merchant” when customer already said they couldn’t. Human passed — Hind added new info (transaction can’t be cancelled).
Hind R M.	Solve the Request	passed	partial	AI flagged that Hind didn’t address customer’s inability to contact merchant. Human credited her for adding the cancellation info as new value.
Ahmed K M.	Solve the Request	passed	partial	AI penalized Ahmed for not responding to customer’s follow-up messages (16:37–16:39). Human passed — likely considered Ahmed’s shift had ended.

Root cause: The system prompt says “use full conversation context” and penalize agents for failing to leverage prior context. But the human reviewer evaluates each agent’s contribution more charitably — if the agent added value relative to before, that’s a pass even if they didn’t address every subsequent message. The AI applies the “full context” rule more literally, creating stricter attribution.

AI over-applies escalation exemption (1 check)

Agent	Check	Human	AI	Why they diverged
Ahmed K M.	Create a Human Bond	failed	n/a	Human failed Ahmed for overly long, poorly digestible solution. AI marked n/a because the ticket was later escalated, citing “check is not applicable in case of reassignment or escalation.”

Root cause: The check’s additional_instructions say “not applicable in case of reassignment or escalation.” The AI interpreted the later AM→PS escalation as exempting Ahmed entirely. The human reviewer understood this exemption is for when the agent initiates an escalation before providing a solution — not when the solution was already delivered and someone else later re-routes the ticket.

Structural observations

System prompt is calibration-heavy but ambiguous in application

Contains 62 “do not penalize” directives, 15 “auto-pass” rules, 12 “automatically score” overrides.
These rules are necessary (they encode real human calibration) but the AI applies them inconsistently — sometimes too broadly (escalation exemption), sometimes not at all (tone nuance).
Bumping effort from low to medium did not fix this; it gave the AI more confidence in its existing (mis)interpretations.

Scorecard XML is hard to parse reliably

Check descriptions are HTML-escaped with nested <ul>/<li> structures.
Cross-criterion dependencies exist (e.g., “Solve the Request is fail, partial” as a justification option in Be Efficient).
The AI needs to cross-reference KB documents by @mention references embedded in HTML.

Human gold set has gaps

14 of 22 human check summaries are “No summary provided” — making it hard to audit why the human chose a particular result for many checks.
This asymmetry means some disagreements might reflect human under-documentation rather than AI error.

Recommendations

Tone evaluation needs KB grounding. The AI should be required to read and cite the Qonto Tone of Voice document before scoring tone checks. Currently it appears to skip it. Adding a scorecard-level instruction like “You MUST read the @Qonto Tone of Voice KB doc before evaluating any Expression criterion” would help.
Clarify “full context” vs “agent contribution” in multi-agent tickets. The system prompt’s instruction to “use full conversation context” conflicts with the human practice of evaluating each agent’s marginal contribution. A rule like “evaluate each agent based on the information available when they were active, not messages that arrived after their handling ended” would align AI closer to human behavior.
Tighten the escalation exemption rule. The “not applicable in case of reassignment or escalation” instruction needs a qualifier: it should only apply when the agent being evaluated escalated before delivering a solution, not when a later agent re-routes the ticket.
Expand the eval set. One case is insufficient to draw conclusions. Running 5–10 cases across different ticket types and languages would reveal whether these patterns are systematic or case-specific.
Effort level: revert to low for now. Medium effort costs 4x more and takes 2x longer without measurably improving accuracy. The gains need to come from prompt/scorecard changes first.