Qonto Q3 QA recalibration

Final verdict

Qonto’s pre-cutover Q3 evaluations are sufficiently calibrated to the post-cutover evaluator. Stop broad re-evaluation here.

After the final replay wave, the weekly overall scores align with the immediate post-cutover range:

July 20 and July 27 now sit directly inside the August range. Their call scores match August 3–24, and July non-call scores match August 10–24. The lower September scores should be treated as a subsequent performance or population-mix trend, not evidence that more pre-cutover criteria require blanket replay.

Scope

• Customer: Qonto EU
• Rails organization ID: 33
• Scorecard: Stellar Support Scorecard, ID 57
• Original target: Qonto’s Tone of Voice, criterion ID 433
• Original criterion public ID: scorecard_criterion_GDjBm1KqvloXhEx9x7dJlkz2
• Pre-cutover period: July 1 through August 4, 2026, Europe/Paris
• Working cutover boundary: August 5, 2026 at 00:00 Europe/Paris
• Included population: completed, active, effective AI evaluations
• Initial prioritization excluded evaluations containing an auto-failed criterion because changing an ordinary criterion would not change their zero overall score
• Critical Errors remained out of scope pending confirmation from Qonto

Why this happened

The original signal appeared as a sharp score change around early August, most visibly on calls. Internal meeting notes showed that the QA model/provider migration was underway around the same boundary:

• A planned Bedrock-to-OpenAI move was discussed on August 6.
• Complexity-based QA model routing shipped around August 10.
• By August 13, evaluation workflows had switched from Bedrock to OpenAI.

The exact cutover timestamp was not documented, but the timing and criterion-level discontinuities strongly support a model-calibration change rather than a sudden improvement in agent performance.

Tone of Voice was real but did not explain the full bump. Once TOV was replayed, a weighted criterion/channel decomposition showed that Solve the Request on calls had the greatest expected impact per replay. It carried 30% of the scorecard and retained a roughly 15-point pass-rate gap against the stable post-cutover baseline.

Population

Prioritization findings

The residual-gap analysis separated aggregate chart impact from impact per replay:

1. Solve the Request, calls — best impact per replay. There were 1,027 eligible current passes from July 13 through August 2.
2. Solve the Request, non-calls — largest aggregate remaining population, but lower expected movement per replay.
3. Be Efficient for the Client, non-calls — meaningful mainly in the final two July weeks.
4. Be Efficient for the Client, calls — real but much smaller after the first replay batch.
5. Understand the Request — a visible pass-rate discontinuity but a smaller score contribution because the criterion is worth 10%.
6. Tone of Voice — small residual contribution after the completed TOV waves.

The observed reruns validated this ordering:

• Be Efficient call passes moved by −3.24 points in the selected cohort.
• Solve Request call passes moved by −16.14, −19.85, and −20.18 points across the three waves.

Replay log

All 1,027 eligible Solve Request call passes from July 13 through August 2 were replayed successfully.

TOV first-batch detail

The initial deliberately sensitive TOV batch contained 500 calls from July 27 through August 4 with no auto-failed result. It included 473 TOV passes and 27 partials, selected round-robin across day and agent strata and covering 189 agents.

• Overall score: 59.07 → 54.48
• 218 evaluations decreased, 277 were unchanged, and 5 increased
• Original passes: 214 stayed Pass, 230 became Partial, 28 became Fail, and 1 became Auto-fail
• Original partials: 17 stayed Partial, 9 became Pass, and 1 became Fail

Across the six initial call TOV runs, 1,769 non-auto-failed calls moved from 53.30 to 51.39. Including earlier runs, all 1,928 effective non-auto-failed calls from July 1 through August 4 were covered.

Decision and follow-ups

• Do not continue broad pre-cutover re-evaluation based solely on the lower September trend.
• Leave auto-failed calls deprioritized; changing ordinary criteria does not change their zero overall score.
• If a future discrepancy reappears, compare against the immediate August 3–24 post-cutover baseline before using later months.
• Preserve the run IDs above for auditability.

References

• ENG-3554 
• Qonto recalibration meeting notes 
• Source notebook