Qonto stale QA score aggregates — July 2026

Status

Customer-visible symptom understood and bounded.
The original July 16 repair covered 501 scorecard-result rows across 459 evaluations/tickets.
The July 13 customer extract contained 59 changed evaluation rows across 57 unique tickets.
A later July 20 extract audit found 29 additional stale scorecard aggregates. Those 29 were repaired in production on July 20 and verified.
The full-payload duplicate callback path was guarded by PR #8280 .
The incremental criterion-save path that remained open was guarded by PR #8372 and deployed on July 20.
Hourly read-only consistency monitoring was added by PR #8378 . It was merged but had not yet reached release-prod at the time this note was written.

This incident concerned denormalized/cached score totals. The saved check- and criterion-level results were the source of truth for the affected stale-total cases. Do not describe every historical score change as this incident: some old-export/current-state differences are legitimate later re-evaluations.

Customer symptom

Qonto reported that ticket 6260067 was scored 15 in the Monday, July 13 extract but later displayed 40 in Rulebase. There was no auto-fail and no intentional re-evaluation explaining that change.

For this ticket:

The July 13 XLSX headline score was 15.
The criterion/check results already summed to 40 in that original export.
The current stored score became 40 after aggregate recalculation.
The correction did not change the criterion decisions.
Braintrust showed two runs roughly 2 minutes 36 seconds apart, consistent with duplicate/concurrent evaluation execution, but the database state was sufficient to establish the stale aggregate independently.

The accurate product explanation is: the check-level results were retained, while the denormalized overall score had been calculated from an earlier/intermediate state and was later recalculated from those saved results. It was not an auto-fail or a reviewer changing the evaluation.

Data model and invariant

Relevant tables:

qa_agent_evaluation_scorecard_criterion_results: saved criterion results.
qa_agent_evaluation_scorecard_results.score: cached scorecard total.
qa_agent_evaluations.overall_score: cached evaluation total.
Conversation QA score columns: downstream cached conversation aggregates.

For completed scorecards, the core invariant is:

Select saved root criterion results.
Apply the scorecard’s scoring type, auto-fail behavior, and cap rules.
The resulting score must equal qa_agent_evaluation_scorecard_results.score.
The evaluation and conversation aggregates must then agree with their completed scorecard/evaluation children.

Do not implement this check as a naïve SQL SUM only. Production behavior includes weighted scoring, auto-fail options, and ScorecardCapRules::Evaluator.

Confirmed chronology

All times below are UTC unless otherwise noted.

July 13 export

Scheduled export completed at approximately 2026-07-13 07:00 (09:00 Europe/Paris).
It contained 1,142 rows for evaluations from July 7 through July 12.
Ticket 6260067 appeared with headline 15, although its saved criteria totaled 40.

July 16 first repair and guard

04:23:50–04:26:49: the first production repair recalculated 501 affected scorecard-result rows representing 459 evaluations/tickets. The immediate post-repair signature scan returned zero.
04:41:54: PR #8280, [ENG-1799] Guard completed QA imports and collapse empty timelines, merged as a60c8b14bd1d3358c3577efa7951aa0635812a43.
05:03: the commit reached release-prod via PR #8281.
Approximately 05:10: Rails API/worker revision 67ead718a72edbe1391703bee220dfa9fab40ea3 deployed with the guard.

PR #8280 changed QAAgentEvaluationScorecardResult#import_sandbox_agent_payload! to:

acquire the scorecard-result row lock with with_lock;
ignore a replacement full payload if the scorecard result was already completed;
prevent a duplicate full callback from deleting and replacing committed criterion rows after completion.

The regression spec covered that full-payload import method.

Why #8280 was necessary but insufficient

The production model had two persistence paths:

Full-payload import: import_sandbox_agent_payload!.
Incremental leaf persistence: save_intermediate_leaf_criterion_result!.

PR #8280 protected only the full-payload path. The incremental path could still save ordinary leaf payloads after completion and was not serialized on the scorecard-result row. Completion also counted all criterion rows instead of completed AI-evaluable leaf rows, allowing parent aggregates to help satisfy the threshold.

This explains why later records could exhibit the same stale-total signature even after #8280 deployed. It was not evidence that the row lock inside #8280 failed; another write/finalization path was outside that lock and guard.

July 17 separate always-pass re-evaluation batch

A separate, intentional re-evaluation batch followed PR #8321 , [ENG-1800] Support always-pass AI checks, which merged at 10:10:09 UTC on July 17.

99 pre-existing Information Sharing Guidelines check results across 99 unique tickets were updated between 10:46:09 and 10:53:23 UTC (06:46–06:53 America/New_York).
Every affected evaluation had an initial score of 0 because it had previously auto-failed.
The rule result stated: Automatic scoring rule: This check is configured to always be marked as Pass.
Ticket 6201025 was included and updated at 10:50:20 UTC.
The post-batch overall scores ranged from 0 to 70.
The enriched audit contained ticket ID, Zendesk agent name, evaluation-time team, initial score, new score, direct Rulebase URL, and re-evaluation timestamp, with no missing agent/team metadata.

This batch was not another instance of the stale aggregate defect. It deliberately re-evaluated previously auto-failed evaluations after the always-pass rule became supported. The observed database window was Friday morning, not Thursday around 08:00 America/New_York.

July 20 audit and repair

07:00:08: the scheduled July 20 export completed with 1,356 rows.
The export faithfully reproduced the stored scorecard totals for every row.
29 stored totals disagreed with the saved criteria/rules:
- 28 were stored/exported as 0 but derived to scores between 5 and 25.
- Ticket 6281712 was stored/exported as 35 but derived to 25.
- 28/29 had criterion timestamps later than the scorecard-total timestamp, matching the incomplete-finalization/stale-cache signature.
At approximately 12:13:56, the exact 29 rows were repaired in one guarded production transaction.
At approximately 12:14, a full read-only audit of all four captured exports returned zero current stored-total mismatches.

July 20 durable follow-ups

12:42:06: PR #8372, [ENG-1860] Guard incremental QA criterion imports, merged as e713f4c3c491d72011356684e7fff07f1bf27d1a.
13:29:36: #8372 reached release-prod via PR #8375.
Approximately 13:35: production Rails task definition 1403, commit 75b524b00411cdc2247bf8d2178d8ad7ee665635, was running with #8372.
#8372:
- serializes incremental leaf saves behind the scorecard-result row lock;
- ignores ordinary incremental payloads after completion;
- counts only completed AI-evaluable leaf results for completion;
- leaves the intentional feedback re-evaluation path intact because that path explicitly recalculates aggregates.
15:07:48: PR #8378, [ENG-1864] Add QA score consistency monitoring, merged as 42b51eca7a7da5bce9b6bff1b25f91988b255c31.
At note time, #8378 was merged to main but not yet present in the observed release-prod history.
#8378 adds an hourly, read-only audit of recently updated completed scorecards and evaluations, uses production cap/auto-fail calculations, reports mismatches/truncated scans to Sentry, and adds bounded-query indexes.

Blast radius

July 16 broad repair population

The 459 affected evaluation dates were:

Evaluation date	Tickets/evaluations
July 8	33
July 9	70
July 10	38
July 11	7
July 12	4
July 13	246
July 14	58
July 15	3
Total	459

This population was not limited to a single Monday extract. Qonto’s scheduled export cadence determined which stale totals became customer-visible in a particular file.

July 13 extract intersection

157 affected evaluation rows / 148 tickets from the broad repair intersected the captured July 13 export population.
59 export rows / 57 unique tickets had a changed headline score when compared with the corrected current state.
19 of those rows matched the strict “criteria already represented the corrected result; only headline total was stale” pattern.
Do not present all 89 July 13 export-vs-current differences from the later broad audit as this incident. That count also includes unrelated legitimate changes after the export.

The 57 ticket IDs supplied in the customer-facing CSV were:


6087297, 6099071, 6102551, 6140948, 6156548, 6157833, 6168679,
6169731, 6198557, 6205144, 6206866, 6223522, 6225116, 6228746,
6232732, 6242591, 6248400, 6256309, 6256566, 6256858, 6259294,
6260067, 6261127, 6262082, 6262799, 6264999, 6265469, 6267985,
6268423, 6273057, 6273134, 6273958, 6274151, 6274503, 6274739,
6275082, 6275687, 6275909, 6276910, 6277032, 6277202, 6277342,
6277365, 6278426, 6280559, 6280757, 6282447, 6284653, 6286642,
6288211, 6291677, 6293718, 6295711, 6297465, 6298091, 6300010,
6302919

The enriched report contains 59 evaluation rows because tickets 6140948 and 6169731 each had two affected agents. A row-level report must retain both evaluations rather than collapsing them to one ticket row. The report columns were:

ticket ID;
Zendesk agent name;
evaluation-time team snapshot;
initial July 13 exported score;
corrected score;
direct Rulebase URL containing conversation, agent, and QA evaluation IDs.

July 20 exact repair population


6268059, 6281712, 6313035, 6313637, 6316073, 6323758, 6326240,
6329331, 6329521, 6330550, 6330584, 6332808, 6334258, 6335097,
6335787, 6335855, 6335956, 6336428, 6337185, 6337718, 6339105,
6340234, 6340891, 6340955, 6341802, 6342562, 6344321, 6345222,
6347100

Export evidence

Captured scheduled exports:

Export	Rows	Export vs current score differences
June 30	1,259	9
July 6	1,773	19
July 13	1,142	89
July 20	1,356	29

Interpretation:

Current stored-total mismatches = 0 means the present scorecard cache agrees with present criteria/rules.
Export vs current does not by itself prove an incident. It also captures later re-evaluations and legitimate updates.
The July 20 file matched the stored values at export time; after the repair, a newly generated extract would show the corrected values while the historical file remains unchanged.
Exact claims about whether June 30 or July 6 contained stale totals at their original export times require point-in-time snapshot audits, not present-state comparison.

Production repair design

The July 20 repair intentionally did not restore a whole database or overwrite criterion decisions.

Safety properties:

Scoped to Qonto (organization_id = 33) and verified organization name.
Embedded the exact 29 scorecard-result public IDs and ticket IDs.
Defaulted to dry-run and required both APPLY=1 and a one-use confirmation token to commit.
Locked all exact scorecard-result rows before validation.
Verified candidate count, public IDs, ticket IDs, completed status, observed score, and criteria-derived expected score.
Validated every row before updating any row.
Ran the full repair in one database transaction; any mismatch aborted all changes.
Left criterion/check result rows untouched.
Recalculated only the scorecard cache, then used the normal model path to refresh evaluation and conversation aggregates.
Re-verified each score immediately after update.

The dry-run was executed against production inside a transaction and deliberately rolled back because the July 13 recovery database predated the 29 later evaluations. The apply run then committed all 29 guards successfully.

Post-repair read-only verification of all 1,356 July 20 export rows found zero remaining inconsistencies. A four-export audit also found zero current stored-total mismatches.

Recovery and historical reconstruction

Do not restore an old snapshot over production. Use a private point-in-time restore as a read-only evidence source, compare exact records, and migrate only explicit guarded rows if a repair is warranted.

Private RDS point-in-time restores created during the investigation:

rulebase-recovery-qonto-20260630-191900
rulebase-recovery-qonto-20260706-070000
rulebase-recovery-qonto-20260713-070000

All three were available when this note was written. They are isolated recovery resources, not production replacements. They must be deleted after the historical audit/evidence-retention need is resolved to avoid ongoing cost.

Private recovery bucket:

rulebase-recovery-qonto-20260719-879381255132
Public access blocked.
AES-256 server-side encryption.
Seven-day lifecycle configured.

Important object groups used during the investigation:

audit/: captured audit inputs, scripts, and results.
repair/: the bounded recalculation script.
report/: the 57-ticket/59-evaluation stale-score report and the 99-ticket always-pass re-evaluation report metadata.

An additional restore, rulebase-recovery-qonto-20260717-104500, was requested while reconstructing the always-pass batch. It was cancelled once the confirmed initial score of 0 made historical reconstruction unnecessary. A regional RDS inventory confirmed that this extra instance no longer existed; the three earlier historical-audit restores remained available.

Local transient investigation artifacts lived under .codex/recovery/qonto-20260713/ and are intentionally untracked. Do not assume they will exist in another checkout; use this note and durable PRs as the long-term record.

Customer communication guidance

Use calm, precise language:

State that the per-check/criterion results were retained and the overall score was recalculated from them.
State that there was no auto-fail or intentional re-evaluation where that has been verified.
Explain that a transient finalization issue left the exported overall total on an intermediate calculation.
Acknowledge if an old extract already contained the stale total and say that a fresh extract contains the corrected total.
Supply the bounded list of affected evaluation rows with initial and corrected scores.

Avoid:

claiming tickets were re-evaluated when they were not;
silently restoring an old headline score that conflicts with saved criteria;
saying “no previous or future batches were affected” before exact audits support it;
treating every old-export/current-score difference as the same incident;
describing a cached overall score as the source of truth when its component results disagree.

Engineering learnings

Finalization must be idempotent across every persistence path. Guarding one callback endpoint is insufficient when full-payload and incremental saves share the same aggregate.
Use one lock boundary for child writes and aggregate finalization. Ordinary late writes after completion should be ignored; intentional re-evaluation should use a separate explicit recalculation path.
Completion predicates must count the intended units. Count completed AI-evaluable leaf criteria, not every criterion row, because parent aggregate rows are not independent evaluation work.
Cached score columns need continuous invariant checks. An hourly read-only consistency audit is appropriate for production denormalized totals.
Reuse production scoring logic in audits. Cap rules, auto-fail, weighted scoring, and manual-only criteria make hand-written sums unsafe.
Repair components and all dependent aggregates together. Updating only the scorecard cache can leave evaluation/conversation caches stale.
Historical exports are evidence. Preserve the exact XLSX and export timestamp; comparing only with current database state cannot distinguish stale totals from later legitimate changes.
PITR is for reconstruction, not wholesale rollback. Restore privately, query read-only, diff narrowly, and write to production only through a bounded guarded transaction.
Report at evaluation granularity. One ticket can have multiple agents/evaluations, so a unique ticket list can hide distinct score changes.
Separate immediate repair verification from post-deploy verification. Run the invariant scan after the data repair and again after the durable guard has been deployed long enough to observe new traffic.

Follow-up checklist

Verify PR #8378 reaches release-prod and that its hourly job runs successfully.
Confirm Sentry alert routing/ownership for QA score consistency mismatches and truncated scans.
Run a post-#8372 Qonto scan over newly updated completed evaluations, not only rows from captured exports.
Complete the June 30 and July 6 point-in-time audits if an exact historical customer statement is still needed.
Delete the three recovery RDS instances when no longer needed.
Retain customer-shared CSV/XLSX evidence in the approved durable location if the seven-day recovery bucket is not sufficient.