Qonto stale QA score aggregates

Incident dates: July 8-21, 2026

Environment: EU production

Customer: Qonto (organization 33)

Status: Data repaired; production rollout in progress

Severity: Medium

Summary

Some completed Qonto evaluations stored a stale score on the scorecard result and, in several cases found during the latest audits, on the parent agent evaluation and conversation. The completed leaf/check results were present, but a second completion path could recalculate parent criteria after another path had already finalized the cached aggregate. This made the exported overall score disagree with the sum implied by the stored criterion results.

Qonto first reported ticket 6260067: its Monday export showed an overall score of 15, while the application later showed 40. The ticket was not re-evaluated and did not auto-fail. Its criterion-level results supported 40; 15 was a stale aggregate captured by the earlier export.

The first review covered the 459 evaluations in the export period (July 8-15) and produced a 57-ticket stale-score report. A separate 99-ticket always-pass/auto-fail recalculation was investigated at the same time but was not this failure mode. After the first repair and partial concurrency fixes, the consistency monitor found nine additional stale scorecard aggregates on July 20-21. All nine were repaired, and a full follow-up audit found zero mismatches across 2,949 scorecard results and 2,949 agent evaluations. A later alert exposed incomplete cap metadata on one of those repaired records, and a fresh production sweep found one further stale aggregate created before the preventative fix reached production. Both follow-up records were repaired with guarded transactions.

New-chat handoff

This document is the canonical starting point for this incident. A new investigation should not infer current production state from an old Slack alert; follow the verification checklist below.

Canonical identifiers

• Customer: Qonto, Rails organization ID 33, timezone Europe/Paris.
• Alert: QA score consistency audit failed; Sentry fingerprint qa_score_cache_consistency.
• Schedule: hourly at :30; the audit considers scorecard/evaluation records touched in the previous three hours.
• Original customer example: ticket 6260067, exported score 15, criterion-supported score 40.
• Metadata-only follow-up: ticket 6314769, scorecard result qa_agent_evaluation_scorecard_result_OVBP7QrDwl7Uxp8YKw3JEnj1.
• Last pre-deployment recurrence: ticket 6396915, scorecard result qa_agent_evaluation_scorecard_result_Q5VkXG2NQLOTExRMK86Wlwep.
• Preventative code fix: commit 5f181387ea63be28fd29b58de24e65d2adf2f72f, PR #8451 .
• Production release: PR #8457 , merged July 21 at 11:41 AM ET. At 11:44 AM, Flightcontrol was still deploying and the live EU API and worker containers still had the old code.

Code map

• Completion and aggregate calculation: rulebase-api/app/models/qa_agent_evaluation_scorecard_result.rb, especially check_completion, check_completion_while_locked, calculate_parent_criteria, calculate_overall_score, recalculate_scorecard_score!, and score_cap_attributes.
• Background completion caller: rulebase-api/app/jobs/evaluations/check_agent_scorecard_completion_job.rb.
• Consistency calculation: rulebase-web/rulebase-api2/src/lib/qa-score-consistency/audit.ts.
• Slack alert formatting: rulebase-web/rulebase-api2/src/lib/qa-score-consistency/slack-notification.ts.
• Cap calculation and aggregate refresh helpers: rulebase-web/rulebase-api2/src/lib/qa-feedback-re-evaluation.ts, especially calculateExpectedQaScorecardAttributes, recalculateScorecardScore, recalculateQaEvaluationScore, and updateConversationQaScores.
• Scheduler and three-hour audit window: rulebase-web/rulebase-api2/src/worker.ts, maintenance task audit-qa-score-consistency.
• Historical bounded repair pattern: .codex/recovery/qonto-20260713/recalculate_current_stale_scores.rb. Its hard-coded candidates are historical; do not reuse them for a new alert.
• Original 57-ticket output: .codex/recovery/qonto-20260713/qonto-57-affected-tickets-enriched.csv.

Interpreting an alert

The audit’s actual -> expected direction is authoritative. Do not describe a metadata-only warning as a changed score, and do not describe an auto-fail-masked subtotal as a headline-score error.

Read-only investigation checklist

1. Record the alert time, environment, mismatch count, audited counts, record IDs, and exact differences.
2. Resolve each scorecard-result public ID to organization, ticket ID, scorecard, evaluation, stored score/cap fields, and all relevant updated_at timestamps.
3. Recompute the raw score from completed root criterion results. Separately check for any selected auto_fail option.
4. If the scorecard has cap rules, match each rule’s criterion_id and result_score_types against the selected leaf option. Compare all seven score attributes: score, max_score, score_cap_applied, score_cap_value, score_before_cap, score_cap_reason, and score_cap_criterion_id.
5. Inspect leaf and root criterion timestamps. A second cluster of writes after scorecard completion is the signature seen on tickets 6365947, 6394747, and 6396915.
6. Estimate the blast radius using the same three-hour window as the scheduled audit, then widen only if evidence requires it.
7. Before calling anything a post-fix recurrence, confirm both live EU Rails services contain with_lock { check_completion_while_locked }:
   • API: bin/aws/ecs-shell.sh api-prod-eu
   • Worker: bin/aws/ecs-shell.sh api-worker-prod-eu
8. Use the next scheduled :30 audit as the recovery signal. A merged release PR is not evidence that the live tasks have rolled over.

All ad hoc production queries should run in a read-only transaction first. The API2 production shell normally starts in /app/rulebase-api2 with DATABASE_URL already present; Rails containers start in /rails.

Repair checklist

1. Never repair from the Slack sample alone. Recompute the expected state from current completed criteria and current cap rules.
2. Use one bounded transaction with short lock and statement timeouts.
3. Lock exact rows with FOR UPDATE; guard on organization ID, ticket ID, public record ID, processing status, and every expected before-value.
4. Validate every candidate before changing any candidate. Roll back the entire transaction on a count or state mismatch.
5. Leave criterion decisions untouched. Refresh the scorecard cache, then the agent-evaluation cache, then the conversation QA caches.
6. Persist the complete score-cap attribute set together. The first repair of ticket 6314769 updated only numeric scores and caused the later metadata alert.
7. For a metadata-only repair, require that the numeric score and headline already equal the derived values and update only the missing metadata.
8. Print the before-state, derived evidence, and returned rows. Commit only after in-transaction verification, then perform a separate read-only verification after commit.
9. Do not restore a database snapshot or overwrite criteria to fix this incident.

Conditions for closure

• Both live EU Rails API and worker containers contain the central completion lock.
• The latest hourly audit completes with zero mismatches after both services have rolled over.
• No newer completed scorecard shows criterion writes after its locked completion.
• This status block and the deployment corrective action are updated with the verification time.

Customer impact

• The Monday evaluation export could contain an incorrect overall-score total even though its check-level results were correct.
• Ticket 6260067 appeared to change from 15 to 40 without a re-evaluation, creating understandable concern about evaluation stability.
• The July 20-21 repeats affected ten tickets. Six had stale scorecard subtotals hidden by an auto-fail, while four had an incorrect headline score.
• No criterion decisions or written evaluation evidence were lost. The issue was in denormalized/cached aggregates derived from those results.

Latest affected tickets and repair

Detection

The initial issue was customer-reported after comparing the Monday extract with the current UI. We then added an hourly consistency audit in PR #8378 . It compares stored scorecard/evaluation aggregates with values reconstructed from completed criterion results and reports mismatches to Slack and Sentry under the fingerprint qa_score_cache_consistency.

That monitor detected the July 20-21 repeats. It was functioning as designed, but it detects bad cached totals after they have been written; it does not prevent the competing writes.

The next hourly alert reported one mismatch on ticket 6314769. Its numeric score and headline were already correct at 25, but the earlier repair had not populated score_cap_applied, score_cap_value, score_before_cap, score_cap_reason, or score_cap_criterion_id. A fresh read-only sweep then found ticket 6396915, which had not yet appeared in the alert: its cached score was 0 while its criteria supported 35 before a configured cap of 25.

Timeline

All times are Eastern Time unless noted otherwise.

• July 8-15: The 459 evaluations in the original export period were evaluated. The subsequent stale-score report contained 57 tickets.
• July 13: Qonto’s Monday extract captured ticket 6260067 with an overall score of 15.
• July 16: We repaired the known stale aggregates. A newly generated extract would show totals consistent with the stored criterion results.
• July 20, 8:42 AM: PR #8372  shipped a row lock around incremental criterion imports.
• July 20, 9:14 AM (13:14 UTC): Ticket 6365947 finalized with a cached score of 0; its root criteria were updated roughly three seconds later, making the correct total 20.
• July 20, 11:07 AM: PR #8378  shipped hourly QA score consistency monitoring.
• July 20, 12:11 PM: PR #8381  made API2 final-result submissions idempotent.
• July 21, 9:51 AM (13:51 UTC): Ticket 6394747 finalized with a cached score of 0; leaf and root results were updated roughly 2.7 seconds later, making the correct total 50.
• July 21: We repaired all nine newly detected records using guarded transactions and ran a full Qonto audit. It found zero remaining mismatches among 2,949 scorecard results and 2,949 agent evaluations.
• July 21, 9:58 AM (13:58 UTC): The repair changed ticket 6314769 from 35 to the correct capped score of 25, but updated only the numeric caches and omitted the cap-provenance columns.
• July 21, 10:30 AM: The hourly audit correctly reported the missing cap metadata on ticket 6314769. This was not a new incorrect customer-facing score.
• July 21, 10:58 AM: PR #8451  merged the central row-locked completion boundary from ENG-1906 . The commit was not yet present on release-prod.
• July 21, 11:05 AM: Ticket 6396915 completed on the old production code with a cached score of 0. Its leaf and root criteria were updated approximately 24 seconds later and supported 35 before the configured cap of 25.
• July 21, after 11:05 AM: We repaired ticket 6396915 from 0 to 25 across the scorecard, evaluation, and conversation caches, including full cap provenance. We then applied a metadata-only repair to ticket 6314769; its score and headline remained 25.
• July 21, 11:41 AM: Production release PR #8457  merged into release-prod and triggered the Flightcontrol rollout.

Root cause

QAAgentEvaluationScorecardResult#check_completion was the shared finalization method, but it did not acquire a database row lock. It also checked processing_status_completed? on the model instance already held by the caller.

There are two ways into completion:

1. the synchronous incremental-result save path; and
2. Evaluations::CheckAgentScorecardCompletionJob.

GoodJob’s concurrency key serializes background jobs with one another, but it cannot serialize a job against the synchronous request path. Two callers could therefore load the same scorecard result while it was pending and enter completion. One caller would calculate and store the overall score, while the other still held an old in-memory pending state and could run parent/cross-criteria calculation again. The production timestamps on tickets 6365947 and 6394747 show the second criterion update landing approximately three seconds after aggregate finalization. Ticket 6396915 showed the same ordering with a wider interval: the scorecard finalized at 11:05:46 AM and its criterion results were rewritten around 11:06:10 AM.

PR #8372  protected the incremental import method with with_lock, but the background job called check_completion directly and bypassed that lock. The lock therefore guarded only one entrance rather than the completion critical section itself.

Contributing factors

• Aggregate scores are denormalized across criterion results, scorecard results, agent evaluations, and conversations.
• Completion had multiple callers with different concurrency controls.
• The early return used an in-memory status and did not force a reload from the database.
• Cross-criteria and parent-criteria calculations can update results after leaf completion, widening the race window.
• The consistency audit initially acted only as detection; repair remained a manual operation.
• The initial incident overlapped with a separate always-pass/auto-fail recalculation, which made it harder to distinguish changed criterion decisions from stale totals.

Resolution

Data

We repaired only records whose current completed criterion results deterministically disagreed with their cached aggregate. Each repair ran in a guarded transaction and was verified afterward. We did not restore a whole database snapshot or overwrite criterion decisions.

For ticket 6396915, the repair locked the exact scorecard, evaluation, and conversation rows; verified a 35/100 uncapped criterion total, no auto-fail, and the matching 25-point partial-result cap; then updated the scorecard from 0 to 25 and the evaluation/conversation caches from 0% to 25%. For ticket 6314769, the repair locked only the scorecard result, verified that its existing 25 score and 25% headline were already correct, and populated only the missing cap metadata. Independent post-commit reads verified both records.

During repair, abandoned Rails runner sessions held row locks and caused later writes to queue. We identified the exact database backends and orphan runner processes, terminated only those sessions, allowed their uncommitted work to roll back, then reran compact guarded transactions. No application service process or unrelated database session was terminated.

Code

ENG-1906 moves the row lock to the public check_completion boundary. Every external completion caller now reloads and locks the scorecard result before checking whether it is already complete. Import methods that already hold the same lock call a private lock-aware helper to avoid nesting locks.

This makes the completion decision and aggregate calculation one database critical section shared by both the synchronous import and background-job paths. A deterministic regression test loads a pending result, completes it through another instance, and proves that the stale caller reloads under lock and does not recalculate or update the completed record.

What went well

• Qonto provided a concrete ticket and before/after values that made the inconsistency reproducible.
• Criterion-level results let us reconstruct the intended totals without guessing or restoring production wholesale.
• The hourly monitor found the repeat quickly and confirmed the initial recovery after repair.
• The cap-metadata alert was precise enough to distinguish an incomplete repair from another incorrect score.
• Database timestamps gave direct evidence of the ordering race.
• Repairs were scoped by record ID, guarded, transactional, and followed by a full read-only audit.

What did not go well

• The first concurrency fix protected one writer rather than the shared finalization boundary, so another entry point remained unsafe.
• We initially treated the changed UI score as possible re-evaluation behavior rather than first testing cached totals against the immutable check-level evidence.
• The customer-facing export had no visible indication that its overall score disagreed with its own criterion rows.
• Manual production repair through long-lived Rails runners created avoidable lock contention and complicated verification.
• The first repair of ticket 6314769 corrected the numeric score but did not use the complete score-cap attribute calculation, leaving the record internally inconsistent and generating a later alert.
• Detection did not include automatic safe remediation, so repeats still required an incident response.

Corrective actions

• Add an hourly score-cache consistency audit with Slack and Sentry reporting (ENG-1864 ).
• Guard incremental criterion imports with a scorecard-result row lock (ENG-1860 ).
• Make API2 final-result submission idempotent (ENG-1867 ).
• Repair the nine initially detected July 20-21 mismatches and verify all 2,949 Qonto evaluations.
• Repair ticket 6396915 and complete the missing cap metadata on ticket 6314769.
• Merge the central Rails completion lock and regression coverage in PR #8451  (ENG-1906 ).
• Complete the production rollout from PR #8457  and verify subsequent hourly audits.
• Complete the larger atomic API2 persistence design so criterion and aggregate finalization have one owner (ENG-1870 ).
• Add direct ticket IDs and before/expected values to consistency alerts so an operator does not need a second lookup.
• Add a guarded repair command that recomputes selected mismatches with a dry-run default, bounded batches, lock timeouts, and post-write verification.
• Decide whether safe score-only mismatches should self-heal after the monitor records sufficient evidence.
• Add export-time validation so an export cannot silently emit an overall score inconsistent with its included criterion results.

Lessons

Concurrency controls must live at the shared state transition, not at one caller. A queue-level uniqueness guarantee is insufficient when synchronous code can perform the same transition. For denormalized aggregates, consistency audits are normal and valuable, but they are the backstop: the write path still needs a single atomic owner and exports should validate the totals they publish.