CX-risk snapshot completion fan-in

ENG-2249  transfers the normal completion signal and retry loop from Rails GoodJob to API2 BullMQ. Evaluation check execution and the final CX-risk transaction remain Rails-owned.

Contract and idempotency

• Every terminal check transition emits the snapshot ID, organization ID, and a SHA-256 key derived from ordered evaluation-check IDs, statuses, and update timestamps. Duplicate state reuses one BullMQ job; changed state receives a new job.
• API2 validates tenant identity and retains failed jobs. The worker calls only the authenticated Rails completion endpoint; it does not reproduce Rails result aggregation or downstream effects.
• Rails reloads the snapshot under a dedicated advisory lock. Completed and failed snapshots return without side effects, incomplete snapshots return waiting, and only an all-terminal snapshot enters the existing finalizer.
• The finalizer derives the persisted snapshot churn level from the terminal customer-issue check’s score before validating the snapshot. This keeps the stored enum within low, medium, and high and lets completion recover from an invalid pre-finalization value without weakening other validations.
• Immediate enqueue rejection or transport failure falls back to Evaluations::CheckCompletionJob. The fallback enters the same locked method, so an ambiguous request accepted by API2 cannot finalize twice.

Deploy and observe

Deploy Rails and API2 from the same release. No feature flag or production data repair is required. For 24 hours in both regions, record:

• completed, retrying, and retained failed jobs on conversation-cx-risk-completion;
• Rails internal-finalizer status counts and latency;
• GoodJob fallback volume and lock retry exhaustion;
• snapshots still running after every evaluation check is terminal;
• repeated notifications, summaries, assignments, or other finalization effects.

The package is production-ready only when real completions occur in each region, the queue drains, fallback stays bounded, no snapshot is stranded, and no final effect repeats.

Kuda’s September 7 audit found 179 snapshots created in the preceding 24 hours stranded with raw churn values outside the Rails enum. After deploying ENG-2837, require new Kuda snapshots to complete without HTTP 422 callbacks and confirm the failed-queue count stops increasing. Existing terminal jobs and stranded rows require a separately authorized repair or replay; deployment alone does not mutate them.

Rollback and retirement

Revert the revision to restore GoodJob as the normal delivery path. Do not purge BullMQ or GoodJob: late API2 calls are safe because the Rails finalizer checks terminal state under the advisory lock. Keep the Rails endpoint, model method, job class, and retry behavior until the regional observation passes and a later package moves the final transaction itself. This package alone does not make Rails unavailable for CX-risk completion.