Evaluation-check execution

ENG-2817  transfers the normal Evaluations::RunEvaluationCheckJob delivery and retry lifecycle from GoodJob to API2 BullMQ. ENG-2831  adds native complaint, customer-issue and sales-call compliance bodies behind independent organization-scoped flags. Rails remains the fail-closed body and code-revert fallback.

Contract and failure behavior

• Rails emits the numeric evaluation-check ID, organization ID and a SHA-256 key derived from the check ID, status and microsecond-precision update time. Ambiguous duplicate delivery reuses the same retained BullMQ job.
• BullMQ and the immediate GoodJob fallback share one non-blocking PostgreSQL advisory lock per check. An ambiguously accepted enqueue therefore cannot run the expensive check body twice; the delivery holding the lock retains retry responsibility.
• API2 retries execution five times, matching the retained GoodJob attempt budget. The long-running worker lock protects the LLM-backed checks from false stalled-job redispatch.
• The worker evaluates one PostHog flag for the check’s organization and type. A disabled, unavailable or unsupported result uses the synchronous Rails executor. An enabled result loads and persists the check body in API2.
• Native complaint execution records the detected complaint and follow-up result. Native customer-issue execution records the issue and schedules topic assignment after commit. Native compliance execution records criterion-group results, references and terminal check state.
• Native bodies preserve Rails-compatible prompt text, applicability checks, tenant scope and the per-check advisory lock. Completed, failed and not-applicable checks remain idempotent. A retry may resume a running check, matching the legacy GoodJob path after an interrupted attempt.
• After the last failed execution attempt, API2 enqueues a separate deterministic evaluation-check-run-failure job. Its eight attempts durably retry the Rails failure transition without rerunning the expensive check body.
• The failure transition takes the same per-check advisory lock as execution. It preserves an already-terminal check, retries while execution owns the lock, and otherwise marks both the check and its CX-risk snapshot failed.
• Immediate queue rejection or transport failure falls back to Evaluations::RunEvaluationCheckJob. The retained job and internal endpoint preserve the legacy implementation for disabled flags and rollback.

Rollout flags

All three flags fail closed. Their owner is Platform. The code gates were introduced September 6, 2026, and the PostHog flags are enabled in both regions. Remove or review them by December 5, 2026.

Deploy and observe

Deploy Rails and API2 from PR #11312 in the same release before enabling a native-body flag. No data repair is required. Start with one organization and one type. For 24 hours in both regions, record:

• completed, retrying and retained failed jobs on evaluation-check-run and evaluation-check-run-failure;
• native and Rails executor status counts and latency by check type;
• GoodJob fallback volume for Evaluations::RunEvaluationCheckJob;
• evaluation checks or CX-risk snapshots stranded in running;
• duplicate complaint, customer-issue or compliance result rows and effects;
• terminal failures whose Rails failure transition did not complete;
• customer-issue topic-assignment delivery and CX-risk completion delivery.

The package receives production ownership credit only after real checks of each traffic-bearing type complete in both regions, the queue drains, fallback stays bounded and no result or downstream effect repeats.

The rollout gate passed on September 8, 2026 at 22:01 UTC. The final audit found no new terminal run or failure-queue job, duplicate active AI evaluation, or newly stranded CX-risk snapshot. Historical retained jobs and eight older inactive/running snapshots remain cleanup evidence rather than blockers. Keep the Rails executor and per-type flags for rollback until their separate removal change lands.

Rollback and retirement

Disable the affected native-body flag to send later attempts through Rails under the same advisory lock. Revert the delivery revision only when GoodJob must resume as the normal producer. Do not purge either queue: late API2 calls are safe for terminal checks, and pending GoodJob fallbacks remain valid. Keep the Rails endpoints, check implementations and job class through the final regional observation window. This package does not make Rails unavailable for evaluation execution.