QA evaluation backfill execution family

Issue: ENG-3036  (child of ENG-2681). Owner: engineering (Chidi). Reviewer: Ronald (independent review required before merge).

Complete CloudWatch worker logs after the earlier ownership release showed US normal-path GoodJob starts from 2026-09-11T20:00:03Z through 20:07:19.776Z for QAEvaluationBackfill::PreviewJob, ConfirmJob, QueueReplacementJob, BackdateJob, and ReconcileJob: two automatic runs (run 171 for organization 19, run 172 for organization 25), request 5899294 and replacement run-agent 1512 from run 171, 11 attempts / 11 unique jobs. EU had none. Those rows reset the zero-GoodJob clock; backfills are not maintenance and block the milestone.

Reopened 2026-09-16. The first package shipped API2 execution behind a default-off PostHog organization gate (qa_evaluation_backfill_api2_owner) with the Rails GoodJob classes as the gate-off path. The exact EU Rails-worker CloudWatch window 2026-09-15T15:06:59Z → 2026-09-16T15:06:59Z still contained QAEvaluationBackfill::PreviewJob for backfill run 1156, organization 33, because the gate was false for that organization. The run was cancelled, but the Rails preview kept planning for roughly two hours and ended in a PostgreSQL statement timeout. A default-off flag therefore does not meet the completion target; the target is zero normal-path Rails jobs.

This correction makes API2/BullMQ the permanent normal owner of the complete five-job family for every organization. There is no organization gate, no environment override, and no Rails fallback on the normal path. Rails keeps its admin/report HTTP entrypoints and model/request callbacks as compatibility enqueuers only, and they never perform_later these classes. The five Rails job classes remain solely as operator recovery tombstones: their enqueue raises QAEvaluationBackfill::OperatorRecoveryTombstone::NormalPathDisabledError, so perform_later, set(wait:).perform_later, and automatic retries all fail; only an explicit perform_now from an operator console runs them.

Ownership boundary

Rails normal-path entrypoint inventory

Every Rails producer routes unconditionally through QAEvaluationBackfill::Api2Dispatcher (rulebase-api/app/services/qa_evaluation_backfill/api2_dispatcher.rb). The inventory below is exhaustive for app/, lib/, config/, and db/data/; spec/lint/qa_evaluation_backfill_normal_path_spec.rb fails the build if any of them (or a new one) reaches a job class directly.

The dispatcher posts POST /internal/qa_evaluation_backfills/enqueue (bearer RAILS_INTERNAL_API_KEY) and treats only a 202 as acceptance. Any non-2xx or transport error raises Api2Dispatcher::DeliveryError — there is no GoodJob fallback; the calling request/callback fails loudly and is retried by its own caller. The dispatcher never consults PostHog, Flipper, or Organization.

Queue and ids

• Queue qa-evaluation-backfill, worker concurrency 4, one organization + one run/run-agent per job (failure isolation at the run-agent / conversation boundary inside confirm and replacement).
• Deterministic ids: qa-eval-backfill-preview-<run>, qa-eval-backfill-confirm-<run>, qa-eval-backfill-replacement-<run_agent>-<replaced_request|manual>, qa-eval-backfill-backdate-<run>, qa-eval-backfill-reconcile-<org>-<date>-<wait_attempt>. Finished jobs with the same id are replaced (addReplacingFinishedJob); an active job wins. The report handoff reuses the existing deterministic organization report job id.
• 5 attempts, exponential backoff from 30s (GoodJob parity); run/run-agent rows checkpoint progress so every kind is safe to re-drive.
• The worker executes for every organization. There is no execution-time owner check, no gate_off outcome, and no QA_EVALUATION_BACKFILL_OWNER environment override.

Cancellation during preview

runPreview passes a checkpoint into PlanBuilder, awaited before each agent plan and before each eligibility batch. The checkpoint enforces a 30-minute wall-clock budget (PREVIEW_WALL_CLOCK_BUDGET_MS; each statement is still capped by the 5-minute SET LOCAL statement_timeout) and, at most every 5 s (PREVIEW_CANCELLATION_CHECK_INTERVAL_MS), re-reads the run row. Under READ COMMITTED the read sees an admin cancel committed after planning started. A run that is no longer previewing throws PreviewCancelledError, which rolls back the planning transaction, returns { outcome: "skipped", reason: "cancelled_during_planning" } without marking the run failed, and is not retried. Budget exhaustion throws PreviewBudgetExceededError, marks the run failed with a note, and rethrows so BullMQ records the attempt. The final locked previewing check before the run-agent write stays as the last line of defense. The Rails tombstone PreviewJob has the same checkpoint (CANCELLATION_CHECK_INTERVAL, PREVIEW_WALL_CLOCK_BUDGET) for operator recovery runs.

Ownership (no feature gate)

The qa_evaluation_backfill_api2_owner flag, Organization#qa_evaluation_backfill_api2_owner?, API2 resolveQaEvaluationBackfillOwner, POSTHOG_FEATURE_FLAGS.QA_EVALUATION_BACKFILL_API2_OWNER, and QA_EVALUATION_BACKFILL_OWNER are removed. The flag was never created in production and nothing in this correction creates, reads, or deletes it. If a stale definition exists in PostHog it is inert and may be deleted at any time.

Rollout gates

1. Merge only after independent review and green CI (Rails suite including spec/lint/qa_evaluation_backfill_normal_path_spec.rb; API2 pnpm typecheck and the PGlite suite).
2. Deploy Rails and API2 together (the dispatcher already exists on both sides; the change removes the Rails branch and the API2 gate, so ordering is not load-bearing, but do not leave one side undeployed across a daily report tick).
3. Immediately after deploy: run the outcome queries below for the previous and current target dates in both regions; expect no GoodJob rows for the five classes with created_at >= :deployed_at and every auto_proceed run reaching backdate_status = 'completed' within 120 minutes.
4. Strict CloudWatch verification (required before closing ENG-3036): after at least one full daily cycle per region, query the Rails worker log group for the exact 24-hour window with the filter "QAEvaluationBackfill::" ("PreviewJob" OR "ConfirmJob" OR "QueueReplacementJob" OR "BackdateJob" OR "ReconcileJob") and require zero job-start rows. Record the exact window boundaries, region, and result in the migration log. An operator perform_now recovery is the only acceptable match and must be linked to an incident note.

Rollback trigger

Any of: a DeliveryError spike in Rails (Sentry), an organization/date failing the invariant query after 120 minutes, duplicate active requests for one conversation, a run stuck in queueing/running for > 30 minutes, a preview skipped with cancelled_during_planning for a run that is still previewing, or cross-tenant ids in API2 logs.

Rollback (code revert or explicit operator-only recovery)

There is no flag to flip. Two options, never combined on the same run:

1. Code revert. Revert this PR (Rails and API2 together). That restores the gated dispatcher; with the flag still absent in production the gate evaluates false and Rails GoodJob execution resumes for every organization. Queued BullMQ jobs then complete as gate_off. Use this only for a systemic defect in the API2 executor.
2. Explicit operator-only recovery for individual stranded runs, from a Rails console, within ActsAsTenant.with_tenant(run.organization):
   • re-drive on API2 first: previewing → QAEvaluationBackfill::Api2Dispatcher.preview_later(run); queueing → confirm_later(run); backdate_status: running → backdate_later(run); a missing report → reconcile_later(run.organization, run.target_date).
   • only if API2 is unavailable: QAEvaluationBackfill::PreviewJob.perform_now(run.id) (or the matching class) — synchronous, foreground, in the console. perform_later raises by design. Record the run id, class, operator, and reason in the migration log; that entry justifies the matching CloudWatch row.

Invariant and evidence queries (read-only)

Invariant: for each eligible organization/date, the target shortfall and planned eligible/standby conversations reconcile to terminal requests/evaluations plus explicit excluded, active-request-blocked, no-gap, or standby-exhausted reasons within 120 minutes of the run’s creation.

-- Outcome per run-agent for a coverage date. Replace :org and :date.
WITH ra AS (
  SELECT ra.id AS run_agent_id, r.id AS run_id, r.status, r.backdate_status, r.backdated_at,
         r.created_at, ra.employee_id, ra.target_count, ra.consumed_count_at_queue,
         ra.eligible_count_at_queue, ra.requested_count, ra.excluded, ra.standby_exhausted_at,
         jsonb_array_length(ra.eligible_conversation_ids) AS eligible_planned,
         jsonb_array_length(ra.standby_conversation_ids) AS standby_remaining,
         ra.observed_in_flight_request_ids
  FROM public.qa_evaluation_backfill_runs r
  JOIN public.qa_evaluation_backfill_run_agents ra ON ra.qa_evaluation_backfill_run_id = r.id
  WHERE r.organization_id = :org AND r.target_date = :date
),
req AS (
  SELECT q.qa_evaluation_backfill_run_agent_id AS run_agent_id,
         array_agg(q.id ORDER BY q.id) AS request_ids,
         array_agg(q.replaces_qa_evaluation_request_id) FILTER (WHERE q.replaces_qa_evaluation_request_id IS NOT NULL) AS replacements,
         array_agg(q.qa_evaluation_id) FILTER (WHERE q.qa_evaluation_id IS NOT NULL) AS evaluation_ids,
         count(*) FILTER (WHERE q.status IN ('pending','ready','processing')) AS active,
         count(*) FILTER (WHERE q.status = 'completed') AS completed,
         count(*) FILTER (WHERE q.status IN ('skipped','failed','cancelled')) AS non_completed,
         min(q.completed_at) AS first_completed_at, max(q.completed_at) AS last_completed_at
  FROM public.qa_evaluation_requests q
  WHERE q.organization_id = :org AND q.qa_evaluation_backfill_run_agent_id IN (SELECT run_agent_id FROM ra)
  GROUP BY 1
)
SELECT ra.*, req.request_ids, req.replacements, req.evaluation_ids,
       req.active, req.completed, req.non_completed, req.last_completed_at,
       GREATEST(ra.target_count - ra.consumed_count_at_queue, 0) AS shortfall_at_plan,
       CASE
         WHEN ra.excluded THEN 'excluded'
         WHEN GREATEST(ra.target_count - ra.consumed_count_at_queue, 0) = 0
              AND jsonb_array_length(ra.observed_in_flight_request_ids) > 0 THEN 'active_request_blocked'
         WHEN GREATEST(ra.target_count - ra.consumed_count_at_queue, 0) = 0 THEN 'no_gap'
         WHEN ra.standby_exhausted_at IS NOT NULL THEN 'standby_exhausted'
         WHEN COALESCE(req.active, 0) = 0
              AND COALESCE(req.completed, 0) >= GREATEST(ra.target_count - ra.consumed_count_at_queue, 0) THEN 'filled'
         WHEN COALESCE(req.active, 0) > 0 THEN 'in_progress'
         ELSE 'shortfall_unexplained'
       END AS outcome,
       ra.backdate_status = 'completed' AND ra.backdated_at IS NOT NULL AS backdated,
       now() - ra.created_at > interval '120 minutes' AS past_window
FROM ra LEFT JOIN req USING (run_agent_id)
ORDER BY ra.run_id, ra.run_agent_id;

Rows with outcome = 'shortfall_unexplained' or 'in_progress' and past_window = true violate the invariant and are the rollback trigger.

-- Report handoff: one post-backfill daily report per completed auto run.
SELECT r.id AS run_id, r.organization_id, r.target_date, r.backdated_at,
       (SELECT count(*) FROM public.qa_evaluation_backfill_runs x
         WHERE x.organization_id = r.organization_id AND x.target_date = r.target_date
           AND x.status NOT IN ('cancelled','failed') AND x.backdate_status <> 'completed') AS unsettled_runs
FROM public.qa_evaluation_backfill_runs r
WHERE r.auto_proceed AND r.backdate_status = 'completed' AND r.target_date = :date;

Pair it with the daily-agent-evaluation-report BullMQ job source = "backfill" for (organization, date) — exactly one expected.

-- Zero-GoodJob assertion after the deploy timestamp (per region).
SELECT job_class, count(*) AS executions, count(DISTINCT active_job_id) AS unique_jobs
FROM good_jobs
WHERE job_class IN ('QAEvaluationBackfill::PreviewJob','QAEvaluationBackfill::ConfirmJob',
                    'QAEvaluationBackfill::QueueReplacementJob','QAEvaluationBackfill::BackdateJob',
                    'QAEvaluationBackfill::ReconcileJob')
  AND created_at >= :deployed_at
GROUP BY 1;

The expected result is zero rows. Any row is either an operator perform_now with a logged justification or a regression to fix.

-- Exact outcome check for the reopened incident shape: the most recent run per
-- organization/date, its terminal state, and whether a cancel was honored
-- (no run-agents written after cancellation). Replace :since.
SELECT r.id AS run_id, r.organization_id, r.target_date, r.status, r.backdate_status,
       r.auto_proceed, r.created_at, r.updated_at,
       (SELECT count(*) FROM public.qa_evaluation_backfill_run_agents ra
         WHERE ra.qa_evaluation_backfill_run_id = r.id) AS run_agents,
       (SELECT max(ra.created_at) FROM public.qa_evaluation_backfill_run_agents ra
         WHERE ra.qa_evaluation_backfill_run_id = r.id) AS last_run_agent_at,
       r.status = 'cancelled'
         AND EXISTS (SELECT 1 FROM public.qa_evaluation_backfill_run_agents ra
                      WHERE ra.qa_evaluation_backfill_run_id = r.id
                        AND ra.created_at > r.updated_at) AS wrote_after_cancel,
       r.status = 'previewing' AND now() - r.updated_at > interval '35 minutes' AS stuck_previewing,
       r.notes
FROM public.qa_evaluation_backfill_runs r
WHERE r.created_at >= :since
ORDER BY r.organization_id, r.target_date, r.id;

wrote_after_cancel = true or stuck_previewing = true fails the gate (the preview wall-clock budget is 30 minutes).

-- Exactly-once terminal effects: no conversation with two active backfill
-- requests and no run-agent with more requests than its planned shortfall.
SELECT q.conversation_id, count(*) AS active_requests
FROM public.qa_evaluation_requests q
WHERE q.qa_evaluation_backfill_run_agent_id IS NOT NULL
  AND q.status IN ('pending','ready','processing')
GROUP BY 1 HAVING count(*) > 1;
 
SELECT ra.id AS run_agent_id, ra.requested_count,
       GREATEST(ra.target_count - ra.consumed_count_at_queue, 0) AS shortfall_at_plan
FROM public.qa_evaluation_backfill_run_agents ra
JOIN public.qa_evaluation_backfill_runs r ON r.id = ra.qa_evaluation_backfill_run_id
WHERE r.created_at >= :since
  AND ra.requested_count > GREATEST(ra.target_count - ra.consumed_count_at_queue, 0);

Both return zero rows when the gate passes.

Side-effect inventory

Tests

• rulebase-web/rulebase-api2/src/lib/qa-evaluation-backfill/execute.db.test.ts (PGlite): Rails-entry-shaped job data → the real executeQaEvaluationBackfillJob with real SQL, PlanBuilder port, callbacks and routing; only the BullMQ/report transport edges are recorded. Fixtures (test-fixtures.ts) are anonymized, production-shaped tenants 19 (US) and 25 (EU) with prefixed ids, fixed-per-period rules, employees, completed AI evaluations, in-flight requests, scorecard/ticket-scope conditions; scenarios cover no-gap, skip → replacement, completed-net-zero replacement, standby exhaustion, active-request collision, partial retry, duplicate and concurrent delivery, cancellation before and during planning (PlanBuilder checkpoint), wall-clock budget exhaustion, retry of a failed preview, backdate, reconcile/report, an organization whose old gate was false (run 1156 shape), and the run-171-shaped end-to-end lifecycle across all five stages.
• routes/internal-qa-evaluation-backfills.test.ts.
• Rails: spec/lint/qa_evaluation_backfill_normal_path_spec.rb (source-level gate: no direct QAEvaluationBackfill::*Job call, no owner-flag reference in app/ lib/ config/ db/data/; all five classes include the tombstone; a perform_later raises), spec/services/qa_evaluation_backfill/api2_dispatcher_spec.rb (API2 for every organization regardless of PostHog, delivery errors raise, no GoodJob rows, explicit perform_now recovery still works), spec/requests/admin/qa_evaluation_backfills_spec.rb (manual admin create/confirm/backdate/cancel handoffs, double-submit exactly once, old gate false), spec/jobs/qa_evaluation_backfill/preview_job_spec.rb (cancellation observed at the next checkpoint, wall-clock budget), model, request-callback, daily-report and data-migration specs asserting the API2 handoff instead of GoodJob.

Known gaps

• The Rails PlanBuilder remains the reference; the API2 port is verified by the PGlite suite, not by a live dual-run diff. Run the canary’s first automatic run with both PlanBuilder.new(...).call (read-only, Rails console) and the persisted API2 run-agents before expanding.
• The cancellation checkpoint observes cancels at agent/batch boundaries; a single 5-minute statement that is already running finishes (or times out) before the checkpoint fires.
• A Rails DeliveryError (API2 unreachable) fails the admin request or the model callback; the caller retries. There is intentionally no queueing fallback on Rails.
• Not covered here: TranslateConversationJob and the remaining QA lifecycle packages.

State