QA post-processing and citation repair

ENG-2934  moves the QA evaluation post-processing work that ran after the native QA runner finished from Rails/GoodJob to API2/BullMQ. Production EU+US rolling 12-hour evidence ending 2026-09-09 10:13 UTC: SyncQAAgentEvaluationScorecardCriterionResultChunksJob 85 executions, ReviewAICheckCitationsForOrganizationJob 54 executions but ~37 minutes of Rails worker time (the single largest LLM-bound consumer left on GoodJob).

Class coverage

The Rails job classes for the migrated work remain deployed so queued rows can drain and so the rollback path (below) works without a deploy.

Remaining deferred feedback generator

GenerateExpandedCommentJob (the comment-level expanded-comment generator under EvaluationScorecardCriterionResultComment) stays on GoodJob — it was not part of this migration’s job set.

The evaluation-level generators and the per-criterion generator (GenerateQAAgentEvaluationCriterionFeedbackJob) are ported fully natively: src/lib/qa-feedback-generation/ contains the prompt constants (copied verbatim from the Rails GENERATE_EXPANDED_COMMENT_INSTRUCTIONS heredocs), a Blueprinter-equivalent render layer under render/ that produces the same JSON the *_Blueprint classes emit, the [KB_DOC ...]/[MSG_REF ...] mention enrichment, and the OpenAI call (model_reasoning = gpt-5.2, reasoning effort low, service_tier: flex, max_output_tokens: 2000, same generate_expanded_comment prompt_id metadata). Deliberate render differences, none of which affect prompt meaning: Active Storage URLs are presigned object-storage URLs rather than rails_blob_url paths (same convention as employee-payload.ts), Time.current in the metadata context is volatile anyway, and mention spans are scanned with a targeted regex instead of Nokogiri (the source HTML bytes are preserved rather than re-serialized). The generate_expanded_comment! call sites on the models remain in place for the GoodJob rollback path.

Contract

Criterion-result chunk/embedding sync

• Producer: the existing after_commit hook calls Api2QACriterionResultChunkSyncEnqueuer.enqueue! → POST /internal/qa_criterion_result_chunk_syncs/enqueue (bearer RulebaseConfig.internal_api_key, 1 s open / 3 s read timeout). A 202 means the BullMQ add was awaited, i.e. the job is durably in Redis.
• Fallback: any non-2xx or transport error falls back to SyncQAAgentEvaluationScorecardCriterionResultChunksJob.perform_later and logs API2 QA criterion result chunk sync enqueue failed; falling back to GoodJob. Rails-side ineligible cleanup (delete chunks, clear chunks_synced_at) is unchanged and still runs inline.
• Job id qa-criterion-result-chunk-sync-<criterion_result_id>, 5 attempts, exponential backoff from 10 s. Every accepted enqueue is durable: the enqueue atomically sets a Redis dirty marker (qa-criterion-result-chunk-sync:dirty:<id>) and checks for an open run (qa-criterion-result-chunk-sync:open:<id>); an open run re-reads the row before its atomic final clean check closes it, a finished job under the deterministic id is removed and re-added, a waiting/delayed job is absorbed, and an id that is still active after its run closed gets a -successor-<n> job so the edit cannot be lost across BullMQ completion.
• Worker semantics mirror #sync_chunks + ChunkEmbeddingSync: missing row or blank expanded_comment → delete document_chunks for source_type = 'QAAgentEvaluationScorecardCriterionResult' and clear chunks_synced_at; organization memory enabled → skip; otherwise chunk the comment as Markdown (chunkKnowledgeBaseMarkdown, same splitter the embed function used), embed with the "<scorecard name> > <criterion description>" context prefix, require 3072-dimension vectors, and replace chunks inside one transaction under SELECT … FOR UPDATE on the criterion result (parity with with_lock). Chunk ids come from document_chunks_id_seq so Rails and API2 never collide.

Daily citation review and repair

• Scheduler qa-citation-review-tick, cron 0 5 * * * UTC, data { kind: "tick", date: "yesterday" } — identical to the GoodJob entry.
• The tick resolves today/yesterday once in UTC (Rails default zone), lists organizations, and enqueues one organization job per org with id qa-citation-review-<org_id>-<YYYY-MM-DD> (3 attempts, exponential backoff from 60 s). Tick retries dedupe against completed jobs and checkpoint completedOrganizationIds, so a tick retry never replays an org.
• Each organization job interprets the date in the org’s own timezone with half-open bounds [date 00:00 local, date+1 00:00 local), selects the same candidates as ReviewAICheckCitationsForOrganizationJob (AI-source evaluations only; criterion reasons with PartLink on fail/partial/auto_fail options with at least one kept check, then check results with fail/partial + PartLink summaries; ordered, deduplicated), and processes each scorecard result in that order.
• Per scorecard result: deterministic PartLink repair first (same-conversation ids, unique Levenshtein ≤ 5 correction, ambiguous/unmatched stripped, attribute rewritten as part_id='…', scores untouched), then the verify_check_citation verifier (gpt-4.1, 4000 max output tokens, same prompt file, nullable structured output). Corrections replace the criterion result and check results, then re-run cross-criteria rules, parent criteria, scorecard score, evaluation score and conversation QA scores in that order.
• Bounded/resumable: processedScorecardResultIds, partLinkRepairs and misattributionFixes are checkpointed into BullMQ job data after every scorecard result, so a retry resumes after the last completed result and cannot double-mutate.
• Slack: repairs first, then misattributions; sections truncated at 3000 characters; no webhook or no changes → no send; the send is guarded by an outbound_delivery_claims key so a retry after an ambiguous outcome never re-posts (see “Resolving uncertain delivery claims” in the routine workload runbook). A definitive Slack rejection rethrows for BullMQ retry.
• Admin re-drive: POST /internal/qa_citation_reviews/enqueue { organization_id, date } replaces a finished job for that org/date.

Evaluation feedback generation (fully native)

• Producer: QAAgentEvaluation#generate_expanded_comment_later / #generate_criterion_feedback_for_differences_later call Api2QAFeedbackGenerationEnqueuer.enqueue! → POST /internal/qa_feedback_generation/enqueue { organization_id, evaluation_id, kind, force } (bearer RulebaseConfig.internal_api_key, 1 s open / 3 s read timeout). A 202 means the BullMQ add was awaited, i.e. the job is durably in Redis. The route only accepts expanded_comment and criterion_feedback; the criterion_result kind is produced internally by the diff job.
• Commit boundary: enqueue_feedback_generation wraps the entire delivery — kill-switch check, API2 enqueue, and GoodJob fallback — in ActiveRecord.after_all_transactions_commit. This fixes a production race observed on 2026-09-09 (~20:12 UTC): mark_as_completed! called trigger_feedback_generation_if_human inside an open transaction, API2 picked the job up within milliseconds, loadEvaluation could not see the uncommitted row, and UnrecoverableError suppressed retries — six failed jobs (evaluation ids 255855, 255865, 255880, org 25) that had to be re-driven manually. Delivery now happens only after the outermost commit, and never after a rollback.
• Snapshot render compatibility: customer_churn_risk_level is a Rails model method, not a database column. API2 derives it from customer_churn_risk_score (none for null, high at 0.7, medium at 0.4, otherwise low) and tenant-qualifies the snapshot lookup. This fixes a second production failure observed after the commit-boundary release: new evaluation-level jobs could see their rows but exhausted retries while selecting the nonexistent method name as a column.
• Fallback: any non-2xx or transport error falls back to the retained GoodJob class (GenerateQAAgentEvaluationExpandedCommentJob.perform_later / GenerateCriterionFeedbackForDifferencesJob.perform_later) and logs a warning; Flipper good_job_qa_feedback_generation (per-org actor supported) bypasses API2 entirely.
• Job id qa-feedback-generation-<kind>-<evaluation_id>-<normal|force> (and ...-criterion_result-<criterion_result_id>-<ai_id|none>-<normal|force> for the per-criterion fan-out), 3 attempts, exponential backoff from 10 s. force is part of the identity so an admin force re-drive never collapses onto a waiting normal job; a finished job under the id is removed and re-added, a waiting/delayed job is absorbed.
• Worker semantics: expanded_comment runs generateEvaluationExpandedComment (port of QAAgentEvaluation#generate_expanded_comment!) and criterion_feedback runs generateCriterionFeedbackForDifferences (port of #generate_criterion_feedback_for_differences), which diffs each leaf criterion result against the AI criterion-results map (latest AI evaluation for the same subject employee, else the snapshot’s own results) and enqueues one criterion_result job per difference — the same per-criterion isolation GenerateQAAgentEvaluationCriterionFeedbackJob provided. Each criterion_result job runs generateCriterionResultExpandedComment (port of QAAgentEvaluationScorecardCriterionResult#generate_expanded_comment!).
• Error parity: a missing evaluation/criterion result completes unrecoverable, matching discard_on ActiveRecord::RecordNotFound; the eval-level flow logs and swallows OpenAI errors (its Rails rescue did the same), while the criterion flow raises on empty responses and API errors so BullMQ retries (mirroring the retired job’s retry_on list). Both flows short-circuit on qa_organization_memory_enabled? and on an existing expanded_comment unless force.

Rollout controls

Rollback order

1. Citation review: enable pause_api2_qa_citation_review, confirm the next tick logs paused, then enable the review_ai_check_citations cron in the GoodJob dashboard. Organization jobs already in flight finish their current scorecard result and checkpoint; they do not re-run under Rails because Rails selects candidates by created_at window and the repaired rows no longer contain broken links.
2. Chunk sync: enable good_job_qa_criterion_result_chunk_sync (globally or for one organization). API2 jobs already queued still run; they are idempotent against the same row.
3. Re-drive a missed citation day after rollback with ReviewAICheckCitationsJob.perform_later('YYYY-MM-DD') (Rails) or one POST /internal/qa_citation_reviews/enqueue per org (API2).
4. Feedback generation: enable good_job_qa_feedback_generation (globally or for one organization). API2 jobs already queued still run and execute the same Rails code path, so there is no double-write risk.

Bake gates

Ownership is asserted only after, in both regions:

• Two consecutive 05:00 UTC ticks complete with enqueued = organizations and zero failed organization jobs (or every failure is a tenant-isolated Sentry event with a checkpoint that resumed on retry).
• Zero new normal-path GoodJob rows for SyncQAAgentEvaluationScorecardCriterionResultChunksJob, ReviewAICheckCitationsJob, ReviewAICheckCitationsForOrganizationJob, GenerateQAAgentEvaluationExpandedCommentJob and GenerateCriterionFeedbackForDifferencesJob for 24 hours (retained rows inside the 12-hour cleanup window are historical; fallback rows are identified by the warning log above).
• Zero UnrecoverableError: qa_agent_evaluation <id> not found failures on the qa-feedback-generation queue since the commit-boundary fix deployed — that error signature means the enqueue still outran the producing transaction and the fix is incomplete.
• No uncertain outbound delivery claims with qa-citation-review keys, and the team Slack channel shows one post per org/day at most.
• chunks_synced_at continues to advance for eligible criterion results and document_chunks counts per result match the Markdown chunker output.

After 24 clean hours, remove the GoodJob fallback branches in sync_chunks_later and QAAgentEvaluation#enqueue_feedback_generation, delete SyncQAAgentEvaluationScorecardCriterionResultChunksJob, GenerateQAAgentEvaluationExpandedCommentJob, GenerateCriterionFeedbackForDifferencesJob, both citation job classes, the cron entry and the flags.

Observation queries

GoodJob normal-path traffic (must trend to zero):

SELECT serialized_params->>'job_class' AS job_class,
       count(*) AS executions,
       sum(extract(epoch FROM (finished_at - performed_at))) / 60 AS worker_minutes
FROM good_jobs
WHERE serialized_params->>'job_class' IN (
  'SyncQAAgentEvaluationScorecardCriterionResultChunksJob',
  'ReviewAICheckCitationsJob',
  'ReviewAICheckCitationsForOrganizationJob',
  'GenerateQAAgentEvaluationExpandedCommentJob',
  'GenerateCriterionFeedbackForDifferencesJob')
  AND created_at > now() - interval '24 hours'
GROUP BY 1;

Chunk-sync health (results with a comment but stale/missing chunks):

SELECT count(*) FILTER (WHERE chunks_synced_at IS NULL) AS never_synced,
       count(*) FILTER (WHERE chunks_synced_at < updated_at - interval '10 minutes') AS stale
FROM qa_agent_evaluation_scorecard_criterion_results
WHERE expanded_comment IS NOT NULL AND btrim(expanded_comment) <> ''
  AND updated_at > now() - interval '24 hours';

Citation repair effect (broken links remaining for yesterday’s evaluations):

SELECT count(*)
FROM qa_agent_evaluation_scorecard_criterion_check_results ccr
JOIN qa_agent_evaluation_scorecard_criterion_results cr
  ON cr.id = ccr.qa_agent_evaluation_scorecard_criterion_result_id
JOIN qa_agent_evaluation_scorecard_results sr
  ON sr.id = cr.qa_agent_evaluation_scorecard_result_id
JOIN qa_agent_evaluations e ON e.id = sr.qa_agent_evaluation_id
WHERE e.source = 'ai'
  AND ccr.result IN ('failed', 'partial')
  AND ccr.summary LIKE '%PartLink%'
  AND ccr.created_at >= (current_date - 1)::timestamp AT TIME ZONE 'UTC'
  AND ccr.created_at <  current_date::timestamp AT TIME ZONE 'UTC';

Slack delivery claims for the scan:

SELECT delivery_key, claimed_at, delivered_at
FROM outbound_delivery_claims
WHERE delivery_key LIKE 'qa-citation-review:slack:%'
  AND claimed_at > now() - interval '48 hours'
ORDER BY claimed_at DESC;

BullMQ (redis-cli): ZCARD bull:qa-citation-review:completed, ZCARD bull:qa-citation-review:failed, ZCARD bull:qa-criterion-result-chunk-sync:failed; failed counts must stay at zero or be explained by Sentry events tagged with the organization id.

Verification

API2 unit tests cover PartLink parsing/repair parity with the Rails specs (including the part_id='…' rewrite), verifier prompt byte-equality with config/prompts/verify_check_citation.txt, model/token/schema settings, suspect classification and 500-character truncation, Slack ordering and 3000 character truncation, organization checkpointing/resume/dedupe, tick date resolution, pause gate, fan-out and deterministic job options, chunk-sync context prefix and Markdown chunking, internal route auth/enqueue, and worker and scheduler registration in worker.ts. A PGlite test runs the real candidate SQL against org-local and UTC day bounds. Rails specs cover the enqueuer, the API2-first callback, the GoodJob fallback and the Flipper rollback, and the disabled cron entry. pnpm typecheck and scoped RuboCop are required before push.