Routine GoodJob workload families

Summary dispatch and expired-evaluation sweep

ENG-3000 removes three additional normal-path GoodJob classes. Every Conversation#summarize_later call now uses Api2ConversationSummaryEnqueuer.enqueue_or_later!: successful API2 acceptance creates no GoodJob row, while a proven delivery error logs and reports a warning before enqueuing the retained SummarizeConversationJob. The job class also drains work queued by older revisions.

API2 registers mark-expired-evaluations-failed-hourly at :05 UTC on the maintenance queue. Its job calls POST /internal/expired_evaluations/process, where Rails executes the same snapshot and QA-request timeout transitions used by the retained jobs and returns candidate/failure counts. BullMQ retries the authenticated request three times with exponential backoff. The two GoodJob cron entries are disabled by default.

For expiry rollback, set Flipper pause_api2_expired_evaluation_cleanup, then enable mark_expired_evaluations_as_failed and mark_expired_qa_evaluation_requests_as_failed in GoodJob. Re-drive once after the last missed hour; both operations are idempotent over already terminal records. Remove the pause switch, endpoint and retained jobs only after both regions show successful BullMQ runs and 24 hours of zero new normal-path rows.

ENG-2920  removes normal-path Rails/GoodJob ownership of the four remaining routine workload families in the September 8 queue snapshot: SyncJobs::SyncUnit21Job, ConversationUploads::CheckNewUploadsJob and the downstream ProcessDirectUploadJob path, SendDailyAgentEvaluationReportJob plus SendDailyAgentEvaluationReportForOrganizationJob, and FixCounterCachesJob. API2 owns the schedulers, durable execution, retries, and business behavior; Rails keeps only narrow authenticated /internal/* endpoints that execute model-bound work synchronously.

Contract

• Register one BullMQ job scheduler per family on the exact existing cadences: Unit21 every ten minutes, upload polling every thirty minutes, counter-cache repair every three hours, and daily agent evaluation reports at 04:00 UTC (yesterday) and 20:00 UTC (today).
• Enqueue per-unit work with deterministic job IDs (unit21-sync-source-<id>, conversation-upload-check-<id>, conversation-upload-process-<id>, daily-agent-eval-report-<org>-<date>-<email|no-email>-<bf|no-bf>, counter-cache-fix) so BullMQ dedupe preserves single-run concurrency and idempotent outbound delivery.
• Keep the Rails cron entries in config/initializers/good_job.rb but set enabled_by_default: false, with each entry naming its Flipper pause_api2_* kill switch; a code revert re-enables Rails cron without racing the API2 schedulers.
• Gate every API2 tick and worker behind the matching pause_api2_* check so the kill switch is effective even while schedulers remain registered.

Unit21 sync

• Select the same sources as SyncJobs::SyncUnit21Job: non-discarded OrganizationDataSource records whose data source is Unit21.
• Sync one source per BullMQ job through POST /internal/organization_data_sources/:id/sync_unit21, which runs the existing sync_unit21! path and returns the sync payload or the Rails error classification. Per-source failures are isolated in the tick and reach Sentry with the organization data source ID.

Conversation uploads

• Poll conversation_uploads rows that are pending inside the recent window, plus direct uploads that are failed or stuck in processing beyond the 30-minute threshold — matching CheckNewUploadsJob and the periodic-retry scan.
• Route pending rows by upload_type: presign rows go to the check job (legacy blob_key lookup, 30-minute expiry). direct rows are never sent through check_status — ConversationUpload#check_status now returns early for them as a belt-and-braces guard. A direct row whose active_storage_attachments file is still present and that is older than the five-minute controller grace window is queued as a deterministic conversation-upload-process-<id> job (which collapses onto an already queued controller hand-off); a direct row with no retained file is left pending for the durable-delivery reconciler rather than expired.
• Check uploads through POST /internal/conversation_uploads/:id/check; process direct uploads through .../process_direct and record transient retries through .../retry_direct; mark terminal failures through .../mark_failed.
• Preserve the Rails retry classification: retryable errors rethrow inside the five-attempt budget and use the polynomially_longer equivalent backoff (attemptsMade^4 * 1000 + 2000); non-retryable errors and exhausted attempts mark the upload failed via the same model method and capture the Sentry exception with the Rails fingerprint.
• ConversationUploadsController#create and any other direct-upload producer enqueue the API2 process job via Api2ConversationUploadEnqueuer instead of ProcessDirectUploadJob.perform_later; the upload path never depends on GoodJob analysis jobs.
• XCALLY completion (ENG-2948): once Xcally#process has durably committed the recording attachment, it hands conversation completion to API2 through POST /internal/provider_conversation_sync_completions/enqueue (Api2ProviderConversationSyncCompletionEnqueuer, PostHog flag api2_conversation_upload_sync_completion). The BullMQ provider-conversation-sync-completion job is pinned by sync_run_id = xcally-upload-<conversation_upload.id> and the route keeps the completed job for a 7-day retention window so the pinned id still dedupes a retry that arrives after the first run finished.
• Single-execution identity: dispatch is claimed durably in public.outbound_delivery_claims under conversation-upload/sync-completion/<upload_id> before the enqueue call. The API2 hand-off and the GoodJob fallback share that claim — whichever dispatch lands marks it delivered, and a retried upload processor pass dispatches nothing. ActiveStorage::AnalyzeJob still runs in Rails but is no longer coupled to completion — the retired batch’s wait on it was implicit capture, not a required ordering for transcription discovery.
• Failure contract follows the claim rules: a proven rejection (API2 answers non-2xx) takes the GoodJob batch fallback and marks the claim, with a warning log and a Sentry warning. An ambiguous failure (timeout or transport error that may follow an accepted BullMQ job) must not fall back — the claim is left unresolved and surfaced to Sentry for human reconciliation per ### Resolving uncertain delivery claims below. Resolve an uncertain key by checking whether a provider_conversation_sync_completion_xcally-upload-<id> BullMQ job exists or ran; if not, release! the claim and re-drive the upload. Flag off (api2_conversation_upload_sync_completion) restores the legacy batch completion exactly as before; there is no normal-path CompleteConversationSyncJob enqueue once the flag is on.

Native direct-upload processing (ENG-3248, default off)

PostHog flag api2_conversation_upload_native_processing (owner: platform; created 2026-09-18; review/remove by 2026-12-17; objective: zero Rails process_direct/retry_direct calls). With it off the conversation-upload-process job keeps calling Rails exactly as above. With it on for an organization, direct uploads whose source is xcally or outreach_kaia are processed by src/lib/conversation-uploads/native-processing/:

• run.ts mirrors ConversationUpload#process_direct_upload / #retry_direct_upload!: periodic-retry eligibility (MAX_PERIODIC_RETRIES, minimum interval, stuck-processing and stored-source-file checks), the processing → completed status transitions, and clearing the periodic_retry_count / last_processing_error* metadata on success. Failures raise ConversationUploadProcessingError(retryable, errorClass) and flow into the existing worker classification (mark_failed parity for terminal errors, five-attempt backoff for retryable ones).
• xcally.ts / outreach-kaia.ts reproduce the Rails processors: the same organization-data-source, employee (email → integration profile → name), client (phone / email / name), conversation (organization_data_source_id
  • external_id, external_type IS NULL), part and call-detail identities, membership linking, agent-replacement reconciliation with QA-eligibility and conversation-team refresh, prefix_id and thread creation. Metadata, media and ZIP validation errors are terminal; storage, ffmpeg spawn and enqueue failures are retryable.
• Media (GSM 8 kHz mono → WAV, MP4 → WAV, ffprobe duration → duration_ms and handling_time) runs on the API2 image, which now ships ffmpeg. Downloads, conversion and object uploads happen outside any transaction; the attachment is inserted under a short FOR UPDATE on the exact call-detail row, re-checked for an existing recording, and an orphaned WAV object is deleted if that insert fails. Non-GSM recordings share the upload’s Active Storage blob, as in Rails.
• Completion reuses the native upload-driven sync completion (ENG-3242): the sync_completion delivery row is inserted in the persistence transaction with sync_run_id = xcally-upload-<upload_id> (reopened when failed), the BullMQ job is enqueued after commit and after the recording attachment, and the row moves to accepted; the completion worker marks it completed. An enqueue failure marks the row uncertain and retries the job.

Rollback is the flag: turning it off returns the organization to the Rails processors with no deploy. Replays are safe in either direction because both sides use the same external-id identities and the same delivery keys.

Durable Rails→API2 delivery ledger (ENG-2948, default off)

PostHog flag api2_conversation_upload_durable_delivery (owner: platform; created 2026-09-13; review/remove by 2026-12-12; objective: every direct upload hand-off and XCALLY completion hand-off is recoverable after a commit-then-timeout). With it off nothing below runs: the controller calls Api2ConversationUploadEnqueuer synchronously and the XCALLY processor uses the outbound_delivery_claims path above. With it on:

• ConversationUploadsController#create computes conversation_uploads.idempotency_key = direct:sha256(source | canonical metadata | file bytes) and reuses the existing tenant row (unique index index_conversation_uploads_on_org_id_and_idempotency_key) on a client retry, so a retried POST after a lost response creates no second upload, attachment, or hand-off.
• One conversation_upload_deliveries row per (conversation_upload_id, kind) is inserted in the same transaction as the upload (kind process, delivery_key = conversation-upload/process/<upload_id>) or as the created conversation (kind sync_completion, delivery_key = conversation-upload/sync-completion/<upload_id>, sync_run_id = xcally-upload-<upload_id>). The HTTP call happens after commit, so a Redis/HTTP failure leaves a discoverable row.
• States: pending (Rails-owned; a proven API2 rejection parks the row until next_attempt_at, after which the API2 reconciler picks it up; five Rails rejections are terminal) → sent (Rails saw a 2xx) / uncertain (timeout or transport error; never blindly resent by Rails) → accepted (API2 wrote the ledger after the BullMQ add succeeded, so a lost response still proves acceptance) → completed (process worker saw Rails report completed; completion worker finished complete_conversation_sync) or failed (terminal: Rails rejected five times, the Active Storage file is gone, the target conversation was deleted, or the replay budget of three is exhausted). owner records which side is expected to move the row.
• Reconciliation runs inside the 30-minute conversation-upload tick (reconcileConversationUploadDeliveries) over rows in sent, uncertain, or accepted that have not moved for ten minutes. Order per row: (1) business outcome from the durable Rails tables — process rows with conversation_uploads.status = 'completed' and sync_completion rows whose conversation has last_synced_at set are marked completed without touching Redis; (2) a live deterministic BullMQ job (conversation-upload-process-<id> / provider_conversation_sync_completion_xcally-upload-<id>) in waiting/delayed/active/prioritized means wait; (3) only when the job is missing or finished-without-outcome and the file/conversation still exists does the tick take an optimistic claim (UPDATE ... WHERE state = $old AND replay_count = $n) and re-add the job, replacing a finished job of the same id. A sync_completion row is only replayed once its upload is completed (Rails completes the upload after the recording is attached and the hand-off dispatched); while the upload is pending/processing the Rails processor still owns it, and a failed/expired upload marks the row failed until the periodic retry reprocesses it and ConversationUploadDelivery.record! reopens the row; (4) missing file / missing conversation / replay_count >= 3 → failed with last_error.
• API2 ledger marks that follow an already-durable effect (job added, upload completed, completion run finished) are best-effort: a failed mark is logged to Sentry and left for the reconciler, never surfaced as a 5xx or a failed job, so Rails cannot read a queued job as a rejection and a completed upload cannot be flipped to failed. Rails-side dispatch holds a two-minute lease (last_attempted_at) so a client retry racing an in-flight send does not double-post.

Read-only validation queries (both regions):

-- Pending direct uploads that still hold a file: must be picked up by the
-- scanner, never expired. Anything older than ~45 minutes here is a bug.
SELECT u.id, u.organization_id, u.status, u.created_at
FROM conversation_uploads u
WHERE u.upload_type = 'direct'
  AND u.status = 'pending'
  AND EXISTS (SELECT 1 FROM active_storage_attachments a
              WHERE a.record_type = 'ConversationUpload'
                AND a.name = 'file' AND a.record_id = u.id)
ORDER BY u.created_at;
 
-- Direct uploads expired by the presign path after the fix ships: must be 0.
SELECT COUNT(*) FROM conversation_uploads
WHERE upload_type = 'direct' AND status = 'expired'
  AND updated_at > '<deploy timestamp>';
 
-- Ledger rows still in flight past the reconcile grace window.
SELECT d.id, d.kind, d.state, d.owner, d.attempts, d.replay_count,
       d.conversation_upload_id, d.conversation_id, d.sync_run_id,
       d.last_error, d.updated_at
FROM conversation_upload_deliveries d
WHERE d.state IN ('pending', 'sent', 'uncertain', 'accepted')
  AND d.updated_at < now() - interval '10 minutes'
ORDER BY d.updated_at;
 
-- Terminal failures to triage by hand (no automatic replay).
SELECT id, kind, conversation_upload_id, last_error, failed_at
FROM conversation_upload_deliveries
WHERE state = 'failed'
ORDER BY failed_at DESC;
 
-- Completion delivered but transcription/sync never landed: must be 0.
SELECT d.id, d.conversation_id
FROM conversation_upload_deliveries d
JOIN conversations c ON c.id = d.conversation_id
WHERE d.kind = 'sync_completion' AND d.state = 'completed'
  AND c.last_synced_at IS NULL;

Deployment, canary, acceptance, rollback:

1. Server first. Deploy the Rails migrations (ledger table and idempotency_key — additive, transactional, no backfill) and API2 with the flag off. Readiness: the scanner’s next tick logs skippedDirect/reconciled counts, the first validation query returns only rows younger than the grace window, and the ledger table is empty.
2. Canary. Enable api2_conversation_upload_durable_delivery for one organization that already has api2_conversation_upload_sync_completion on. Gates over 24 hours: every new ledger row reaches completed; zero failed; zero uncertain rows older than the grace window; the last_synced_at IS NULL query is 0; no duplicate conversation_uploads for one idempotency_key; CompleteConversationSyncJob GoodJob rows for that org stay at 0.
3. Acceptance. Repeat the gates region-wide, then widen the flag. The legacy owners (synchronous controller enqueue, outbound_delivery_claims and the GoodJob completion batch) stay in code until acceptance; only then is the flag baked in and the claim path retired in a follow-up.
4. Rollback. Turn the durable flag off: new uploads take the legacy path immediately. Rows already in the ledger keep reconciling — the reconciler only acts on durable ledger/business state and is safe to leave running; pause_api2_conversation_upload_checks stops the whole tick (scanner and reconciler) if needed, and api2_conversation_upload_sync_completion off restores GoodJob completion as documented above. No production rows are repaired or replayed as part of shipping this change; historical outbound_delivery_claims are resolved by the manual procedure below.

Zero-new-GoodJob bake for XCALLY completion

After the flag is enabled for an organization in a region, new CompleteConversationSyncJob rows attributable to XCALLY uploads must stay at zero (retained rows inside the 12-hour GoodJob cleanup window are historical):

SELECT COUNT(*)
FROM good_jobs
WHERE job_class = 'CompleteConversationSyncJob'
  AND queue_name = 'default'
  AND created_at > now() - interval '12 hours';

Attribute any surviving rows to conversations.organization_data_source_id → data_sources.key = 'xcally' via the serialized conversation GlobalID on the job arguments; the converse assertion is that BullMQ provider-conversation-sync-completion accepted each hand-off. A warning log line XCALLY upload completion API2 hand-off failed marks a proven-rejection fallback; ... hand-off outcome is uncertain marks an unresolved claim that needs reconciliation. The upload-driven completion job now claims the same native completion steps as provider syncs and skips the Rails complete call when all completion keys are native; the rollback is the shorter PROVIDER_SYNC_NATIVE_EFFECTS list. Both warning outcomes also surface in Sentry.

Daily agent evaluation reports

• Resolve yesterday/today in each organization’s own time zone, select the same reportable organizations (non-demo orgs with a fixed-per-period ticket rule or no ticket rules), and enqueue one per-org BullMQ job with sendEmail: false and queueBackfill: true from the cron ticks.
• Row collection is org-gated: with PostHog api2_daily_agent_evaluation_report_collect on for an organization, the tick’s pre-collection and the per-org job run the ported QAEvaluationReport::RowCollector natively (one transaction, statement_timeout = 5min, no Rails call, no CollectRowsJob row). With the flag off — or PostHog unreachable — the per-org job collects rows through POST /internal/daily_agent_evaluation_reports/collect as before, and that endpoint plus the collector and job class stay until bake. A native failure is never retried through Rails inside an attempt; BullMQ retries replay the same engine so a flag flip mid-incident cannot double-collect.
• The per-org job then queues the backfill preview through .../backfill_preview (which returns the existing QAEvaluationBackfillRun when one is active, preserving backfill-preview-before-notification ordering and active-run recovery), posts the Slack report, and only then sends email. Each completed step is persisted in the BullMQ job data so a retry re-runs only the remaining steps — the duplicate-delivery protection GoodJob could not offer.
• The tick job resolves and persists every org’s coverage date (resolvedDates) before the first external call, so a BullMQ retry across a tenant’s local midnight cannot shift today/yesterday coverage.
• Every outbound send is claimed durably in public.outbound_delivery_claims before the irreversible call: email deliveries carry a delivery_key the Rails endpoints honor, and the Slack post claims the key API2-side. A proven non-delivery — the provider rejected the request with a response (e.g. a Slack non-2xx) — voids the claim (OutboundDeliveryClaim#release! on Rails, releaseOutboundClaim in API2) so the retry re-claims and re-sends instead of stalling on a permanently uncertain key. Ambiguous outcomes — a deliver_now raise or a dropped fetch response can both follow a landed delivery — keep the claim unresolved: uncertain is never resent automatically and is surfaced to Sentry for manual resolution instead.
• Preserve the per-org Slack payload section limit and the combined email: the cron tick collects rows for every org and sends one email with CSV attachments for the orgs that produced rows.
• Admin-triggered and post-backfill report sends go through Api2DailyAgentEvaluationReportEnqueuer with source: 'admin' | 'backfill' instead of the legacy job classes.

Counter caches

• Run all seven repair steps (assignment_queue_memberships, group_employees, employees, credit_usages, conversations, sheet_rows, evaluation_sampling_rule_counters) through POST /internal/counter_caches/fix, which executes the existing FixCounterCachesJob step implementations.
• Hold a session-level advisory lock (pg_try_advisory_lock(hashtextextended('fix_counter_caches', 0))) on a dedicated connection so no idle transaction is held across the seven Rails HTTP repair calls and the same lock is visible to the Rails FixCounterCachesJob rollback path. A contended API2 run re-parks itself (DelayedError, five minutes) instead of being consumed; a contended Rails run raises LockContentionError and retries via retry_on rather than reporting success.
• Persist completedSteps in the job data so a retry resumes after the last completed step instead of re-running bounded repairs.
• Employee#merge_into! enqueues the repair through Api2CounterCacheFixEnqueuer; an enqueue failure is logged and reported to Sentry without failing the merge.

Counter-cache regional rollout

Create the PostHog flag api2_counter_cache_native_repairs with a release condition for one deployment region (region = eu-central-1 or region = us-east-1) at 100%, with no percentage cohort. Wait for one three-hour tick; verify counter cache fix completed with zero Internal::CounterCachesController#fix calls in that region and unchanged counter parity. Then add the second region and repeat the same verification. Rollback by disabling the PostHog flag (the next run returns to Rails) or by enabling the Flipper pause_api2_counter_cache_repairs switch.

Rollback and rollout assertion

The kill switches pause_api2_unit21_sync, pause_api2_conversation_upload_checks, pause_api2_daily_agent_evaluation_reports, and pause_api2_counter_cache_repairs stop the API2 side immediately; reverting the enabled_by_default: false lines restores the Rails cron entries. For XCALLY completion, api2_conversation_upload_sync_completion off is the non-racing revert: uploads processed while the flag is off complete through the GoodJob batch exactly as before. Every worker re-checks its flag at execution time, so queued BullMQ jobs report skipped/paused and complete rather than racing the reverted Rails path. After rollout, assert that the six named classes receive zero new normal-path GoodJob rows — retained rows inside the 12-hour cleanup window are historical, not new traffic.

Re-driving work skipped while paused

Skipped BullMQ jobs complete as skipped — nothing resumes them automatically after rollback. Re-drive each family explicitly once Rails is re-enabled:

• Daily reports — for a same-day rollback, enqueue the aggregate job with the sentinel that was skipped so per-org dates still resolve in each tenant’s timezone: SendDailyAgentEvaluationReportJob.perform_later('yesterday') (04:00 UTC tick) or perform_later('today') (20:00 UTC tick). For a rollback after the local day has moved on, pass literal dates instead — either the aggregate perform_later('YYYY-MM-DD') (one date for all orgs) or, for exact per-tenant coverage, one SendDailyAgentEvaluationReportForOrganizationJob.perform_later(org, 'YYYY-MM-DD', false) per reportable org. The API2-side equivalent is one POST /internal/daily_agent_evaluation_reports/enqueue per org with an explicit date.
• Counter caches — run FixCounterCachesJob.perform_later; a contended run retries on the shared lock rather than reporting a false success.
• Unit21 sync and conversation uploads — the next cron tick covers them; both are idempotent sweeps, so no manual catch-up is needed beyond re-enabling the Rails cron entries.

Resolving uncertain delivery claims

A claim row with no delivered_at means one send attempt’s outcome could not be confirmed. Find them with OutboundDeliveryClaim.unresolved (Rails) or SELECT * FROM outbound_delivery_claims WHERE delivered_at IS NULL, and resolve each by hand rather than re-sending blindly:

1. Check whether the delivery actually happened — the monitoring Slack channel for slack keys, the mail provider’s log for email/combined keys (the key encodes channel, job id, and enqueue timestamp).
2. If it landed: OutboundDeliveryClaim.find_by(delivery_key:).mark_delivered!. The next retry sees already_sent and skips.
3. If it did not: OutboundDeliveryClaim.find_by(delivery_key:).release! (or delete the row), then re-drive the caller — the BullMQ job’s retry re-claims and re-sends.

Because senders release their claim only on proven non-delivery (a provider rejection response), uncertain appears for genuinely ambiguous sends — raised sends, lost responses, unwritable checkpoints — and stays rare; each occurrence surfaces in Sentry as a ... delivery outcome uncertain warning.

Verification

Focused API2 unit tests cover tick fan-out, failure isolation, job-ID dedupe, report date resolution and step ordering; PGlite tests cover the real candidate-selection SQL, reportable-organization eligibility, and the counter-cache advisory-lock run. Rails request specs cover every new /internal/* endpoint (auth, not-found, success, retry, and failure marking) and every rewired producer’s API2 enqueue. API2 typecheck and scoped RuboCop are required before push.