Zendesk daily ticket reconciliation

ENG-2931  moves the 00:00 UTC SyncZendeskTicketsJob sweep and its ImportZendeskConversationJob fan-out from Rails/GoodJob to the API2 zendesk-ticket-reconciliation BullMQ queue. Normal Zendesk webhook ingestion already belongs to API2 (ENG-2692, ENG-2917); the daily sweep stays as the safety net for tickets whose webhook never arrived or failed terminally, and it is not narrowed until webhook coverage has a measured miss rate.

Why

Rolling 12h EU+US GoodJob evidence ending 2026-09-09 10:13 UTC showed ImportZendeskConversationJob at 5,029 executions for 2,738 unique jobs with 2,291 rate-limit attempts, all produced by the midnight sweep and with no real backlog. The Rails job retried ImportRateLimitedError up to 60 times with a random wait, so one throttled tenant amplified into thousands of executions.

Contract

• One scheduler, zendesk-ticket-reconciliation, fires a scan job at 0 0 * * * (the existing cadence). The scan lists non-discarded Rails Zendesk sources joined to an API2 organization, skips unmapped tenants with a log line, and enqueues one source job per mapped source. An enqueue failure for one source is recorded and does not stop the others.
• A source job re-checks the kill switch, then walks /api/v2/incremental/tickets.json from last_synced_at (24h fallback on the first run, matching Rails) with a 10-second gap between pages and a 200-page cap. Each page’s ticket jobs are enqueued before its checkpoint is written; last_synced_at only moves forward (update_last_synced_at_checkpoint semantics), and the final watermark is set to the scan start only after end_of_stream. A capped or failed source keeps its window so the next sweep resumes from the last delivered page.
• A ticket job fetches the ticket, upserts the Rails-compatible conversation through the shared zendesk/ticket-import.ts module (same code as webhook ingestion), and enqueues the source-owned provider sync with a deterministic run ID. A 404 marks the ticket obsolete and completes.
• Deduplication IDs are zendesk_reconciliation_source_<org>_<ods> and zendesk_reconciliation_ticket_<org>_<ods>_<ticketId>, so a webhook and a sweep for the same ticket, or two overlapping sweeps, produce one live job.
• Ticket jobs have 3 attempts, source/scan jobs 5, exponential backoff from 60s. A Zendesk 429 does not consume an attempt: the worker calls moveToDelayed for Retry-After (default 60s) and rethrows DelayedError. Worker throughput is capped at 30 jobs / 10s on top of the shared per-connection limiter.
• Failed ticket jobs are retained (removeOnFail: false) for manual redrive; scan/source failures are removed after logging because the next sweep retries them from the retained window.
• The Rails sync_zendesk_tickets cron stays in good_job.rb with enabled_by_default: false. SyncZendeskTicketsJob and ImportZendeskConversationJob are a documented recovery-only shim.

Kill switch and rollback

Flipper pause_api2_zendesk_ticket_reconciliation is checked at scan, source, and ticket execution time; paused jobs complete as paused without touching Zendesk or last_synced_at. Rollback order:

1. Enable pause_api2_zendesk_ticket_reconciliation in the affected region.
2. Only then re-enable the sync_zendesk_tickets cron in the GoodJob dashboard (or revert enabled_by_default: false). Never run both owners: both advance last_synced_at on the same row.
3. Leave the API2 worker and queued BullMQ jobs alone; they drain as paused. Do not delete the scheduler unless the migration is being abandoned.

Work skipped while paused is covered by the next Rails sweep because the watermark did not move.

Bake criteria

Deploy the API2 worker/scheduler and the Rails revision together (overlap is safe: conversations are unique per external identity and provider sync deduplicates). Then observe two midnight ticks in US and EU and require:

• every mapped source’s last_synced_at advanced past the tick, and each unmapped source is explicitly listed as retained Rails-only coverage;
• zero new SyncZendeskTicketsJob / ImportZendeskConversationJob GoodJob rows created after the deploy;
• no capped source results, no growing waiting/delayed backlog on zendesk-ticket-reconciliation, delayed count returning to zero within the hour after the tick;
• no new duplicate Zendesk external identities and healthy downstream provider syncs;
• ticket failed-set count is zero or each entry is explained.

Observation queries

Rails (each region):

SELECT ods.id, ods.organization_id, ods.last_synced_at
FROM organization_data_sources ods
JOIN data_sources ds ON ds.id = ods.data_source_id
WHERE ds.key = 'zendesk' AND ods.discarded_at IS NULL
ORDER BY ods.last_synced_at NULLS FIRST;
 
SELECT job_class, count(*) AS jobs, sum(executions_count) AS executions
FROM good_jobs
WHERE job_class IN ('SyncZendeskTicketsJob', 'ImportZendeskConversationJob')
  AND created_at > now() - interval '24 hours'
GROUP BY job_class;

The second query must return no rows after the deploy. API2: open /admin/queues, select zendesk-ticket-reconciliation, and read waiting, delayed, failed counts and the scheduler’s next time. Worker logs are tagged zendesk-ticket-reconciliation with status, pageCount, ticketCount, cursorBefore, and cursorAfter per source.

Recover a failed ticket

The watermark records queue delivery, not import success. A terminally failed ticket keeps organizationId, organizationDataSourceId, and ticketId in the failed set. With production authorization: match the job to its final-attempt log/Sentry event, fix the credential/provider/persistence cause, and retry that individual job from /admin/queues. Do not reset last_synced_at, edit the job identity, or purge the failed set to clear an alert. For a source whose outage predates the retained window, run the Rails shim manually for that source only while the API2 flag is paused, then unpause.

Verification

src/lib/zendesk/ticket-reconciliation.test.ts covers source mapping and skips, per-source enqueue isolation, 24h fallback, enqueue-before-checkpoint, pagination spacing, page cap retention, non-advancing cursors, kill switch at every kind, idempotent upsert plus provider-sync enqueue, 404 handling, and Retry-After deferral. fetch.test.ts covers the incremental export URL and 429 mapping; reconciliation-worker-wiring.test.ts asserts the dedupe/retention options, scheduler registration, and the paused Rails cron. Rails GoodJob configuration specs assert the disabled cron entry.