Zendesk API2 cutover operations

Current state

API2 is the only owner of normal Zendesk source sync. ENG-2506 deleted the Rails Conversation::Sync::Zendesk synchronizer, the API2 shadow mode and schedulePostRailsProviderSync, the exhausted-retry Rails fallback, and the Rails enqueue-error fallback to SyncConversationJob. Rails gates SyncConversationJob, CompleteConversationSyncJob, EnqueueBotHandoffSummaryJob, ImportProviderConversationPartImagesJob and ProviderConversationSync::EffectsApplier completion batches for Zendesk with ZendeskRailsSyncRetired::RetiredPathError. There is no flag; rollback is a code revert deployed to Rails and API2 together. The behavior inventory, post-deploy US/EU outcome queries and residual risks live in the rollout tracker. api2_zendesk_source_sync is orphaned and its Flipper records can be deleted.

Webhook intake: Rails still receives Zendesk webhooks on api.rulebase.co and commits the coalesced zendesk_webhook_ticket_syncs row as durable intent, but no GoodJob is involved. ZendeskWebhookTicketSync.record_event! hands the row to POST /internal/zendesk_webhook_ticket_syncs/enqueue synchronously with run id rails-zendesk-<event id>; API2 applies the 30-second debounce, BullMQ retries, and pending-event continuation on the deterministic identity (job id = run id, deduplication id = <organization_id>_<row id> with BullMQ keepLastIfActive, so an event or continuation that lands while the row’s job is active becomes exactly one debounced successor instead of being dropped). A handoff that fails or times out is logged and left for the 5-minute recover-stranded-zendesk-webhook-ticket-syncs sweep, which re-arms any tenant/provider-fenced row pending for more than 20 minutes on the same identity. ProcessZendeskWebhookTicketSyncJob is a tombstone: enqueue raises, and perform only drains jobs serialized before the deploy by handing their row to API2.

See the Rails-to-API2 migration tracker for the current weighted progress, rollout inventory, and dated decisions.

Run the read-only comparator in production through Infisical; a direct ECS Exec invocation does not provide the application secrets required by API2:

AWS_PROFILE=AdministratorAccess-879381255132 \
  bin/aws/ecs-shell.sh api2-prod-us \
  "bash -lc 'infisical run --projectId=\$INFISICAL_PROJECT_ID --env=\$INFISICAL_ENV -- /app/node_modules/.bin/tsx /app/rulebase-api2/scripts/check-zendesk-parity.ts 1'"

Use scripts/check-zendesk-parity.ts <sample-size> for read-only parity audits; it compares API2’s plan against persisted rows and writes nothing. The sample size is applied independently to every active, non-discarded Zendesk organization data source, so the output reports activeZendeskSources and selected records across all sources. Fresh conversations are reported as skipped; use scripts/check-zendesk-parity.ts <sample-size> --force for a read-only comparison that bypasses freshness skipping.

The comparator samples only conversations matching organization_data_source_id = ?, source_deleted_at IS NULL, and external_created_at IS NOT NULL, ordered by external_created_at DESC. This matches the existing idx_conversations_source_external_created_active index. If a source has fewer eligible conversations than requested, the comparator reports the smaller sample and does not fall back to an unindexed query.

The comparator waits for Redis before making Zendesk requests and deterministically awaits its Redis and PostgreSQL cleanup on success and failure. Field parity reads Rails conversation_field_values joined to conversation_fields, matching planned nonblank values by stable conversation_fields.external_id. Values are compared using the exact JSON representation API2 persists; missing rows and different values are reported with the field name and label for diagnostics. Extra Rails rows for fields omitted from the plan are ignored because blank Zendesk values are skipped and existing rows are preserved.

The comparator exits non-zero when any configured mismatch-rate threshold fails. The JSON output includes checks for parts, authors, status, tags, assignee, provider status, work schedule, and canonical fields, plus normalization diagnostics. Unsupported Zendesk Change events include their fieldName so unsupported provider behavior can be classified without writing data.

Rollback is a revert of the ENG-2506 merge commit deployed to both services; there is no Rails fallback to fall back to. After a revert, watch sync failure rate, retry exhaustion, Zendesk 429s, part presence/type mismatches, and canonical field mismatches, and keep the outcome queries in the rollout tracker at zero Rails Zendesk sync jobs before re-landing.