Rollouts and flags

Temporary migration flags select an owner while a new API2 path proves parity. They are not permanent configuration. After a clean global bake window, remove the selection branch in code and deploy it. Delete the unused flag only after the exact bake-in revision is verified live and healthy in both US and EU.

Native public media routes

pause_api2_public_media_routes is a Flipper kill switch owned by platform engineering. It was created 2026-09-17, has safe state off, and should be removed by 2026-12-16. Roll it out and roll it back per region because Flipper state is stored in each region’s database. When enabled, public media requests use the retained Rails fallback where available.

Native Zendesk sidebar

pause_api2_zendesk_sidebar is a Flipper kill switch owned by platform engineering. It was created 2026-09-17, has safe state off, and should be removed by 2026-12-16. Roll it out and roll it back per region. When enabled, /zendesk-app/sidebar requests use the retained Rails fallback where available. See the native Zendesk sidebar runbook.

Program-wide GoodJob traffic gate

Use the GoodJob traffic baseline when evaluating queue retirement. The GoodJob dashboard retains only 12 hours of completed rows, so its “today” count cannot prove a full-day trend or a zero-traffic bake. Use equal 24-hour regional windows for rollout decisions and distinguish unique jobs from performed attempts when retries amplify traffic.

For each migrated class, require no new normal-path GoodJob enqueue in either region, healthy BullMQ acceptance and drain, no duplicate business effects, and no unapproved delivery fallback or callback to Rails. Webhook ingress is not complete while the provider still depends on Rails business behavior: during the temporary relay, Rails must acknowledge only after API2 durably accepts the raw event; after observation, change the provider URL to API2 and bake out the relay. Rails-operational Active Storage, CloudWatch, autoscaling, and heartbeat jobs are tracked for service deletion but excluded from application workload ordering.

The September 8 implementation-readiness milestone does not waive these gates. Regional cutovers remain due September 11, followed by at least 72 hours of clean final operation on September 12–14 before approved Rails/GoodJob retirement on September 15.

September 10 Kuda gate cohort and ENG-3000 cutover

Kuda is the first production cohort for the three organization-scoped gates needed to exercise the remaining native paths. The PostHog flags are active at 100% only where organization $group_key = kuda-pilot; a control organization evaluates false:

Disable an affected flag to return only that organization’s next execution to the Rails fallback. Do not widen the cohort until a natural Kuda report burst and relationship event show healthy BullMQ acceptance, correct persistence and no new corresponding GoodJob row.

ENG-3000 is revision-selected rather than organization-selected. Summary handoff uses direct API2 enqueue with a retained GoodJob fallback only after a proven delivery failure. Expiry cleanup runs hourly at 5 * * * * UTC on the API2 maintenance queue and calls Rails synchronously; Flipper pause_api2_expired_evaluation_cleanup pauses the BullMQ execution. Roll back expiry ownership in order: enable the pause switch, enable both retained Rails crons, and re-drive the missed hour if needed. A shared PostgreSQL advisory lock serializes the API2 endpoint and both retained Rails jobs, so a re-drive that meets an in-flight sweep fails and retries instead of duplicating terminal effects. Roll back summary ownership by reverting the revision. Gate credit requires successful regional BullMQ expiry runs, zero new normal-path rows for both expiry jobs and summaries, and no summary handoff warning or duplicate effect.

Rulebase webhook event forwarding

One per-organization PostHog flag, api2_rulebase_webhook_events (safe state off; owner platform engineering; created 2026-09-09; review/remove by 2026-12-08), gates the conversation_assigned hand-off in ConversationAssignment#forward_assignment_webhook_to_api2_later. Roll out per region after a clean day on early tenants; the flag fails closed to GoodJob when PostHog is unreachable, and while it is off ForwardRulebaseWebhookJob remains the producer.

Baseline: the rolling-hour evidence on 2026-09-09 showed 3 ForwardRulebaseWebhookJob executions in the US and 0 in the EU, all successful — expect roughly tens of rows per day, not thousands.

Bake gate: in each region, 24 hours of zero new normal-path ForwardRulebaseWebhookJob rows, healthy acceptance and drain on the rulebase-webhook-events BullMQ queue, near-zero EnqueueApi2RulebaseWebhookEventJob bridge rows, no Api2RulebaseWebhookEventEnqueuer delivery warnings in Sentry, and no duplicate assignment side effects. Roll back in the order: turn the flag off → the next committed assignment falls back to ForwardRulebaseWebhookJob → let retained EnqueueApi2RulebaseWebhookEventJob rows drain or discard them (each carries the deterministic event_id, so a drained retry cannot duplicate an already-accepted BullMQ job). After 24 clean hours in both regions, remove the flag and the GoodJob fallback branch; the ForwardRulebaseWebhookJob class stays until its retained rows drain.

QA post-processing and citation repair

Two Flipper kill switches (safe state off) own the revert for the QA post-processing package: pause_api2_qa_citation_review makes the API2 05:00 UTC tick complete as paused, after which the disabled review_ai_check_citations GoodJob cron can be enabled in the dashboard without racing; good_job_qa_criterion_result_chunk_sync (global or per organization actor) routes the criterion-result after_commit back to SyncQAAgentEvaluationScorecardCriterionResultChunksJob. Both were created 2026-09-09, owned by platform engineering, and must be removed with their fallback branches by 2026-12-08. Deploy with both flags off in each region.

Bake: two consecutive clean ticks per region with zero failed organization jobs, 24 hours of zero new normal-path GoodJob rows for the three migrated classes (fallback rows are visible via the falling back to GoodJob warning), no uncertain qa-citation-review:slack:* delivery claims, at most one Slack post per organization per day, and chunks_synced_at advancing for eligible criterion results. Roll back in the order pause API2 → enable Rails cron → re-drive missed dates with ReviewAICheckCitationsJob.perform_later('YYYY-MM-DD'). After the bake, delete the fallback branch, the two citation job classes, the chunk-sync job class, the cron entry and both flags in one revision.

Daily report row collection gate

One per-organization PostHog flag, api2_daily_agent_evaluation_report_collect (safe state off), selects the API2-native row collector for the daily agent evaluation report in place of the Rails POST /internal/daily_agent_evaluation_reports/collect endpoint that enqueues QAEvaluationReport::CollectRowsJob on GoodJob latency_30s. Roll out per region to individual organizations after the deploy, then widen; the flag fails closed to the Rails endpoint when PostHog is unreachable. There is no automatic Rails retry inside an attempt — a native failure raises through the org job’s BullMQ retry on the same path — so flipping the flag off mid-incident is the rollback, and it cannot double-run a collection already in flight. Owner: platform engineering; created 2026-09-09; review/remove by 2026-12-08.

Gates for bake credit: in each region, the 04:00 and 20:00 UTC ticks enqueue all reportable org jobs; daily-agent-evaluation-report BullMQ jobs complete with healthy success/failure counts and no growing backlog; row parity holds on sampled organizations against the Rails collector; zero new normal-path QAEvaluationReport::CollectRowsJob rows for 24 hours (retained rows inside the 12-hour window are historical); and database load around both scheduled bursts stays bounded — the report worker’s concurrency: 3 and the 5-minute statement_timeout carry the GoodJob-era limits. Rollback order: disable the PostHog gate for the affected organizations (the next BullMQ retry or the next org job resumes the Rails path), and only if API2 is the problem, re-drive missed dates with POST /internal/daily_agent_evaluation_reports/enqueue or the retained Rails perform_later calls documented in the routine workload runbook. After the bake, delete the selection branch and flag; the Rails collector, endpoints, and CollectRowsJob class stay until the retained rows drain and the combined bake for the report package closes.

Relationship construction producer hand-off

Two per-organization PostHog flags gate the Rails commit callbacks that enqueue relationship construction: api2_conversation_relationships covers Conversation subject/create and ConversationFieldValue string commits (one conversation-wide BullMQ pass per commit, keyed by a per-commit change key), and api2_conversation_part_relationships covers reference-bearing ConversationPart creates (one BullMQ job per reference kind). Roll out per region at 100% after a clean day on early tenants; the flags fail closed to GoodJob when PostHog is unreachable, and an explicit API2 rejection falls back to CreateConversationRelationshipsJob / CreateConversationPartRelationshipsJob automatically.

Gates for bake credit: no new normal-path GoodJob rows for the two job classes in either region’s retained window, healthy acceptance and drain on provider-conversation-field-value-link plus the three part link queues, near zero EnqueueApi2Conversation*RelationshipsJob bridge rows, and no duplicate thread items or link writes. The PROVIDER_SYNC_NATIVE_EFFECTS hand-back is independent (via_api2: false): a kind returned to Rails executes on GoodJob no matter the flag state. After 24 clean hours in both regions, remove the flags and the GoodJob branches; the GoodJob job classes stay until their retained rows drain. Flags created 2026-09-09; review/remove by 2026-12-08. See the relationship construction runbook.

Knowledge Library sync ownership

The Knowledge Library pipeline is revision-selected: deploying the candidate makes API2 the owner of the daily scan, provider sync, crawl, document sync and chunking for every non-Intercom, non-session-token-Scribe source, and turns every Rails producer into an API2 hand-off. The Rails cron sync_knowledge_base_sources ships enabled_by_default: false. Flipper pause_api2_knowledge_base_sync is the kill switch (safe state off; owner engineering; created 2026-09-09; review/remove by 2026-12-08): while on, Rails producers enqueue the retained GoodJob jobs directly and API2’s tick and workers complete queued jobs as paused instead of running the native providers. Provider execution is native to API2; Rails keeps only the image-rehosting endpoint plus its provider classes for the rollback path.

Bake gate: in each region, one 01:00 UTC API2 scan completed, sources’ last_synced_at advancing, zero new normal-path rows for the six *KnowledgeBase*Job classes after the deploy timestamp (retained 12-hour rows are historical), zero Api2KnowledgeBaseSyncEnqueuer::DeliveryError fallback warnings, no duplicate document_chunks sets, no cross-tenant Sentry events, bounded knowledge-base-sync/knowledge-base-chunks queues, and 24 clean hours. Rollback order: enable the flag, enable the GoodJob cron, then re-drive skipped work once the flag is off again (next tick, manual runKnowledgeBaseSyncTick(), or KnowledgeBaseSyncDispatch.sync_source). Queries and re-drive commands are in the Knowledge Library sync runbook.

Weekly agent performance reports ownership

Moves the weekly agent performance report pipeline to native API2/BullMQ — the daily 08:00 UTC generation tick, Monday 09:00 UTC preview tick with the 30-minute auto-send, per-organization report preparation, native OpenAI content generation, citation persistence, and Resend delivery all run in api2; no Rails /internal/* domain endpoints execute the work. The kill switch pause_api2_agent_performance_reports (Flipper, owner platform-eng, created 2026-09-09, review by 2026-12-09) stops API2 ticks and org workers; content jobs do not check it so a pause cannot strand pending reports. The two retired crons stay registered with enabled_by_default: false as the non-racing revert path.

Deploy API2 and Rails from the same revision: BullMQ schedulers register at worker boot while Rails stops producing the retired classes. Confirm in both regions: exactly two registered schedulers (agent-performance-reports-generate, agent-performance-reports-preview), real per-org generate-organization and content jobs completing to completed/failed report statuses, no new normal-path GoodJob rows for GenerateWeeklyAgentPerformanceReportsJob, GenerateWeeklyAgentPerformanceReportsForOrganizationJob, GenerateAgentPerformanceReportContentJob, SendAgentPerformanceEmailsPreviewJob, and SendAgentPerformanceReportEmailsForOrganizationJob, and on Mondays one claimed preview/send per report via public.outbound_delivery_claims. Pre-cutover EU+US evidence (rolling 12h ending ~2026-09-09 10:13 UTC): 53 org-fan-out and ~52 content executions, ~107 for the theme. Hold 24 clean hours — at least one daily generate tick and one Monday preview/auto-send — before ownership credit.

Rollback: enable the kill switch, then re-enable the two crons in the GoodJob dashboard or revert the enabled_by_default: false lines. Report rows are API2- and Rails-compatible (same statuses, same generate_later handoff), so the Rails path resumes without duplicate reports; the emails_already_sent_for_period gate prevents a second send for a delivered period. Re-drive skipped work through the documented enqueue* routes rather than manual perform_later. See the weekly agent performance reports runbook.

Native QA conversation-part caption advancement

ENG-2885 is a flagless ownership refinement inside the already-selected API2 QA request worker. For a leased caption_images stage, API2 processes one eligible conversation-part image and records its attempt before the next advance. Attachment work falls back to the Rails stage primitive without a duplicate native broadcast. Existing Rails caption context/persistence calls remain visible and must not be counted as stage fallbacks.

After deployment, observe image-bearing QA requests in both regions. Require one attempted ID and provider call per candidate, no cross-tenant selection, no repeated failed/no-op candidate, correct started/completed events, and zero Rails execute_stage calls for part-only caption work. Count attachment fallbacks and caption context/result calls separately, then hold 24 clean hours. Roll back by reverting the application revision; preserve request progress and never clear attempted IDs or replay evaluations manually.

Native QA attachment caption advancement

ENG-2887 is stacked on ENG-2885 and removes the remaining Rails execute_stage call for caption_images. API2 processes one pending attachment under the request lease, checkpoints its ID, reuses only tenant-local matching binaries, fails closed when cached text cannot be redacted, and deletes a tombstoned R2 object only after its unshared blob row is removed. Legacy Rails/GoodJob execution remains unchanged for rollback.

After deployment, observe actual image/PDF attachment candidates in both regions. Require persisted completed/unsupported/failed outcomes, transient attempts that do not starve later items, safe cache reuse and tombstones, matching stage events, and zero Rails execute_stage calls for caption_images. Hold 24 clean hours before production credit; never clear attempted IDs or replay evaluations manually.

Native QA related-conversation refresh

The QA request worker automatically uses API2’s provider-sync engine for kept Intercom, Zendesk, Freshdesk, Jira, Outreach and xCally related conversations. The request lease contains a durable pending conversation and sync-run ID, so retries must reuse the existing provider outbox. Unsupported, discarded or timestamp-incompatible records fall back to Rails without emitting a duplicate native stage broadcast.

After deployment, observe an actual supported related refresh in each region. Require one attempted ID and provider-sync run per related conversation, successful started/completed broadcasts, no tenant/source mismatch, no retained provider effect, and no Rails execute_stage call for the supported stage. Separately query source keys and discarded-state coverage before removing the fallback. Roll back by reverting the application revision; preserve request stage progress and provider-sync outboxes, and never clear or replay them manually.

WorkOS event polling ownership

ENG-2858 is a flagless scheduler cutover. The API2 worker registers sync-workos-events-ten-minute at */10 * * * * UTC with five total attempts. It uses the existing singleton public.workos_event_cursors row in each region, applies raw events serially through API2’s transactional native effect layer, and updates the cursor only after the entire page succeeds.

Deploy API2 and Rails from the same revision. Confirm exactly one BullMQ scheduler and one initialized cursor in each region, no newly scheduled SyncWorkOSEventsJob GoodJob executions or Rails event callbacks, and successful processing of real identity, membership and RBAC events without payload conflict, duplicate effect or retained BullMQ failure. Observe at 30, 60 and 120 minutes and for 24 hours before recording ownership.

Rollback removes the persisted BullMQ scheduler while compatible API2 workers are still running, lets active work drain, and then reverts the revision so the Rails cron resumes. Do not manually run both pollers, change cursors, delete event rows or purge queued work. Rails payload classes remain only as code-revert rollback until the observation gate closes.

Native headless authentication

The global Flipper headless_auth flag selects the first-party email-first screens. With ENG-2842 deployed, every headless mutation and CSRF setup checks that same flag in API2; a false or unavailable result sends new users through hosted AuthKit. The public feature probe remains reachable so the UI can make that choice.

Before leaving the flag on, verify API2 has the regional WorkOS client, secret, 32-byte cookie password, exact app Origin and Redis connection. Observe actual Magic Auth, password fallback, signup, email-verification and password-reset flows in both regions, including tenantless and organization-scoped sessions. Require successful subsequent session reads and logout, no Rails headless requests, bounded error/rate-limit events and 24 clean hours.

Disable headless_auth for immediate hosted-AuthKit rollback. Keep native code deployed for existing sessions; no session/data/queue cleanup is needed. Revert the UI/server revision for code rollback while Rails endpoints remain. Remove the Rails actions only after the clean bake. Owner: Platform. Existing flag; review/remove the compatibility branch by December 6, 2026.

Native conversation-QA completion

native_conversation_qa_completion selects API2 terminal completion only for result runs already accepted by native_conversation_qa_result_import. Both flags are organization-scoped PostHog gates and fail closed. The completion flag has no effect while native result import is disabled.

Deploy PR #11312 and apply its Drizzle migration in both regions before changing either flag. Verify that api2.conversation_qa_result_imports.completion_owner exists, is non-null and defaults existing or unspecified rows to rails. Then:

1. Confirm the target organization already has clean native-import evidence.
2. Enable native completion for one organization without changing other organizations or regions.
3. Observe accepted imports with completion_owner = 'api2' through success, skip and failure outcomes before expanding the cohort.
4. Require correct request and agent terminal state, one parent score, current override lineage, one workload link, stable conversation scores, bounded billing jobs and one conversation-summary dispatch.
5. Require no Rails effects callback for API2-owned imports, no payload or owner conflict, no duplicate result rows and no retained finalization failure.
6. Repeat in the other region and hold the final cohort for 24 clean hours.

Disabling the flag changes ownership only for result runs accepted afterward. An existing marker with completion_owner = 'api2' continues through API2 on replay; do not edit that marker or send the run through Rails. Keep the native completion code deployed until those runs drain. For a broader rollback, disable native completion first, then disable native result import for new runs.

Gate status: passed September 8, 2026 at 22:01 UTC. Result import and completion are enabled in both regions. The final audit found no marker conflict, duplicate result, retained finalization failure, or repeated effect. Keep the rollback flags and Rails effects path until a separate bake-in change removes them.

Owner: Platform. The code gate was introduced September 6, 2026; the PostHog flag is enabled in both regions. Remove or review by December 5, 2026.

Bake-in (ENG-3465 ). The finalizer no longer reads native_conversation_qa_result_import or native_conversation_qa_completion. Native import and completion are the production default for every newly accepted result run; Rails writes results only while Flipper pause_native_conversation_qa_result_import is enabled or unreadable (fail-closed rollback, no deploy needed). Markers already recorded with completion_owner = 'rails' keep replaying through the Rails effects callback, and a finalization payload without an organization slug now fails the job instead of guessing ownership. Archive both PostHog flags after the carrying SHA is COMPLETED on all four API2 services. Owner: Platform; switch created 2026-09-25; remove with the Rails evaluations#results / evaluations#result_effects endpoints, review by 2026-12-24.

Native evaluation-check bodies

The evaluation-check BullMQ worker selects a body independently by organization and type:

All three flags default off and fail closed to the retained, tenant-scoped Rails executor. Enable one check type for one organization at a time. For each type, record completed, retrying and retained-failed jobs; Rails fallback executions; check and CX-risk snapshot terminal state; and duplicate result or downstream effect counts. Customer-issue rollout must also verify one topic-assignment job when an issue is created. Require real traffic for every enabled type in each region and 24 clean hours before expanding or removing Rails.

If a native body regresses, disable only its type flag for the affected organizations. A later retry re-evaluates the flag and can resume through the Rails implementation under the same per-check advisory lock. Do not purge run or failure queues and do not rewrite running or terminal check rows.

Gate status: passed September 8, 2026 at 22:01 UTC. All three native-body flags are enabled in both regions. The final audit found no new terminal run or failure-queue job, duplicate active AI evaluation, or newly stranded CX-risk snapshot. Keep the Rails executor and per-type rollback flags until a separate bake-in change removes them.

Owner: Platform. The code gates were introduced September 6, 2026; all three PostHog flags are enabled in both regions. Remove or review by December 5, 2026. The delivery, retry and terminal-failure contract remains in the evaluation-check runbook.

Bake-in (ENG-3453 ). The worker no longer reads the three PostHog flags. Native execution is the production default; Rails runs a check only while Flipper pause_native_evaluation_check_run is enabled or unreadable (fail-closed rollback, no deploy needed). A missing check returns missing natively and a type Rails never creates fails the job instead of calling Rails. Archive the three PostHog flags after the carrying SHA is COMPLETED on all four API2 services. Owner: Platform; switch created 2026-09-25; remove with the Rails evaluation_check_runs#run endpoint, review by 2026-12-24.

Evaluation-schedule execution ownership

ENG-2808 is a flagless scheduler cutover. The API2 worker registers one evaluation-schedule-tick BullMQ scheduler at */15 * * * * UTC, scans at most 5,000 active due schedules in stable window/ID order, and enqueues deterministic schedule/window jobs with eight attempts. Each job sends its immutable tenant, schedule, run date, and expected window to the authenticated Rails boundary; Rails locks the schedule and returns a no-op if that window already advanced or changed.

Deploy API2 and Rails from the same revision in both regions. Require exactly one BullMQ scheduler per region, successful real due windows, no new RunEvaluationSchedulesJob GoodJob executions, no duplicate evaluation batches, no retained BullMQ failure, and stable schedule advancement at 30/60/120 minutes and through 24 hours. A schedule failure must not block later due schedules.

Roll back by first removing the persisted evaluation-schedule-tick scheduler while compatible API2 workers are still running, then letting the already queued tick/window jobs drain. Revert the revision only after that drain so the Rails GoodJob cron is restored without the BullMQ scheduler creating another tick. Do not delete BullMQ work or edit schedule/batch rows manually. Native API2 batch construction is a later package.

Provider post-transcription continuation

ENG-2803  and PR #11223  transfer the normal provider-sync continuation branch into API2. API2 applies the exact Rails-compatible hourly organization throttle, selects eligible call and image parts, joins deterministic transcription children to the provider-sync run, rebuilds the transcript, marks terminal state, and enqueues translation. Rails keeps only narrow CX-risk dispatch and an explicit GoodJob translation fallback when native delivery fails.

This package has no rollout flag and receives no production credit before deployment. After the exact revision reaches both regions, observe at 30, 60, and 120 minutes and for 24 hours. Require bounded provider-sync and transcription queues, one continuation per provider run, no transcript or translation source-version drift, no duplicate provider requests, expected CX-risk dispatch, successful fallback evidence where exercised, and declining normal traffic to the retained broad Rails analysis and completion endpoints. Roll back by reverting the revision; do not delete BullMQ or GoodJob work or edit production conversation state manually.

Direct Intercom webhook ingress

ENG-2800 prepares a flagless provider-domain cutover. In production US API2, an external delivery must pass Intercom’s HMAC check, then synchronously forward the exact raw body to https://eu.api2.rulebase.co/webhooks/intercom with the shared internal bearer and loop-prevention header before any US processing. EU accepts only that internally authenticated copy and never forwards it again. If the required EU request fails or its target is missing, US returns a retryable 500 without applying local effects. Non-production and EU deployments do not fan out.

Deployment alone does not change ownership. Verify the exact revision in both regions, test the external-signature and internal-forward contracts without provider mutation, and confirm normal queue and error health. Changing the Intercom webhook destination from Rails to API2 is a separate external action requiring explicit authorization. After that change, require real signed events to produce one accepted delivery and expected downstream work in each region, with no forwarding loop, duplicate effects, HMAC failure, or growing retry rate. Roll back first by restoring the Rails provider URL; keep Rails ingress and its forwarding code until the observation gate is complete.

Direct Zendesk and Jira webhook ingress

API2 serves provider-facing Zendesk and Jira ingress at POST /webhooks/zendesk and POST /webhooks/jira, folding external authentication into the same legacy-webhook-ingestion pipeline that already receives Rails-forwarded deliveries on /internal/webhooks/*. Zendesk deliveries carry the shared bearer token each subscription was registered with (ZENDESK_WEBHOOK_BEARER_TOKEN, the value ZendeskConfig#webhook_bearer_token reads on Rails); tenant resolution keys off X-Zendesk-Webhook-Id as before. Jira deliveries carry the HS256 JWT the app signs per delivery — issuer pinned to JIRA_CLIENT_ID, context.cloudId selecting the kept organization data sources — and X-Atlassian-Webhook-Identifier stays the dedupe key with the body digest as fallback. Both endpoints answer non-2xx before the BullMQ add resolves so the provider retries, and issue_property_set / issue_property_deleted remain acknowledged no-ops. Session authentication steps aside for the exact paths only.

Deployment alone does not change ownership: Rails stays the registered destination and keeps authenticating and forwarding. Cutover requires, in order:

1. Provision ZENDESK_WEBHOOK_BEARER_TOKEN into the API2 Infisical environments (staging, prod, production-eu) with the value of /rulebase/<env>/ZENDESK_WEBHOOK_BEARER_TOKEN, and confirm the API2 JIRA_CLIENT_ID/JIRA_CLIENT_SECRET pair matches the OAuth app credentials that sign Jira webhook JWTs — a mismatch fails closed with 401.
2. Verify the exact revision in both regions, probe bearer/JWT rejection and acceptance without provider mutation, and confirm normal queue and error health.
3. Repoint provider registrations — a separate external action requiring explicit authorization: update each Zendesk webhook subscription’s endpoint (currently https://{region}.api.rulebase.co/webhooks/zendesk, written by OrganizationDataSource#setup_zendesk_webhook) through the Zendesk API, and each Jira connection’s webhook URL (written by Clients::Jira#create_webhook) through the Jira REST API. New registrations continue writing the Rails URL until the registration code is switched in a follow-up.
4. Require real signed events to produce one accepted delivery and expected downstream work per region, with no duplicate effects, auth failures, or growing retry rate.

Roll back first by restoring the Rails provider URLs; keep Rails ingress, its forwarding code, and the legacy_webhook_rails_processing Flipper kill switch until the observation gate is complete. Intercom follows the same contract via POST /webhooks/intercom once its dashboard destination is repointed.

Public-schema writer gate

Before any ENG-2682 move migration, regenerate the tracked writer inventory at the exact candidate revision. The September 6 0a076d5d639 result covers 194 tables, 12 sequences and 367 affected foreign keys and reports 23 executable references. Resolve its seven executable table bindings, retain the reviewed assignment-routine wrapper until all four caller sites drain, and update the operational ACL probe. Nine catalog/lock/drop references belong to the four-table legacy screening-resource retirement command; run it only after every old worker has drained, or update it for the reviewed schema location if the tables still exist at cutover. Re-run after every caller change. Do not treat the 180 relations with no active qualified literal as unused; complete stored routine, dynamic SQL and external-client review.

The current result has no active named-constraint upsert or direct sequence caller, but that finding is source-only. Restore fresh regional schema and ACL exports, reconcile default grants and identities, and review lock/load budgets before generating a move migration. AWS SSO renewal is the immediate dependency for that regional evidence. Keep productionCutoverReady false until all gates pass in both regions.

Native authentication rollout

WorkOS authorization reconciliation (ENG-2815)

The candidate removes Rails’ sync_workos_authorization GoodJob cron and ensures sync-workos-authorization-daily on API2’s maintenance queue at 03:15 UTC with four total attempts. Keep the Rails job/service deployed for queued work, remaining remote-write callers and code rollback.

After deployment, verify one scheduler in each regional Redis instance and the absence of a newly registered Rails recurring entry. Observe one real run per region: permission and role totals must be plausible, assignable mappings and tenant scope must match WorkOS, additive local permissions must remain intact, and every retry must either succeed or remain visibly retained. Require no unresolved reconciliation Sentry event and 24 hours after the real run before ownership credit. Roll back by removing the BullMQ scheduler first, draining active work, then reverting to restore the Rails cron. Exact contract: authorization reconciliation runbook.

WorkOS membership reconciliation (ENG-2809)

The candidate removes Rails’ sync_workos_memberships GoodJob cron and ensures sync-workos-memberships-daily on API2’s maintenance queue at 03:00 UTC. The legacy Rails job class remains available for already-enqueued jobs and a code revert. Do not manually enqueue both owners during rollout.

After deployment, verify the scheduler exists in both regional Redis instances and that Rails no longer schedules a new recurring entry. Observe one real daily run per region: page and organization totals must be plausible, membership and role changes must match WorkOS, failed organizations must reach a successful retry or remain visibly retained, and no Sentry membership-reconciliation error may remain unresolved. Require 24 hours after the actual run before ownership credit. Roll back by removing the BullMQ scheduler before restoring the Rails cron; keep API2 workers available until any retained attempt drains. Exact contract and evidence: membership reconciliation runbook.

ENG-2741/#11073 merged at September 3 10:46 UTC. The user completed the regional WorkOS configuration copies; HMAC-only comparisons at 10:40 proved equality with each region’s Rails values and 32-byte cookie passwords. Release b55ce46e727bc12e20afdc9b2af2e326074b372a succeeded EU 11:05:10 / US 11:06:05 and contains the logout merge. Fresh runtime reads found the native logout module and all three WorkOS settings. This supersedes the 09:57 missing-setting gate. Actual logout/revocation/redirect evidence and 24-hour observation remain.

ENG-2751 switched existing-member sidebar actions through API2 in #11091 and release e74224443f completed both regional API2 deployments. Its regional UI caller also deployed successfully through GitHub Actions run 33789129052 to both regional UI Workers. Both regional Rails configurations confirm 30-day cookies/retention and no idle timeout; shared cookies/origins and WorkOS parity are already proved. Require actual switch-to-target/old-cookie rejection/new-cookie logout observations. Login and new identity flows remain Rails-owned. Revert the sidebar caller first if needed; the renewed Marshal/WorkOS envelope is readable by Rails. See the switch runbook.

ENG-2758’s hosted AuthKit server compatibility is deployed in both regions. Fresh US/EU tasks confirm the API2 callback bases and app/CORS origins, and the user confirmed both regional callback URLs are registered in WorkOS. The caller stage moves login, signup, password reset and same-region regional-auth entry points to API2. For email-first SSO, API2 selects the active domain connection and falls back to AuthKit on a missing, inactive or failed lookup, matching Rails. After deployment, require actual tenantless, existing-member and SSO logins, replay rejection, logout/switch compatibility and 24 hours without regression. Invitation and new local-membership flows remain Rails-owned. See the hosted-login runbook.

Work-schedule caller gate

ENG-2742’s shared Rails/API2 schedule writer lock deployed through release e74224443f. ENG-2752’s three UI caller changes are merged, and GitHub Actions run 33789129052 deployed the same revision to both regional UI Workers. Require actual create/default/update/discard events and 24-hour parity. Roll back callers before removing native writers or the shared lock. No production schema/data repair is included.

Cloudflare’s separate dashboard Git builds still read build/server/wrangler.json, but build:cloudflare runs as an SPA and removes that directory. The canonical production path is the GitHub Actions workflow, which uses the checked-in wrangler.jsonc and wrangler.eu.jsonc; its successful exact-revision deployment is the evidence used for caller observation.

ENG-2738 session retention

Native trim-sessions-daily runs at midnight UTC with persisted cutoff/cursors, ten-page jobs and five bounded retry attempts. Remove the Rails recurring entry in the same release; preserve the legacy class. No production data mutation or new runtime flag is part of preparation. Verify scheduler presence in each environment, actual midnight deletion and 24-hour regional observation before certifying ownership. Rollback must remove the persistent scheduler and drain only its continuations with compatible workers before restoring Rails. Exact steps and local PostgreSQL evidence: rulebase-web/rulebase-api2/docs/session-retention.md.

September 15 execution gates

Pull ready implementation and rollout preparation forward; dates are latest milestones, not reasons to wait. Native auth, QA stage/failure ownership and schema compatibility rehearsal are the first priorities. Foundation gates are September 4, retained implementation parity September 8, and final regional cutovers September 11. Preserve at least 72 hours of clean final operation during September 12–14, including actual provider events and required Sunday/Monday schedules, before approved Rails retirement September 15.

Complete or explicitly park non-actionable observation waits with an owner, next check time/event and unresolved gate before reusing a slot. Parked work is not complete. Keep at most three active packages including waits. Continue independent implementation while affected packages await review or observation. A missed gate requires a concrete recovery sequence and updated forecast; required CI/reviews, package-specific bake windows and production authorization boundaries still apply.

Simulation is retired by product decision: no replacement or feature bake is required. OPS/REALTIME must remove obsolete callers/routes/subscriptions and stop reminder scheduling/queued execution before Rails shutdown. Stored data and migration history remain preserved. Shared call-analysis and native coaching contracts remain retained.

Native existing-session validation (ENG-2730)

The initial merged query referenced nonexistent accounts.discarded_at and fails authentication with a server error. The incident owner is correcting the query and schema fixture. Require the repaired revision and actual regional auth recovery before further runtime migration rollout or observation credit.

The auth package runbook requires Ruby fixture parity, tenant/revocation/expiry tests, CI and review before merge. A read-only probe decoded 1,000 retained sessions per region at September 3 01:18 UTC; nine sampled sessions were authenticated, so this is format compatibility evidence only. After automatic deployment, observe actual signed-in activity, 401/403/500 rates and database latency/pool pressure at 30/60/120 minutes and through 24 hours in both regions. Revert code on regression; existing Rails session tables and Ruby-compatible payloads remain in place. No production session repair, manual sign-out or infrastructure operation is authorized here.

QA lifecycle and schema preparation

The assignment-interval routine follow-up is now deployable as migration0344: move the table, owned sequence, primary-key index and rewritten physical function together; retain the old table alias and function wrapper; retire all old callers before removing aliases; reverse function/table together if required. The migration preserves OIDs/owners, requires the reviewed default ACL state and aborts on catalog or body drift. Fresh September 7 13:17 UTC US/EU preflight and the exact forward/reverse rehearsal pass. Roll out through the normal regional migration deployment, verify both routine signatures and new interval writes, then keep aliases through old-task drainage and rollback observation.

The September 6 caller package makes the four execution paths dual-schema aware before that move: Rails Jira/Zendesk sync and API2 provider persistence/backfill prefer public when installed and otherwise call api2. Deploy and observe this compatibility layer before physical movement. Keep the api2 signature wrapper until old application revisions drain and rollback observation is complete.

The following Rails compatibility package removes the remaining hard-coded model bindings. Keep RULEBASE_API2_PUBLIC_RELATIONS empty until a reviewed domain move creates its public relation and the legacy alias is ready for old revisions. Then add only that relation’s key to the comma-separated allowlist and restart Rails in the ordered regional rollout. Invalid keys fail closed. Verify the selected table name, representative read, status-incident organization join where applicable, and rollback to the legacy binding before advancing the next relation. The query-data ACL diagnostic uses public-first catalog resolution and requires no environment selection.

ENG-2682’s expanded rehearsal is local-only and reports production cutover as not ready. It restores the unchanged repository structure and verifies forward and reverse moves, all 1,425 restored foreign keys, old/new table/view reads and synthetic reader access. Before generating production moves, obtain fresh regional schema/ACL exports, account for six absent manifest objects, and review exceptional writers and lock/load budgets. No grants or schemas were changed in production.

The full writer matrix proves that old named-constraint upserts fail through compatibility aliases. The 26 colliding index renames also rename PK/unique constraints, so writers using direct target tables must update those names too. Eleven restored old sequence names disappear after movement. Inventory and fix these exceptional callers before any move; planner success for 323 inferred conflict targets does not close the runtime writer or production-ACL gates.

ENG-2739 requires actual related-conversation progress after deployment in each region, no recurring aborted-transaction errors, and timely failure-outbox delivery. Three US EasyPay backfill requests exhausted retries on September 3; all three failure effects delivered first attempt. No replay is authorized. One provider call can still time out; do not infer full recovery from local savepoint tests or successful deployment.

ENG-2731 requires the public Drizzle outbox and compatible Rails stage/finalizer endpoints before native ownership credit. Missing primitive deployment delays jobs without terminal failure. Account for old GoodJob/Rails owners and verify actual trigger, retry and finalization activity in both regions, including oldest pending outbox age. For the native preparation-complete slice, require a real regional request to persist prep_completed_at, become ready and advance to the retained Rails eligibility step without a Rails execute_stage call for preparation completion. Verify eligibility separately. Preserve the table/endpoints through rollback; the persisted failure scheduler requires separately authorized handling before older workers take over. Full contract: rulebase-web/rulebase-api2/docs/qa-request-lifecycle.md.

The conversation-result handoff uses a separate BullMQ finalization queue keyed by the immutable result-run UUID. ENG-2801 makes API2 the result-tree writer behind PostHog native_conversation_qa_result_import; the flag defaults off and must fail closed. Owner: QA platform. Purpose: move result persistence and aggregation out of Rails. Created September 4, 2026; remove or formally review by September 12, 2026.

Deploy the migration and both service revisions with the flag off. Verify legacy success, no-scorecard skip and terminal failure first. Enable one internal or low-volume tenant, then expand one region at a time only after matching agent, scorecard, QA-evaluation and request terminal state. For native runs, require one marker per evaluation/run, no duplicate criterion/check rows, no tenant mismatch, and successful effects-only delivery. Observe native-import, native-replay and legacy outcomes; result-finalizer waiting/delayed/failed counts; effects 409/5xx; oldest pending agent/request age; and conversation/usage parity at 30, 60 and 120 minutes and through 24 hours.

Disable the flag immediately on a persistence, aggregate, tenant or effects regression. New unmarked jobs return to the full Rails writer. A marked run must continue through API2 and the Rails effects-only endpoint even after disablement, because switching writers mid-retry can lose completion effects. Keep the marker table, finalization worker and effects endpoint deployed until every marked and retained job drains; do not delete or replay queue data. Rails lifecycle effects remain required after native import and are the next ownership boundary.

ENG-2682 is preparation only. Its local rehearsal verifies OID-preserving moves, bounded lock rollback, ordinary view upserts, exact reader grants/tenant filters and lossless reverse movement. Named-constraint upserts, table-only commands, routine bodies and default grants still need per-writer review. No production table-move migration or permission change is included. Require current full schema/ACL restoration and mapping reconciliation before generating those moves.

Native live QA evaluation billing (ENG-2766)

The reviewed package makes API2/BullMQ the normal owner of the idempotent Grant qa_evaluation event and its usage-ledger mirror. Rails completion sends a tenant-qualified enqueue only. Immediate delivery errors fall back to the unchanged BillQAEvaluationToGrantJob; do not remove that fallback before the regional bake completes.

Require the exact API2 route/worker and Rails caller revisions in US and EU. For a real flag-enabled AI evaluation in each active region, verify one native job, one Grant usage ID, positive billed credits, matching Rails billed markers, one tenant-correct API2 ledger event, a mirrored timestamp and no normal-path GoodJob billing row. Check waiting, delayed and failed BullMQ work; GoodJob fallbacks; PostHog/Grant errors; duplicate conflicts; oldest billed-but-unmirrored age; and Settings > Usage parity at 30/60/120 minutes and through 24 hours.

Roll back by restoring the Rails GoodJob enqueue while leaving the native worker deployed until queued jobs drain. Do not delete BullMQ jobs or reverse billed markers or ledger events. Grant idempotency makes temporary overlap safe. Keep hourly native reconciliation running as the independent accepted-charge backstop.

Native connection-log retention (ENG-2723)

The reviewed implementation replaces Rails midnight cleanup with trim-connection-logs-daily on the existing maintenance queue. No rollout flag, provider request or schema migration. Premerge read-only regional index and DB capacity checks passed at September 3 00:48 UTC. Require the deployed worker and scheduler, an actual midnight sweep covering all eligible sources, no unexplained failed source/batch jobs, and a 24-hour observation window before bake credit.

Keep all deletion batches tenant/source-qualified and at most 10,000 rows, with the initial per-source cutoff preserved. A code revert cannot restore deleted logs or remove the persisted BullMQ scheduler. Separately authorized operators must remove that scheduler and inspect only retention jobs before older workers take over; never clear the shared queue. Restore Rails cron with the reviewed revert. The full contract and rollback are in rulebase-web/rulebase-api2/docs/connection-log-retention.md.

Active incident gate: Zendesk intake contention

ENG-2707  is a code-only mitigation. New workers set ZENDESK_WEBHOOK_TICKET_SYNC_GLOBAL_CONCURRENCY (default 4) before starting intake. At 64 outstanding provider jobs, an intake job yields for 30 seconds only if the bounded active-job sample contains a Zendesk sync with its exact organization and source. Missing/obsolete durable rows skip before this check; unrelated sources and tenants are not gated. Delayed work preserves the BullMQ job, per-row deduplication, failure budget, and durable receipt. The gate is a pressure signal, not an atomic hard limit on provider queue size.

Approval and observation: the September 2 standing authorization covers scoped reviewed/green migration and regression merges, including main’s automatic production deployment. It does not permit manual deployment dispatch, review bypass, or production data/flag/queue/infrastructure changes. ENG-2707 is already live in both regions; observe its actual rollout rather than inventing a staged US-then-EU release that the automatic deployment did not perform. Take a pre-deploy baseline, then inspect at 30, 60, and 120 minutes and 24 hours in both regions. After old workers drain, webhook active jobs must stay at or below four. EU provider depth and oldest wait must fall at both 60 and 120 minutes; treat zero provider completions for 10 minutes, new lost/duplicate receipts, or any new terminal fallback failure as a stop. If depth or age rises at both observations, stop expansion and investigate source-level throughput; do not declare this mitigation a complete root-cause fix.

Rollback order: revert the code first, retain all queued work and receipts, and initially leave the safe global cap in place. The cap persists in Redis across a code rollback. Only after every new worker has stopped, and with a separate explicit production approval, call the webhook queue’s removeGlobalConcurrency() if restoring the old unlimited fleet setting is necessary. Do not delete deduplication keys, flush queues, replay receipts, or change provider concurrency as part of this rollback.

At 15:04 UTC, Flipper still had Zendesk source sync and QA sampling recovery true in US/EU, while ticket-closed QA had no enabled gate. PostHog transcription was untargeted 100%. All controls were read-only. Pause expansion that adds load to the affected provider resources and deletion of their controls until that workload is healthy through the bake window. Independent accounting and synchronous-route implementation can continue.

Independent package rollout contracts

CX-risk snapshot completion fan-in (ENG-2249)

Rails emits a state-bound completion event after every terminal evaluation-check transition. API2 deduplicates the event by snapshot ID and a digest of ordered check IDs, statuses, and update timestamps, then retries the tenant-qualified Rails finalization call eight times. Rails reloads under a dedicated per-snapshot advisory lock and returns without repeating completed or failed snapshots. An immediate Rails-to-API2 delivery failure uses the retained GoodJob; ambiguous overlap is safe under the same lock.

This package does not move evaluation-check model execution or CX-risk result writes out of Rails. Deploy Rails and API2 together, then require 24 hours in US and EU with real completed jobs, no retained BullMQ failures, no rising GoodJob fallback, no snapshots stranded in running after every check is terminal, and no repeated finalization effects. Roll back by reverting the revision to restore GoodJob delivery. Leave queued BullMQ jobs intact: the locked Rails endpoint accepts late calls safely. Remove the legacy job only after the exact revision passes the regional window and Rails finalization has its own replacement package.

• ENG-2720 replaces existing hourly QA credit reconciliation without calling Grant. Before merge, verify tenant/mapping and idempotent ledger parity plus current pending-mirror coverage. The 23:04 UTC read-only gate found zero overdue rows in both regions, with all 27 last-day EU charges mapped/mirrored and no US charges. Observe two hourly ticks and 24 hours in both regions: scanned/written/already-present/failed/unmapped counts and oldest pending age. Reverting code restores the Rails cron. Before older workers without the native handler take over, remove the persisted API2 scheduler with separate authorization and inspect already-queued instances. Overlap must remain idempotent.
• ENG-2721 removes deletion forwarding from authenticated Intercom ingestion. Preserve source/tenant/external-ID fan-out and last-writer tombstone behavior; provider API usage does not increase. Verify deletion success/failure signals and zero corresponding Rails forwarding on actual events. A quiet window with no deletion event is not positive bake evidence. Revert the code on parity failure.
• ENG-2722 switches four first-party tag mutations to API2. Verify permission, tenancy, origin/CSRF, payload/status and concurrent idempotency parity. Observe regional route/error evidence on the deployed client/server revision; revert compatible client/server changes on regression. This does not add provider load.

Current controls

At 19:04 UTC the ENG-2707 revision was live in both regions and the global webhook concurrency was four. EU provider wait fell to 5,703, but its oldest sampled wait rose to 230 minutes and 1,906 webhook jobs were delayed. Do not remove the cap or expand load on the affected provider resources until depth and age fall through the observation thresholds. This is not a gate on unrelated streams.

Freshdesk recurring imports are revision-selected, not flag-selected. Their deployment and rollback prerequisites are recorded below; this branch does not deploy, change flags, or mutate production state.

A September 8 22:01 UTC audit confirmed that all five native QA/evaluation flags are enabled in both regions. The organization-scoped ticket-closed actor sets also match the six regional tenants representing the five production customers.

The Intercom PostHog flags were created at 22:29:00Z and 22:29:05Z. The transcription flag is api2_conversation_transcription.

Freshdesk recurring ticket imports

A post-deploy source scan exposed Rails-compatible nullable sync_data: SQL NULL or JSON null failed before the reconciliation window could be checkpointed. The follow-up normalizes both forms for source reads and tenant-scoped checkpoint writes. Require a successful scan and advancing watermark for every mapped source after that revision deploys before assigning production credit. Both regional active Freshdesk sources had SQL NULL metadata and no checkpoint at 19:07 UTC. The 19:10 UTC mapping check found one mapped US source and one unmapped EU source. The null-metadata fix restores the US scan; it does not make the EU source eligible. Resolve the EU mapping or explicitly retain Rails recurring coverage for that source before declaring the migration complete.

Recovery approval: authorize the reviewed fix revision for US, then EU, and observe two 15-minute scans plus the full 24-hour bake. The initial 15-minute lookback cannot recover the pre-fix outage gap because no window was saved. Before deployment, approve a bounded tenant/source historical import from the last verified successful Rails import (last retained scans: 17:00 UTC in both regions on September 2) through the first successful API2 scan; confirm that interval and source identity read-only before executing the existing Rails recovery path. Do not reset a new watermark to force replay. Stop on any new null-metadata event, non-advancing watermark over two scans, growing ticket backlog, or unexplained duplicate external identity. Roll back the code first if necessary, preserving windows and failed tickets; restoring Rails recurring ownership and removing the persisted API2 scheduler require separate explicit approval.

ENG-2696  registers freshdesk-ticket-reconciliation every 15 minutes and removes only the Rails recurring cron. Source and ticket jobs retry 5 times with exponential backoff and use tenant/source/ticket deduplication. Terminal source/scan failures are reported through logs and Sentry, then only that job is removed; the next scheduled scan retries delivery using the persisted source window. Failed ticket jobs have no age/count eviction and remain until successful manual redrive or explicit removal. Do not add age/count cleanup to this shared failed set: BullMQ would also trim retained ticket failures. The first scan looks back 15 minutes; later scans use the last delivered watermark minus 5 minutes. A fixed window is saved before network requests and removed only after all valid ticket IDs are durably queued.

Before production deployment:

1. Verify every active regional Freshdesk source has an API2 organization mapping and usable credentials. Unmapped sources are deliberately excluded; unresolved sources block removing their Rails recurring coverage.
2. Confirm native Freshdesk sync and the shared Redis queue are healthy. Check that the conservative 15-jobs-per-minute import limit can drain normal traffic within the 15-minute cadence.
3. Deploy the API2 worker and scheduler before the Rails revision removes its cron. Temporary overlap is safe because conversation identities are unique with NULLS NOT DISTINCT and downstream sync jobs deduplicate.
4. Observe US and EU for 24 hours. Require advancing source watermarks, no unexplained skipped sources or invalid IDs, no growing import backlog, no new duplicate external keys, and successful native syncs. Failed ticket jobs remain individually redrivable after their cause is fixed.

Recover a failed ticket

The source watermark records queue delivery, not successful ticket processing. A terminally failed ticket remains in BullMQ with its original organizationId, organizationDataSourceId, and ticketId, even after later scans advance the watermark. There is no infinite automatic retry and no source-wide wait for a poison ticket.

With separate production authorization:

1. Open the affected region’s API2 /admin/queues board using an authenticated Rulebase staff account. Select freshdesk-ticket-reconciliation and its failed jobs. Match kind: ticket, job ID, and all three identity fields to the final-attempt Sentry event and worker log before acting.
2. Fix the provider, credential, or persistence failure, then use the individual job’s retry action. Retry the retained job; do not edit its identity, reset the source watermark, or bulk-retry unrelated tenants.
3. Confirm the job completes, its conversation keeps one null-type external identity, and the downstream native sync succeeds. Check failed-job count and oldest age, waiting/active backlog, Redis memory, and final-attempt Sentry errors until the incident is resolved. Retention prevents eviction; it does not replace alerts or operator follow-up.
4. Remove a retained failed job only after verifying successful recovery through this or an equivalent authorized import, or documenting an intentionally retired ticket/source. Never purge the failed set to clear an alert.

A source that hits the 300-page / 30,000-ticket list limit fails with its window retained. Investigate the source and plan a bounded historical recovery; do not move its watermark forward to hide undelivered work.

Rollback requires restoring the Rails cron while retaining the API2 worker until queued work drains. An operator must remove the BullMQ scheduler definition and disable its registration in the rollback revision; removing registration code alone does not remove a scheduler already stored in Redis. Do not delete queued jobs or alter source watermarks. These production actions require separate authorization and are not performed by this code-only task.

Intercom rollback before bake-in deploy

If production evidence requires rollback before the selection branches are baked, use this order so API2 never receives work it cannot own:

1. Disable api2_intercom_webhook_ingestion. API2 forwards incoming events to Rails instead of processing them.
2. Disable api2_intercom_conversation_import and api2_intercom_ticket_import. New imports return to GoodJob.
3. Disable api2_intercom_webhook_forward. Rails resumes direct webhook handling.

Do not delete queued API2 or GoodJob work during rollback. Let idempotency and the documented fallback paths drain it.

Intercom after bake-in

After the bake-in revision is deployed, the four flags no longer select an owner:

• Rails hands conversation and ticket imports directly to API2. An immediate delivery error is logged and falls back to the matching GoodJob.
• Rails forwards Intercom webhook deliveries whenever an API2 base URL is configured. ENG-2998 makes a failed forward return a retryable 503 without creating ProcessIntercomWebhookEventJob; local Rails processing remains available only when the API2 base URL is absent.
• API2 owns conversation, ticket, and deletion events. Ticket-part notifications identify their owning ticket under data.item.ticket.id; ordinary ticket items keep using their direct ID.
• The obsolete authenticated /internal/intercom_webhook_events hand-back route is removed. Repeated deliveries remain safe because API2 import job IDs are deterministic and deletion writes are idempotent.

Delete the four PostHog/Flipper records only after the recovery revision is live and the queue, fallback, duplicate, stuck-work, and Sentry checks remain clean through at least one rolling deployment in both regions. A behavioral rollback after that point requires reverting the bake-in revision; changing an orphaned flag has no effect.

The earlier forwarding recovery completed that regional window with zero matching Sentry events in the latest 24 hours. Flag deletion remains blocked on the reconciliation source-selection guard: the September 1 audit found retained BullMQ failures from three legacy regional Rails sources without API2 organization mappings. PR #10888  merged with the guard. After it deploys, require one clean 24-hour window with no new reconciliation failures and advancing mapped-source watermarks before deleting the four orphaned records.

ENG-2998 is a flagless ownership correction. Roll back by reverting its revision to restore the Rails timeout fallback; do not change provider destinations, flags, queues, or stored webhook state. After deployment, require a complete 24-hour US/EU window with no newly created ProcessIntercomWebhookEventJob, bounded API2 ingress errors and no growing Intercom retry rate before assigning retirement credit.

Transcription and Zendesk after bake-in

PR #10741  is deployed and verified live in US and EU. The conversation-level transcription branch and the Zendesk branch no longer select API2 or Rails:

• transcription always hands off to API2, while an immediate API2 delivery error is captured and falls back to TranscribeConversationJob;
• supported Zendesk sync always hands off to API2; the one-shot SyncConversationJob enqueue-error fallback was removed by ENG-2506 (see below);
• disabling the old flags has no effect, so delete their PostHog/Flipper records after deployment verification;
• a behavioral rollback requires reverting the bake-in code, not toggling an orphaned flag.
• The 2026-09-09 rolling-hour GoodJob baseline reported four successful SyncConversationJob executions in US and zero in EU. The regional serialized argument census was unavailable during this audit, so those four executions remain unclassified rather than being attributed to the known legacy Zendesk source. The Rails producer now keeps source-mismatched or missing-subdomain connections out of the API2 retry path; valid mapped sources remain API2-owned.

Zendesk Rails synchronizer retirement (ENG-2506)

ENG-2506  removes the Rails Zendesk synchronizer and every compatibility path that let Rails and API2 both own a normal Zendesk sync. No flag is used: the rollback control is a code revert of the package.

Why: in the EU Rails-worker window 2026-09-15T15:06:59Z to 2026-09-16T15:06:59Z one normal Zendesk conversation produced Rails EnqueueBotHandoffSummaryJob and CompleteConversationSyncJob at 2026-09-16T12:33:23Z. API2 had run that conversation in zendeskSourceSyncMode=shadow (persist only, then schedulePostRailsProviderSync into the Rails completion chain) immediately before, and a source_owned pass a minute later applied botHandoff and syncCompletion natively. Two owners, one ticket; the strict zero-GoodJob clock reset.

Behavior inventory after this package:

Webhook identity model: the BullMQ job id is the producer’s run id (a redelivered ambiguous handoff, a Rails-side transport retry, or a repeated recovery sweep for the same row state collapses onto itself) and the BullMQ deduplication id is the row identity <organization_id>_<row id> (concurrent webhooks for one ticket and the recovery sweep keep at most one queued snapshot per row while it is waiting or delayed). The deduplication uses BullMQ keepLastIfActive: a handoff that lands while the row’s job is active is parked as the single successor and BullMQ enqueues it, still debounced 30 seconds, when the active job completes or fails — it is never dropped and never waits for the sweep. The worker consumes only the event count it observed, so a snapshot that ran while more events arrived leaves the remainder pending and re-enters the same identity with run id <run id>-follow-up (collapsing with any mid-run handoff into that one successor). Proven against real Redis in rulebase-api2/src/lib/zendesk/webhook-ticket-sync-successor.redis.db.test.ts. If Rails dies after commit, or API2/Redis is unavailable during the synchronous handoff (Api2ZendeskWebhookTicketSyncEnqueuer::DeliveryError is logged and captured at warning level, nothing is retried on GoodJob), the committed row is picked up by the recovery sweep. Asserted in rulebase-api2/src/lib/zendesk/webhook-ticket-sync-handoff.db.test.ts (publication failure before acceptance, ambiguous redelivery, concurrent receipt, event during processing, provider failure/retry, idempotent replay, tenant fence, zero public.good_jobs) and in rulebase-api/spec/models/zendesk_webhook_ticket_sync_spec.rb, spec/jobs/process_zendesk_webhook_ticket_sync_job_spec.rb, and the source gate spec/services/zendesk_rails_sync_retired_source_gate_spec.rb (fails on any normal-path ProcessZendeskWebhookTicketSyncJob enqueue, on Conversation::Sync::Zendesk, shadow mode, or the Rails internal enqueue route returning).

Overlap proof: API2’s outbox row api2.provider_conversation_sync_effects is keyed by syncRunId. Concurrent runs with distinct ids each persist once and dispatch one deterministic completion; a concurrent duplicate of the same id loses on the primary key, rolls back without persisting, and its retry returns already-applied. This is asserted in rulebase-api2/src/lib/provider-conversation-sync/zendesk-source-owned-run.db.test.ts together with zero Rails bridge calls and zero public.good_jobs rows.

Rollout gate: deploy Rails and API2 together (Rails now rejects the retired paths that older API2 code could still call). Watch for 24 hours per region.

Post-deploy outcome queries (run read-only in each region’s Rails database through the production shell; replace the window with the deploy time):

-- 1. Zero Rails ownership of Zendesk sync since deploy. Expect 0 rows.
WITH jobs AS (
  SELECT j.job_class,
         j.serialized_params -> 'arguments' -> 0 ->> '_aj_globalid' AS gid
  FROM good_jobs j
  WHERE j.created_at >= '2026-09-16T15:06:59Z'
    AND j.job_class IN (
      'SyncConversationJob',
      'CompleteConversationSyncJob',
      'EnqueueBotHandoffSummaryJob'
    )
)
SELECT jobs.job_class, count(*)
FROM jobs
JOIN conversations c
  ON jobs.gid LIKE 'gid://%/Conversation/%'
 AND c.id = split_part(jobs.gid, '/', 5)::bigint
JOIN organization_data_sources ods ON ods.id = c.organization_data_source_id
JOIN data_sources ds ON ds.id = ods.data_source_id
WHERE ds.key = 'zendesk'
GROUP BY jobs.job_class;
 
-- 1b. Same for ImportProviderConversationPartImagesJob, whose argument is a part.
SELECT count(*)
FROM good_jobs j
JOIN conversation_parts p
  ON p.id = split_part(j.serialized_params -> 'arguments' -> 0 ->> '_aj_globalid', '/', 5)::bigint
JOIN conversations c ON c.id = p.conversation_id
JOIN organization_data_sources ods ON ods.id = c.organization_data_source_id
JOIN data_sources ds ON ds.id = ods.data_source_id
WHERE ds.key = 'zendesk'
  AND j.job_class = 'ImportProviderConversationPartImagesJob'
  AND j.created_at >= '2026-09-16T15:06:59Z';
 
-- 1c. The webhook wrapper itself: zero new ProcessZendeskWebhookTicketSyncJob
--     rows since deploy (its argument is a ZendeskWebhookTicketSync, so no
--     conversation join is needed; every row is Zendesk). Expect 0.
SELECT count(*)
FROM good_jobs
WHERE job_class = 'ProcessZendeskWebhookTicketSyncJob'
  AND created_at >= '2026-09-16T15:06:59Z';
 
-- 2. Retired-path discards (should be 0; any row is a caller to remove).
SELECT job_class, count(*)
FROM good_jobs
WHERE created_at >= '2026-09-16T15:06:59Z'
  AND error LIKE '%ZendeskRailsSyncRetired::RetiredPathError%'
GROUP BY job_class;
 
-- 3. Webhook intake still drains: coalesced Zendesk ticket syncs are deleted
--    once API2 consumes them, and the 5-minute recovery sweep re-arms any row
--    older than 20 minutes, so nothing should sit pending past ~30 minutes.
--    Expect 0.
SELECT count(*)
FROM zendesk_webhook_ticket_syncs
WHERE pending_events_count > 0
  AND last_event_at < now() - interval '30 minutes';

-- 4. API2 side: one delivered outbox row per Zendesk run, none undelivered
--    for longer than the completion retry budget. Expect 0 undelivered.
SELECT count(*) FILTER (WHERE delivered_at IS NULL) AS undelivered,
       count(*) AS total
FROM api2.provider_conversation_sync_effects
WHERE payload ->> 'provider' = 'zendesk'
  AND created_at >= '2026-09-16T15:06:59Z'
  AND created_at < now() - interval '30 minutes';
 
-- 5. Bot handoff stamped at most once per conversation is structural
--    (`bot_handoff_at` is a single column); confirm handoff coverage kept moving.
SELECT count(*)
FROM conversations c
JOIN organization_data_sources ods ON ods.id = c.organization_data_source_id
JOIN data_sources ds ON ds.id = ods.data_source_id
WHERE ds.key = 'zendesk'
  AND c.bot_handoff_at >= '2026-09-16T15:06:59Z';

CloudWatch (API2 worker log group, both regions): zendeskSourceSyncMode and zendesk_rails_sync_fallback must not appear after the deploy; provider_conversation_sync_completion job ids must be unique per syncRunId; stranded webhook ticket sync recovery tick done should report enqueued: 0 in steady state (a non-zero count is a lost wakeup to investigate, not an error). Rails log group: the GoodJob class filter for this package is SyncConversationJob, CompleteConversationSyncJob, EnqueueBotHandoffSummaryJob, ImportProviderConversationPartImagesJob and ProcessZendeskWebhookTicketSyncJob; the only expected ProcessZendeskWebhookTicketSyncJob lines are goodjob-drain-* handoffs from jobs serialized before the deploy, and API2 handoff deferred to recovery sweep warnings must be rare and followed by a matching recovery enqueue.

Rollback is a code revert of the ENG-2506 merge commit, deployed to Rails and API2 together. Reverting API2 alone would re-arm schedulePostRailsProviderSync against a Rails that discards the chain; reverting Rails alone would restore Rails ownership without the shadow persist step. Do not reintroduce api2_zendesk_source_sync or any dual-path flag.

Residual risks: legacy jobs already in GoodJob at deploy time are discarded with a Sentry capture rather than run (expected, bounded by queue depth), except ProcessZendeskWebhookTicketSyncJob rows which drain by handing their durable row to API2 once; the webhook debounce now depends on API2/Redis availability at receipt time, and a sustained outage delays first processing by up to the 20-minute stranded threshold plus the 5-minute sweep cadence instead of the previous 30 seconds (events are never lost because the row is committed first); non-Zendesk providers still use Conversation::Sync and ImportProviderConversationPartImagesJob on Rails and are unchanged by this package.

Provider daily field-definition sync

ENG-3010 transfers the 02:00 UTC provider field-definition sweep and its per-source fan-out to the provider-fields-sync BullMQ queue. No PostHog flag is used; the Flipper kill switch pause_api2_provider_fields_sync (safe state “on” for API2, checked at scan and source execution) is the rollback control, and the Rails cron stays in good_job.rb with enabled_by_default: false. Rollback order is pause the Flipper flag first, then re-enable the Rails cron; never let both owners upsert the same field rows. Queued BullMQ jobs drain as paused and must not be deleted.

Deploy API2 and Rails together, then observe two 02:00 UTC ticks in US and EU: every kept Zendesk, Freshdesk, and Intercom source completes with no category error, last_synced_at advances on its field rows, zero new SyncProviderFieldsJob/SyncProviderFieldsForOrganizationDataSourceJob GoodJob rows, no duplicate (organization_data_source_id, external_id) rows, and an empty or fully explained failed set. Queries and the failed-source redrive procedure are in the package runbook.

Zendesk daily ticket reconciliation

ENG-2931 transfers the 00:00 UTC ticket sweep and its per-ticket imports to the zendesk-ticket-reconciliation BullMQ queue. No PostHog flag is used; the Flipper kill switch pause_api2_zendesk_ticket_reconciliation (safe state “on” for API2, checked at scan, source, and ticket execution) is the rollback control, and the Rails cron stays in good_job.rb with enabled_by_default: false. Rollback order is pause the Flipper flag first, then re-enable the Rails cron; never let both owners advance last_synced_at. Queued BullMQ jobs drain as paused and must not be deleted.

Deploy API2 and Rails together, then observe two midnight ticks in US and EU: every mapped source’s watermark advances, zero new SyncZendeskTicketsJob/ImportZendeskConversationJob GoodJob rows, no capped sources, delayed count returns to zero within the hour, no duplicate Zendesk external identities, and an empty or fully explained ticket failed set. Unmapped Rails-only sources are skipped and must be listed as retained Rails coverage before the shim is removed. Queries and the failed-ticket redrive procedure are in the package runbook.

Zendesk stuck-conversation recovery scheduler

ENG-2684 transfers the six-hour recovery sweep from the Rails GoodJob cron to a BullMQ scheduler on the existing provider-conversation-sync queue. The API2 scan preserves the one-hour import grace window, 500-conversation per-source cap, chronological ordering, tenant/source qualification, source-owned sync mode, retries, and provider-sync deduplication. It additionally excludes discarded sources and organizations without an API2 mapping before enqueue so this producer cannot amplify impossible provider-sync retries.

The Rails job class remains in the deploy as a code-revert rollback path, but its cron entry is removed so the two schedulers cannot race. No rollout flag or production data change is required. After deploy, observe four scheduled ticks in US and EU. Require successful scheduler completion, no duplicate Rails sweep, no new unmapped-organization error attributable to this producer, bounded cap warnings, and no growing provider-sync backlog. Roll back by reverting ENG-2684 to restore the Rails cron; do not delete queued BullMQ work. Remove the retained Rails job and specs only after the regional window passes.

Zendesk webhook ticket execution

PR #10960  and ENG-2692  do not add a rollout flag. Rails continues to receive Zendesk webhooks and durably coalesce each ticket’s events, but the debounced GoodJob now hands the snapshot to a tenant-qualified BullMQ worker. The stable ActiveJob ID becomes the BullMQ run ID, making ambiguous delivery retries safe even when a newer webhook updates the durable row between attempts.

API2 fetches the ticket through the shared per-connection limiter, imports the same top-level conversation attributes, enqueues the existing source-owned provider sync, and consumes only the observed event count. Per-row deduplication keeps snapshot processors serial, while the same stable run ID makes a retry after provider enqueue replay its outbox. Events arriving during work schedule another debounced handoff.

Superseded by ENG-2506 (above): the GoodJob debounce wrapper, the Rails implementation fallback on enqueue rejection, GoodJob-side ambiguous-delivery retries, and the terminal Rails fallback are all removed. Rails commits the row and hands off synchronously; API2/BullMQ owns debounce, retry, continuation and stranded-row recovery.

QA sampling recovery bake-in

The September 1 audit found the recovery gate globally enabled. API2 completed at least 1,766 retained jobs across the latest US and EU 12-hour windows with no failures, no waiting work, and no active work. The Rails global scheduler still woke 48 times per region over the same period, then skipped every organization because API2 owned them.

The bake-in makes API2 the permanent durable owner: it lists every organization on its 15-minute BullMQ cadence and retains per-organization retries and deduplication. Rails continues to perform the bounded, authenticated domain operation selected by API2, but no longer has a global cron or enqueues a new GoodJob sweep after every terminal ticket-closed request.

After deployment, observe both regions for 24 hours. Require fresh successful API2 recovery completions, zero failed or stuck recovery jobs, no material rise in QA sampling Sentry errors, and no unexpected drop in ticket-closed request creation. If a threshold fails, revert the bake-in revision to restore the Rails scheduler and selection branch; toggling the old flag has no effect after the code deploys. Delete api2_qa_sampling_recovery only after the exact revision passes that window.

Direct call-detail transcription

The standalone call-detail handoff temporarily reuses the existing api2_conversation_transcription organization gate, which is already at 100% in US and EU. When enabled, Rails posts one tenant-qualified call to the authenticated API2 internal route, and API2 enqueues it on the existing conversation-transcription call queue with a deterministic ID derived from the call and its current revision. It has no completion parent because these entry points do not own a conversation-level fan-in.

When the gate is disabled or PostHog is unavailable, Rails enqueues TranscribeConversationPartCallDetailJob. An explicit API2 rejection captures a warning and uses the same fallback. Ambiguous transport failures instead retry the deterministic API2 handoff through GoodJob, preventing a timed-out request that API2 already accepted from racing an independent Rails transcription. The conversation-level Rails fallback deliberately enqueues that job directly inside its GoodJob batch; it must not use the standalone API2 handoff or its success callback could run before the independent BullMQ job finishes.

No production flag change is required for initial ownership because the reused gate is already at 100%. Roll back by disabling it; do not delete queued BullMQ or GoodJob work. After a clean regional bake window, remove this last selection branch and delete the PostHog record. Credit the package only after both regions show healthy BullMQ throughput, no unexpected transcription failures, bounded delivery fallbacks, and a material decline in residual Rails job runtime.

Post-sync conversation image captioning

ENG-2689  reuses api2_conversation_transcription to select provider ownership for image captions. An organization outside the rollout, including a failed-closed PostHog lookup, continues to enqueue GenerateConversationPartImageCaptionJob. An explicit API2 rejection also uses that provider fallback. An ambiguous handoff instead retries through GoodJob with the same deterministic API2 job ID so two providers cannot race.

For conversation transcription, eligible image parts become children of the existing BullMQ completion flow. Caption failures retain the current Rails batch-success behavior and block post-sync analysis; call failures keep their existing independent completion behavior. Direct CX-risk snapshot captions use the same queue without a completion parent because their caller never waited for the legacy GoodJob result.

Observe 30, 60, and 120 minutes, then 24 hours in US and EU. Require comparable caption content, no unredacted derived text, no tenant mismatch, no stale or duplicate write, bounded retries/fallbacks, and no conversation stranded before post-sync analysis. Roll back before bake-in by disabling the existing PostHog flag, leaving queued BullMQ and GoodJob work intact. A later code bake must keep the explicit-delivery fallback until it has independent regional evidence.

QA request preparation ownership

PR #10926  moved user_requested preparation to the existing API2 BullMQ orchestrator. PR #10941 , tracked in ENG-2680 , extends unconditional API2 ownership to admin_forced and scheduled requests. ticket_closed remains the only trigger behind the organization Flipper gate while its historical population is repaired.

Seven-day production evidence contained 4 completed US and 62 completed EU user-requested requests, with no active user-requested backlog. The latest EU failure was August 28; ten of twelve failures were an already-repaired schema drift. The historical ticket-closed backlog was subsequently drained. Read-only checks at 2026-09-06 17:21–17:22 UTC found zero active ticket-closed requests older than 15 minutes and zero conversations with duplicate active requests in both regions. The partial unique index over active conversation requests was valid, ready, and unique in both regions. A per-organization staged rollout started at 17:30–17:31 UTC for EasyPay in both regions and Nala in EU. The initial US Rho actor incorrectly selected rho (organization 18); it was atomically corrected to the real production tenant rho-tnkw (organization 19) at 2026-09-07 00:23:23 UTC. The unintended US Nala actor (48) was removed at 00:30:15 UTC; Nala’s production tenant is EU organization 41. US kuda-pilot (25) and EU Qonto (33) then passed regional preflight and were enabled at 00:33:21 and 00:35:05 UTC, respectively. Qonto’s 124 active rows were all scheduled in the future, so none were overdue; both new tenants had zero duplicate active conversations, and the regional partial indexes were valid, ready, and unique. The exact enabled sets are now US organizations 19, 25, and 38, plus EU organizations 33, 38, and 41. The staged gate passed on September 8 at 22:01 UTC after the post-recovery audit found no overdue or duplicate active request and no new terminal queue failure. US Kuda request 5863910 is retained as a non-blocking deployment-time incident repaired by PRs #11508 and #11512; no recurrence appeared after the corrected workers were live.

API2 uses tenant-qualified request data, a deterministic request job ID, five exponential-backoff attempts, bounded Rails stage calls, and the existing Rails terminal-failure callback. Scheduled requests preserve their delay in BullMQ, and immediate delivery failure still falls back to GoodJob. Rails locks the conversation while creating an active request in normal, admin, and backfill paths. The active-row validation applies only to active records, so historical duplicates can transition to failed or cancelled instead of trapping each other in the active set.

Maintenance::ReconcileStaleQAEvaluationRequestsTask is the approval-only data repair. It requires organization_slug, cutoff_at, and a maximum request count, scopes its preview to one organization, locks each conversation, rechecks activity and age, and marks only the selected stale rows failed without emitting one Sentry event per row. Do not run it globally or infer a cutoff. For each region and organization:

1. Deploy the code revision and preview the task collection with an explicitly approved slug, cutoff, and maximum.
2. Confirm every selected row is active and older than the cutoff, then run the task. Verify the selected cohort terminalized and current or future-scheduled requests were unchanged.
3. Confirm duplicate active conversations and requests older than 15 minutes are zero before enabling api2_qa_evaluation_request_processing for that organization.
4. Observe at 30, 60, and 120 minutes and again at 24 hours. Require bounded delivery fallbacks, no terminal BullMQ failure, no callback 422, no duplicate evaluation, and no active request older than 15 minutes.

Roll back ticket-closed ownership by disabling the organization gate. Roll back admin and scheduled ownership by reverting the revision. Do not delete BullMQ or GoodJob work. The repair is intentionally terminal and is not automatically reversed; retry only eligible conversations through the established recovery flow after the underlying cause is safe.

ENG-2698  makes the bounded caption stage request-resumable. Rails stores per-request conversation-part and attachment cursors plus attempted IDs in stage_progress; retries continue past invalid or transiently failing items, cursor wrap prevents a newly eligible lower ID from being skipped, and a request-scoped advisory lock serializes overlapping bounded advances. This state does not alter the GoodJob rollback path or make a pending attachment ineligible for a later evaluation request.

After deployment, observe at least 24 hours in both regions. Require zero orphaned user-requested rows older than 15 minutes, no duplicate evaluations, bounded fallback and retry rates, and no new request-processing regression. Roll back by reverting the revision; do not toggle the automated-processing flag because it does not select ownership for the user-requested lane.

QA-stage call transcription continuation

ENG-2686  removes synchronous provider execution from API2-owned QA preparation. When Rails reaches transcribe_calls, it returns one tenant-qualified call payload instead of calling the provider. API2 enqueues the existing deterministic call job, moves the QA job to BullMQ’s delayed set for 15 seconds, and asks Rails to advance the same stage again. A retained failed call job fails the QA attempt instead of creating an endless delayed loop. The GoodJob-owned processor keeps its synchronous path, so a code revert restores the prior behavior.

No flag or production-data change is part of this package. After deployment, observe at 30, 60, and 120 minutes, then again at 24 hours. Require no QA job delayed for more than 15 minutes, no increase in terminal call-transcription or QA-request failures, no duplicate provider request for the same call revision, and normal progression from transcribe_calls to caption_images. Also compare Rails TranscribeConversationPartCallDetailJob runtime with the API2 call queue; credit two production points only when US and EU evidence is available and clean. Roll back by reverting the revision. Do not delete queued BullMQ work, change api2_qa_evaluation_request_processing, or mutate pending QA rows.

QA result-callback recovery observation

The September 1 post-deploy audit found an EU Rails result-import regression that supersedes the normal bake window. Sentry RULEBASE-API-5S7 recorded 153 validation failures in the first observed window, and retained BullMQ jobs exhausted all five retries with HTTP 422 responses. Failures continued after the latest regional deployment.

The API2 runner validates that every failed check selects a scored justification. Rails then applied the customer-specific deterministic PII override from PR #10614  after that validation. For the affected production scorecard, the regular fail option has no justification choices while auto_fail has two. The override preferred the unusable regular option, replaced the accepted result with failed + nil justification, and triggered the model validation.

The recovery merged in PR #10933  and PR #10935 , reverting the #10616 scoring follow-up and the original #10614 override. The release containing both reverts deployed successfully to US and EU at about 2026-09-02T00:42Z. Sentry’s last matching event was at 00:27:53Z, before the recovery deploy. Ten US and nine EU ticket-closed requests completed after the deploy, with no later matching event. Keep ticket-closed ownership gated through the full observation window and stale-row repair. Roll back the recovery only by reverting the reverts; no flag, queue, or production-data mutation is required.

Legacy Zendesk source configuration

One active legacy Zendesk source has no stored subdomain. API2 cannot load its credentials and exhausts bounded provider-sync retries before the existing Rails fallback; the queue was current at the last regional audit, but the retained failure and Sentry event recur with new sync attempts. Rails cannot safely infer the missing account identity either.

Do not silence the exception, skip Zendesk globally, or invent a subdomain. Inspect and repair that one tenant-scoped source only with explicit production data approval and evidence from the owning integration account. After repair, require a successful sync, no new matching Sentry event, and no retained failure for 24 hours. If the source is obsolete, discard it through the normal product workflow rather than a direct database mutation.

Manual-export bake-in

api2_manual_exports has an untargeted 100% rollout. The September 1 audit found two successful manual QA evaluation exports in EU during the preceding 24 hours, no failed manual export rows in either regional database, and zero waiting, active, delayed, or paused export-runs work. The deterministic workbook harness matched all eight supported Rails/API2 cases after ignoring only the generated export timestamp.

ENG-2677 makes API2 unconditional for Evaluations and All interactions, removes their Rails client functions and feature schema key, and leaves the Tickets manual action as the only one-off Rails export owner.

ENG-2683 adds the classic Tickets workbook to API2: one row per conversation and agent, latest eligible AI and Auditor evaluations side by side, merged KPA headers per scorecard, organization and membership scope, existing BullMQ retries, and the same email delivery. The deterministic comparator now passes 10/10 cases, including both Tickets cases, with identical values, sheet names, row counts, and merge ranges. The UI queues qa_tickets through /export-runs without a column picker. The Rails endpoint and ExportQATicketsJob remain only as a code-revert rollback path during the bake window.

After ENG-2683 deploys, require at least one fresh successful Tickets export in US and EU, zero failed or stuck export-runs, correct multi-scorecard headers, and no related Sentry regression for 24 hours. Roll back by reverting ENG-2683; do not delete queued BullMQ work. The old PostHog flag no longer selects an owner. Delete it only after both ENG-2677 and ENG-2683 pass their exact regional windows, then remove the unused Rails manual endpoint and job in a follow-up.

Sales-call compliance bake-in

US has globally enabled api2_sales_call_compliance_pipeline; EU has no flag record and no sales-call traffic. In the latest 24-hour audit, US persisted 70 API2 result snapshots with 70 distinct result-run IDs. The retained BullMQ queue had no waiting work and its latest failure was August 30. EU had no sales-call rows, API2 result snapshots, completed jobs, or failed jobs.

PR #10932  removes the regional Flipper decision from both Rails completion and API2 native completion. Every eligible sales call now advertises API2 compliance ownership, subject to the unchanged evaluation-credit and reduced-ingestion guards. API2 retains deterministic job IDs, tenant qualification, source-version conflict detection, bounded retries, Rails persistence callbacks, and terminal Sentry reporting. Non-sales-call post-sync analysis continues through Rails.

After deployment, observe both regions for 24 hours. Require no duplicate result-run IDs, no new retained or terminal queue failure, no analysis stuck in progress for more than 30 minutes, stable transcription completion, and no unexpected Rails sales-call analysis work. Roll back by reverting the bake-in revision. Do not delete BullMQ work. Delete the orphaned US Flipper record only after the exact revision passes the window; EU has no record to delete.

Sales-call compliance credit reconciliation

ENG-2687  and PR #10949  move the six-hour credit-reconciliation cadence from the Rails GoodJob cron to the API2 maintenance queue. The bounded task scans at most 5,000 eligible sales-call conversations in 250-row, oldest-ID pages. It requires completed tenant-matched compliance evidence, chooses the newest active evaluation credit for the same organization, inserts one idempotent usage row, and updates the credit’s cached usage counter in the same transaction. A candidate error is isolated so later rows run, then rethrown as an aggregate failure so BullMQ can retry the tick.

This package has no feature flag or production-data mutation. After deployment, observe at 30, 60, and 120 minutes, the first six-hour run, and 24 hours. Require exactly one scheduler per region, no retained maintenance failure, no duplicate usage for a conversation, matching credit_usages and credits.credit_usages_count deltas, and no cross-organization assignment. Credit one production point only after US and EU database and queue evidence is available and clean. Roll back by reverting the revision, which restores the GoodJob schedule while retaining the unchanged Rails job implementation. Do not delete BullMQ work or edit reconciliation rows manually.

Workspace search bake-in

api2_workspace_search has an untargeted 100% rollout. Command K currently keeps a Rails /search branch plus separate API2 company and work-item queries as a disabled rollback fan-out. The bake-in removes that branch and makes one API2 workspace-search request the permanent owner for every entity result.

The PostHog project does not retain feature-flag evaluation events, so direct flag-call volume is not available as parity evidence. Keep the flag record until the exact revision is deployed. Then verify representative identifier, name, and cross-type searches in both regions, require no new workspace-search Sentry regression through the observation window, and delete the orphaned record. Roll back after deployment by reverting the bake-in revision; changing the old flag no longer affects ownership.

Read-flag cleanup

PR #10558  baked a group of globally enabled API2 reads into the UI. The following active PostHog keys had no remaining code reference in the August 30 audit and are cleanup candidates:

• api2_current_actor_reads
• api2_roles_reads
• api2_member_directory_reads
• api2_org_structure_reads
• api2_scorecard_reads
• api2_scorecard_criteria_reads
• api2_settings_reads
• api2_config_reads
• api2_rule_reads
• api2_slack_channel_reads
• api2_object_field_reads
• api2_conversation_taxonomy_reads
• api2_knowledge_library_reads

Reconfirm absence in the deployed UI before deleting them. Workspace search is being baked independently in PR #10921 . api2_manual_exports is handled by its independent bake-in and production verification window, not this read-flag cleanup batch.

Jira assignment intervals for SLA reporting

sla_assignment_intervals selects the live resolution-SLA adjustment source. Its purpose is to replace repeated JSON assignment-history reconstruction with canonical api2.conversation_assignment_intervals rows while preserving the exact widget result. Data Platform owns the flag. The rollout objective is to make EasyPay’s 30-day Support-team SLA query use indexed tenant-scoped interval lookups without changing saved dashboard or chart configuration.

The first target is EasyPay / Lesaka (organization 38, Jira connection 46, Support team 88, dashboard db_01HBT5Q3YW9WM0AJJ7RRQN7395, chart 110). The target team remains excluded in the existing chart configuration; the flag changes only how the excluded duration is calculated. A missing or unavailable PostHog evaluation fails closed to the legacy raw-event query. Frozen sla_adherence_weekly sources never use the new path.

Roll out in this order:

1. Deploy the Drizzle interval table, indexes, rebuild function, and dual Jira sync writes with the flag disabled.
2. Run Conversation assignment interval backfill from the API2 admin maintenance dashboard for the EasyPay organization slug. The BullMQ task rebuilds from stored Jira assignment parts, reports a conversation-ID resume cursor, and is safe to rerun while live syncs continue.
3. Compare interval counts, open intervals, employee mappings, team snapshots, and live resolution-SLA widget results with the legacy path over 30 days. Record EXPLAIN (ANALYZE, BUFFERS) for both paths.
4. Enable the PostHog organization-group flag only for EasyPay. Observe one complete reporting cycle for result parity, latency, database load, and new Sentry errors before adding another organization.
5. When the new path is globally proven, remove the raw-event branch and flag. If parity or performance regresses before bake-in, disable the flag; interval writes and the resumable backfill can remain in place.

Review or remove the flag by 2026-11-30. Do not leave a globally enabled flag as permanent configuration.

EasyPay production plan evidence (2026-09-01)

A read-only US production comparison used chart 110’s exact 30-day window, timezone, resolution metric, conversation-field and policy dimensions, Jira connection 46, and excluded Support team 88. The interval path was modeled with 10,074 rows in a transaction-local temporary table populated from the same stored Jira events; the transaction was rolled back and no production rows were changed.

Both queries returned the same 25 grouped values. On warm buffers, EXPLAIN (ANALYZE, BUFFERS) measured 957.571 ms for the legacy raw-event query and 232.228 ms for the interval query: a 75.75% reduction, or about 4.1x faster. Shared buffer hits fell from 320,914 to 106,039. The old correlated assignment subquery ran 7,570 times and repeatedly joined assignment parts, profiles, and employees; the new correlated lookup ran 7,570 times using conversation_assignment_intervals_conversation_time_idx at about 0.002 ms per lookup.

The planner chose the tenant/conversation/time B-tree and did not need a range operator or cross-conversation interval search. Do not add a GiST/range index for this rollout. Revisit only if post-backfill plans stop using the B-tree or a future query introduces range-overlap predicates across many conversations.

Rollout evidence checklist

Before advancing or baking any migration, record:

• US and EU request/job volume and error rate;
• BullMQ and GoodJob backlog, failed, delayed, retry, and fallback counts;
• data-parity or idempotency evidence appropriate to the workload;
• new Sentry issues and changes to known issue frequency;
• success thresholds, observation duration, and exact rollback order;
• the PR and deployment that remove the temporary selection branch;
• the flag deletion or documented reason it remains.