Progress log

Add new entries above older entries. Record material state changes only; routine audits with no change belong in the automation run summary rather than this log.

September 16, 2026 — QA evaluation backfill family: API2 made the permanent owner (ENG-3036 reopened)

Correction: the exact EU Rails-worker CloudWatch window 2026-09-15T15:06:59Z → 2026-09-16T15:06:59Z contained QAEvaluationBackfill::PreviewJob for backfill run 1156, organization 33, while the qa_evaluation_backfill_api2_owner gate was false. The run was cancelled, yet the Rails preview kept planning for roughly two hours and ended in a PostgreSQL statement timeout, so ENG-3036’s default-off gate did not meet the zero-normal-path-GoodJob target. The gate, the Rails dispatcher’s organization-gated GoodJob fallback, Organization#qa_evaluation_backfill_api2_owner?, the admin backdate perform_now branch, API2 resolveQaEvaluationBackfillOwner, the PostHog flag constant and the QA_EVALUATION_BACKFILL_OWNER override are removed; QAEvaluationBackfill::Api2Dispatcher now posts every stage to API2 for every organization and raises on non-202. The five Rails job classes stay only as operator recovery tombstones whose enqueue raises OperatorRecoveryTombstone::NormalPathDisabledError (explicit console perform_now only). Both previews (API2 runPreview and the Rails tombstone) gain a PlanBuilder checkpoint that re-reads run status every 5 s and enforces a 30-minute wall-clock budget, so a cancelled preview stops at the next agent or eligibility batch instead of at the final write. A source-level RSpec gate fails on any reintroduced direct job call; PGlite and Rails DB-backed specs cover the old-gate-false organization, manual admin backfill, cancellation during planning, retries, concurrent delivery and all five stages. Code preparation only: no flag, data, queue, scheduler, deployment or provider configuration changed. Closing ENG-3036 still requires the strict zero-row CloudWatch verification over one full daily cycle per region recorded here; see the runbook.

September 15, 2026 — Dashboard, catalog and search reads moved to API2

Progress: ENG-3131 (owner: Chidi) removes the Rails client from the eight remaining dashboard, catalog and global-search read roots in rulebase-ui (Notion pages, Zendesk brands, customer-issue statistics, both scorecard performance breakdowns, check results, data sources, legacy search). API2 gains native GET /notion_pages, /customer_issues/{id}/statistics, /scorecard_criteria/{id}/statistics, /check_results and /legacy_search; brands, data sources and workspace search reuse the existing routes. Rails tenant/policy scope, Pagy envelopes, filters, ordering, organization time-zone date bounds and aggregate denominators are ported one-for-one, with no API2-to-Rails call or fallback. PGlite acceptance uses org 25 / org 41 shapes and an ENG-3131 source gate walks the runtime import graph. Reconciliation queries and rollback triggers are in the package runbook. Bake runs through 2026-09-15 20:30 America/New_York; no merge or deploy in the package.

September 13, 2026 — Knowledge Library writes and Rails client prepared for API2-only UI

Progress: ENG-3048 (owner: Chidi) removes the Rails client from the ten Knowledge Library boundary files in rulebase-ui and gives API2 native document create/update/duplicate/tag/discard/sync and source create/update/delete/sync under /knowledge-library/* with knowledge_base:manage. The callback-equivalent effects (validations, normalized tags, pg_search refresh, chunk deletion/resync markers, provider source-identifier resync, section/source counters, discarded timestamps, organization-memory events, crawl-task deletion and document nullification) reuse the ENG-2940 primitives; queue handoff is a durable outbox committed with the write and swept by the maintenance queue, creates honour an Idempotency-Key, and there is no API2-to-Rails call or GoodJob bridge. The dead import-task poll is removed. PGlite acceptance uses org 25 / org 41 shapes; UI tests cover query keys, invalidation, navigation and errors, and a scoped source gate asserts zero Rails reachability. The rulebase-ui census drops from 240 source/test (229 runtime) to 233 (222 runtime); Beam stays 0/0. No flag, no fallback: the server-first PR (#11935) and the UI cutover PR (#11940) are separate, and the cutover PR is based on the server branch so it cannot merge before the API2 revision is verified live; rollback is code revert. Nothing deploys from these branches; the readiness and evidence gates are in the package runbook.

September 11, 2026 — conversation exploration and QA reads prepared for API2-only UI

Progress: ENG-3037 (owner: Chidi) removes the Rails client from the 14 conversation exploration and QA read files in rulebase-ui. API2 gains native GET /conversation_threads, GET /conversations/{id}/qa_evaluations, GET /conversations/{id}/conversation_cx_risk_snapshot_scorecard_results, GET /conversation_cx_risk_snapshots/{id}, GET /conversations/search, and GET /conversation-filters/{id}; the existing /conversations, /tags, and /conversation-filters reads are wired as-is. Tenant scope, prefixed IDs, filters, sort, pagination, thread aggregation, QA/CX-risk ordering, signed attachment URLs, and error/unauthorized behavior are reproduced and covered by PGlite acceptance with US org 25 / EU org 41 fixtures, UI client tests, and a scoped source-gate test. The rulebase-ui census drops from 253 source/test (242 runtime) to 239 (228 runtime); Beam stays 0/0. No flag, no fallback: deploy API2 before UI, rollback is code revert. Nothing deploys from this branch; the runtime gate and observation query are in the package runbook.

September 11, 2026 — QA evaluation backfill execution family prepared for API2

Progress: ENG-3036 moves the five-job QAEvaluationBackfill family (PreviewJob, ConfirmJob, QueueReplacementJob, BackdateJob, ReconcileJob) from GoodJob to a native API2 qa-evaluation-backfill BullMQ worker. The trigger was the post-release CloudWatch window 2026-09-11T20:00:03Z–20:07:19.776Z: US automatic runs 171 (organization 19, request 5899294, replacement run-agent 1512) and 172 (organization 25) produced 11 attempts / 11 unique normal-path GoodJob jobs and reset the zero-GoodJob clock; EU had none. The plan builder, coverage period, preview/confirm/replacement/backdate/reconcile orchestration and their run, run-agent and request state machines are ported natively with deterministic job ids per run, run-agent/replaced request, and reconcile wait attempt; created requests hand off to the existing API2 QA orchestration queue and the post-backfill report to the existing daily report queue, so no request lifecycle side effect is duplicated. Every Rails admin/report entrypoint and model callback routes through QAEvaluationBackfill::Api2Dispatcher; the default-off PostHog organization gate qa_evaluation_backfill_api2_owner selects the executor, an API2 non-202 raises rather than falling back, and the Rails job classes remain only as the gate-off rollback/manual recovery path. A PGlite suite drives the real worker composition with anonymized production-shaped tenants 19 and 25. Code preparation only: nothing is deployed and the flag is not created in production. See the QA evaluation backfill runbook.

September 11, 2026 — evaluation-contest submission moved to API2; production schema gap fixed

Progress: ENG-3033 (#11888 ) moved conversation and work-item contest submission plus reviewer email/Slack notification to API2, with the Rails create path reduced to a pending outbox row. Submission is not flag-gated: rulebase-ui calls POST /conversations/{id}/evaluation_contests on API2 directly. Notification ownership stays behind PostHog evaluation_contest_notifications_api2_owner (safe state Rails; canaries US org 25, EU org 41).

Production gap: immediately after deploy every contest insert failed with column "outcome" of relation "evaluation_contests" does not exist (Sentry RULEBASE-API2-DA ), so agents could not contest for roughly 20 minutes. The API2 Drizzle table and both PGlite acceptance fixtures were modelled on rulebase-api/db/structure.sql, which lists outcome text[] and assignee_id bigint on evaluation_contests; neither column has a Rails migration and neither exists in production, so the acceptance suite passed against a relation production does not have. #11894  dropped both columns from the Drizzle schema; the follow-up removes them from the PGlite fixtures and asserts the fixture’s column list matches production so the submission test exercises the real boundary. structure.sql still carries the phantom columns and should be regenerated from a production-shaped database.

Evidence query (per region, API2 database; expect zero rows after the fix deploys, and confirm the column list has no outcome/assignee_id):

SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = 'evaluation_contests'
ORDER BY column_name;
 
SELECT count(*) AS contests_last_hour
FROM public.evaluation_contests
WHERE created_at > now() - interval '1 hour';

Pair with Sentry: RULEBASE-API2-DA must record no new events after the #11894 release, and contests in the hour window must be non-zero during business hours for the canary orgs.

Rollback / current gate: submission has no runtime gate; rollback is a revert of the rulebase-ui contest mutation to the Rails endpoint (Rails EvaluationContestsController#create remains deployed) or a redeploy of the previous API2 release. Notification delivery rolls back via the PostHog flag or EVALUATION_CONTEST_NOTIFICATIONS_OWNER=rails as documented in the notification runbook . Retirement credit requires the focused DB suites (conversation-contest-writes.db.test.ts, work-item-contest-writes.db.test.ts) and API2 typecheck green, plus a clean 72-hour window on the runbook invariant.

September 11, 2026 — organization directory writes prepared for API2-only UI

Progress: ENG-3031 adds native API2 team create/update/discard, partner create/update/delete, membership update/remove, and invitation revoke/import, then switches the 13 remaining Rails-client runtime importers in the rulebase-ui Teams, Partners, Memberships, and Invitations features to rulebaseApi2. The routes reproduce Rails’ tenant checks, kept-name uniqueness, partner-domain normalization, scope replacement, full_workspace_access derivation, WorkOS-after-commit ordering, discard side effects, and audit_events rows. A PGlite acceptance suite with an anonymized production-shaped tenant covers positive, negative, rollback, ordering, retry, and replay paths. The rulebase-ui census drops from 266 source/test (255 runtime) to 253 (242 runtime). Nothing deploys from this branch; the 72-hour reconciliation query and rollback trigger are in the package runbook.

September 10, 2026 — provider field-definition sync prepared in API2

ENG-3010  moves SyncProviderFieldsJob and SyncProviderFieldsForOrganizationDataSourceJob off GoodJob. The 24-hour CloudWatch census ending 22:48 UTC recorded 12 US and 6 EU per-source attempts plus one 02:00 UTC sweep per region. API2 now owns the daily scan of kept Zendesk, Freshdesk, and Intercom sources, deduplicated per-source jobs that reuse the existing provider clients, Rails-compatible value-type and enum mappings, and idempotent upserts of conversation, customer, employee, and company fields with per-category and per-field error isolation. The Rails cron is kept but disabled by default and the job classes are a documented recovery-only shim behind pause_api2_provider_fields_sync. This branch deploys nothing; bake criteria and observation queries are in the package runbook.

September 10, 2026 — Knowledge Library document reads moved to API2

Progress: ENG-3003 moves the document list, memory list/count, detail and filtered navigation queries to one API2 read path. API2 reproduces Rails’ tenant scope, visible-record rules, recursive section selection, Zendesk locale and brand behavior, search membership/ranking, pagination, summary/detail payloads, and created-at adjacency. Focused PGlite route tests and UI client tests pass, as do API2 and Rulebase UI typechecks. The Rulebase UI Rails-client census drops from 291 to 289 files overall and from 281 to 279 runtime files; Beam stays at zero. Merge, regional deployment, live attribution and a clean observation window remain required before retirement credit. Details are in the package runbook.

September 10, 2026 — final Kuda gates enabled; summary and expiry cutover prepared

Progress: PostHog flags api2_daily_agent_evaluation_report_collect (273642), api2_conversation_relationships (273643), and api2_conversation_part_relationships (273644) are active only for Kuda ($group_key = kuda-pilot) and evaluate false for a control organization. These join the already Kuda-scoped XCALLY completion and Rulebase webhook flags. Natural runtime evidence is still required before widening.

The exact 24-hour GoodJob census found 48 US summary attempts / 47 unique and 56 EU summaries, plus 17 executions of each expiry class in each region. ENG-3000 routes new summary requests directly to API2 with a Rails delivery fallback, disables both Rails expiry crons by default, and adds one API2 hourly maintenance sweep using the shared Rails expiry logic through an authenticated synchronous endpoint. The expiry pause switch plus retained crons is the non-racing rollback. Focused Rails and API2 tests and API2 typecheck pass; merge, regional deployment and post-deploy queue evidence remain open.

September 10, 2026 — Rails Intercom timeout fallback removal prepared

Progress: a complete regional CloudWatch census over the rolling 24 hours ending at 16:04 UTC found 59 US ProcessIntercomWebhookEventJob attempts for 52 unique deliveries and none in EU. The correlated Rails requests had successful API2 traffic with zero database queries, while four API2 timeout/error requests performed four Rails queries and immediately created the legacy GoodJob. This identified the configured-error fallback as an active second owner rather than stale queue work.

ENG-2998 removes that fallback: configured Rails ingress now returns a retryable 503 when API2 delivery fails, so Intercom remains the retry owner. It also removes the obsolete API2-to-Rails internal webhook endpoint and teaches API2 to read the owning ticket ID from ticket-part notifications. Local Rails handling remains available only when no API2 base URL is configured. Deployment plus a clean regional window with no new legacy Intercom jobs remain required before retirement credit.

September 10, 2026 — globally enabled UI reads baked into API2

Progress: ENG-2554 removes the unreachable Rails read branches guarded by the four universally enabled API2 settings helpers. The Rulebase UI census drops from 311 to 291 Rails-client importer files overall and from 300 to 281 runtime importers; no rollout-helper references remain. Existing Rails-owned mutations and reads without a migrated API2 contract remain in place. Typecheck, lint and focused query tests cover the source cutover. Merge, deploy, live attribution and a clean observation window remain required before retirement credit. Details are in the package runbook.

September 10, 2026 — QA statistics read moved to API2

Progress: the QA overview, back-office overview, Radar and Analytics score cards no longer call Rails QAAgentEvaluationsController#statistics. API2 now owns the endpoint with the existing QA membership-policy scope and filter SQL, plus parity coverage for Rails’ local-day grouping, daily and period-level flooring, equal-length previous period, effective/active/completed rows, conversation/work-item date selection, and all UI filter dimensions. Focused API2 PGlite and UI client tests pass. The September 10 source census drops to 311 Rails-client importers (300 runtime); after ENG-2554 also lands, the projected combined census is 291 (281 runtime). API2 must deploy before the UI, then Sentry attribution must show this transaction absent before retirement credit is claimed.

September 10, 2026 — Beam prepared for API2-only runtime

Progress: ENG-2971 removes Beam’s eight remaining Rails-client importers and all Rails runtime/build configuration. Its session, organization, conversation, Zendesk import and Jira ticket flows now terminate on API2. Existing pending Rails Jira drafts are migrated idempotently into API2 before submission and the legacy pending row is canceled; the direct Zendesk import route remains tenant-scoped and hands the imported ticket to the existing native provider-sync queue with a stable run id. Focused API2 tests and the Beam production build cover the compatibility paths. Merge, regional API2 deployment, Cloudflare Beam deployment and a real sidebar login/ticket/Jira smoke remain required before production-retirement credit. Details and rollback are in the package runbook.

September 9, 2026 — Rails table ownership map published; legacy public.dashboard_* retired

Progress: rails-table-ownership classifies all 166 Rails-owned tables declared in api2 rails-schema.ts: 107 dual-written or Rails-model-coupled, 44 Rails-writer (API2 read-only), 5 framework-owned, and the 8 dead public.dashboard_* instance tables retired by renaming them to deprecated_dashboard_* (db/migrate/20260909225243). Declared dependency surface drops to 156. dashboard_widget_types and dashboard_widget_type_groups stay live — Rails still serves the widget-type catalog. Next package: WP-1 zendesk_webhook_ticket_syncs single-writer cutover behind api2_owns_zendesk_webhook_ticket_syncs.

September 9, 2026 — Rulebase webhook forwarding handed to API2

Progress: ForwardRulebaseWebhookJob leaves the normal path. It was a GoodJob bridge rebuilding {eventType, organizationId, conversation} for API2 /webhooks/rulebase; its only live producer is the conversation_assigned event from ConversationAssignment, reached through Conversation#assign_to_account! and all callers. Rolling-hour evidence on 2026-09-09: 3 US executions, 0 EU, all successful.

ENG-2958 defers delivery past the outermost transaction with ActiveRecord.after_all_transactions_commit, then Api2RulebaseWebhookEventEnqueuer POSTs {eventId, event} to POST /internal/rulebase_webhook_events/enqueue (1s open / 3s read). API2 durably enqueues rulebase-webhook-events before returning 202, deduped on jobId = rulebase_webhook_<event_id> (conversation_assignment_<id>). Classification: 4xx is a proven pre-enqueue rejection → retained ForwardRulebaseWebhookJob fallback; 5xx/transport is ambiguous → EnqueueApi2RulebaseWebhookEventJob retries the same event_id so BullMQ dedupe collapses retries onto the accepted job. The api2_rulebase_webhook_events PostHog flag (safe state off) is the rollback; /webhooks/rulebase and the worker share processRulebaseWebhookEvent, so payload semantics are unchanged. Flag rollout, deployment evidence, and the 24-hour zero-new-rows US/EU bake remain open; see the rollout gate and the traffic baseline.

September 9, 2026 — daily report row collection moved into API2

Progress: QAEvaluationReport::CollectRowsJob is the last normal-path GoodJob row in the daily agent evaluation report pipeline. The rolling-hour baseline (2026-09-09T19:56–20:56Z) showed 104 executions (70 US / 34 EU, all completed, zero errors) — the scheduled 20:00 UTC burst.

Row collection is now native to API2 behind the per-organization PostHog gate api2_daily_agent_evaluation_report_collect (off by default): both the tick’s pre-collection and the per-org job run the ported QAEvaluationReport::RowCollector semantics inside a single transaction with statement_timeout = 5min, bounded by the report worker’s concurrency: 3. The flag’s off state keeps the Rails /collect endpoint as the explicit rollback path — there is no automatic Rails fallback on native errors, so retries never produce an ambiguous outcome across two engines. Bake gate: both regions, BullMQ successes/failures/backlog at both 04:00 and 20:00 UTC bursts, parity sampling of rows, zero new normal-path CollectRowsJob rows for 24 hours, and DB load around both scheduled bursts. See rollouts and the routine GoodJob workloads runbook.

September 9, 2026 — API2 gains direct Zendesk and Jira webhook ingress

ENG-2967. Provider-facing Zendesk and Jira deliveries can now land directly on API2: POST /webhooks/zendesk verifies the shared registration bearer token (ZENDESK_WEBHOOK_BEARER_TOKEN, newly required in the API2 environment) and POST /webhooks/jira verifies the per-delivery HS256 JWT with the issuer pinned to JIRA_CLIENT_ID and context.cloudId resolving the tenant. Both feed the existing legacy-webhook-ingestion BullMQ pipeline with unchanged deterministic dedupe ids, retry policy, and ignored-event rules, and the session-auth exemption list gained only the two exact paths. Nothing about the registered provider destinations changed: Rails keeps its public ingress and forwarding code, legacy_webhook_rails_processing still reverts processing to GoodJob, and repointing each Zendesk subscription and Jira connection is a separate authorized ops step enumerated under Direct Zendesk and Jira webhook ingress. Intercom’s direct ingress contract (/webhooks/intercom with ordered US→EU fanout) is unchanged.

September 9, 2026 — XCALLY upload completion handed to API2

Progress: the last live CompleteConversationSyncJob producer leaves the normal path. Production evidence from the rolling 12-hour window on 2026-09-09: ~765 executions in the US from organization 25 / data source xcally, zero in the EU; every fresh XCALLY completion callback belonged to a GoodJob::Batch whose sole child was ActiveStorage::AnalyzeJob, produced by ConversationUpload::Processors::Xcally#process. Two Jira callbacks existed before the latest Rails deploy and none afterward — Jira stays out of scope.

ENG-2948 decouples the recording attach (ActiveStorage analysis stays in Rails) from completion: after the attach commits, Xcally#process claims conversation-upload/sync-completion/<id> in outbound_delivery_claims and hands completion to Api2ProviderConversationSyncCompletionEnqueuer, which POSTs /internal/provider_conversation_sync_completions/enqueue. API2 tenant-checks the target and enqueues the existing provider-conversation-sync-completion BullMQ job, pinned by sync_run_id = xcally-upload-<id> with the completed job retained 7 days. The claim is the single-winner contract across both dispatch paths: proven API2 rejection falls back to the GoodJob batch once; ambiguous delivery leaves the claim unresolved for Sentry-surfaced reconciliation instead of racing a Rails completion against a possibly-accepted BullMQ job. The GoodJob batch on_success also remains under the api2_conversation_upload_sync_completion flag’s off state. The verified-implicit wait on ActiveStorage::AnalyzeJob is dropped: nothing in complete_conversation_sync consumes analyzed blob metadata. Flag rollout, deployment evidence, and the zero-new-GoodJob bake remain open; see the traffic baseline and the routine workloads runbook.

September 9, 2026 — QA chunk sync and citation review/repair prepared for API2

Criterion-result chunk/embedding sync and the daily organization citation review/repair scan move to API2 BullMQ ownership. The Rails after_commit producer enqueues through /internal/qa_criterion_result_chunk_syncs/enqueue with a deterministic per-result job id and falls back to SyncQAAgentEvaluationScorecardCriterionResultChunksJob only on delivery failure or under the good_job_qa_criterion_result_chunk_sync Flipper switch. The review_ai_check_citations cron is disabled by default and replaced by the qa-citation-review-tick scheduler (05:00 UTC, yesterday); the pause_api2_qa_citation_review Flipper gate is the non-racing revert.

The API2 port keeps the exact verify_check_citation prompt (byte-equality tested), gpt-4.1/4000-token settings, candidate SQL with org-local half-open day bounds, deterministic PartLink repair, score recalculation order and Slack formatting. Organization scans checkpoint each processed scorecard result into job data and guard the Slack post with an outbound delivery claim, so retries resume without duplicate mutations or notifications.

The criterion feedback, expanded-comment and difference-feedback generators stay on GoodJob pending a prompt-parity harness for their Blueprinter-rendered context. This is code preparation only; deployment, two clean scheduled ticks per region and 24 hours of zero new normal-path GoodJob rows remain required. See the QA post-processing runbook.

September 9, 2026 — Zendesk daily reconciliation prepared in API2

ENG-2931  moves SyncZendeskTicketsJob and ImportZendeskConversationJob off GoodJob. The rolling 12-hour EU+US window ending 10:13 UTC recorded 5,029 executions for 2,738 unique import jobs with 2,291 rate-limit attempts, all created by the midnight sweep and with no real backlog: the Rails job retried throttles up to 60 times. API2 now owns the daily scan, per-source incremental export with enqueue-before-checkpoint watermarks, deduplicated ticket imports through the shared webhook import module, and provider-sync fan-out; a 429 re-delays the same job for Retry-After rather than consuming an attempt. The Rails cron is kept but disabled by default and the job classes are a documented recovery-only shim behind pause_api2_zendesk_ticket_reconciliation. This branch deploys nothing; bake criteria and observation queries are in the package runbook.

September 9, 2026 — relationship construction producers moved to API2

The two remaining GoodJob producers of relationship construction — CreateConversationRelationshipsJob (conversation subject and string field-value commits) and CreateConversationPartRelationshipsJob (reference-bearing part creates) — now hand off to API2 BullMQ through /internal/conversation_relationships/enqueue and /internal/conversation_part_relationships/enqueue behind PostHog flags. Delivery is ambiguous-retry-safe: a transport failure retries the same deterministic job ids through bridge jobs, and an explicit API2 rejection falls back to the GoodJob pass so a commit is never dropped.

This is code preparation only. Ownership requires deployment, a clean 12-hour window with no new normal-path GoodJob rows for the two classes in either region, healthy BullMQ acceptance and drain on the four link queues, and the bake observation in the relationship construction runbook.

September 9, 2026 — Knowledge Library sync prepared for API2

The Rails Knowledge Library synchronization pipeline moves to API2 BullMQ ownership: the 01:00 UTC source scan, per-source provider sync, document sync, crawl start and per-URL crawl, and native chunking/embeddings/search indexing. The trigger was the rolling 12-hour window ending ~10:13 UTC, in which SyncKnowledgeBaseDocumentJob produced 2,629 executions for 158 unique jobs and 2,488 US GoodJob throttle failures — GoodJob’s perform_throttle retrying throttled jobs as failures, not provider load. API2 replaces that with Redis fixed-window provider limits (Notion 4/minute, Scribe 50/minute, other documents 3,000/minute, source sync Notion 4/minute and others 20/second) that park a job in BullMQ’s delayed set instead of consuming a retry attempt, a 60/minute queue limiter on the chunk worker, deterministic kb-* job ids with waiting/active dedupe, and three bounded attempts per unit.

Every provider path is ported natively to API2 (src/lib/knowledge-base-sync/providers/*): Zendesk brands/translations/ recursive sections, Notion search and block rendering, Confluence pages and blog posts, API-key Scribe listings and guide/page rendering, and the URL crawler (HTML sanitising, same-host link discovery, LLM article extraction with the copied Rails prompts, Redis visited set and urls_pending counters). Runners write the Rails Knowledge Library tables directly, tenant-scoped, and return follow-up work (document syncs, chunk syncs, child crawl URLs) for API2 to enqueue under deterministic ids, so a retried unit cannot fan out recursively. The only Rails boundary left is ActiveStorage image rehosting (now also for Zendesk attachments with the data-source token); the Rails provider classes stay only as the GoodJob rollback path. Every Rails producer now routes through KnowledgeBaseSyncDispatch: an API2 202 is the only durable acceptance, and any failure or the Flipper pause_api2_knowledge_base_sync kill switch enqueues the retained GoodJob job instead. The Rails cron stays disabled by default; Intercom and session-token Scribe keep their existing API2 owners; API2-native agent/workflow knowledge tables are not involved.

This is code preparation only. Deployment, one clean daily tick per region, zero new normal-path GoodJob rows for the six *KnowledgeBase*Job classes (retained 12-hour rows are historical), no fallback warnings, and 24 clean hours remain required for ownership. See the Knowledge Library sync runbook.

September 9, 2026 — weekly agent performance reports move to native API2

Moves the full weekly agent performance report pipeline to API2/BullMQ: scheduling (daily 08:00 UTC generation, Monday 09:00 UTC preview + 30-minute auto-send), per-organization fan-out, report lifecycle, native OpenAI content generation (pmpt_687b..., gpt-4.1), citation persistence, and Resend email delivery. Report preparation, citations, and emails run natively in src/lib/agent-performance-reports/ — no Rails /internal/* domain endpoints execute the work. Pre-cutover evidence (EU+US rolling 12h ending ~2026-09-09 10:13 UTC): 53 GenerateWeeklyAgentPerformanceReportsForOrganizationJob and ~52 GenerateAgentPerformanceReportContentJob executions, ~107 for the theme.

Per-org progress markers persist in BullMQ job data and every outbound email is claimed in public.outbound_delivery_claims, so retries cannot duplicate reports, content, citations, or notifications. The two Rails cron entries stay registered but enabled_by_default: false, and pause_api2_agent_performance_reports gates every tick and org worker; generate_later, admin actions, and rake tasks post to the api2 /internal/agent_performance_reports/enqueue* routes instead of perform_later. Legacy job classes remain for already-enqueued rows and rollback. This is code preparation only — deployment, real executions in both regions, zero new GoodJob rows for the five named classes, and 24 clean hours remain required. See the weekly agent performance reports runbook.

September 9, 2026 — simulation scheduling and notifications retired

ENG-2926 applies the approved simulation product retirement to Rails runtime ownership. It removes the configured daily reminder cron, stops assignment and simulation updates from creating notification/reminder jobs, and removes the mailer execution path from reminder records. Four job classes stay deployed as successful no-op tombstones until legacy queued and delayed GoodJob rows drain.

No simulation data or reminder history is deleted. Deployment must prove the cron is absent, no new simulation notification work is created, and existing rows drain without delivery. UI routes, Rails controllers, Action Cable and the remaining generation/report jobs stay explicit follow-up cleanup before Rails source retirement.

September 9, 2026 — routine GoodJob workload families prepared for API2

ENG-2920 moves the four remaining routine workload families from the September 8 queue snapshot to API2 BullMQ ownership: the ten-minute Unit21 source sync, thirty-minute conversation-upload polling plus controller-triggered direct-upload processing, the 04:00/20:00 UTC daily agent evaluation reports and their per-organization fan-out, and the three-hourly counter-cache repair. API2 owns schedulers, durable execution, retries, and per-tenant failure isolation; Rails keeps only narrow authenticated /internal/* endpoints that execute the model-bound work.

The five Rails cron entries are paused in the same candidate via enabled_by_default: false and guarded by pause_api2_* Flipper kill switches, keeping a non-racing code-revert path. Per-org report jobs persist step markers in BullMQ job data, so retries skip completed delivery steps and cannot double-deliver Slack or email. Every legacy producer — upload create, admin report trigger, post-backfill notification, and employee-merge counter repair — now enqueues through API2 rather than perform_later.

This is code preparation only. Deployment, real scheduled executions in each region, zero new normal-path GoodJob rows for the six named classes (retained 12-hour rows are historical), bounded queues, and 24 clean hours remain required for ownership. See the routine workload runbook.

September 8, 2026 — implementation milestone cleared; runtime queue gate remains

PR #11533  merged to main at 7111e1ac219f9f0b65b923ac12c0f998369da30f with all four API2 test shards, aggregate CI, build, formatting, Codecov, and automated review green. ENG-2908 and ENG-2909 are done. This clears the September 8 parity-ready implementation gate for the scoped retained contracts; it does not certify zero GoodJob traffic or Rails retirement.

Restored AWS production access enabled fresh read-only regional database and worker-log queries. Full-day Rails worker logs establish the missing historical comparison. Combined US/EU executions averaged 1,885,736 per UTC day on August 1–7 and 239,684 on September 1–7, down 87.3%; medians fell from 2,007,890 to 178,121, down 91.1%. Exact selected-day unique jobs fell from 1,301,257 on August 4 to 94,093 on September 7, down 92.8%. A Zendesk import retry storm raised September 7 executions to 311,989 from only 94,093 unique jobs. See the GoodJob traffic baseline for the daily series and query semantics.

The September 8 GoodJob snapshot retained roughly 34,000 EU and 16,000 US completions, but production cleanup preserves only 12 hours. The leading application classes were Zendesk ticket handoff (30,182 combined), Jira webhook processing (5,554), CX risk (2,759), dispute acknowledgement (2,520), legacy Zendesk events (1,005), and conversation completion (877). Active Storage analysis and Rails-only CloudWatch publication were explicitly excluded from application prioritization; autoscaling and heartbeats are likewise worker operations rather than product workloads.

Zendesk ticket work normally reaches API2 already, but Rails still owns the initial 30-second GoodJob debounce and API2 can schedule follow-up work back through Rails. Jira still validates, resolves, and applies issue/comment event semantics in Rails; the Jira Assistant webhook is not a compatible replacement. Two implementation handoffs are in progress: one moves both providers through a temporary synchronous Rails-to-API2 durable-accept relay and then to direct API2 URLs; the other moves dispute acknowledgement scheduling and processing to API2 while filtering unchanged disputes before enqueue. These receive no completion credit until their PRs, regional deployments, actual work, and bake windows pass.

The next ordered candidates are the remaining conversation-completion residue, then the broader CX-risk entry point and state transitions. The target remains regional cutover by September 11, clean final operation on September 12–14, and approved Rails/GoodJob retirement on September 15.

September 8, 2026 — native QA ownership rollout gate passed

The staged native QA ownership gate passed at 22:01 UTC. The exact ticket-closed actor sets remain US rho-tnkw (19), kuda-pilot (25), and EasyPay (38), plus EU Qonto (33), EasyPay (38), and Nala (41). All five native QA PostHog flags are enabled in both regions: result import, completion, and the complaint, customer-issue, and compliance evaluation-check bodies.

Since 2026-09-07 21:59 UTC, native result-import markers totalled 167 for US Rho, 289 for US Kuda, 2,027 for US EasyPay, 923 for EU Qonto, and 13 for EU Nala. EU EasyPay was quiet in that window; its customer had substantial native traffic through the US tenant. The final audit found no overdue active ticket-closed request, duplicate active request, duplicate active AI evaluation, marker inconsistency, new stranded CX-risk snapshot, or new terminal queue failure. Eight older inactive/running CX-risk snapshots remain the unchanged cleanup baseline and do not block this gate.

US Kuda request 5863910 stalled during worker replacement and Redis churn before the recovery release, then timed out at 18:00 UTC. PRs #11508  and #11512  added graceful worker shutdown, longer BullMQ locks, terminal-stall replacement, expired-lease recovery, and wiring coverage; corrected workers were live by about 18:09 UTC. Later transient Rails 500/504 and Redis connection errors completed on their second BullMQ attempt. The recovered deployment-time incident is recorded as non-blocking because the post-fix checkpoint was clean and showed no recurrence. Keep rollback flags and retained fallbacks until their separate bake-in removal changes land.

September 8, 2026 — native QA attachment captions prepared

ENG-2887 removes the Rails request-stage callback for the remaining attachment portion of caption_images. Under the active request lease, API2 selects one tenant/conversation-qualified attachment, reuses and re-redacts same-binary captions when safe, otherwise signs the shared R2 object and runs the existing attachment caption executor. Completed, unsupported and failed outcomes persist directly; transient exceptions stay pending but are checkpointed for this request.

Redaction-changed output tombstones the row and its ActiveStorage link in one transaction. The underlying blob row and object are removed only when no other attachment references them. Production deployment, actual US/EU attachment candidates and 24 clean hours remain open; legacy Rails/GoodJob behavior remains the rollback path.

September 8, 2026 — native QA conversation-part captions prepared

ENG-2885 removes the Rails request-stage callback when caption_images is advancing ordinary conversation-part image candidates. API2 selects one tenant/request/conversation-qualified part under the active stage lease, runs the existing caption executor, and checkpoints attempted IDs even when the provider fails or the author is intentionally ineligible. A bad part therefore cannot starve later caption candidates.

Attachment candidates deliberately retain Rails stage ownership so cache reuse, tombstoning and ActiveStorage deletion are not split across runtimes. The existing caption context and stale-write persistence endpoints also remain. Deployment, actual US/EU candidates, bounded fallback counts and 24 clean hours remain open; this is a partial dependency removal rather than full QA parity.

September 8, 2026 — native QA related refresh prepared

ENG-2731 moves normal kept/supported related-conversation refreshes from the Rails QA stage endpoint to the existing native provider-sync engine. API2 selects the same root plus five preceding/five subsequent thread conversations, persists one conversation ID and stable sync-run ID under the request lease, and checkpoints the attempt after the provider outbox completes. Ambiguous delivery therefore reuses one provider-sync run instead of duplicating work.

Missing sources and provider failures preserve Rails’ best-effort behavior. Cross-tenant conversations or sources fail closed. Unsupported, discarded or timestamp-incompatible sources deliberately continue through the Rails compatibility primitive until their production population is censused. Deployment, actual supported-source traffic in US/EU, provider-outbox and stage ledger parity, zero supported-stage Rails calls and 24 clean hours remain open. Captioning, eligibility/sampling, evaluation creation and native terminal failure effects are still required for the Rails-unavailable QA gate.

September 8, 2026 — QA preparation completion prepared for API2

ENG-2731 moves mark_prep_complete from the Rails stage adapter to native API2 SQL. The request row is selected by numeric tenant, request and active stage lease, then prep_completed_at and ready are persisted in one transaction. Rails-compatible started/completed/failed events use the shared Action Cable transport, with terminal events after commit or rollback.

Focused unit and PostgreSQL-backed tests cover native routing, tenant/lease rejection, state persistence and broadcast ordering. Captioning, related-provider sync, eligibility/sampling, evaluation creation and terminal failure effects remain explicit Rails boundaries. Deployment and one actual stage transition in each region remain required; this does not close the QA domain.

September 7, 2026 — WorkOS event processing prepared for API2

ENG-2858 transfers the ten-minute WorkOS Events poll and retry owner from Rails GoodJob to API2’s maintenance BullMQ queue. API2 requests the exact repeated 17-type filters used by the Ruby client and parses raw JSON so snake-case keys reach native handlers unchanged. A transaction-scoped event lock and the database-unique ledger reject an ID reused with different content and process a matching retry at most once.

All effects now run in API2: organization bootstrap and Grant customer setup, account updates, membership/employee lifecycle, invitation acceptance, permission/role synchronization and idempotent system audit history.

The shared regional cursor advances optimistically only after every event in a page succeeds; a partial failure therefore replays through the idempotent event boundary. Read-only production preflight found exactly one initialized cursor in each region. The Rails job class and payload effects remain dormant for queued work and rollback, but its recurring cron entry is removed in the candidate.

This is code preparation only. Deployment, one real event in each region, no new Rails recurring execution or event callback, bounded queues, no duplicate effects or payload conflicts, and 24 clean hours remain required for ownership.

September 7, 2026 — assignment intervals ready for public-schema cutover

ENG-2682 migration0344 moves the physical assignment-interval table, owned sequence, primary-key index and rebuild function from api2 to their canonical public.conversation_assignment_intervals* and public.rebuild_conversation_assignment_intervals names. It preserves the existing object OIDs and installs an automatically updatable table view plus a routine wrapper at the old api2 names so old Rails and API2 tasks remain safe during rolling deployment.

Fresh 13:17 UTC read-only preflight passed in both regions. The migration0336 routine body, owner, volatility and security mode match; explicit and default ACLs are absent; all five indexes are valid and ready; target names are unoccupied; and there were no lock waiters or blocked sessions. The physical tables were about 60 MB/148k estimated rows in US and 8.5 MB/28k in EU. The move is catalog metadata only: it does not copy rows or rebuild indexes.

The PostgreSQL 15.10 rehearsal applies migrations 0326, 0336 and the deployable move verbatim. It proves legacy and native routine calls, legacy view insert/update/delete, preserved object identities, routine behavior, atomic failure, and a lossless reviewed reverse transaction. The migration refuses catalog/body/ACL drift and uses 2-second lock and 30-second statement timeouts. Regional deployment and post-deploy traffic observation remain open; aliases stay until old tasks drain and the rollback window closes.

September 7, 2026 — native headless authentication prepared

ENG-2842 and PR #11358 move the full first-party email-first WorkOS surface to API2: feature detection and CSRF setup, method/Magic Auth, password login and signup, email verification and password reset. API2 persists anonymous attempts with private Rack IDs and Rails-compatible masked CSRF, reuses native hosted-login provisioning and session rotation, scopes organization deep links, recovers raced one-time-code completions and preserves enumeration-safe errors and Rails-equivalent rate limits.

The existing headless_auth Flipper gate protects CSRF setup and every mutation; it selects hosted AuthKit when off or unavailable. Rails keeps the same endpoints during observation for code rollback. This package makes no provider, flag, schema, data or queue change. Regional deployment, actual login/signup/verification/reset evidence, zero Rails headless requests and 24 clean hours remain required before deleting Rails actions or claiming production ownership.

September 7, 2026 — WorkOS regional 404 parity restored

The first post-cutover production audit found the daily API2 WorkOS authorization reconciliation retained one failed BullMQ job in each region. US organization 36 referenced a WorkOS organization missing from the US environment; seven EU local organizations referenced WorkOS organizations missing from the EU environment. Retries repeated the same terminal 404s while the separate membership reconciliation completed normally.

Rails explicitly skipped WorkOS::NotFoundError for this regional drift, but API2 classified every organization-role listing error as retryable. ENG-2815 now restores that contract by skipping only the WorkOS SDK’s typed not-found exception, incrementing a separate skipped-organization count and continuing with remaining organizations. Transient and unexpected failures still fail the aggregate attempt and retain the existing three-retry policy.

This is a code-only parity correction. After deployment, require the next real 03:15 UTC run in US and EU to complete without a retained maintenance failure; the skipped counts should match the known regional drift unless the underlying WorkOS references change.

September 7, 2026 — invalid Kuda churn values isolated

The US CX-risk completion queue was actively retaining HTTP 422 failures for Kuda. A read-only database audit found 179 snapshots created in the preceding 24 hours still running: 162 stored raw churn_risk_level = 3 and 17 stored churn_risk_level = 4, while the Rails snapshot enum accepts only 0 through 2. Rails logs identified ActiveRecord::RecordInvalid: Churn risk level is not included in the list; every sampled snapshot had terminal evaluation checks and a valid customer-issue churn score.

ENG-2837 makes the Rails finalizer recompute and persist the canonical snapshot churn enum from that terminal score in the same transaction. This repairs the invalid in-memory value before validation and preserves the existing thresholds. After deployment, require new Kuda snapshots and their BullMQ callbacks to complete without 422s. The existing stranded rows and terminal queue jobs need a separately authorized production repair; this package performs none.

September 7, 2026 — native QA completion and evaluation-check bodies prepared

PR #11312  merged at 59c615b15e7 with native organization context loading, native conversation-QA completion and native complaint, customer-issue and sales-call compliance checks. The final CI run passed the API2 build, all four API2 shards, all four Rails shards, aggregate CI, coverage, Prettier, RuboCop and Rails lint.

Native QA imports now record completion_owner as api2 or rails. That choice is made once when the result run is accepted and remains authoritative for replay even if rollout flags later change. API2 completion handles terminal request and agent state, override lineage, parent and conversation scores, workload linkage, billing dispatch and conversation-summary dispatch. It rejects payloads that combine failure with skip or terminal state with result rows before importing anything.

The evaluation-check worker can execute complaint, customer-issue and compliance bodies without the synchronous Rails executor. It preserves the per-check advisory lock, terminal idempotency, CX-risk completion identity and customer-issue assignment effect. Organization lookup preserves Rails’ slug-first behavior before falling back to a numeric internal ID.

This is code preparation only. A read-only audit of PostHog project 63353 at 01:41 UTC found no feature flags matching native_; the earlier result-import flag and all four new flags are therefore absent and evaluate through their fail-closed paths. Flag creation, staged enablement and 24-hour regional observation remain open.

September 7, 2026 — ticket-closed ownership expanded to Kuda and Qonto

The remaining production cohort entered the organization-scoped rollout after regional preflight. US kuda-pilot (organization 25) was enabled at 00:33:21 UTC; EU Qonto (organization 33) was enabled at 00:35:05 UTC. Kuda had zero overdue active ticket-closed requests and zero duplicate active conversations. Qonto’s 124 pending requests were all deliberately scheduled between 01:01 and 23:52 UTC, leaving zero requests overdue by 15 minutes; it also had zero duplicate active conversations. The partial active-request index was valid, ready, and unique in both regions.

The exact enabled actor sets are now US rho-tnkw (19), kuda-pilot (25), and easypay (38), plus EU qonto (33), easypay (38), and nala (41). The final 24-hour observation checkpoint is September 8 at 00:36 UTC.

September 7, 2026 — Rho and Nala rollout targets corrected

The September 6 US actor rollout targeted rho (organization 18), but Rho’s real production tenant is rho-tnkw (organization 19). At 2026-09-07 00:23:23 UTC, the api2_qa_evaluation_request_processing Flipper actor was atomically changed from organization 18 to organization 19. The exact US cohort is now rho-tnkw (19) and easypay (38). The unintended US nala actor (48) was removed at 00:30:15 UTC; Nala’s production cohort is EU organization 41. The exact EU cohort remains easypay (38) and nala (41). The higher-risk kuda-pilot US tenant (25) and Qonto EU tenant (33) remain disabled, as do rho (18), rho-uq0j (20), and the other similarly named Kuda records.

Preflight found zero active ticket-closed requests older than 15 minutes, zero conversations with duplicate active requests, and a valid, ready, unique partial active-request index for organization 19. The observed Rho evaluation created at 00:00 UTC completed through the Rails GoodJob fallback before the corrected flag enablement, so it is not API2 ownership evidence. EasyPay and EU Nala retain their September 6 observation clock; Rho’s 24-hour clock starts at the corrected enablement time and ends September 8 at 00:23 UTC.

September 6, 2026 — remaining Rails bindings made cutover-selectable

ENG-2682 removes the final four hard-coded Rails model bindings and the direct agent-view ACL probe from the executable schema-cutover blocker set. A shared Api2::RelationBinding maps every current relation to its reviewed public destination and selects public relations independently through RULEBASE_API2_PUBLIC_RELATIONS. The empty default preserves all current api2 bindings; unknown relation names fail application boot instead of silently binding the wrong table. Status-incident queries resolve their organization relation through the same map, including the collision-safe public.api2_organizations destination.

The query-data diagnostic now resolves public.agent_conversations first and falls back to agent_api.conversations for kind, schema-usage, table-access and owner probes. The regenerated exact-source census falls from 14 to nine executable blockers. Every remaining blocker belongs to the guarded work-item-resource retirement script and intentionally stays parked until old workers drain. This package changes no default binding, schema object, grant, queue or production data; setting the allowlist remains a separately ordered cutover operation.

September 6, 2026 — assignment-routine callers made dual-schema compatible

ENG-2682 removes the six direct api2. assignment-routine caller references from Rails Jira/Zendesk sync and API2 provider persistence/backfill execution. Both runtimes now prefer public.rebuild_conversation_assignment_intervals(bigint,bigint,bigint) when the physical move installs it and fall back to the current api2 signature on old-schema deployments. Rails preserves its pre-install no-op; API2 fails a job rather than falsely reporting a completed rebuild when neither signature exists. Focused Rails and PGlite tests cover public preference, old-schema fallback and missing-routine behavior. This deploys no schema change and does not remove the required api2 wrapper.

The production release at ef7193ab530 contains evaluation-check delivery commit d231a0ae7ee. Initial read-only observation found 1,547 completed US and 9 completed EU BullMQ runs, no waiting/delayed/failed run jobs, no failure-queue jobs and zero GoodJob fallbacks. The US snapshot had one active run. Complaint, compliance and customer-issue checks completed in both regions. Keep the full 24-hour observation gate open.

September 6, 2026 — ticket-closed API2 ownership staged for three customers

The organization-scoped api2_qa_evaluation_request_processing Flipper gate was enabled for Rho, EasyPay, and Nala after the repair and uniqueness prerequisites passed. US initially enabled rho (organization 18), easypay (38), and nala (48) at 17:30:42 UTC; the Rho actor was corrected to rho-tnkw (organization 19) on September 7. EU enabled easypay (38) and nala (41) at 17:31:02 UTC; Rho has no production EU organization.

The writes were guarded by exact organization IDs and slugs, a serializable transaction, zero stale ticket-closed requests older than 15 minutes, and zero duplicate active conversations. Application-level Flipper checks returned true only for those production organizations. The Rho sandbox organizations plus all Kuda and Qonto organizations remain disabled.

The zero-minute observation found no new target ticket-closed requests, no target BullMQ failure, no stale target request, and no duplicate active target conversation. Both regional queues had zero waiting, active, delayed, or paused jobs. US retained seven older failed jobs unrelated to the enabled cohort; EU retained none. Continue the 30-, 60-, 120-minute, and 24-hour checkpoints before expanding or baking in ownership.

September 6, 2026 — automated-QA repair and invariant verified in production

Read-only ECS-exec checks at 17:21–17:22 UTC closed the data and database prerequisites for staged ticket-closed ownership. US and EU both have zero active ticket_closed evaluation requests older than 15 minutes and zero conversations with duplicate active requests.

The exact partial unique index public.idx_qa_eval_requests_active_conversation_unique is present in both regions with indisvalid, indisready, and indisunique all true. Its predicate covers pending, ready, and processing rows by conversation_id, matching the reviewed invariant.

No production row, flag, queue, schema, or infrastructure state changed during the verification. The per-organization staged enablement followed at 17:30–17:31 UTC and remains in its observation window.

September 6, 2026 — evaluation-check run delivery prepared

The evaluation-check package moves normal Evaluations::RunEvaluationCheckJob delivery and its five-attempt retry lifecycle from GoodJob to API2 BullMQ. Rails emits a deterministic state-event key and falls back to the retained job only when immediate API2 delivery fails. API2 retains exhausted jobs and calls the authenticated Rails failure transition through a separate deterministic, eight-attempt BullMQ recovery job after the final execution attempt.

Rails temporarily keeps the tenant-scoped complaint, customer-issue and compliance check bodies. Its run and failure endpoints share a per-check PostgreSQL advisory lock; failure retries on contention and both endpoints no-op for terminal or missing checks, making ambiguous delivery and late retries safe. Focused API2 and Rails tests cover durable identity, payload conflicts, authentication, tenant isolation, terminal idempotency and fallback. This records no production credit until both regions observe real check types and a clean 24-hour window.

September 6, 2026 — WorkOS authorization reconciliation prepared

ENG-2815 moves the daily 03:15 UTC WorkOS permission and role import from the Rails GoodJob cron to a deterministic API2 maintenance scheduler. API2 imports only assignable, non-widget permissions; persists WorkOS identifiers and slugs; creates global environment roles and tenant-matched custom roles; and adds assignable role permissions without removing local permissions from partial remote payloads.

Transaction advisory locks and database uniqueness make ambiguous retries idempotent. Individual permission, environment-role, organization-list and organization-role failures are isolated, counted and surfaced through a failed aggregate BullMQ attempt with four total attempts. Rails retains its job and authorization service for queued work, remote-write callers and code rollback. No production ownership is recorded before regional deployment, a real 03:15 run, parity and 24-hour observation.

September 6, 2026 — regional captures reconciled; routine re-pinned to migration0336

ENG-2682 took fresh full-catalog/ACL captures from both production regions via ECS exec at ~15:00 UTC (US PostgreSQL 15.17, EU 15.14). The organization identity bridges are clean in both regions: 0 missing numeric mappings, 0 missing legacy rows, 0 slug mismatches, 0 duplicate numeric-mapping groups.

The live api2.rebuild_conversation_assignment_intervals body in both regions now matches immutable migration0336 (body sha256 496d0c19…), not migration0326 that the pinned evidence targeted. The routine rehearsal and assignment-routine-production.json were re-pinned to 0336; the rehearsal now applies 0326 then 0336 in deployment order and reruns green locally, with VOLATILE/SECURITY INVOKER, no settings and default ACLs unchanged.

The manifest was reconciled with the captures: the three ENG-2775 tables (api2.bot_topic_escalation_analyses, bot_topic_escalation_reasons, bot_topic_runs, collision-free, live in both regions) and their four foreign keys were added, and the two fk_qa*_work_item keys removed this morning as stale were re-added — they are validated live in both regions and absent from the checked-in structure dump only because the Rails migration adds them conditionally. Totals move to 197 tables, 719 indexes and 373 affected foreign keys per region. The four api2.work_item_resource* tables and their nine foreign keys stay in the manifest with a deploy-pending note; live captures show 193 tables and 364 foreign keys until migration0327 reaches production. Live production has also dropped legacy public.conversation_assignment_intervals, so 28 table/26 index collisions exist live versus the 29/27 pinned against the checked-in structure; the api2_ destination stays pinned.

The writer inventory was regenerated at b7a6e34c4bb: 20 executable blockers remain after the three operational/API2 dynamic bindings (sla-outcomes.ts weekly/target selection, holiday-calendar refresh) were made schema-agnostic by interpolating Drizzle table objects. The structure rehearsal now reports manifest foreign keys absent from the dump instead of failing on them, and all three local rehearsals rerun green. No production schema, data, role, grant or service changed; productionCutoverReady remains false pending ACL/default-grant review, caller drain and lock/load budgets.

September 6, 2026 — current-main schema rehearsal restored

ENG-2682 refreshed the public-schema plan from 1baf2de490 through current main 0a076d5d639. The previous full rehearsal failed before movement because it treated the already-reviewed assignment-interval routine as an unsupported new object. The fixed check now matches routine identity arguments exactly, reports every unplanned relation and index collision together, and compares all API2-owned or API2-referencing foreign keys with the checked manifest.

The manifest adds 14 native tables and 16 foreign keys, removes two stale QA-to-work-item foreign keys, and records the new public collision for the conversation-assignment table, owned sequence and primary key. PostgreSQL 15.10 now passes forward, old-alias, physical-name and reverse plans for all 193 tables in the checked Rails structure, 347 unique indexes, 207 named constraints, 12 sequences and 367 affected foreign keys. The newly merged Drizzle-owned QA result-import table has a reviewed destination and is reported as absent from that structure snapshot. The exact-main writer inventory grows from 14 to 23 blockers; the nine additions are confined to the separately guarded legacy screening-resource retirement command.

No production schema, data, role, grant or service changed. Regional catalog, ACL/default-grant, identity and lock/load proof is still blocked on restored AWS access, and productionCutoverReady remains false.

September 5, 2026 — WorkOS membership reconciliation prepared

ENG-2809 moves the 03:00 UTC organization-membership reconciliation from Rails/GoodJob to API2’s maintenance BullMQ queue. The native task preserves WorkOS pagination and statuses, kept-row lookup by workos_membership_id, tenant mismatch protection, status/role no-op behavior, and isolation between memberships and organizations. Missing organization roles are imported with only the permission keys Rails currently permits for assignment; shared environment roles are reused read-only and remain owned by the separate authorization sync.

Per-membership and per-organization errors remain isolated for progress, but make the BullMQ attempt fail so three bounded retries can recover transient WorkOS or database errors. Aggregate logs and Sentry counters contain internal IDs and counts only. Unit and PGlite coverage verify pagination, failure isolation, persisted role and status updates, and permission filtering. This adds no production credit; regional deployment, a real daily run, retry/queue health, parity and 24-hour observation remain open.

September 5, 2026 — evaluation-schedule BullMQ ownership prepared

ENG-2808 moves the 15-minute due-schedule scan and durable per-window delivery from GoodJob to API2/BullMQ. The bounded scan orders due active schedules by window and ID, attempts every enqueue before retrying failures, and uses a schedule/window job identity with retained exhausted failures.

Rails temporarily retains its production-proven batch-construction transaction behind an authenticated tenant-scoped endpoint. The endpoint locks and rechecks the expected window, making delayed retries safe after schedule edits and making ambiguous replays no-ops after a batch advances. Focused API2 and Rails tests cover query selection, bounds, retries, tenant isolation, replay, and rollback wiring. This records no production ownership until the revision is reviewed, deployed, and observed on real windows in both regions.

September 5, 2026 — CX-risk completion fan-in prepared

ENG-2249 moves normal CX-risk snapshot completion signals from Evaluations::CheckCompletionJob to a deterministic BullMQ queue. The event ID is bound to the ordered evaluation-check state, so duplicate delivery reuses one job while a later terminal transition produces a new one. Eight API2 attempts and retained failures recover transient Rails finalizer outages without rerunning the evaluation models.

Rails still owns the final database and model effects behind an authenticated, tenant-qualified endpoint. Both API2 delivery and the retained GoodJob fallback enter that finalizer through the same per-snapshot advisory lock, preventing an ambiguous enqueue response from duplicating completion. This is a bounded fan-in transfer, not native CX-risk result ownership. Production credit requires real successful events, bounded fallback/failure counts, no stranded running snapshots, and a clean 24-hour US/EU observation window.

September 5, 2026 — provider post-transcription continuation prepared

The latest Sentry census ranks provider post-sync analysis as the largest ready internal Rails bridge after external ingress and conversation summary work. ENG-2803 and ENG-2962 move the provider-sync transcription-versus-continuation decision, exact fixed-window throttle, eligible call/image selection, deterministic BullMQ fan-in, terminal transcription state, transcript rebuild, and translation enqueue into API2.

API2 now applies the responder, Jira customer-field, and stale-ticket guards and enqueues CX-risk starts directly. Rails retains a narrow translation fallback endpoint when native delivery fails; existing broad provider analysis and transcription-completion endpoints remain compatibility-only for rolling deploy and code-revert rollback. Focused API2 assertions and changed-file RuboCop pass; full typecheck and Rails request/rswag runs remain environment dependent. No flag, schema, queue, provider URL, deployment, replay, or production-data mutation is included, and no production credit changes before the exact revision bakes in both regions.

September 4, 2026 — native conversation-QA result import prepared

ENG-2801  makes API2 the tenant-scoped result-tree writer behind a default-off PostHog flag. The finalization worker resolves only the agent/scorecard/criterion rows pre-created for that evaluation, rejects missing or duplicate pairs, imports criterion and check results, derives parent/scorecard/QA aggregates, and claims the immutable result-run ID in one transaction.

A new api2.conversation_qa_result_imports marker records the owning run and payload digest. This keeps a committed native import on the native retry path if the retained Rails effects callback fails or the flag later changes; an already-accepted unmarked run stays on the legacy Rails writer. Rails receives a new effects-only callback that cannot import rows and retains agent/request completion, sampling, notifications and billing. Focused API2 database/unit tests and Rails request tests cover atomic rollback, tenant and run conflicts, legacy fallback, retry idempotency, success, skip and terminal failure. No flag or production data changed; migration deployment and cohort evidence remain open.

September 4, 2026 — direct Intercom webhook ingress prepared

The fresh Sentry transaction census ranks Intercom webhook receipt as the largest remaining Rails web surface at about 209 requests per minute. ENG-2800 therefore prepares the existing API2 /webhooks/intercom route to become the provider destination without changing Intercom configuration in this package.

Production US API2 now synchronously sends the exact authenticated raw delivery to EU API2 before local parsing or effects. The regional copy uses the existing shared internal bearer and loop-prevention header; EU and already-forwarded deliveries never fan out. A failed or missing required regional target returns 500 before US processing so Intercom can retry the complete delivery. Focused tests cover topology selection, exact-body forwarding, authentication ordering, loop prevention, and fail-closed behavior.

This records no production ownership or percentage change. After review and regional deployment, probe both authentication paths. The Intercom provider URL change remains separately authorized external work; retain Rails ingress as the rollback until real regional events and the clean observation window pass.

September 4, 2026 — schema runtime writer inventory prepared

ENG-2682 now scans tracked Rails, API2, operational, deployment, test, documentation and migration source against 271 non-index manifest relations and the api2.rebuild_conversation_assignment_intervals routine. The deterministic result classifies 2,029 exact references; 619 are active runtime or operational references and 149 are ordinary writes. It found no active named-constraint upsert or direct sequence caller.

Fourteen executable references remain cutover blockers: four Rails model table bindings, three operational/API2 dynamic table bindings, six assignment-routine references across four caller sites, and one operational ACL probe. Comment-only mentions are classified but do not count as blockers. Dynamic SQL, stored routines, external clients and 177 manifest relations without an active literal remain manual gates. Fresh regional schema/ACL evidence could not be captured because AWS SSO expired; no production schema, data, role or grant was changed, and production cutover readiness remains false.

September 3, 2026 — native live QA billing prepared

ENG-2766 moves the normal qa_evaluation Grant charge from Rails GoodJob to a deterministic API2 BullMQ job. API2 rechecks the organization-scoped PostHog flag, validates the evaluation/conversation/native-organization tenant chain, uses the evaluation prefix ID for Grant idempotency, and atomically persists the Rails billed markers with the API2 usage event and mirrored timestamp. Existing billed-but-unmirrored rows are recovered without calling Grant again.

Rails completion now only delivers the native enqueue. An immediate API2 delivery error falls back to the unchanged BillQAEvaluationToGrantJob; Grant and ledger idempotency make ambiguous overlap safe. The hourly native reconciliation remains the independent backstop. Focused route, real-PostgreSQL and Rails fallback checks cover the contract. Production ownership still requires the exact revision in both regions, real flag-enabled charges, no normal-path GoodJob billing, no duplicate charge, ledger parity and 24 hours of queue/error observation. Native preparation primitives and result finalization remain separate QA blockers.

September 3, 2026 — QA result finalization dispatch prepared

Conversation QA inference now hands validated success, no-scorecard skip and terminal-failure payloads to a separate BullMQ finalization queue. The immutable result-run UUID is the queue and Rails idempotency key; a payload digest rejects conflicting reuse. Eight delivery attempts and retained failures let Rails importer outages recover without rerunning the expensive model/tool loop.

This is a bounded execution transfer, not native result ownership. Rails still imports and aggregates result trees, repairs citations and runs completion, sampling, notification and billing effects. Production credit requires real regional finalization events and queue/request parity; no queue replay, data, flag or schema mutation is included.

September 3, 2026 — hosted login caller cutover prepared

The user confirmed the US and EU API2 callback URLs are registered in WorkOS, removing the external gate on ENG-2758. The server implementation from #11123 is deployed in fresh regional tasks, and read-only container inspection confirms the hosted-login route is present in both regions. The caller package moves login, signup, password reset and same-region regional-auth entry points to API2 while leaving dev sessions, headless auth, invitations and identity provisioning on Rails.

The caller audit found one pre-cutover parity gap: email-first enterprise SSO passes sso=1, while the native route previously always selected AuthKit. The package now resolves the active WorkOS SSO connection by email domain and falls back to AuthKit on a missing, inactive or failed lookup, matching Rails. Focused route, WorkOS-selection and UI URL tests cover the contract. Production credit still requires the caller revision to deploy, actual tenantless/member/SSO logins and the 24-hour observation gate.

GitHub Actions run 33789129052 successfully deployed exact release e74224443f to both regional UI Workers. The failing Cloudflare dashboard Git builds still invoke the obsolete build/server/wrangler.json command, but they are redundant and do not block the canonical GitHub deployment path. This corrects the earlier deployment assessment below.

September 3, 2026 — hosted login server compatibility prepared

ENG-2758 adds native hosted AuthKit entry/callback compatibility without moving first-party callers. Private pre-auth Rack sessions bind a 15-minute state and PKCE verifier; callback completion resolves one account, requires an existing kept membership for organization responses, produces a Rails-readable WorkOS session and atomically rotates the cookie. Focused PGlite tests prove rollback, tenantless account creation, identity-conflict denial and single-use concurrent callbacks. The callback also mints the shared Rails CSRF session/cookie pair; Ruby OpenSSL independently verifies its mask and global-token HMAC. Direct invites, new membership/employee provisioning, headless auth, external token contracts and identity webhooks remain on Rails.

Fresh regional API2 tasks at 19:00 UTC contain the correct API2 callback bases, app/CORS origins, WorkOS settings and 32-byte cookie passwords. The caller cutover was held until both regional API2 callback URLs were allowlisted in WorkOS; the user confirmed that registration later on September 3. This remains server preparation, not AUTH completion or production ownership credit.

Organization switching #11091, schedule callers #11093 and assignment-routine compatibility #11094 merged. Release e74224443f deployed Flightcontrol services successfully in both regions, and GitHub Actions run 33789129052 deployed the same revision to both production UI Workers. Separate Cloudflare dashboard Git builds failed because they still invoke build/server/wrangler.json, while the SPA build removes build/server; those redundant builds are not the canonical production deployment path and do not block caller verification.

A subsequent dashboard Git build of main d56ffb339a7 also failed, confirming the redundant dashboard command remains stale rather than indicating a GitHub Actions deployment failure.

September 3, 2026 — deployed assignment routine compatibility proved

ENG-2682 now includes an isolated PostgreSQL 15.10 rehearsal of immutable migration0326. Read-only US/EU catalogs at 11:38 UTC match its function body SHA-256 ab1a780d5957a16c88bd9287bdf3408dc67a8fc3d90b0ea12f83be8917e93622. Seven checks exercise real interval writes, historical team retention, reassignment/removed-part cleanup, handling totals and tenant-separated rows through alias/native/reversed phases. Function OID/owner/ACL/security settings survive, and native execution works after both legacy aliases disappear. The temporary database/role are removed. Synthetic input tables and explicit local writer grants do not prove full production schema/role parity. The full regional export, current manifest, identity and lock/load gates remain open. No shipped migration or production object changed.

September 3, 2026 — native organization switching; regional logout deployed

ENG-2751 transfers the sidebar’s existing-member organization switch to API2. Focused validation covers tenant/role denial, expired/revoked/password-changed sessions, WorkOS identity mismatch and rollback. Real PostgreSQL 15.10 proves concurrent switches consume one refresh token, logout committed first prevents rotation, and provider failure rolls back activity/session writes. Installed WorkOS Ruby 6.2.0 reads the native sealed response. UI tests cover cache cleanup, duplicate submission and preserved state on failure; no browser tests ran.

The user resolved the prior configuration gate. Both regional WorkOS values matched Rails at 10:40; logout #11073 merged 10:46 and its containing release succeeded EU 11:05 / US 11:06. Runtime modules/configuration are verified, but actual logout events and observation remain. Schema #11072 merged 10:17 as a local rehearsal prerequisite, not a production schema cutover.

The daily baseline cuts off at September 3 11:00 UTC. Both regions have no overdue QA credit mirrors; EU org41 has 16/16 latest-window billed mirrors versus 21/21 prior-window, with no US billed events. Existing QA failures remain six US EasyPay requests since original rollout, all delivered failure effects; post-repair requests complete in both regions. EU org33 has 652 pending requests in the latest created-at cohort. GoodJob/BullMQ retention prevents complete throughput comparison. EU Zendesk queue wait/delay was 2,003/2,996 at 11:06, oldest sampled wait 64 minutes; provider wait/active 50/16. Hold load-increasing provider cutovers while checking drain; independent auth/settings remain safe. The older outbox backlog remains 594 EU/one US row; Freshdesk EU mapping and historical recovery remain explicit decisions.

September 3, 2026 — logout conflicts resolved; WorkOS configuration missing

At 09:57 UTC, read-only checks confirmed both regional API2 environments are missing WORKOS_CLIENT_ID, WORKOS_SECRET_KEY and WORKOS_COOKIE_PASSWORD. Both Rails deployments have all three, with 32-byte cookie passwords. Their actual shared-cookie configuration and API2 regional app origins match. No secret values were exposed or changed. Caller cutover remains held for a separately authorized copy of the existing settings within each region.

Conflict resolution preserves both composed-route authentication tests and all schema, schedule and auth ledger entries. Schema #11072 is first in the requested merge order; its normal merge request waits for repository code-owner approval.

September 3, 2026 — native logout prepared; runtime evidence blocked by SSO

ENG-2741 moves the logout contract and both first-party controls to API2. It revokes existing Rack sessions before provider I/O, supports Ruby WorkOS AES-GCM seals and expired signed access tokens, and retains local logout on provider failure. Origin checks reject untrusted or missing browser origins. Tests use actual Rails session DDL and a Ruby-generated synthetic seal. Regional WorkOS configuration is an explicit merge gate; no production mutation was performed.

QA repair release 908e86dabbb340385e1c1790ae7304af22697aad reports successful US deployment at 03:54:19 UTC and EU at 03:55:29. Targeted Sentry reads through 07:03 found no new auth/QA lifecycle issue in the returned delta. Actual stage progress/outbox recovery is unverified because AWS SSO expired. Session and connection retention still require the first native September 4 midnight run. Schema #11072 remains green and required-review pending. This is not a bake.

September 3, 2026 — work-schedule native contracts and writer compatibility

ENG-2742 adds native work-schedule mutation contracts and a transaction advisory lock shared with Rails before default selection. Local tests use the Rails schema and preserve tenant/source scope, timezone labels, omitted fields, provider-managed restrictions and soft deletion. A PostgreSQL rehearsal proves contention against the Rails lock protocol and twelve concurrent native default selections. UI callers remain on Rails pending verified server compatibility; this is a staged implementation, not production ownership closure.

September 3, 2026 — full schema writer matrix and constraint-name gate

The local PostgreSQL 15.10 rehearsal now plans all 323 unique-index conflict inferences (including 18 partial indexes) on all 178 restored tables before, after and after reversing the move. Basic INSERT/UPDATE/DELETE and RETURNING also plan through each old/new binding. Actual-row verification remains the existing representative write test plus future per-domain parity.

The first expanded run found that ALTER INDEX RENAME also renames the owning constraint: conversations_pkey no longer resolves on public.api2_conversations. The corrected matrix proves all 26 mapped names work and the old names fail; 192 named constraints fail through old aliases, and 11 explicit sequence names vanish until rollback. Both successful and failed runs clean up their isolated local database/role. These are explicit deployment prerequisites, not reasons to mutate production or weaken the September 4 gate. Production ACL exports, six missing manifest objects and dynamic/stored writer classification remain.

September 3, 2026 — full repository schema move rehearsal passes

ENG-2682’s expanded PostgreSQL 15.10 rehearsal restores the unchanged Rails structure (SHA-256 4badeabaa221b781e0c97e939ceab22938bfabfd66c0146c775ec2864cfb57fc). All 178 native tables, 76 agent views and 11 sequences move and reverse. The checks preserve catalog identities, columns, indexes, sequences, ACLs and all 1,425 restored foreign keys, and retain writes made through an old table alias. The temporary database and synthetic reader are removed after the run.

Six production manifest objects are absent: assignment intervals and their sequence/rebuild function, work-item evaluation contexts, message-chunk agent view and US-only thread-membership view. Production grants are also absent. The checked-in report explicitly withholds production readiness. Fresh regional schema/ACL restoration, writer classification, identity gaps and lock budgets remain open; no production migration or app binding change is included.

QA repair #11069 merged externally with green CI at 03:43:01 UTC as 05ec75129a2; actual regional recovery remains unverified. Session retention’s release ba08815d91b succeeded EU 03:38:20 and US 03:39:06 UTC. Retention still requires its first native midnight run and 24-hour observation.

September 3, 2026 — QA related-sync transaction regression repaired locally

ENG-2739 isolates the adapter failure to US EasyPay historical-backfill requests 5799716, 5799754 and 5799796 on release 26c9abdc50a between 03:02 and 03:17 UTC. All stopped after related-conversation detection; Rails reported an aborted transaction, and API2 exhausted retries with 504 responses. Read-only SQL at 03:20 confirmed all three failure effects delivered on the first attempt. The initiating provider SQL error remains unidentified.

A real PostgreSQL test reproduced the aborted request transaction before the fix. Native calls now persist one related-record attempt at a time and wrap provider sync in a savepoint, preserving tenant/source checks and the lease lock. All 48 focused regression/legacy examples pass. CI, review, deployment and actual regional recovery remain separate gates. No production jobs, source guards or backfill state were changed.

Session retention #11066 merged externally at 03:23:46 UTC as d557aab681f. Tests/build passed, but the API2 project coverage check disagreed with its later report and remained failed at the merged head. Preserve that gap. First native midnight execution plus 24 hours remains open.

September 3, 2026 — native auth retention and repaired-release checkpoint

ENG-2738 transfers daily session retention to bounded native execution. Regional sessions tables have 6.75M/6.93M rows and no creation-time index, so the scanner uses existing primary keys and deletes only expired rows from bounded pages. Actual Rails DDL, timestamp boundaries, retry persistence and PostgreSQL concurrency/locking are verified locally. Runtime retention observation remains open; no new domain completion or weighted credit is claimed.

Auth repair #11052, QA lifecycle #11054, schema preparation #11050 and connection retention #11044 are merged. Current release 26c9abdc50 succeeded in EU at 02:56:52 and US at 02:59:31 UTC. At 03:06 neither region had overdue billed QA mirrors; EU had 19/19 recent mirrored rows, US had no billed rows in that window. US showed four persisted native QA stage leases after deployment. These are limited workload observations, not full lifecycle parity or a completed bake. The earlier auth incident below is historical; full auth recovery/observation, QA preparation/finalization/billing and full schema/writer gates remain open.

September 3, 2026 — native auth schema regression requires repair

After #11048 merged, the production-regression task identified a reference to nonexistent public.accounts.discarded_at in native session validation. The PGlite fixture incorrectly defined that column, so green fixture tests missed the SQL error. The failing query prevents authentication with a server error; it does not produce another tenant’s session. A separate existing owner is fixing the query and fixture in rulebase-native-session-schema-fix.

Auth deployment/observation credit remains withheld. Hold further runtime migration rollout until the repair lands and recovery is observed, while continuing safe QA/schema preparation. Validate future auth SQL against the actual Rails schema as well as adversarial tenant fixtures. The earlier session format probe only decoded stored data and did not certify the security query.

September 3, 2026 — auth merged; QA lifecycle and schema rehearsal prepared

Native auth #11048 merged at 01:44 UTC as 64198ab5d4. A follow-up isolation review found no cross-tenant access path in the changed authentication boundary; 39 focused persisted-session, middleware and identity-mapping tests passed. That review does not certify every downstream route or a production bake.

ENG-2731 owns request stage leases/completion and atomic terminal failure with a public outbox. Twenty-five API2 tests, twenty Rails adapter/finalizer tests and seven existing request-completion examples passed, plus a real PostgreSQL two-connection lease/failure race. Drizzle migration 0330 was generated and applied to an isolated local Rails-schema fixture. Rails preparation primitives, result finalization and live billing remain explicit follow-ups, so the full Rails-unavailable QA foundation gate is still open.

ENG-2682’s collision/identity manifest and local PostgreSQL 15.10 rehearsal passed seven forward/reverse compatibility checks. Physical OIDs, tenant ACLs, foreign keys, sequences and writes made after movement survive. PostgreSQL supports column-inferred upserts through simple updatable views; named-constraint upserts and table-only commands need separate treatment. This is representative synthetic rehearsal, not a full production-schema rehearsal or production move.

September 3, 2026 — native existing-session validation prepared

ENG-2730 moves existing Rack/Devise cookie validation, tenant authorization and identity hydration into API2. Every request validates the persisted session and authorization revision before cache reuse. Bounded Ruby Marshal compatibility and compare-and-swap activity updates preserve Rails interoperability and prevent logout resurrection. Login/WorkOS/logout issuance and retention remain explicit follow-ups. No schema or production mutation is included in preparation.

Focused codec, middleware and database tests cover revocation, expiry, tenant isolation, concurrent refresh/logout and permission changes. Ruby accepted 26 API2-written fixture documents. At 01:18 UTC, read-only probes decoded all 2,000 sampled US/EU retained sessions; seven US and two EU samples had authenticated Warden state, and session indexes were valid. This is sampled compatibility proof, not deployment, active-user parity or bake completion.

The previously approved September 15 policy and simulation retirement edits are included with this implementation. Newer retention ledger entries are preserved when integrating current main.

September 2, 2026 — pull foundations forward for September 15 retirement

The user approved an ambitious full-retirement target and explicitly asked to encode it in the scheduled automation and durable planning. Roadmap dates are latest target milestones, not start dates. Begin native auth, bounded QA stage/failure ownership and schema compatibility rehearsal immediately when slots and dependencies are ready. Keep three workers/packages maximum and work directly without required subagents. Continue to tested, reviewable capabilities and subsequent ready work instead of stopping after an audit or arbitrary PR quota. Existing review, validation and production-operation boundaries remain.

Required gates: September 4 foundations, September 8 retained implementation parity, September 11 regional cutovers, September 12–14 final observation and September 15 approved Rails/GoodJob service and source retirement. Report a missed gate and its recovery sequence immediately; preserve verification time. The original one-month target and weekly weighted-points goal are superseded.

The completed source census at 1baf2de490 records 1,041 routes, 414 literal Rails callers, 262 job/support files and 40 configured cron entries. Runtime certification remains open. Chidi retired simulation, completing one of 24 inventoried domains by product disposition; 23 migration domains remain. Simulation cleanup is assigned to OPS/REALTIME. Shared call analysis and native coaching remain retained. Source counts and legacy percentages are not new production completion evidence.

This update changes the existing active four-hour automation, roadmap, board, all stream policy headers and coordinated ledgers. It does not start another migration package or alter runtime/production state. Canonical ledger edits are included with ENG-2730; historical evidence below is retained.

September 2, 2026 — native connection-log retention prepared

ENG-2723 transfers the existing midnight UTC TrimConnectionLogsJob workload to API2 maintenance jobs. It retains the newest 1,000 logs per tenant/source with a fixed microsecond-precise cutoff and at most 10,000 deletes per transaction. Queued continuations preserve retries and isolate source failures; public tables and existing log writers/readers remain explicit Rails dependencies. No retention policy, schema, provider load or ad hoc production cleanup changes.

At September 3 00:48 UTC (September 2 New York), both regions had valid retention indexes, no DB lock waiters, and successful midnight Rails cleanup. US source coverage is 53 sources/16 tenants; EU is 21/eight, including discarded sources. The native scheduler needs deployment, an actual midnight run and a 24-hour bake. QA #11039 and tag writes #11040 merged at 00:39 UTC; their deployments/bakes remain unverified in this entry. Intercom deletion observation is parked while platform retention occupies the third package slot. Weighted credit remains unchanged.

September 2, 2026 — prepare native hourly QA credit reconciliation

ENG-2720 removes the hourly GoodJob scan and its per-evaluation mirror jobs from normal recurring execution. API2 scans up to 5,000 rows in 250-row pages and atomically inserts the tenant-correct ledger event and confirms the billed row. It preserves original charge time/amount, prefix idempotency, live metadata, 30-minute exclusion, and no lower history cutoff. A fixed-ceiling rotating cursor revisits bad/unmapped rows without starving later rows. Live Grant billing and RecordQAEvaluationCreditUsageJob remain explicit Rails dependencies. No schema migration, new flag, provider request or production repair is part of this package.

At 23:04 UTC no overdue mirrors existed in either region; all 27 EU charges in the prior day were mapped/mirrored, while US had none. Both databases had no lock waiters. Provider wait/active and delayed Zendesk intake reached zero in both regions; two provider jobs remained delayed in each. Retention-limited completions are not full throughput. US Freshdesk source 25 had a 23:00 checkpoint, while EU source 25 remained unmapped. No full audit was repeated (daily cutoff 19:00 UTC). Intercom deletion #11030 merged at 20:31; regional release statuses succeeded at 21:23, with actual-event bake still pending. No new weighted points are assigned; full retirement census and a defensible ETA remain outstanding.

September 2, 2026 — make delegation optional and cap concurrency at three

The user removed the subagent requirement and reduced concurrency to three. Work directly by default; the limit includes the coordinator if additional workers are used. Keep at most three active packages and rotate the five logical streams. At this revision no subagents remained live. QA and HTTP drafts and Intercom PR #11030 remain the active package queue; platform retention and Freshdesk bake/recovery are parked with their worktrees and evidence preserved. Schedule, merge authorization and validation requirements are unchanged.

September 2, 2026 — activate parallel ownership packages

The user authorized five durable streams with at most five active packages and standing reviewed/green merges, including main’s automatic production deployment. Required reviews/checks remain; direct deployment, repair/replay, flag/queue/source and infrastructure operations require separate authorization. Four runtime slots permit coordinator plus three workers, so platform work rotates through freed slots.

New packages are ENG-2720 native QA credit reconciliation, ENG-2721 native Intercom deletion and ENG-2722 tag mutation/client routing. Ordinary summaries remain next pending plain-text transcript, related-context and stale-write parity. These are independent of the congested Zendesk provider limiter; the former global stop and one-new-PR limit are superseded. The coordinator owns the shared ledgers and merges.

#11026 merged at 19:29:35 UTC as 83c4b73375 and reached release-prod as f08d4196f9. At the 19:38 targeted check, production was still rolling the prior release, so deployment/bake of the fix was unverified. The last matching Freshdesk null error was 19:30 UTC before that rollout. US org/source 31/25 is mapped; EU 31/25 is not. Both still lacked a source checkpoint. Historical recovery after the last retained successful 17:00 UTC Rails scan and EU coverage need specific approval.

EU provider wait was 3,987 / 16 active / oldest sampled wait 216 minutes, compared with 5,703 / 16 / 230 minutes at 19:04. Delayed intake rose from 1,906 to 2,305; US provider wait remained zero. Old undelivered provider effects remained 594 EU and one US. No eligible overdue QA credit mirrors existed in either region. This reuses the full daily audit cutoff at 19:00 UTC and adds only targeted deltas. Retained BullMQ/GoodJob counts are not complete throughput; sampled Sentry traffic is not ALB request volume. No production mutation was performed by this run.

Progress stays 64% production / 84% code-ready, delta 0/0, under the legacy mixed-scope denominator. The prior async label is corrected because the table also weights reads, writes and auth. Full retirement remains unbaselined; no precise new ETA or points are invented before the reproducible census closes. Earlier entries retain their observation-time evidence and superseded operating model.

September 2, 2026 — repair nullable Freshdesk reconciliation metadata

At 19:04 UTC all eight Rails/API2 services were at desired capacity and the ENG-2707 global intake cap was four in US and EU. EU provider wait fell from 9,939 to 5,703, but its oldest sampled wait rose from 159 to 230 minutes while all 16 provider slots stayed active; 1,906 webhook jobs were delayed. US had no provider wait. The incident is improving in depth but not yet healthy.

Sentry RULEBASE-API2-BJ found the newly deployed Freshdesk source worker reading its watermark from null legacy sync_data. A read-only database query at 19:07 UTC confirmed the sole active Freshdesk source in each region has SQL NULL metadata, with neither a watermark nor a saved window. The 19:10 UTC follow-up found US mapped and EU unmapped: only US reaches the failing worker; EU is excluded by the existing mapping guard. Both last retained Rails scan successes were at 17:00 UTC. EU mapping or explicitly retained Rails coverage needs an owner decision before its cutover can be considered safe. The scoped repair normalizes SQL NULL and JSON null to an empty object for source selection, reload, initial checkpoint, and completion while retaining tenant guards and unrelated metadata. Focused database tests cover both legacy forms and durable resume behavior.

Equal Sentry windows ending September 1 and 2 at 19:00 UTC show Rails web p95 falling from 868 to 498 ms and API2 rising from 166 to 237 ms; Rails sampled failure rate fell from 0.74% to 0.38%, with no sampled API2 web failures. Retained GoodJob execution rows still rank EU QA preparation (134,762 seconds) and US evaluation checks/completion polls (149,633 seconds) above summary work (21,620 seconds combined). These are retry-inclusive retained totals, not unique workload counts or complete day-over-day throughput. Undelivered provider effects remain 594 in EU and one in US; all are older than an hour. Flipper Zendesk sync and QA recovery remain globally true in both regions, while ticket-closed QA has no global gate; PostHog transcription is untargeted 100%.

Progress remains 64% production / 84% code-ready (+0 / +0), leaving 36/16 points. Freshdesk and the repair receive no new credit. No net-new package is safety-cleared while the EU backlog remains old and the regression is active. No production data, queue, flag, deployment, or infrastructure was changed by this audit.

September 2, 2026 — reconcile Zendesk intake with the completed migration queue

PR #10991  incorporates main through the Freshdesk merge bc1777379a. Tickets #10945, Zendesk recovery #10946, sales-call credit reconciliation #10949, and Freshdesk #11009 are merged. Earlier entries retain their observation-time status and evidence. The recovered unfinished Tickets-era merge and all newer ledger history are preserved; the intake mitigation retains its existing behavior.

Progress stays 64% production / 84% code-ready, with 36/16 points remaining. ENG-2707 is a regression mitigation; this reconciliation adds no credit and includes no new production audit. The last recorded regional observations are from 15:05 UTC, not a fresh health claim. Exact-revision bake evidence remains required before production credit or cleanup.

This preparation is limited to merge resolution, local validation, and PR/CI handoff. Main automatically syncs to release-prod; the coordinating owner handles merge and production authorization. No deployment, flag change, queue mutation, or data repair is part of this preparation.

September 2, 2026 — prepare recurring Freshdesk imports in API2

ENG-2696  was recovered into a persistent worktree after interruption and reconciled with main through the sales-call credit merge 18b51823d0. This is the existing Freshdesk scope, not a new portfolio selection. All earlier history is retained. Progress remains 64% production / 84% code-ready; no new weighting or production credit is claimed.

The package moves the 15-minute recurring scan and per-ticket imports from GoodJob to BullMQ. It keeps the allowed Freshdesk source codes and minimal Rails import attributes, creates conversation threads, and hands full sync to the existing API2 provider queue. Null-aware uniqueness prevents replayed imports from creating duplicates; optional metadata does not clear fields already written by full sync.

Source failures and ticket failures retry independently. Scans stream bounded pages and save a fixed window before network I/O. Invalid ticket IDs are logged with tenant/source/page/index only and do not starve later valid items. Redis delivery errors and the 300-page cap prevent watermark advancement.

Review tightened terminal-failure recovery: ticket jobs keep their tenant/source/ ticket payload without automatic age/count eviction after five attempts. They can be manually redriven after the source watermark advances. Terminal source/ scan jobs are removed individually because their fixed window survives separately; shared age/count trimming would incorrectly evict failed tickets. Queue-admin recovery and alert/backlog checks are documented without changing production.

The earlier read-only Sentry audit found no Freshdesk-specific error in its 14-day window. Expired AWS SSO prevented regional queue, database, and ECS verification. That earlier evidence is not a fresh deployment audit. Regional mapping and throughput checks remain deployment prerequisites. No production data, flags, deployments, or evaluation runs were changed.

September 2, 2026 — reconcile sales-call credits with the serial landing queue

PR #10949  now includes main through the Zendesk recovery merge 0bdf11bbb6. Production completion remains 64% and code-ready completion remains 84%, leaving 36 production and 16 code-ready points. ENG-2687’s one-point contribution is already included; this reconciliation adds no further migration credit.

All earlier migration entries and rollout contracts are preserved. Sales-call credit reconciliation retains its six-hour cadence, 250-row pages, 5,000-row cap, tenant-qualified eligibility, lossless database IDs, idempotent insertion, and atomic counter update. This preparation does not deploy or claim new regional production evidence.

September 2, 2026 — reconcile Zendesk recovery with the serial landing queue

PR #10946  now includes main through the Tickets merge 2fab3c8b4a, including the separately merged midnight QA sampling spec fix in PR #10947. Production completion remains 64% and code-ready completion remains 84%, leaving 36 production and 16 code-ready points. ENG-2684’s original one-point contribution is already included; this reconciliation adds no further migration credit.

All earlier migration entries and rollout contracts are preserved. Zendesk recovery retains its one-hour grace window, 500-row source cap, tenant mapping guard, isolated failures, and code-revert rollback. This preparation does not deploy or claim new regional production evidence.

September 2, 2026 — reconcile the intake fix after Tickets merged

PR #10991  now incorporates main through 2fab3c8b4a, preserving Tickets and automated-QA history in all four ledgers. Tickets #10945 merged at 14:52 UTC. Regional release 626ea889cb reported success at 15:01 UTC, with new API2 worker revisions; that is deployment evidence, not a completed Tickets workbook bake.

The fresh read-only audit confirms the incident remains active: EU provider wait grew from 9,033 at 14:16 to 9,939 at 15:05 UTC, with 16 active jobs and an oldest sampled wait of 159 minutes. Of the oldest 100 sampled waits, 95 were Zendesk and five Intercom. US provider wait remained zero. In the complete 14:30–15:00 bucket EU completed 1,154 provider jobs and 1,689 webhook jobs; p95 runtimes were 28.7 and 286.6 seconds, versus US provider p95 of 2.87 seconds. Transcription and QA recovery had no new retained failures; Intercom retained missing-organization failures in both regions. US provider sync still retained 292 missing-subdomain failures, and EU still had 594 overdue provider effects. EU durable webhook receipts rose to 13,280; historical rows predate this incident.

Production/code-ready estimates remain 64% / 84% (+0 / +0), with 36/16 points remaining and low forecast confidence. There are two open implementation- complete migration PRs (#10946 and #10949), both conflicting with main and awaiting review, and zero net-new safety-cleared packages. No new migration PR is opened while the EU regression is active. The conditional September 14–18 code-complete and September 28–October 2 baked ETAs remain at risk. The next three domains and their Rails dependencies are unchanged: automated-QA repair/invariant/ticket-closed ownership, post-sync summary/completion callbacks, and the synchronous route/client census plus first bounded cutover.

The reconciliation preserves the unchanged mitigation and all newer main code. API2 typecheck, internal-docs lint, formatting, and focused executor/backpressure tests passed. Main automatically syncs to release-prod; explicit production authorization is required before merging the mitigation. No production mutation occurred.

September 2, 2026 — contain EU Zendesk intake contention

Progress: 64% production / 84% code-ready, both unchanged; 36 and 16 points remain. ENG-2707  is a regression fix, not net-new migration. New ownership work is unsafe while EU queue depth is growing and the original throughput bottleneck remains only partly diagnosed. The mitigation is PR #10991 . No production mutation or deployment was performed.

• AWS SSO is restored. All eight US/EU Rails/API2 web and worker services were healthy at desired capacity, with five BullMQ replicas per region. Infrastructure health does not imply queue health. Automated-QA PR #10941 merged at 13:06 UTC; the latest unrelated release 0023ec654d has successful US/EU deployment records at 14:00/13:59. API2 worker revisions were unchanged from the 12:55/12:58 rollout.
• EU provider wait rose from 6,367 at 13:05 to 6,625 at 13:06 and 7,874 at 14:03, with 16 active jobs and a 119-minute oldest sampled wait. 99 of the oldest 100 sampled waits were Zendesk. Webhook active jobs were 160 in the earlier snapshot and 35 at 14:03; both paths use the same per-source request limiter. US provider wait/active were zero. The backlog predates the webhook cutover, so contention is an amplification risk, not a proven sole cause. At 14:16 UTC, EU wait rose again to 9,033; US remained at zero waiting.
• CloudWatch’s complete 13:30–14:00 UTC bucket recorded 1,805 EU webhook completions versus 1,221 provider completions, with p95 runtimes of 216 and 54.5 seconds. The 12:30–13:00 bucket had 1,647 / 1,191 completions and 104 / 53.4 seconds p95. US provider p95 was 1.98 seconds in the latest complete bucket. Retained transcription, Jira reconciliation, and QA recovery queues had no new failures in either region at 14:16; their retained completion counts are capped and must not be treated as full-day throughput.
• Sentry’s equal 24-hour windows show Rails webhook processing at 56,667 versus 38,667 sampled spans, 5.83 versus 6.61 worker-hours, and 745 versus 1,127 ms p95. Its 8.24% versus 1.72% sampled failure rate includes GoodJob concurrency/retry errors; it is not a terminal lost-event rate. Fresh retained GoodJob queues had no jobs overdue/locked for over an hour in either region.
• US provider sync retains 292 missing-subdomain failures; its source repair still needs approval. Intercom reconciliation retains missing-organization failures in both regions, so its old clean-bake claim must be revalidated. Provider outbox inspection found 594 EU rows overdue an hour, the oldest dating to August 19, plus one US overdue row. EU/US durable webhook tables contain 13,166/419 pending rows, including much older events; those totals are not attributed wholesale to today’s cutover.
• Ticket-closed QA remains disabled. Existing QA failures and translation delivery errors remain watch items. Transcription remains at untargeted 100% in PostHog. No flag was changed or deleted.

The mitigation sets webhook global concurrency to four and defers same-source intake for 30 seconds while provider work is congested. It checks the tenant-qualified receipt first, bounds active-job inspection, preserves attempts/deduplication, and leaves event acknowledgement and Rails fallback untouched. See the rollout and rollback contract.

Validation passed: 23 focused API2 tests, API2 typecheck with a larger local Node heap, internal-docs lint, and whitespace checks. A temporary isolated Redis harness ran two workers, completed 13 jobs with no failures, proved the global active cap of four, and verified that deferral retains the retry budget and per-row deduplication until completion. No browser test or schema change was needed.

Three existing implementation-complete PRs (#10945, #10946, #10949) need main conflict resolution and human review; current required CI passed. No second migration PR is opened during this active incident. Next: (1) deployed automated-QA repair and uniqueness invariant before ticket-closed rollout; (2) provider post-sync summary/completion callback parity; (3) synchronous route/client census and its first bounded cutover. Generic evaluation checks remain a large Rails runtime surface requiring a parity audit.

Schedule risk: verified production completion has stayed at 64% since August 30, below the 8–10-point weekly target. The September 14–18 code-complete and September 28–October 2 baked estimates are conditional, not supported finish commitments. EU sync, stale QA rows, three conflicted PRs, retained callback outboxes, the uncensused synchronous surface, and public-schema consolidation are the exact blockers. Rails deletion remains unestimated.

September 2, 2026 — reconcile Tickets exports with the serial landing queue

PR #10945  now includes main through the automated-QA merge 9bd4a8cc57, including the QA-stage transcription and caption-resumability packages. Production completion remains 64% and code-ready completion remains 84%, leaving 36 production and 16 code-ready points. Tickets’ original three-point contribution is already included in that aggregate; this merge does not add it again.

The reconciliation preserves every earlier migration entry and the current regional rollout contracts. Tickets retains exact Rails workbook parity and a code-revert rollback. No production audit, deployment, flag change, data repair, or queue mutation is part of this landing preparation.

September 2, 2026 — make automated QA ownership repairable and race-safe

Progress: async weighted production completion remains 64% and async code-ready completion remains 84% after reconciliation with the newer Jira, QA-stage transcription, image-captioning, and Zendesk webhook-execution work now on main. This automated-QA stack originally moved the portfolio from 72% to 73%; its one point is already included in the cumulative 84% and is not counted again. Work is tracked in PR #10941  and ENG-2680 .

Production evidence and regression lane

• The PII override recovery is holding: the last matching Sentry 422 occurred before the 00:42Z deploy, while ten US and nine EU ticket-closed requests completed afterward.
• US has 2,059 stale active ticket-closed requests and 1,027 conversations with duplicate active rows. EU has 6,170 and 2,754 respectively. No production row, flag, queue, or infrastructure state changed during this audit.
• QA request BullMQ queues had no waiting, active, or retained failed work at the last regional audit. Scheduled recovery remained bounded, and successful post-recovery callbacks showed the Rails/API2 contract live in both regions.
• A separate legacy Zendesk source with no stored subdomain continues to exhaust bounded API2 provider-sync retries before Rails fallback. Neither runtime can safely infer the missing account identity, so the corrective action is an explicitly approved tenant-scoped integration repair, not an exception- suppressing code change.

Ownership change and repair contract

Scheduled and admin-forced QA preparation now select the existing API2 BullMQ orchestrator unconditionally, joining user-requested preparation. Ticket-closed requests remain behind api2_qa_evaluation_request_processing. Immediate API2 delivery failure still falls back to GoodJob, scheduled delays are preserved, and the existing terminal callback remains unchanged.

Rails locks each conversation while creating active requests in normal, admin, and backfill paths. The uniqueness validation runs only while a request is active, so historical duplicates can transition to terminal states. The new maintenance task requires an explicit organization slug, cutoff, and maximum, previews only that tenant’s stale active cohort, rechecks each row under the conversation lock, and marks it failed without per-row Sentry noise.

This is the first stack, not the final domain cutover. After production repair removes the historical duplicates, a follow-up migration must add the database uniqueness invariant; only then can ticket-closed ownership stage toward 100% and its normal Rails path be removed.

Deploy the revision before any repair. Each production run requires approval of one organization, cutoff, and maximum, collection review, post-run verification, then a staged ticket-closed gate with 30-, 60-, 120-minute, and 24-hour checks. Disable the gate to roll back ticket-closed ownership; revert the revision to restore admin and scheduled selection. Do not delete queued work or automatically undo terminal repair results.

September 2, 2026 — make bounded QA image captioning resumable

Progress: async weighted production completion remains 64% and code-ready completion remains 84%. This is a correctness follow-up to the existing API2-owned QA preparation path, tracked in ENG-2698 .

API2 advances QA preparation through bounded Rails requests. The caption stage previously restarted at the first pending row on every request, so one invalid conversation part or transiently failing attachment could consume the entire 500-advance job budget and starve later valid items.

Each QA evaluation request now persists separate conversation-part and attachment cursors plus attempted IDs in its existing stage_progress. A bounded retry skips work already attempted for that request, continues in ID order, and wraps around its cursor for newly eligible lower IDs. Transient attachments remain pending for a future evaluation, while the current request can finish after attempting each candidate once. Overlapping bounded advances serialize on the request so they cannot select or caption the same row. The unbounded Rails GoodJob path, source rows, production data, rollout controls, redaction, and provider ownership are unchanged.

September 2, 2026 — move post-sync image captioning to API2

Progress: async weighted production completion remains 64%. Including the open Jira, automated-QA, Tickets, Zendesk-recovery, QA-stage transcription, sales-call credit reconciliation, and this image-captioning package, code-ready completion advances from 79% to 81%; 36 production points and 19 code-ready points remain. Work is tracked in ENG-2689 .

Production audit and prioritization

• No migration PR merged after the prior audit. The unrelated daily-report revisions from PRs #10950 and #10952 deployed successfully to US and EU by 07:32Z; neither changes migration ownership.
• AWS SSO remains expired. Regional ECS, BullMQ, GoodJob, database, and rollout evidence is unavailable, so this run does not infer US/EU queue health or data parity and makes no production mutation.
• In Sentry’s latest 24-hour sample, Rails image captioning increased from 667 spans, 1.8 provider-hours, and a 9.6-second p95 in the prior day to 1,333 spans, 6.1 provider-hours, and a 27.3-second p95. Both windows had zero measured failures. The volume, runtime growth, and bounded provider surface made it the highest-impact unclaimed code-ready workload.
• The same window showed no new migration-path regression. API2 BullMQ sampled p95 improved from 490.9 ms to 416.5 ms and its measured failure rate fell; known QA and Zendesk issues remain separate tracked surfaces.

Ownership change and safety contract

API2 now owns the gpt-5-mini provider call, minimal reasoning, flex service tier, derived-text redaction, and retry lifecycle on a dedicated BullMQ queue. Rails remains authoritative for tenant-qualified signed URLs, the existing system and caption prompts, five-image and 1,000-token limits, and persistence. The part revision produces a deterministic job ID and a stale persistence callback cannot overwrite a changed or already-captioned part.

Conversation-level transcription adds eligible image parts to its existing BullMQ fan-in, so Rails post-sync analysis runs only after both calls and image captions complete. Direct CX-risk snapshot captioning uses the existing api2_conversation_transcription PostHog rollout. A disabled or unavailable gate stays on the legacy GoodJob provider job; an explicit API2 rejection also falls back, while an ambiguous delivery retries the deterministic API2 handoff to avoid duplicate provider calls.

After deployment, observe 30, 60, and 120 minutes plus 24 hours. Require clean US/EU queue throughput, no tenant mismatch or stale overwrite, provider-result parity, bounded explicit fallbacks, no duplicate provider run, and normal post-sync completion. Roll back by disabling the existing rollout or reverting the revision; do not delete queued work, mutate caption rows, or change regional infrastructure. Production credit waits for restored AWS access and clean regional evidence.

September 2, 2026 — move sales-call credit reconciliation to API2

Progress at implementation: async weighted production completion remained 64%. This package advanced the then-current code-ready portfolio from 78% to 79%. Its one-point contribution is already included in the current 84% aggregate and must not be added again at merge. Work is tracked in ENG-2687 .

Production audit and prioritization

• No PR merged and no release deployed after the prior audit.
• AWS SSO remains expired. Regional ECS, BullMQ, GoodJob, database, and Flipper evidence is unavailable, so this run does not infer US/EU queue health or database parity.
• Sentry’s latest 24-hour window shows no continuing migration-path regression. The PII validation issue has no event after 00:27:53Z; the translation read-timeout burst ended during the prior rolling deploy; and a one-minute API2 rater burst is unrelated to this migration. Known ticket-closed QA and legacy Zendesk issues remain active and continue to block QA expansion.
• PostHog still reports the transcription rollout at 100% and 18 active api2_* flags, including baked controls awaiting cleanup. No flag, production data, queue, or infrastructure was mutated.
• The QA-stage transcription package exhausts the remaining individual call producers, and Tickets export parity is already open for review. The next bounded ownership unit that could be proved without regional access was the six-hour Rails credit-reconciliation schedule in the otherwise API2-owned sales-call compliance domain.

Ownership change and safety contract

API2 now owns the six-hour reconciliation cadence as a retryable BullMQ maintenance task. It preserves the Rails job’s 5,000-row cap, 250-row pages, oldest-ID ordering, completed sales-call and compliance-result eligibility, tenant boundaries, newest active evaluation-credit choice, idempotent usage insert, and atomic cached-counter increment. One malformed candidate is logged without hiding later candidates, and any candidate failure makes the bounded tick retryable. The GoodJob cron entry is removed; the Rails job class remains unchanged for a code-revert rollback.

After deployment, observe 30, 60, and 120 minutes plus the first scheduled six-hour run and 24 hours. Require one scheduler in each region, no duplicate credit usage, matching usage and cached-counter deltas, no tenant mismatch, and no retained maintenance failure. Roll back with a code revert, preserve queued BullMQ work, and do not mutate production rows or rollout flags. Production credit waits for clean US and EU scheduler and database evidence after AWS access returns.

September 2, 2026 — move Zendesk webhook ticket execution to API2

Progress: async weighted production completion remains 64%. Including the Jira cutover plus open automated-QA, Tickets, Zendesk-recovery, QA-stage transcription, sales-call credit reconciliation, image-captioning, and this Zendesk execution package, code-ready completion advances from 81% to 84%; 36 production points and 16 code-ready points remain. Work is tracked in PR #10960  and ENG-2692 .

Production audit and prioritization

• The latest Sentry day contained 57,333 Rails ProcessZendeskWebhookTicketSyncJob spans and 8.1 worker-hours, versus 29,333 spans and 6.1 hours in the preceding day. Volume rose 95% and runtime 33%, while p95 improved from 1.80 seconds to 1.12 seconds. This made the bounded coalesced ticket executor the highest-impact unclaimed migration surface.
• Direct call-detail transcription, post-sync image captioning, Jira sync, sales-call credit reconciliation, Tickets exports, Zendesk recovery, and automated-QA groundwork already have open packages. Automated QA expansion remains unsafe until callback proof and stale-row repair complete.
• The PII callback regression remains recovered: its 160 trailing-24-hour events all ended at 00:27 UTC, before the 00:42 UTC revert deployment. The ticket-closed QA groups ended at 01:04 UTC. The legacy Zendesk source without a subdomain continued its known two-hour cadence with 292 events; it is not a regression in this migration package.
• Rails-to-API2 translation delivery recorded 59 trailing-24-hour failures; the latest single read timeout at 09:09 UTC fell inside a rolling deployment. API2 QA eligibility recorded 14 timeouts, last at 09:33 UTC. Neither issue involved the Zendesk webhook executor, and no continuing migration-path incident was found.
• No migration PR merged during this audit. Unrelated Self-healing UI revisions reached both Production and Production EU successfully by 10:08 UTC; no deployment was initiated by this migration run.
• AWS SSO for the regional production account expired before the run, so live ECS, BullMQ, GoodJob, database, and Flipper checks could not be repeated. Sentry and PostHog evidence is current; no regional infrastructure health is inferred from stale evidence.
• PostHog still has 18 active api2_* flags. Transcription and both Intercom imports remain at 100%; manual exports, workspace search, and the thirteen baked read controls remain active without a recent evaluation timestamp. No rollout control was changed.
• PR #10946  had one unrelated flaky Rails shard. Rerunning its failed jobs passed every required check, so no regression patch or second migration PR was necessary.

Ownership and rollback contract

Rails retains Zendesk webhook ingress and the durable 30-second coalescing row. Its GoodJob hands a deterministic snapshot to a tenant-qualified BullMQ queue. API2 imports the ticket’s top-level conversation record, delegates the complete source-owned sync to the existing provider worker, atomically consumes only the events it observed, and asks Rails to debounce a new run when later events remain. Per-row BullMQ deduplication prevents overlapping snapshot processors, and the original ActiveJob UUID also identifies the provider sync so a retry after enqueue replays the same outbox. A missing upstream ticket cleans up the obsolete snapshot.

An explicit API2 enqueue rejection immediately uses the preserved Rails path. Ambiguous delivery retries the same BullMQ identity, and terminal API2 failure calls Rails with fallback: true. No flag, database migration, production data mutation, or infrastructure change is required. Deploy to both regions, then observe queue throughput, durable-row drain, duplicates, Sentry, and fallbacks for 24 hours. Roll back with a code revert and leave queued work intact. After that window, remove only the Rails normal-path scheduler; retain terminal Rails processing until API2 owns webhook ingress and the final Rails rollback bridge can be deleted.

September 1, 2026 — move recurring Jira issue reconciliation to API2

Progress: production completion remains unchanged until deployment and a regional observation window. The cutover is tracked in ENG-2253 .

A production timeout in SyncJiraIssueJob exposed four recurring Rails-owned paths that remained after Jira conversation sync moved to API2: the 15-minute basic-auth issue scan, 30-minute dispute discovery, and two daily stale linked issue sweeps. API2 now owns those scans through a bounded BullMQ queue that isolates each source and issue, coalesces duplicate work, observes Atlassian retry policy, and preserves the existing 24-hour watermark overlap.

The issue worker persists Jira metadata, comments, service-desk request field values, linked conversation refreshes, and dispute links. Unavailable issues are marked terminally unavailable. Rails retains the manual historical import, post-create fetches, webhook ingress, and the hourly database-only resolver that consumes API2-written done statuses; none of those paths schedules recurring Jira network reconciliation.

After deployment, verify the incremental, dispute, and stale-linked schedulers in US and EU, advancing source watermarks, bounded 429 retries, and zero newly enqueued executions of the removed Rails jobs.

September 2, 2026 — move Tickets manual exports to API2

Progress: async weighted production completion remains 64% until the revision deploys and passes regional verification. Code-ready completion advances from 72% to 75%; 36 production points and 25 code-ready points remain. Work is tracked in ENG-2683  and PR #10945 .

Morning health and ranking evidence

• The PII callback recovery remains quiet in Sentry: RULEBASE-API-5S7 has 160 total events and its latest event remains 2026-09-02T00:27:53Z, before the regional recovery deploy at about 00:42Z.
• Automated QA is not safe to expand. RULEBASE-API-5S4 recorded 325 stale ticket-closed expiration failures through 01:04:02Z, and RULEBASE-API-5SC recorded a separate 45-event scheduling burst through 00:16:40Z. Fresh successful callbacks and the stale-row repair remain required before ENG-2680 proceeds.
• The latest successful regional audit had healthy manual-export queues and two successful EU manual QA workbooks. A fresh September 2 ECS, BullMQ, GoodJob, database, fallback, PostHog, and Flipper refresh was unavailable because the local AWS SSO token expired. Production remained read-only; no flag, queue, data, deployment, or infrastructure mutation was attempted.
• Residual direct call-detail transcription was re-ranked below Tickets because PR #10913 is already deployed in both regions and awaiting only its bake window. Tickets remained a bounded, normal-path Rails-owned user action with a deterministic parity harness and no dependency on the active QA regression.

Ownership change and parity contract

API2 now builds the classic Tickets workbook with one row per conversation and agent, latest eligible AI and Auditor evaluations side by side, scorecard-local criterion columns, merged KPA headers, flat-scorecard handling, organization and membership scope, workbook limits, durable BullMQ retries, stored attachments, and idempotent email delivery. The Tickets UI action queues a qa_tickets export run directly and retains the no-column-picker workflow.

The deterministic Rails/API2 comparator passes all 10/10 cases. Both former Rails-only Tickets cases match values, sheet names, row counts, and merge ranges. Focused API2 database tests cover the combined layout, inactive unoverridden human review selection, and scorecards without child criteria; the focused UI test covers direct API2 queueing with the ticket filter.

Roll back during the bake window by reverting ENG-2683; the Rails endpoint and ExportQATicketsJob remain available to that code revert. Do not delete queued BullMQ work. After deployment, require fresh successful Tickets exports in US and EU, no failed or stuck export-runs, correct multi-scorecard headers, and no related Sentry regression for 24 hours before removing the Rails rollback path or deleting the orphaned api2_manual_exports record.

Pace and next work

ENG-2683 contributes three code-ready points, which supports the 8–10 point weekly target if a second domain starts after production access is restored. The September 14–18 code-complete ETA is at risk while the regional runtime census is unavailable and automated QA remains safety-blocked. The next three ranked migrations are ENG-2680 after callback and stale-row recovery, the highest-runtime provider callback cluster after a fresh census, and the synchronous QA-stage transcription continuation contract.

September 2, 2026 — move Zendesk stuck-conversation recovery scheduling to API2

Progress at implementation: async weighted production completion remained 64%. This package advanced the then-current code-ready portfolio from 75% to 76%. Its one-point contribution is already included in the current 84% aggregate and must not be added again at merge. Work is tracked in ENG-2684 .

Original audit evidence and selection

• The latest pre-ENG-2684 production release is live in US and EU. ENG-2684 is code-ready but not deployed. Sentry shows no matching PII callback failure after that separate regression’s rollback release and no new direct call-transcription regression.
• One legacy EU organization without an API2 mapping produced four handled provider-sync errors in the latest 24 hours, compared with one in the prior 24 hours. A separate missing-Zendesk-subdomain issue remains active and needs an approved tenant repair or discard; neither condition was mutated.
• AWS SSO expired locally, blocking a fresh read-only ECS, regional database, BullMQ, GoodJob, and Flipper census. The migration therefore claims no new production credit and does not infer regional health from missing evidence.
• The six-hour Rails recovery job was the highest-impact bounded scheduler whose behavior and API2 destination were fully inspectable without that census. At that audit, automated QA was safety-blocked and in review; Tickets export and Jira reconciliation also awaited review. Those packages have since merged.

Ownership, parity, and rollback

API2 now owns the six-hour recovery scan and enqueues old, unsynced, live Zendesk conversations through the existing source-owned provider-sync queue. The scan preserves the one-hour grace period, chronological order, 500-row per-source cap, exact tenant/source scope, retries, and deduplication. It excludes discarded sources and missing API2 organization mappings before enqueue. Database-backed tests cover eligibility, tenant/source isolation, the grace window, deletion/sync exclusions, mapping eligibility, and independent per-source caps.

The Rails job class remains for rollback while its GoodJob cron is removed. After deployment, require four successful six-hour ticks in each region, no duplicate Rails sweeps, no new unmapped-organization errors from this producer, bounded cap warnings, and no growing provider-sync backlog. A code revert restores the Rails cron; queued BullMQ work must not be deleted.

September 1, 2026 — revert deterministic PII override after callback failures

Progress: async weighted production completion remains 64% and async code-ready completion remains 72%. No new workload ownership is claimed while the EU evaluation callback recovery awaits fresh successful callbacks. The recovery is tracked in ENG-2572 .

Production evidence and root cause

• Sentry RULEBASE-API-5S7 recorded 153 EU production ActiveRecord::RecordInvalid events in the first observed window. Retained EU conversation-QA jobs exhausted five attempts on the Rails 422.
• API2 accepted the model output and posted it to Rails. The deterministic PII override from PR #10614  then replaced the accepted result after boundary validation. A read-only production query confirmed the affected criterion has a regular fail option with zero justifications and an auto_fail option with two. The override preferred the unusable regular option and emitted failed without a selected justification.

Recovery and release contract

PR #10933  and PR #10935  reverted the #10616 scoring follow-up and the original #10614 override. The release containing both reverts deployed successfully to Production and Production EU at about 2026-09-02T00:42Z. The latest matching Sentry event was at 00:27:53Z, before the recovery deploy; no matching post-deploy event appeared in the first observed window.

Keep new QA ownership paused until fresh EU callbacks complete successfully and the clean window is sustained. The recovery does not change flags, queues, data, infrastructure, or the weighted completion estimate.

September 1, 2026 — bake sales-call compliance ownership into API2

Progress: weighted production completion remains 64% until the revision deploys and passes regional verification. Code-ready completion advances to 72%; 36 production points and 28 code-ready points remain. Work is tracked in PR #10932  and ENG-2679 .

Production and regression evidence

• The carry-over manual-export and user-requested QA PRs merged at 23:35 and 23:25 UTC. Their release revision deployed to regional API2 and Rails workers by 23:56 UTC; every service is at desired capacity with no pending task.
• US has api2_sales_call_compliance_pipeline globally enabled. The latest 24 hours produced 70 API2 result snapshots with 70 distinct result-run IDs. The queue has no waiting, active, or delayed work and no failure after August 30.
• EU has no sales-call compliance flag record, sales-call row in the latest 24 hours, API2 result snapshot, completed job, or failed job. The bake therefore moves future EU sales calls without draining or rewriting existing work.
• Sentry access is restored. The current issue census shows no new transcription, Intercom, export, or sales-call compliance regression. QA eligibility timeouts and older provider data issues remain separate tracked surfaces.
• Residual Rails call-detail transcription stopped after cutover. EU recorded two isolated delivery fallbacks at 18:51 and 20:51 UTC and neither region has a Rails call-detail job after the latest deployment. The corresponding API2 queues have no waiting work or new failure.
• Automated QA remains blocked from expansion. US has 2,060 pending ticket-closed requests and 1,027 conversations with duplicate active rows; EU has 6,817 and 2,754 respectively. This population is excluded from the sales-call ownership change.

Ownership change and rollback

Every eligible sales call now selects the API2 holistic compliance pipeline in Rails webhook completion and API2 native completion without reading Flipper. Evaluation-credit and reduced-ingestion guards, deterministic BullMQ IDs, tenant scope, source-version conflicts, retries, Rails persistence callbacks, and terminal Sentry reporting are unchanged. Non-sales-call post-sync analysis remains Rails-owned.

After deployment, require 24 hours with no duplicate result runs, new retained or terminal sales-call failure, analysis stuck in progress over 30 minutes, transcription regression, or unexpected Rails sales-call analysis work. Roll back with a code revert and do not delete queued BullMQ work. Delete the orphaned US Flipper record only after that window; EU has no record.

Throughput decision

The migration unit changes from one narrow flag or trigger branch per run to one complete ownership domain targeting 4–6 weighted points. A domain may land through a short stacked PR series so each revision stays reversible, but its work includes the data repair, invariant, orchestration triggers, observability, rollback adapter, and final Rails normal-path removal. Regional bake windows overlap instead of serializing code work.

The next domain is ENG-2680 : repair the stale automated QA population, prevent duplicate active requests, and transfer ticket-closed, scheduled, and admin preparation together. At that pace, the working target moves to September 14–18 for code ownership and September 28–October 2 for production-baked completion.

Program end-state decision

The migration completes only when the rulebase-api Rails project and every Rails/GoodJob server are deleted. Before that deletion, API2 must own all required webhook ingress and background work, the first-party app must dual- load and then cut fully to API2, and the API2 public domain must remain stable for external evaluation consumers such as Qonto. The 64% production and 72% code-ready figures measure only asynchronous workloads; synchronous cutover and Rails deletion readiness remain unbaselined until their censuses are complete.

PostgreSQL must also converge on a single public schema. ENG-2682 inventories and moves required api2 objects through new dependency-ordered roll-forward migrations, verifies regional data and query parity, removes every runtime reference, and drops the empty schema before final Rails deletion. The complete program end state is tracked in ENG-2681 .

September 1, 2026 — move user-requested QA preparation to API2

Progress: weighted production completion remains 64% until the revision deploys and passes regional verification. Including the open manual-export bake-in and this bounded ownership transfer, code-ready completion advances to 70%; 36 production points and 30 code-ready points remain. Work is tracked in PR #10926  and ENG-2678 .

Production and regression evidence

• In the latest seven days, US completed 4 user-requested QA requests with no failures. EU completed 62; its 12 failures all predate August 29, and 10 were the repaired consumed_sampling_rule_partitions schema-drift error. Neither region has a pending, ready, or processing user-requested request.
• The stale population is isolated from this change: US has 2,059 and EU has 6,639 pending requests, all with the ticket_closed trigger. Automated traffic remains on Rails while that population is repaired and reconciled.
• Regional ECS, BullMQ, GoodJob, and database evidence remained healthy after the latest deployment. QA sampling recovery and transcription queues were clear, the Intercom reconciliation guard had no post-deploy failure, and the EU provider-sync backlog was drained.
• Fresh Sentry issue verification remains unavailable because the configured token receives HTTP 403 and both browser paths require authentication. This is an explicit observability gap, not an inferred clean Sentry window.

Ownership change and safety contract

User-requested QA preparation now hands off to API2 regardless of the automated processing flag. Ticket-closed, scheduled, and admin requests still require the organization Flipper gate. API2 keeps its deterministic request job ID, conversation deduplication, bounded stage calls, exponential retries, tenant qualification, and Rails terminal-failure callback; immediate delivery failure continues to enqueue the original GoodJob path.

The API2 advance ceiling now raises instead of returning a successful max-advances result. That makes BullMQ retry an unfinished request and invoke the existing terminal failure path after exhaustion rather than silently orphaning a pending row.

After deployment, require a 24-hour regional window with zero user-requested rows orphaned over 15 minutes, no duplicate evaluations, bounded retries and fallbacks, and no new request-processing errors. Roll back by reverting the revision. Automated traffic and its Flipper gate are unchanged.

September 1, 2026 — bake manual exports into API2

Progress: weighted production completion remains 64% until the bake-in deploys and passes regional verification. Code-ready completion advances to 69%; 36 production points and 31 code-ready points remain. Work is tracked in PR #10925  and ENG-2677 .

Production and parity evidence

• PostHog api2_manual_exports (ID 255646) has an untargeted 100% rollout and was last changed on August 21.
• In the latest 24 hours, EU completed two manual QA evaluation export runs. Neither regional database had a failed manual export row, and both export-runs queues had zero waiting, active, delayed, or paused work.
• The local deterministic workbook harness matched all eight supported Rails/API2 cases across QA evaluations and All interactions. Two Tickets cases remain documented expected mismatches because Rails produces the combined AI/Auditor multi-scorecard workbook that API2 does not yet own.
• The freshly deployed QA sampling bake-in completed new work in both regions with no new BullMQ failure or backlog. Intercom reconciliation had no failure after its source-selection guard, residual transcription queues were clear, and the EU provider-sync backlog remained drained.
• Fresh Sentry issue reads were unavailable: the configured read token returned HTTP 403 and neither authenticated browser surface had a Sentry session. Queue, database, and ECS evidence therefore carry this audit; the deployed manual-export revision still requires its normal Sentry observation window.

Ownership change and rollback

Evaluations and All interactions now queue one-off exports directly through API2. The change removes the live owner-selection branch, the migrated Rails client functions, their dead unit tests, and the feature schema key. Tickets remain explicitly on Rails until the combined workbook has API2 parity.

The parity comparator also ignores the generated Export Date value while still comparing every other cell, sheet name, row count, and merge range. This removes a minute-boundary false failure without weakening workbook coverage.

After deployment, require fresh successful manual exports in US and EU, no failed or stuck export-runs, and no related Sentry regression. Roll back by reverting the bake revision; delete the PostHog record only after the exact deployed revision passes the window.

September 1, 2026 — bake workspace search into API2

Progress: weighted production completion remains 64% until the new revision deploys and passes regional verification. Including the merged transcription change, PR #10920 , and this workspace-search bake-in, code-ready completion advances to 68%; 36 production points and 32 code-ready points remain. Work is tracked in PR #10921  and ENG-2674 .

Production and regression evidence

• PostHog flag api2_workspace_search (ID 255632) has an untargeted 100% rollout and was last changed on August 21. The project event schema does not retain flag-evaluation events, so direct call counts are unavailable and are not inferred.
• API2 services and workers were healthy in US and EU with no pending ECS tasks. The regional Sentry audit surfaced no workspace-search issue among the current API2 groups.
• Residual transcription from PR #10913  is deployed. Retained queues contained 986 US and 157 EU successful call jobs with zero new failures or backlog. Three Rails HTTP 404 handoff fallbacks occurred during the rolling API2 deploy and none appeared later.
• The Intercom reconciliation guard from PR #10888  is deployed with no post-deploy failure. Its retained failed sets are old unmapped-source records, and the 24-hour cleanup window remains open.
• EU provider sync drained its earlier waiting backlog. QA evaluation request processing has no active queue growth, but 2,059 US and 6,170 EU old pending requests plus recurring timeout cleanup block an ownership bake-in.

Ownership change and safety contract

Command K now always requests its entity results from API2 workspace search. The change removes the feature check, Rails multisearch, separate company and work-item queries, their result adapters, and the dead test surface. API2’s cross-type relevance order, exact-identifier priority, result cap, URLs, and loading behavior are retained.

After deployment, verify representative exact-ID, name, and cross-type searches in both regions and require no new workspace-search Sentry regression through the observation window. Roll back by reverting the revision. Delete the PostHog record only after the exact deployed revision passes those checks; it no longer selects an owner after the bake-in deploys.

September 1, 2026 — bake QA sampling recovery scheduling into API2

Progress: weighted production completion remains 64% until PR #10920  deploys and bakes; code-ready completion advances to 67% after today’s residual transcription and QA sampling changes. The estimate keeps medium confidence, with 36 production points and 33 code-ready points remaining. Work is tracked in ENG-2545 .

Production evidence

• api2_qa_sampling_recovery was globally enabled in Flipper in US and EU. The US recovery queue completed 1,000 retained jobs in the latest 12 hours; EU completed 766. Both had zero failures, waiting jobs, or active jobs in that window. Their latest completions were at 17:05 and 17:07 UTC.
• Rails still ran QueueEligibleConversationsForQAEvaluationJob 48 times per region in the latest 12 hours, consuming about 5.2 seconds per US sweep and 6.0 seconds per EU sweep before the global flag skipped API2-owned organizations. API2’s latest sampling Sentry event was August 30; matching Rails query-cancel groups last occurred August 21 or earlier.
• The Intercom reconciliation guard is deployed in both regions. The first post-deploy 17:00 UTC schedulers completed without a new retained failure. Its 24-hour deletion window remains open. EU provider sync improved to about 1,486 waiting jobs with a 49-minute oldest job, down from roughly 2,919 and 57 minutes in the preceding audit.
• QA evaluation request processing remains the largest observed Rails workload at 13,068 combined executions in the latest regional 12-hour windows, but its API2 flag is disabled in both regions. It must return through a staged rollout before it is safe to bake.

Ownership change and rollback

API2 now lists every Rails organization on its durable 15-minute BullMQ tick. The Rails global GoodJob cron and per-terminal-request GoodJob backfill fan-out are removed, along with the rollout gate on the authenticated internal domain operation. Rails remains the authoritative bounded QA-selection boundary; API2 owns scheduling, retries, organization isolation, and deduplication.

After deployment, require 24 hours of fresh completions with no failed or stuck recovery jobs, no material sampling-error increase, and stable ticket-closed request creation in both regions. Roll back by reverting this revision; the old flag no longer selects an owner after deployment. Delete the Flipper record only after the exact deployed revision passes the bake window.

September 1, 2026 — prepare residual call-detail transcription ownership

Progress: remains 64% production-verified and rises to 66% code-ready (provisional). ENG-2670  moves direct asynchronous call-detail handoffs to API2. The two-point delta is not credited to production completion until the revision deploys and bakes in both regions.

Production evidence

• In the latest EU 12-hour window, TranscribeConversationPartCallDetailJob ran 1,986 times for 39,130 seconds of worker runtime—about 10.9 worker-hours and the largest retained Rails job by runtime. It had zero error executions and three outstanding jobs.
• EU persisted 1,983 successful call-detail transcriptions with zero terminal failures in that window; US persisted 827 with zero terminal failures and had no outstanding residual Rails jobs.
• The API2 conversation-transcription call queue had no EU backlog or retained failures. US had one active job and one old, expected undersized-audio failure. The completion queues were also clear apart from one US waiting-children parent.
• Sentry showed no new API2 transcription regression: the only API2 call issue had one event in 14 days and none in the latest day. Rails recorded two conversation-level API2 handoff read timeouts in the latest day; the existing delivery fallback handled that boundary.
• The conversation-level PostHog rollout remains 100% for the organization group. The direct-call package temporarily reuses it rather than introducing a second owner flag; false or unavailable evaluations retain Rails ownership.
• The Intercom reconciliation guard in PR #10888  merged during this audit and began regional deployment. The retained failed sets had grown from 148 to 159 in US and 73 to 78 in EU before the guarded revision was verified live, so their 24-hour clean window remains open.
• EU provider sync remained active with 2,919 waiting and 16 active jobs; its oldest waiting age improved from roughly 80 to 57 minutes. This is monitored independently and did not block the isolated transcription migration.

Ownership change and safety contract

When the existing organization gate is enabled, Rails hands ConversationPartCallDetail#transcribe_later to an authenticated single-call API2 endpoint. API2 reuses the existing tenant-scoped transcription worker, retry policy, terminal audio handling, Rails persistence callbacks, and deterministic call job IDs. A disabled or unavailable gate and immediate delivery failures still enqueue the legacy Rails job.

The conversation-level Rails fallback keeps its call jobs inside the original GoodJob batch instead of using the new standalone API2 path. This preserves the fan-in contract and prevents post-transcription analysis from running before an independent BullMQ child finishes. Synchronous QA-stage transcription remains an explicit, separately ranked Rails dependency.

September 1, 2026 — constrain reconciliation to API2 organizations

Progress: remains 64%. The August 31 forwarding recovery is healthy, but the audit found that the new API2 reconciliation scheduler was selecting legacy Rails sources whose organizations do not exist in API2. The estimate does not increase because this run fixes retry amplification without moving additional production workload ownership. Follow-up tracking remains in ENG-2605 , with the source-selection guard in PR #10888 .

Production evidence

• PR #10791  merged and deployed to Production and Production EU on August 31. All five Rails Api2IntercomWebhookForwarder::Error issue groups had zero events in the latest 24 hours; their last events preceded the recovery deployment.
• Mapped reconciliation sources in both regions advanced through the 13:00 UTC scan with no retained page checkpoint. The last 24 hours contained 13,920 new mapped-source conversations in US and 909 in EU, with zero duplicate external keys.
• The Intercom conversation and ticket import queues had zero waiting, active, delayed, or failed jobs in both regions. Relevant Intercom GoodJob classes also had zero unfinished work.
• The EU provider-sync queue was actively processing but still held roughly 2,700 waiting jobs, predominantly source-owned Zendesk syncs; its oldest waiting job was about 80 minutes old. No matching API2 Sentry issue had a production event in the latest 24 hours, so this remains a regional backlog to monitor before Zendesk flag cleanup rather than evidence for changing the reconciliation patch.
• Reconciliation itself retained 148 failed jobs in US and 73 in EU, with two and one delayed retries respectively. Every sampled failure was one of three regional Rails sources whose organizations had no API2 mapping. Sentry issue RULEBASE-API2-8P recorded 144 events in the latest 24 hours versus 75 in the preceding portion of the 14-day window.
• The migration rollout audit was read-only. The two Intercom and transcription PostHog flags remain active at 100%. Workspace search, manual exports, and thirteen baked read flags also remain active at 100%. Intercom forwarding, ingestion, and Zendesk source sync remain globally enabled in Flipper in US and EU; the sales-call compliance gate remains intentionally US-only pending its independent rollout audit.

Cause and recovery

The scan queried every non-discarded Rails Intercom source, then deferred API2 organization lookup until each source job loaded credentials. A source without an API2 organization therefore failed five times every 15 minutes, retained a failed BullMQ record, and could never enqueue conversation imports.

Source discovery now joins through api2.organizations on the Rails organization ID and deduplicates its result. Mapped sources preserve the same tenant-scoped credentials, pagination, checkpoint, deterministic import IDs, retry policy, and rollback behavior. Unmapped legacy sources are excluded from a processor that cannot service them and remain explicit migration inventory gaps; no production data, flags, queues, or infrastructure were changed.

August 31, 2026 — Intercom forwarding deployment recovery

Progress: remains 64%. Intercom ownership is baked into API2, but its Rails ingress exposed a resilience regression that must bake cleanly before the four orphaned flag records are deleted. Recovery work is tracked in ENG-2629  and PR #10791 .

Production evidence

• The bake-in from PR #10757  and reconciliation replacement from PR #10759  deployed to both Production and Production EU.
• Rails Sentry recorded 405 API2 Intercom forward failures in the last 24 hours: 211 read timeouts, 153 connect timeouts, 36 open timeouts, and five HTTP 502 responses. The timeout issues were absent from the prior seven-day window; only one matching 502 preceded this window. Bursts align with rolling API2 deployment windows.
• API2 recorded four BullMQ lock-renewal events in EU during the same window. No payload or tenant-isolation defect was evident in the inspected events.
• Direct regional BullMQ, GoodJob, fallback-log, and reconciliation-watermark checks were unavailable because the local AWS SSO session had expired. PostHog log discovery also returned no service names, so those checks remain deployment-verification requirements rather than inferred successes.
• The two Intercom PostHog records remain active at 100%, but their selection branches are no longer referenced by deployed code. No production flags, queues, data, deployments, or infrastructure were changed during the audit.

Cause and recovery

Rails forwarded the provider request synchronously with a two-second connect timeout and five-second response timeout. Transient API2 unavailability during a rolling deployment therefore became a provider-visible failure even though Rails still had a durable, tenant-scoped processing route.

The recovery keeps direct API2 forwarding as the normal path. On an immediate delivery error, Rails logs the fallback, resolves the matching kept Intercom sources without tenant leakage, and enqueues the existing ProcessIntercomWebhookEventJob. Conversation and ticket events then use their existing deterministic API2 handoff and GoodJob delivery fallback; deletion effects remain intentionally owned by Rails. A timeout after API2 accepted the original request is safe because API2 import IDs are deterministic and the Rails jobs retain concurrency keys.

Deploy and observe this recovery through at least one rolling deployment in each region before deleting the orphaned Intercom flags or starting a new cutover.

August 31, 2026 — Intercom reconciliation recovery

The globally enabled API2 webhook and import paths remained healthy, but the separate US Rails drift-reconciliation cron was not. One Kuda parent job had retried more than 1,000 times from source 19’s 2026-08-24T06:30:01Z watermark. Rails auto-pagination accumulated the full account window in one process, exhausted the 4 GB worker, and never reached the single end-of-run watermark update. Its GoodJob concurrency key then rejected every later 15-minute cron job. EU completed the same cron normally.

The recovery replaces that global Rails job with an API2 BullMQ scan that fans out one deduplicated job per active Intercom source. Each source job:

• fixes an upper time boundary at the start of a scan;
• starts from the existing watermark minus a five-minute overlap;
• requests at most 150 conversations per page and 50 pages per run;
• enqueues IDs onto the existing deduplicated API2 import queue;
• saves the Intercom cursor after every page; and
• advances the watermark only after the fixed window is complete.

The Kuda watermark is not edited by the migration. The first API2 run resumes from August 24 and subsequent bounded runs continue from their durable page checkpoint. The obsolete Rails cron, job class, and periodic-sync model code are removed; the historical/manual Intercom importer remains available.

At 2026-08-31T00:46Z, the US GoodJob cron entry was disabled in production and confirmed to change from enabled to disabled. The stuck parent and its concurrency-retry rows remain in place until the API2 replacement is deployed; the source watermark has not been changed.

August 30, 2026 — Intercom bake-in prepared after regional audit

Progress: remains 64% until the bake-in revision is deployed and verified. The implementation removes four temporary owner-selection branches without removing genuine delivery-error or deletion fallbacks.

Production evidence

• The last 24 hours averaged 582 Rails web requests/minute versus 2,988 in the prior seven-day baseline. API2 increased from 345 to 556 requests/minute; API2 p95 improved from 177 ms to 40 ms, and both services had a zero measured web failure rate in the latest window.
• At roughly 90 minutes after the Intercom cutover, the US BullMQ conversation import queue retained 10,000 completed jobs and the ticket queue retained 17; both had zero waiting, active, delayed, or failed jobs.
• Since cutover, US persisted 315 new and updated 2,813 Intercom conversations; EU persisted 19 new and updated 195. Both regions had zero newly created unsynced conversations older than 15 minutes and zero duplicate external keys.
• US and EU had no failed or outstanding Intercom GoodJobs. US created 25 conversation-import GoodJobs after cutover, but no API2 enqueue-failure log occurred; the only remaining code path was a false/fail-closed PostHog owner selection, which the bake-in removes.
• API2 logged no webhook routing or processing failure across approximately 13,400 regional receipts. Rails had one US forward return a transient 502 during a rolling production deployment; it did not recur, and provider redelivery plus deterministic import IDs preserve idempotency.
• PostHog conversation and ticket flags remained at 100% organization rollout. Both regional Flipper databases had global true forwarding and ingestion records. No production flag or data mutation was performed by this audit.

Decision

Remove the two PostHog import selections and two Flipper webhook selections in code in PR #10757 . Keep the GoodJob fallback for immediate API2 delivery failure, keep Rails deletion ownership, and keep failed webhook responses retryable. Leave all four production flag records untouched until the merged revision is deployed and verified in US and EU.

August 30, 2026 — blockers cleared and Intercom globally enabled

Progress: established a provisional workload-weighted baseline of 64%. Confidence is medium until the full Rails route, GoodJob, scheduler, and callback census is complete.

Production audit and blocker resolution

• Confirmed the Zendesk requester foreign-key failures came from treating provider agents/admins as client requesters. Landed PR #10734 ; the affected Sentry issue’s last event preceded the fix.
• Classified undersized ElevenLabs audio as a permanent provider input failure instead of a retryable server error. Landed PR #10733  and PR #10739 ; the known Sentry issue’s last event preceded the fixes.
• Discarded demo Zendesk organization data source 43 at 22:04:44Z and internal test source 29 at 22:07:48Z. Source 29 was the origin of the expired-credential error stream and belonged to an internal test organization. No customer source was changed.
• Verified api2_zendesk_source_sync was already globally enabled in both US and EU. The earlier assumption that it still needed a rollout was corrected; its next action is bake-in, not enablement.

Intercom cutover

The implementation landed in four PRs:

• PR #10714 : conversation imports;
• PR #10717 : ticket imports;
• PR #10718 : API2 webhook ingestion with Rails forwarding for unowned events;
• PR #10720 : Rails-to-API2 webhook forwarding.

At 2026-08-30T22:29:50Z, the two PostHog import flags were at 100% for the organization group and the forwarding/ingestion Flipper gates were globally enabled in US and EU. A read-only monitor was scheduled for 30, 60, 90, and 120 minutes after cutover, covering regional traffic, queue health, retries, fallbacks, Sentry, duplicates, and stuck work. No automatic rollback is authorized by that monitor.

Bake-ins and operating cadence

• Started PR #10741  to make API2 transcription and supported Zendesk source sync permanent defaults while retaining the GoodJob delivery-error fallbacks. Production flag records stay in place until the merged code deploys and is verified.
• Created a daily 9:00 AM migration automation. It audits deployments and production health, updates this tracker, fixes regressions before new work, implements one bounded migration, and lands the PR only after required checks pass. Production deployments, flag changes, and data mutations remain explicit approval steps.

Decision

Do not overlap a new production cutover with the Intercom observation window. Code-only bake-ins and inventory work may proceed. If Intercom remains healthy, its four gates become the next bake-in; otherwise the first priority is the regression and its rollback recommendation.