Rails runtime retirement archive

Completed October 3, 2026. Rails source and deployment infrastructure are gone. API2 no longer has Rails HTTP clients, proxy fallbacks, effect adapters, workload-owner switches, or rollback environment variables. Everything below this notice is historical rollout evidence; do not follow its rollback or remaining-work instructions in a current environment.

Rails source removal — 2026-10-02

ENG-2681  / PR #13640  removed the Rails source tree, its CI/deploy entrypoints, and local boot path. The public-schema SQL snapshot moved to API2 for fresh local/CI database bootstrap. A local-only native session seed replaced the Rails-backed Playwright setup while retaining the end-to-end CI gate. The PR merged at 2026-10-02T15:55:15Z as 1c32bf66a8b1ca3beaf88c0803ced752d201c2e9; final-head aggregate CI, Playwright smoke, edge routing, and CodeRabbit passed. GitHub shows no submitted Ronald review despite a pending request. Record this as a review exception, not approval. Release PR #13658  merged to release-prod at 18:00:51Z; its UI Workers Builds checks failed, so its overall deployment must not be called green.

At the 2026-10-02T08:29Z checkpoint below, the gateway had no event of any outcome in either region over 24 hours; seven days still included 441 historical unsuccessful forwards and no successful forwards. Historical Rails controller log groups are unavailable, so this is not a fresh controller seven-day census. The full post-cutover seven-day zero window, native QA/effect outcomes and errors, and live development callback/domain ownership are still open. The source removal and already-destroyed service resources do not prove complete irreversible retirement. Keep the remaining API2 Rails adapters until their own gates pass.

2026-10-02T20:30:00Z follow-up (24 h starts October 1 20:30Z; seven days starts September 25 20:30Z). AWS account 879381255132 reports all three Rails ECS clusters INACTIVE with zero services/tasks: US fc-rulebase-api-prod-xv109mm, EU fc-rulebase-api-prod-eu-u56f166n, development fc-rulebase-api-dev-xs8w05rj. US/EU API2 web services are PRIMARY/COMPLETED at 1/1/0 on task definitions 1931/1948; both report running SHA 52194cf674eb586f5e1dc3ea0f5836827d6f9a68, which contains #13640. Canonical US/EU and api2-dev /v1/identity return 401; the legacy api-dev Slack callback returns 530. Live Slack app redirect and development domain consumers remain unverified.

API2 web logs reach 20:29:59.331Z US and 20:30:00.571Z EU (freshness queries 9b5b767c-a281-4b83-89c8-bb7d305803fe, a531dd9f-c4eb-4ba5-93be-7a3edc1ac462). legacy_rails_gateway has zero events of any outcome in either exact 24-hour window (US 65051f24-1d7d-4ded-b467-29e71157c909, EU 7aa6a7c6-9375-45b6-b9af-e930451a185e). The exact seven-day forwarded method/path/status queries have zero successful Rails forwards and 437 historical unsuccessful forwards: US 214 (212 status 503, 2 status 404), EU 223 (208 status 503, 15 status 404); query IDs US 0e300b2d-e4be-4413-9c04-d3ef11bf9a6c, EU d7735628-261e-4f56-9dc5-2a2ce1981809. /admin*, /clients/MyCRL, and /rails/actions are scanner-shaped; the two historical US GET /agent/agent_performance_reports 503s remain unattributed. API2-local rejected/blocked/retired 404s and normalized 308s are separate. Historical Rails controller log groups are unavailable, so no fresh controller method/path census or full post-cutover seven-day zero is credited. Natural native QA/effect outcome parity and development callback/domain ownership also remain open. Do not delete the remaining Rails adapters or claim complete retirement.

Source audit at 2026-10-03T02:31Z. The deployed Rails clusters are gone, but API2 still has reachable Rails error paths: QA sampling recovery dispatches its Rails branch when ownership resolves to rails; evaluation-check execution and terminal failure marking retain Rails rollback branches; expired-evaluation cleanup calls Rails when its native sweep defers rows; call-recording attachment defaults to Rails when its native owner flag is off or unavailable; and provider-sync effects call Rails if its native effect keys are incomplete. These are conditional code paths, not measured live calls. Before removing them, establish native outcomes and error parity, inspect the actual flag/worker configuration, and complete the post-cutover observation window. The source audit does not fill the missing controller logs or certify a zero-volume window.

2026-10-02T14:30:00Z follow-up (24 h starts October 1 14:30Z; seven days starts September 25 14:30Z). AWS account 879381255132 still reports the US, EU, and development Rails ECS clusters INACTIVE, with zero active services and running tasks. API2 web logs are fresh through 14:30:00.999Z US and 14:29:59.792Z EU (CloudWatch freshness queries 53d60b7b-f7e4-4ab6-8e4a-16ac2eec9842, 57d28d73-a1a8-4a0a-af08-b539322a85e1). API2 legacy_rails_gateway has zero events of any outcome in either 24-hour region (US query 00cc977a-7fcc-4fb2-999c-7fbfc246f692; EU 46917520-f2c5-4840-a98b-dedb32bfc69b). The seven-day window has zero successful Rails forwards and 441 historical unsuccessful forwards: 216 US and 225 EU (method/path/status queries 662f4c3b-d914-42a3-90dc-be839bd9f068, e1f1a456-a85c-4c89-9fd6-78a5abc85fb4). Most target /admin*; /clients/MyCRL and /rails/actions are scanner-shaped, while two historical US GET /agent/agent_performance_reports 503s still lack caller attribution. API2-local rejected/blocked/retired 404s and normalized 308s are separate from forwarded traffic. Historical Rails controller log groups are unavailable, so this checkpoint cannot prove a fresh controller method/path census. PR #13640  passed aggregate CI, Playwright smoke, edge routing, and CodeRabbit on its prior head, but Ronald has not submitted a review; this plan update requires new-head checks before any merge. The full post-cutover seven-day zero window, native QA/effect outcome parity, and live development callback/domain ownership remain open.

Shutdown checkpoint — 2026-10-01

ENG-3587  / PR #13576  removed the three Rails web service definitions from flightcontrol.cue (development, US production, and EU production). It also set LEGACY_RAILS_FALLBACK_ENABLED to false in both production API2 environments; development was already false. The CUE export comparison showed no other service changes. Final-head Flightcontrol, internal-docs, migration-freeze, aggregate CI, and CodeRabbit checks passed. The PR merged at 2026-10-01T02:04:33Z as fc58f3e48c2effbaf94985dd41a8c2923ed45ad7 and is included in release-prod. GitHub still reports REVIEW_REQUIRED with no submitted Ronald approval; record this as a review exception, not retroactive approval.

The exact production census ending 2026-10-01T02:29:01Z found zero Rails /internal/* and /conversation_images/* controller calls in each region over 24 hours. Seven days still contain 79 historical internal calls (74 US, 5 EU), all before the production scale-to-zero cutover, and zero image calls. Separate historical GET /up heartbeats in the seven-day window ending 02:29:55Z total 141,393 US and 141,652 EU; they are not live dependencies. API2’s gateway recorded zero successful Rails forwards in either window. Its 24-hour forwarded 503s total 377 (US 190, EU 187), dominated by /admin* scanner paths with two each of /clients/MyCRL and /rails/actions per region. Seven-day unsuccessful forwards total 464 (US 227, EU 237), including historical 404s. Rejected/blocked 404s and normalized 308s are separate from forwards. The full seven-day post-cutover zero window and native QA/effect outcome parity remain open.

At 2026-10-01T02:09Z, fresh ECS reads showed US and EU production Rails desired/running/pending 0/0/0, with staging at 1/1/0. Staging’s ECS autoscaling target was changed from min/max 1/1 to 0/0, then its desired count was set to zero. A 02:27Z three-region read confirmed all Rails web services ACTIVE with desired/running/pending 0/0/0, completed rollouts, and autoscaling min/max 0/0. The Flightcontrol service records and AWS resources remain in place for rollback; Flightcontrol’s development config page recognizes the Rails service is no longer in flightcontrol.cue. Canonical US/EU and development API2 /v1/identity probes returned the expected unauthenticated 401 after the staging scale change. The repository’s Slack manifest still references https://api-dev.rulebase.co/integrations/auth/slack/callback; at 02:27Z that URL returned 503, while development API2 /v1/identity returned 401. API2 owns /connections/oauth/slack/callback, a different path. Resolve the development callback and domain ownership before deleting the retained service or its domain.

Recheck native outcomes, errors, and the exact regional 24-hour and seven-day windows before widening any retirement claim. Preserve a reversible rollback path until those gates pass. Destroying a Flightcontrol service would delete its underlying AWS resources and is a separate step from this operational scale-to-zero shutdown.

08:26Z follow-up. PR #13578 , which recorded the scale-to-zero evidence above, merged at 2026-10-01T07:32:53Z as da2fae8566f0184b3399b5708cc0bafb6bdb3352. It had passing required checks but no submitted Ronald review; this is another review exception, not approval. release-prod cdc192ba15c7da9c5042344b2f509b184fc2ce4a contains both #13576 and #13578. A branch head is not proof of the running Flightcontrol or ECS revision.

At approximately 08:26Z, unauthenticated HTTP probes returned 503 from api-dev.rulebase.co on both /v1/identity and /integrations/auth/slack/callback, while api2-dev.rulebase.co/v1/identity returned the expected 401 and /connections/oauth/slack/callback returned 400 without OAuth parameters. The checked-in Slack manifest lists the dead Rails development callback and does not list the API2 development callback; the live Slack app configuration has not been verified. The current flightcontrol.cue assigns api2-dev.rulebase.co to the development API2 service and also configures api-dev.rulebase.co in API2’s API2_CORS_ORIGIN, RULEBASE_API_BASE_URL, and RAILS_INTERNAL_BASE_URL for both web and worker. API2’s native image-caption parser uses RULEBASE_API_BASE_URL to recognize historical Rails image URLs; native provider sync mints new image URLs on RULEBASE_API2_BASE_URL. The development UI’s live API base URL and runtime consumers of the old api-dev origin remain unverified. Migrate or remove the Slack redirect in the live app, verify the development UI and other api-dev consumers, and settle domain ownership before destroying the retained staging service.

AWS SSO token refresh failed on this follow-up, so ECS state, Flightcontrol deployment state, CloudWatch volumes, and native outcome/error parity could not be refreshed. Retain the 02:27Z ECS verification and 02:29Z census above as the latest exact evidence rather than extending their windows by inference.

14:33:20Z follow-up (fixed cutoff 2026-10-01; 24 h begins September 30 14:33:20Z, seven days begins September 24 14:33:20Z). PR #13585  merged externally at 08:39:35Z with passing required checks but no submitted Ronald approval, a third recorded review exception. CodeRabbit correctly identified the api-dev API2 configuration references corrected above. AWS STS again succeeded in account 879381255132. ECS shows all three Rails services ACTIVE and stable at desired/running/pending 0/0/0: US fc-rulebase-api-prod-xv109mm, EU fc-rulebase-api-prod-eu-u56f166n, and development fc-rulebase-api-dev-xs8w05rj. No service or domain was destroyed. The US and EU API2 web services are stable at 1/1 on task definitions 1915 and 1932; both task definitions report FC_GIT_COMMIT_SHA=27ace5eb96e283b9262b1636b042cb31e1fcdf03, which contains #13576 and #13578. The newer release-prod branch head 8dbf2ff1e0626578e1f7bdee6d06cbe03d9a2700 is not yet the verified running web revision. Canonical US/EU and development API2 /v1/identity returned 401; api-dev.rulebase.co/v1/identity still returned 503.

CloudWatch Rails process_action.action_controller method/path/status queries found zero /internal/* and /conversation_images/* calls in each region over 24 h (US query 064322b4-8a9d-4e5c-b4aa-df5923b323ca; EU 73cad777-8d35-4c4d-a06a-1930797057f0). Seven days contain 72 historical US internal calls, zero EU internal calls, and zero image calls in either region (US 7b4c818a-8005-4944-bfd1-33c0a1b56554; EU abce4e35-4763-4f31-9874-dcf8996f2284). The US 72 are POST evaluation-check run 200 ×45, transcription context GET 200 ×4 and write POST 200 ×4, reconciliation POST 200 ×5, QA sampling recovery POST 200 ×3, report email POST 200 ×3 and preview POST 200 ×2, evaluation results POST 200 ×2, Slack notification POST 201 ×2, Unit21 sync POST 200 ×1, and direct-upload process POST 500 ×1. Latest US internal call was POST /internal/slack_notifications 201 at September 26 13:00:24Z (query 63b9b8e1-0de5-44c4-8bac-31e8b7b79dc8). Separate seven-day empty GET /up 200 heartbeats were 115,369 US and 115,628 EU (queries a289166b-7d69-4944-b3de-44956c681a08, 9f627723-52d8-47cb-8fe0-520e92d4c103).

API2 legacy_rails_gateway method/path/status queries show zero successful Rails forwards in either window. In 24 h, US had no forwarded events; EU had one scanner-shaped POST /admin 503 (path queries US 9f79181b-249e-44eb-b314-185bb19417a9, EU 97f1643c-d8cb-4265-a52e-856f5c02abdc). The other gateway events were API2-local rejected/blocked/retired 404s: US 654 total (647 rejected, 3 blocked, 4 retired), EU 1,059 total (1,056 rejected, 2 retired, 1 forwarded). Seven days include 218 US unsuccessful forwards (212 503, 6 404) and 235 EU (211 503, 24 404), dominated by admin/old-auth scanner probes, with an unresolved historical US GET /agent/agent_performance_reports 503 ×2; there are no successful forwards (summary queries US 414d4703-bbe2-4d11-908e-335b424f730d, EU 4ca97b2f-d62f-4d9b-ab2f-6b3edc95b089; path queries US 65fe274b-dcb5-4c01-856b-c2cb83718866, EU 28946e59-0302-4b57-a5dd-fa065100c326). Rejected/blocked 404s and normalized 308s are separate. The complete post-cutover seven-day zero window, development callback/domain ownership, and native QA/effect outcome parity remain open; retain the reversible zero-replica services.

20:30:00Z follow-up (fixed cutoff 2026-10-01; 24 h begins September 30 20:30:00Z, seven days begins September 24 20:30:00Z). AWS STS succeeded in account 879381255132. ECS again shows all three Rails services ACTIVE, PRIMARY/COMPLETED, desired/running/pending 0/0/0 (US fc-rulebase-api-prod-xv109mm, EU fc-rulebase-api-prod-eu-u56f166n, development fc-rulebase-api-dev-xs8w05rj). No Flightcontrol service, domain, or AWS resource was destroyed. Canonical US/EU and development API2 /v1/identity returned 401; the old development Slack callback at api-dev.rulebase.co/integrations/auth/slack/callback returned 503. The live Slack app redirect and development UI/API base URL remain unverified.

Exact Rails controller method/path/status queries found zero /internal/* and /conversation_images/* calls in either region over 24 h (US dbcc8c56-ce3e-41ce-88b5-c13607225ab3, EU 239d7611-7736-49b3-883e-1895d9c92cd1). Seven days contain 67 historical US internal calls, zero EU internal and zero image calls in either region (US 08dfece1-bbdb-4cd7-a3fb-54752b916880, EU b4e3e832-79a1-4a7a-b119-5722536875e3). US method/status totals are POST 200 ×60, GET 200 ×4, POST 201 ×2, POST 500 ×1; these are pre-scale residue, including evaluation-check runs, transcription, reports, Slack, Unit21, and one failed direct-upload process. Separate seven-day GET /up 200 heartbeats are 102,500 US and 102,757 EU (US 79308d62-c7ff-414c-b2d7-b14f841936ce, EU fdb0343c-493d-4863-a392-ddf550b31c9a); they are historical empty health checks, not live application dependencies.

The API2 gateway has zero successful Rails forwards in either window. The 24 h contains zero forwarded events in both regions; all 314 US events and 227 EU events were local 404 responses (US rejected 311, retired 2, blocked 1; EU rejected 227). Seven days contain 216 US unsuccessful forwards (212 503, 4 404) and 225 EU (211 503, 14 404), dominated by historical /admin* scanner probes (exact summary queries US 297d7d92-cc75-4a47-a15a-b4ab7d6caa33, EU 02d5b8dc-9676-456e-8d7a-00b1743807ae; path queries US b6d1fb47-b2c6-40d2-bf0d-4ee3840dd476, EU bc877208-09d6-468d-92c7-7f2932ed9152). The two historical US GET /agent/agent_performance_reports 503s still need caller attribution; no successful live dependency was observed. Rejected/blocked/retired 404s and normalized 308s are not successful Rails forwards. PR #13590  has green required checks but no submitted Ronald review at this checkpoint and remains open. The full seven-day post-cutover zero window, native outcome/error parity, and development callback/domain ownership still block irreversible retirement; keep the zero-replica services available for rollback.

2026-10-02T02:30:00Z follow-up (24 h starts October 1 02:30Z; seven days starts September 25 02:30Z). PR #13590  merged externally at October 1 20:49:07Z as 1f94deae615c7f4c643c4bd8e9d66dddee22f4a4 with green checks and no submitted Ronald approval; this is a review exception, not approval. Flightcontrol subsequently destroyed the three Rails ECS clusters through its CloudFormation role at October 2 01:45:37Z (US), 01:45:54Z (EU), and 01:45:52Z (development). ECS now reports all three clusters INACTIVE with zero services, superseding the reversible zero-replica posture above. This resource deletion was not performed by this automation. The US/EU canonical API hosts and api2-dev.rulebase.co/v1/identity still return 401; api-dev.rulebase.co/v1/identity returns 530. The checked-in development Slack callback and API2 api-dev URL configuration described above have not been reconciled with the live Slack app, development UI, or domain owner. Treat that dev-host failure as an unresolved operational risk.

The API2 web services remain ACTIVE, PRIMARY/COMPLETED at desired/running/pending 1/1/0: US task definition 1922 and EU 1939 both report FC_GIT_COMMIT_SHA=2ec23e3294dbad401ee1cb092a32b92e4c34ac4e; development task definition 2335 reports be46e98d6c3457259dec0394d03118036afb663a. These are running-task-definition attestations, distinct from branch heads.

API2 gateway logs are fresh through approximately 02:31Z. Fixed-cutoff CloudWatch queries found zero legacy_rails_gateway events of any outcome in either region over 24 h (US 21b07d1f-e484-4d41-9254-9b85bdace5b1, EU b7c8bf5e-1ce2-4595-a366-125ab02de3ca). Seven days contain zero successful Rails forwards; 216 US and 225 EU historical forwards were all unsuccessful (US GET 503 ×155, POST 503 ×57, GET 404 ×4; EU GET 503 ×176, POST 503 ×35, GET 404 ×12, POST 404 ×2). The seven-day forwarded paths are dominated by /admin* probes, with /clients/MyCRL and /rails/actions also present (summary queries US a561e77c-d002-455f-b831-d42eaf5408d2, EU fa71f9d0-31de-4f85-bb93-aba8eca63d7f; path queries US fa290f1d-524f-4518-bb3f-f70307400abb, EU 57b154af-5a18-461d-9f09-c96a20452f9f). Rejected, blocked, and retired API2-local 404s and normalized 308s are separate from forwards. The historical Rails controller log groups are no longer listed in CloudWatch; the latest exact controller census remains the October 1 20:30Z observation above. Do not infer a new controller seven-day zero window from the missing log groups. The checked-in Slack manifest now names the existing API2 development callback instead of the dead Rails callback, but this does not establish or change the live Slack app configuration. Native QA/effect outcome and error parity, the full post-cutover seven-day window, and dev-host ownership remain open; do not claim complete irreversible retirement.

2026-10-02T08:29:00Z follow-up (24 h starts October 1 08:29Z; seven days starts September 25 08:29Z). PR #13624 , which recorded Flightcontrol deletion and corrected the checked-in development Slack callback, merged externally at 02:47:46Z as 3bf7c1c6eee199601ad8f363528f7a7889a513bd. Its final-head CI workflow succeeded, but GitHub lists zero submitted Ronald reviews; this is another review exception, not approval. Fresh AWS ECS reads show the US, EU, and development Rails clusters remain INACTIVE with zero services. Canonical US/EU and api2-dev /v1/identity return 401; api-dev.rulebase.co/v1/identity returns 530, while API2’s development Slack callback returns 400 without OAuth parameters. The live Slack app redirect and development host consumers remain unverified.

The US/EU API2 web log groups contain events through 08:29Z. Fixed-cutoff legacy_rails_gateway queries show zero events of any outcome in either region over 24 h (US abece527-24a6-48ad-955f-4fa9e7183ab1, EU 221b2829-5346-427b-972c-c60c7e897fc0). Seven days still contain zero successful forwards and the same 441 historical unsuccessful forwards: US 216 (GET 503 ×155, POST 503 ×57, GET 404 ×4) and EU 225 (GET 503 ×176, POST 503 ×35, GET 404 ×12, POST 404 ×2); summary queries US c72acf41-d410-436e-a639-35406c4c110d, EU bc653688-89bb-4265-847f-91717afc9cac. API2-local rejected, blocked, and retired 404s and normalized 308s are separate. This is gateway evidence, not a fresh Rails controller method/path census: those historical controller log groups are still unavailable. Native QA/effect outcomes and errors, live development callback/domain ownership, and a complete seven-day post-cutover zero window remain open.

The public API belongs to API2. The remaining migration goal is narrower: nobody should use the Rails application runtime.

This plan is the source of truth for that work. It organizes the remaining dependencies by who calls Rails, measures them from production traffic, and turns the largest coherent families into migration packages.

Current boundary

The migration has crossed the public ownership boundary:

• api.rulebase.co and eu.api.rulebase.co terminate on API2.
• API2 owns the documented /v1 public contract and its OpenAPI document.
• Deprecated public paths are API2-owned compatibility aliases with successor links and sunset metadata.
• Rails sits behind regional legacy-rails.* origins as an implementation dependency. Public documentation and API responses do not expose those origins.

This makes Rails an internal legacy service from a product and API-contract perspective. It does not make Rails unused. API2 still forwards some public requests to Rails and calls Rails controllers for behavior that API2 does not yet own.

The retirement program therefore has two active workstreams:

1. Stop forwarding public calls to Rails.
2. Stop calling Rails for effects.

Residual GoodJob, Active Storage, and Rails operational work are shutdown gates for those workstreams. They are not a separate queue of migration work.

What recently changed

The canonical public API changes entered release-prod in #12352 , and the follow-up release #12358  deployed as 7dd7eef. On September 17, 2026, both regional API2 services were steady on their current task definitions, canonical /v1/identity returned the native API2 authorization response in both regions, and the canonical OpenAPI document and documentation redirect were live. The public-boundary deployment gate is therefore complete.

The same day, attachment purge #12353 , image-caption context/writeback #12355 , and generic tool telemetry plus the first two native tools #12356  reached both regions in release-prod revision 9311bcca. Both API2 and Rails services completed their rollouts by approximately 2026-09-17T13:04:40Z. The first nine-minute post-deploy slice contained zero Rails calls for those three families. This is an initial smoke signal, not the complete production window required to close their retirement gates.

QA sampling recovery #12369  merged into #12373  at 2026-09-17T13:26Z. Keep qa_sampling_recovery_native off: the merged native transcription claim still uses the plain organization counter instead of Rails’ member-aware counter and :members set. A same-hour rollback or redelivery can therefore charge one conversation twice. A follow-up correction and focused counter-compatibility proof are required before any production canary. That correction shipped in #12423  and is live on cb829218; the September 18 rollout below supersedes the “keep off” guidance.

Native public media #12374  reached both regions through #12397 . The regional API2 deployments completed at approximately 2026-09-17T16:40Z. In the first post-deploy slice, API2 served six US conversation-image requests successfully, and neither region forwarded a conversation-image or Active Storage request to Rails. This is enough to confirm that the route is active; the comparable outcome distribution and complete zero-forwarding window remain the retirement gate.

The production census ending around 2026-09-17T17:11Z extends structured gateway coverage to approximately 13 hours. It contains 15,535 events: 8,335 scanner, unknown-path, or Rails 404 events and 7,200 meaningful events. The meaningful tail is now led by Active Storage and conversation images (3,599), Action Cable (1,917), conversation uploads and preflights (850), Unit21 (332), Zendesk (422), Slack (2), and 78 other events. This remains a partial window and must not be extrapolated into a daily rate.

Three clean-current-main implementation packages are active from this census: ENG-3217  corrects the shared member-aware QA throttle while the production gate remains off; ENG-3218  moves public conversation-upload ingress and status ownership into API2; and ENG-3219  starts the Cable retirement with native SSE for ConversationChannel. None is merged or deployed.

Evidence as of 2026-09-18 (trailing windows ending approximately 10:16Z)

Historical: superseded by the 13:12:11Z section below where the two disagree.

Deployment state: the evidence windows below were measured while both regions ran release-prod cb829218 on every API2 process (US web 2/2 and worker 5/5; EU web 1/1 and worker 5/5). At 10:17:31Z release-prod advanced through #12477  to 1d29cccdf5c02a8c3ede5498b045007f43f78e75, which contains release d2188798 (call-recording attachment #12455 , conversation transcription state #12456 , native transcribe_calls #12463 ) plus upload-driven sync completion #12469  and native QA related-conversation sync #12470  (both merged 10:17:19Z). That deployment is in progress: at 10:19:26Z US web/worker still ran cb829218 while EU web/worker had advanced to the intermediate release 9d657672131dde460b927cb6be867bf6304632f8, all steady. Attribute no traffic change to any of those packages until the running task SHAs show 1d29cccd in both regions and the rollout has completed. The API2 worker services are the Flightcontrol logical services rulebase-api2-worker-prod and rulebase-api2-worker-prod-eu; their physical ECS names contain -pr- and must not be read as missing production workers.

Effects boundary, fixed trailing 24 hours: 47,230 API2-to-Rails calls. 6,962 are pre-cutover residue from already-shipped families (attachment purge 3,000; image captions 1,279; generic tools 2,683), every one last seen before 2026-09-17T12:54Z. The remaining 40,268 meaningful live calls are:

Effects boundary, fixed trailing seven days: 417,996 calls, of which 199,570 are shipped-family pre-cutover residue (purge 115,637; captions 37,745; tools 46,188) and 218,426 remain.

Gateway boundary, fixed trailing 24 hours: 18,934 events: 527 blocked at API2, 12,817 forwarded and answered 404 by Rails (scanner/unknown paths), and 5,589 meaningful as an upper bound. The current clear live tail is 2,583 = Action Cable 1,415 + conversation uploads 944 + Outreach 224.

The seven-day gateway query still has only about 30 hours of structured coverage: 24,205 events, 644 blocked, 14,559 Rails 404, 9,000 meaningful; clear live tail 3,926 = Cable 2,486 + uploads 1,216 + Outreach 224. Do not present this as complete seven-day coverage.

QA sampling recovery rollout: qa_sampling_recovery_native was created at 09:36Z as a bounded kuda-pilot canary, expanded at 09:53Z to kuda-pilot, rho-tnkw, qonto, at 10:12Z to easypay and nala, and at 10:16Z advanced to those five exact 100% canaries plus a second 25% organization-level rollout group. Every completed natural cycle (Kuda 09:45 and 10:01, Rho 10:00, Qonto 10:10, Easypay 10:15, NALA 10:15) ran natively with zero candidate or transcription failures; the per-cycle counters are in the rollout runbook . PostHog last_called_at stayed at 09:45Z throughout and is advisory only; the structured worker result is the source of truth.

QA sampling recovery rollback (10:36Z): the first 25% tick at 10:30Z (US 11 native organizations, EU 5; completed results seen so far 1,204 candidates, 10 requested, zero candidate/transcription failures) did not pass. At 10:32:58Z EU BullMQ emitted a terminal qa-sampling-recovery failure for Qonto (organization 33), job qa_sampling_recovery_33_1988585 (the 10:15 bucket): job stalled more than allowable limit, attemptsMade 1 of maxAttempts 5, terminalFailure: true, correlating with the EU worker replacements during the 1d29cccd rollout. At 10:36:16Z the coordinator rolled PostHog flag version 5 back to four exact 100% canaries only (kuda-pilot, rho-tnkw, easypay, nala); Qonto and the 25% cohort are back on Rails. Re-enabling Qonto or any percentage rollout is gated on a durability review of the stalled bucket (partial persisted admissions, audit entries, requests, and Redis throttle consumption) and natural scheduler-cycle resumption evidence after the hardening change is deployed; the automation is not authorized to replay customer jobs, so no manual replay of the stalled bucket is part of the gate; see the incident section of the runbook.

Evidence as of 2026-09-18T13:12:11Z (epoch 1789737131)

Historical: superseded by the 14:06:18Z section below where the two disagree.

This section supersedes the 10:16Z windows above wherever the two disagree. Deployment and QA-cycle facts were verified at 13:04:35Z; the effects and gateway windows below were re-measured with cutoff 13:12:11Z.

Deployment state: release-prod is b96da600faff9d1a29051252b8a4c08250047675 (merged 12:34:22Z; Flightcontrol succeeded 12:40:56Z; new task definitions and images registered 12:37Z to 12:38Z). Running tasks: US web fc-rulebase-api2-prod-5whj12su task definition 1634, 1/1; EU web fc-rulebase-api2-prod-eu-9rfp06k task definition 1652, 2/2; US worker fc-rulebase-api2-worker-pr-zy7q0 task definition 1624, 5/5; EU worker fc-rulebase-api2-worker-pr-m9780 task definition 1641, 5/5. The worker names contain -pr- because of Flightcontrol’s physical naming; they are the production logical services rulebase-api2-worker-prod and rulebase-api2-worker-prod-eu, not preview workers. Both web regions acquired the migration advisory lock and completed startup with no migration failure and no package-specific first-slice error. Pre-existing provider/Intercom warnings continued, deployment churn briefly produced Redis connection/stream errors, and one still-Rails-owned conversation-upload process (conversation-upload-process-648660) returned 504 at 13:02Z.

QA sampling recovery: qa_sampling_recovery_native remains exactly kuda-pilot, rho-tnkw, easypay, nala after the 10:36Z rollback; Qonto and percentage targeting remain off. Every four-canary cycle completed after the b96da600 deployment was clean with zero candidate or transcription failures: US 12:45 Rho 13 candidates / 9 requested / 4 skipped and Kuda 1,903 / 0 / 100 skipped / 1,802 throttled / 1 ignored / 73 counters lowered; US 13:00 Rho 4 / 1 / 3 and Kuda 1,994 / 53 / 99 skipped / 1,841 throttled / 1 ignored / 73 counters lowered; EU Easypay 155 / 1 / 30 skipped / 124 throttled / 15 counters lowered, then 161 / 2 / 35 / 124 / 15; NALA cycles had zero candidates. Qonto is back on Rails and completed its 5,000-candidate cycles there. Do not widen the flag before #12482  (deployment- interruption-safe sweeps, ENG-3249 ) is reviewed and deployed and the normal scheduler’s natural post-deploy cycles show clean resumption; production proof comes from natural traffic only, with no manual customer-job replay.

Other native flags (decision: no widening): conversation_upload_intake_native is Kuda-only with no native traffic and PostHog last_called_at null. conversation_transcription_state_native is Kuda-only: 23 natural Kuda context loads in the inspected slice; the 10 after b96da600 were all terminal already_transcribed, plus one completed conversation; still no native persistence-write proof. call_recording_attachment_native is Kuda-only with no native attachment events, while Rails telemetry showed 14 US provider_sync and 189 EU aircall_import attachment calls.

Effects boundary, fixed trailing 24 hours ending 13:12:11Z: 37,749 API2-to-Rails calls (US 12,884; EU 24,865). The already-cut-over attachment purge, generic tools, and image-caption families are zero in this window.

Effects boundary, fixed trailing seven days ending 13:12:11Z: 412,091 calls (US 157,345; EU 254,746). Shipped-family pre-cutover residue is 195,626 (attachment purge 113,411; tools 45,438; image captions 36,777). Leading remainder: QA sampling recovery 154,276; call recording 15,679; QA stage 13,533; transcription context/write 8,793; reconciliation 7,476; upload processing 6,833; provider-sync completion 6,441.

Gateway boundary, fixed trailing 24 hours ending 13:12:11Z: 16,709 events (US 14,035; EU 2,674): 531 API2-blocked, 12,253 forwarded Rails 404/scanner, 3,925 non-404 as an upper bound. Meaningful live dependencies: Action Cable 1,164 (US last 09:34Z; EU still live through 13:06Z), conversation uploads 1,016, Outreach 224. Older residue: Zendesk 727 (last 2026-09-17T23:45Z), Unit21 338 (last 2026-09-17T21:59Z), public media 427 (last 2026-09-17T14:46Z), and 29 miscellaneous non-404 events.

Gateway coverage caveat: structured gateway telemetry is only about 33 hours old, so the nominal seven-day query is exact over the available coverage and is not a full seven days: 25,809 events; 715 blocked; 15,682 Rails 404; 9,412 non-404. Known families: Cable 2,559; uploads 1,540; Zendesk 914; Unit21 529; Outreach 224; media 3,599; 47 other.

The current live public-route dependencies are still Action Cable, conversation uploads, and Outreach. #12474  owns Outreach ingress plus the explicit allowlist, #12483  owns native XCALLY/Outreach-Kaia direct upload processing, and #12482  owns QA sweep interruption safety; all three are green and await Ronald/human review.

Evidence as of 2026-09-18T14:06:18Z (residual windows) and 14:09:47Z (flag state)

Historical: superseded by the 15:05:47Z section below where the two disagree.

This section supersedes the 13:12:11Z windows above wherever the two disagree.

QA interruption safety deployed: #12482  (ENG-3249 ) was human-merged and is deployed in the current cmu70 API2 worker build: US worker task definition 1628 and EU worker task definition 1645, both steady 5/5 (physical service names still carry -pr-; they are the production logical workers).

Clean natural 14:00Z cycles on the hardened build (structured worker results; every native cycle had zero candidate and zero transcription failures): US org 48 0 candidates; US org 38 1 candidate / 1 requested / 0 skipped; US org 19 4 / 1 / 3; Kuda US org 25 2,406 candidates / 4 requested / 116 skipped / 2,286 throttled / 75 counters lowered; EU org 25 0 candidates; EU org 41 178 candidates / 0 requested / 51 skipped / 127 throttled / 15 counters lowered.

Flag state: at 14:09:47Z qa_sampling_recovery_native was expanded to exact slugs kuda-pilot, rho-tnkw, easypay, nala, qonto only. Percentage targeting stays off; there is no manual replay of the stalled qa_sampling_recovery_33_1988585 bucket, and none is authorized. Rollback remains removing qonto from the exact list.

Qonto’s first natural hardened cycle (clean): at 14:22:23.484Z job qa_sampling_recovery_33_1988601 ran with owner=api2: 5,000 candidates, 9 requested, 130 skipped, 4,860 throttled, 1 ignored, 244 counters lowered, loadMs 2,641, sweepMs 440,679, 0 candidate failures, 0 transcription failures, stoppedEarly null. This is the natural scheduler evidence the rollback gate required. The five exact slugs remain enabled; percentage targeting stays off.

Effects boundary, fixed trailing 24 hours ending 14:06:18Z: 36,952 API2-to-Rails calls (US 12,198; EU 24,754), down 797 from the 13:12:11Z window.

Effects boundary, fixed trailing seven days ending 14:06:18Z: 409,658 calls, of which 193,829 is shipped-family pre-cutover residue (attachment purge 112,671; tools 44,812; image-caption context/write 36,346).

Gateway boundary, fixed trailing 24 hours ending 14:06:18Z: 15,687 events: 530 API2-blocked, 11,537 forwarded Rails 404/scanner, 3,620 non-404 as an upper bound. Meaningful live dependencies: Action Cable 1,077, conversation uploads 1,049, Outreach 221.

Gateway coverage caveat: the nominal seven-day query covers only the structured gateway telemetry available since 2026-09-17 and is not a full seven days: 26,004 events; 734 blocked; 15,690 Rails 404; 9,580 non-404.

Evidence as of 2026-09-18T15:05:47Z (effects census) and 15:09:20Z (flag state)

Historical: superseded by the 16:06:07Z section below where the two disagree.

This section supersedes the 14:06:18Z windows above wherever the two disagree. Gateway windows were not re-measured; the 14:06:18Z gateway figures stand.

Deployment state: release-prod 677ba560 is steady with zero pending and all deployments complete. US web task definition 1638 (2/2), EU web 1656 (1/1); the logical production workers (physical -pr- names) run task definitions 1628 (US) and 1645 (EU), both 5/5. All four task definitions run commit 677ba560.

Repeated natural exact-canary cycles after the first Qonto proof stayed clean (candidates / requested / skipped / throttled / ignored / counters lowered; zero candidate and transcription failures in every cycle):

Flag state: at 15:09:20.750879Z, after those repeated natural cycles, qa_sampling_recovery_native v8 kept the five exact slugs (kuda-pilot, rho-tnkw, easypay, nala, qonto) at 100% and added a separate 5% organization-group rollout (server-only, active). This is the only canary change. The next natural cycles and API2 errors must be verified before any further widening; any regression is an immediate rollback to the five exact slugs. No manual customer-job replay.

Other native flags (still exact Kuda-only, decision: no widening): post-cmu70 through about 15:10Z, conversation_upload_intake_native saw zero native upload events, conversation_transcription_state_native saw zero API2-owned transcription state transitions, and call_recording_attachment_native saw zero native call-recording attachments; all call-recording traffic stayed Rails-owned (US 11 provider_sync, EU 197 aircall_import). No API2 level>=50 errors from 14:25Z through about 15:10Z.

Effects boundary, fixed trailing 24 hours ending 15:05:47Z: 35,204 API2-to-Rails calls (US 11,513; EU 23,691), down 1,748 from the 14:06:18Z window.

Effects boundary, fixed trailing seven days ending 15:05:47Z: 406,808 calls (US 154,855; EU 251,953), of which 192,210 is shipped-family pre-cutover residue (attachment purge 111,889; tools 44,606; image captions 35,715).

Evidence as of 2026-09-18T16:06:07Z (effects census) and 17:09Z (deployment, flag state and cycles)

This section supersedes the 15:05:47Z windows above wherever the two disagree, and the 14:06:18Z gateway figures for the public track.

Gateway boundary, fixed trailing 24 hours ending 16:06:07Z (public track; package-level detail in #12512 ): 13,930 events (US 11,693; EU 2,237): 528 API2-blocked, 10,529 forwarded/rejected Rails 404 scanner/stale, 2,871 non-404/Action Cable upper bound, 2 aborts. Meaningful live dependencies: conversation uploads 989, Action Cable 825, Outreach rolling residue 210 with zero forwards after 15:28Z; older residue Zendesk about 582, Unit21 239. Three signed Outreach deliveries at 15:43Z returned malformed_event 500 — recorded as parity/error evidence, not as accepted natural deliveries; no accepted natural Outreach delivery yet, so Outreach ownership holds.

Gateway coverage caveat: the nominal seven-day query still covers only the structured telemetry available since 2026-09-17 (partial coverage): 27,332 events; 780 blocked; 16,650 Rails 404; 9,898 non-404/Cable; 4 aborts.

Direct upload processing: #12483  is deployed in 623cf7a7; api2_conversation_upload_native_processing is confirmed absent in PostHog, so the path is fail-closed (off) and every direct upload still calls Rails; no package-specific first-slice error. Flag creation and the first canary are held pending Ronald’s review and a natural-traffic plan.

Deployment state: release-prod c47e1113 (#12520, containing only #12517 on top of 623cf7a7) is live; Flightcontrol reports #12520 Success in US and EU, API2 web 1/1 and the logical API2 workers (physical names fc-rulebase-api2-worker-pr-zy7q0 and fc-rulebase-api2-worker-pr-m9780, not preview services) 5/5 in both regions. The EU worker deployment took 9m54s and is service revision rev:1651. Runtime logs at 17:09Z show active natural Qonto (EU org 33) processing across multiple worker hosts, including recovery-33 native effects applied and provider sync completion, with no error/fatal rows in the visible first slice.

Verification boundary for c47e1113 (state precisely, do not assume): PostHog has zero application logs and the AWS SSO session expired, so for this revision the exact task-definition commit SHAs, migration startup, the full first-slice level>=50 error query, and a fresh residual recount after the 16:06:07Z cutoff are unverified. The fixed 24 h / 7 d census ending 16:06:07Z below remains the current measured baseline until a new exact count is taken.

Preceding revisions: 623cf7a7 (#12516) completed in EU at 16:35:56Z and US at 16:36:16Z with zero pending, zero failed tasks, every image and FC_GIT_COMMIT_SHA matching, both web tasks acquiring the migration advisory lock with no migration failure, web TD 1643/1661 and workers TD 1633/1650; before it cf0374d (completed 16:10Z; web TD 1641/1659, workers TD 1631/1648) and c7d66aaa, the running revision during the 15:36Z Qonto stop below.

Organization attribution correction (read-only production DB mapping): EU org 41 is nala, an exact canary, not Spendesk. EU Spendesk is org 39 (slug spendesk) and stayed Rails-owned. The four EU org-41 cycles below are therefore Nala exact-canary cycles, not 5% group cycles; earlier text attributing them to Spendesk is superseded, and earlier sections’ references to “EU org 41” (including rows labelled Easypay (org 41)) describe the same Nala organization.

Nala (EU org 41) natural cycles were clean (candidates / requested / skipped / throttled / counters lowered; zero candidate failures, zero transcription failures, zero unprocessed): 15:16Z 278 / 0 / 147 / 131 / 15; 15:30Z 166 / 0 / 33 / 133 / 15; 15:45Z 178 / 0 / 45 / 133 / 15; 16:00Z 294 / 0 / 52 / 242 / 15.

Qonto (org 33) stopped safely during the c7d66aaa rollout: the 15:36Z cycle ended with 911 unprocessed and zero failures (the #12482 cooperative stop), then the normal scheduler resumed with clean 5,000-candidate cycles at 15:51Z (15 requested / 90 skipped / 4,895 throttled) and 16:07Z (8 / 68 / 4,924), zero failures and zero unprocessed. No manual replay. This is the first production observation of the deployment-interruption path behaving as designed.

Kuda (US org 25) 16:07Z: 5,000 candidates / 6 requested / 117 skipped / 4,876 throttled / 1 ignored / 75 counters lowered, zero failures.

Flag state: at 16:11:54Z qa_sampling_recovery_native v9 advanced the organization group from 5% to 10%, preserving the exact slugs kuda-pilot, rho-tnkw, easypay, nala, qonto and server-only group evaluation. At 16:12:41Z conversation_transcription_state_native v2 added qonto as a second exact canary next to kuda-pilot, because Kuda produced no native persistence writes while Qonto had 18 Rails loaded and 18 Rails transcribed calls since 15:09Z. Hold further widening of both flags until the natural outcomes of these changes are verified; any regression is an immediate rollback to the previous flag version. No manual customer-job replay.

First native transcription-state persistence proof (Qonto, EU org 33): after v2, one native job completed end-to-end: 16:23:45Z owner api2, phase context, outcome loaded for call detail conversation_part_call_detail_mxPBJ8kOGrmtJ8kQbljy9Z4a; 16:24:16Z phase persist, outcome transcribed. Read-only row reconciliation: org_id 33, transcribed_at 16:24:16.644Z, language eng, duration_ms 151,340, attempt_count 0, failed_at nil, 7 transcript-message rows. CreditUsage count is 0 because Qonto has zero active transcription credits, which matches both native and Rails semantics (both skip usage when there is no active credit). No native-path error. Hold at Qonto + Kuda only: the package runbook requires a complete 24-hour distribution/reconciliation window before further expansion.

First natural 10% cycle (16:15Z) was clean. The 10% group newly selected only zero-candidate API2 organizations: US 26 intercom-demo, 40 synctera-demo, 44 novo-quick-test-l5h6, 45 vestwell-demo; EU 26 intercom-demo. Exact canaries: US Rho 8 candidates (3 requested / 5 skipped), Easypay 2 skipped; EU Nala 178 (45 skipped / 133 throttled / 15 counters lowered). All cycles had zero candidate/transcription failures, zero unprocessed, stoppedEarly null. Hold at 10%: the new percentage cohort contributed zero candidates, so it is not yet evidence for widening.

16:30Z cycle under v9 and the 623cf7a7 rollout. US was clean: Kuda 3,055 candidates / 0 requested / 113 skipped / 2,941 throttled / 1 ignored / 78 counters lowered; Rho 9 / 6 / 3; Easypay 3 skipped; percentage-selected organizations again had zero candidates. EU Nala 170 / 0 / 36 / 134 / 15 was clean. Qonto (EU org 33) was interrupted by the rollout and stopped safely: 5,000 candidates, 12 requested, 108 skipped, 2,956 throttled, 245 counters lowered, 1,924 unprocessed, stoppedEarly shutdown, zero candidate/transcription failures. This is the second production cooperative stop. Hold QA at 10%; the next natural Qonto scheduler cycle must resume cleanly (no manual replay).

Flag state at 17:09Z (no widening): qa_sampling_recovery_native is still v9 — exact kuda-pilot, rho-tnkw, easypay, nala, qonto at 100% plus the 10% organization group; PostHog last called 16:45:01Z (advisory). The 17:09Z worker logs show Qonto processing has resumed naturally under c47e1113, but a clean resumed Qonto QA-cycle summary (structured worker result with zero failures and zero unprocessed) and a non-zero percentage-cohort outcome are both still missing, so the group stays at 10%.

Effects boundary, fixed trailing 24 hours ending 16:06:07Z: 33,475 API2-to-Rails calls (US 10,705; EU 22,770), down 1,729 from the 15:05:47Z window.

Effects boundary, fixed trailing seven days ending 16:06:07Z: 403,815 calls (US 153,570; EU 250,245), of which 190,537 is shipped-family pre-cutover residue (attachment purge 111,057; tools 44,248; image captions 35,232), leaving 213,278.

Read-only verification attempt at 2026-09-18T19:12Z (access boundary; no new census)

This attempt refreshed only what read-only PostHog (production project 63353) and Sentry could answer. It does not supersede the 16:06:07Z effects/gateway census or the 17:09Z deployment observation; those remain the current measured baselines.

Access boundary (exact): no AWS/Flightcontrol credentials were available (aws sts get-caller-identity → Unable to locate credentials), and PostHog logs-count over the trailing 24 h and 7 d returned 0. Therefore the following remain unverified after 17:09Z: the task-definition commit SHAs and desired/running counts in both regions (last verified c47e1113, web 1/1, logical -pr- production workers 5/5, EU worker rev:1651), migration startup, the first-slice level>=50 error query, every natural qa_sampling_recovery_native cycle summary after 17:09Z (including the resumed Qonto cycle and any percentage-cohort candidate outcome), and any effects residual recount after 16:06:07Z. Post-17:09Z gateway traffic is likewise unverified; the public runbook update in #12512  (head 71063f926) records the same boundary and the 15:28Z→cutoff split (27 US upload forwards, 11 EU Cable forwards, 0 Outreach forwards; the partial 7 d 27,332 is US 21,327 / EU 6,005). Outreach remains held: the three signed US deliveries at 15:43Z returned malformed_event 500 with Rails parity and no accepted delivery exists, none after 16:10Z.

Exact flag definitions (PostHog production, read at ~19:05Z):

No flag was changed during this verification.

Advisory PostHog event signals (not outcome reconciliation): PostHog holds no QA cycle summaries or package outcome events, so these do not satisfy the QA gates below. $feature_flag_called rows for qa_sampling_recovery_native from 17:30:00Z to 19:00:00Z show the scheduler still evaluating the flag each 15-minute bucket: true for the five exact slugs and for the percentage-selected zero-candidate organizations novo-quick-test-l5h6, synctera-demo, vestwell-demo and intercom-demo (17:30Z, 18:30Z, 18:45Z); qonto evaluated true at 17:45:00.386Z. These prove flag evaluation only, not that the cycles completed cleanly. call.transcribed events split cleanly by emitting library: API2 (posthog-node) emitted 26 for org:qonto from 16:24:16.788Z (the documented first native persist) through 19:13:04.971Z, 14 of them after 17:20Z, while Rails (posthog-ruby) last emitted a Qonto call.transcribed at 15:55:02.747Z — none after qonto was added to the transcription-state flag at 16:12:41Z. rho-tnkw (not a transcription-state canary) stayed Rails-emitted (24 events, last 19:14:16Z). No call.transcription_failed event exists in the project. Kuda produced no API2-emitted call.transcribed; conversation_transcription_state_native evaluated true three times for kuda-pilot/qonto and false once for rho-tnkw (18:08:49Z); call_recording_attachment_native evaluated false once for rho-tnkw (17:48:38Z, the advisory timestamp above) and conversation_upload_intake_native was never evaluated. Row-level reconciliation of these Qonto native persists (transcript rows, credit usage, attempt counts) remains unverified without database access.

Sentry (API2 and Rails, unresolved, seen after 17:20Z): no new API2 issue matches the QA sampling, QA stage, transcription, upload or call-recording paths. One Rails issue is relevant to upload intake: RULEBASE-API-5WT  Api2ConversationUploadEnqueuer::AmbiguousDeliveryError: Net::ReadTimeout in ConversationUploadsController#create (Rails → API2 enqueue), 2 occurrences (09:52:53Z, 17:55:40Z), 0 users. This is the still-Rails-owned upload path calling API2, not a native-intake regression; track it under the upload-intake package.

Triage decision (internal track, ENG-3218  full decision and query; ENG-3248  summary): RULEBASE-API-5WT does not block the Kuda intake or direct-processing canaries and needs no code PR. It is the existing Rails → API2 enqueue timeout on the still-Rails-owned ingress: both occurrences were 3-second read timeouts, the client still received 201, and the durable path recorded an uncertain ledger row for the API2 reconciler under the deterministic job id. The first event predates c47e1113. Native intake removes this hop entirely. Residual verification gap: without production DB access, conversation-upload/process/649026 could not be confirmed to have left the uncertain state; if it remains owner=rails, upgrade this to a blocker. Other unresolved API2 items (RULEBASE-API2-82 dashboard-widget statistics query, RULEBASE-API2-DJ provider-sync N+1 HTTP, RULEBASE-API2-F1 Redis subscriber-mode on channel stream) predate or are unrelated to the QA track.

Decisions (19:12Z): hold qa_sampling_recovery_native at v9 — a clean resumed Qonto structured cycle summary and a non-zero percentage-cohort outcome are still unverified; hold conversation_transcription_state_native at Qonto + Kuda pending the 24-hour reconciliation window and row-level verification; conversation_upload_intake_native and call_recording_attachment_native stay Kuda-only with no native traffic observed; direct upload processing stays fail-closed pending Ronald’s review and a bounded single-organization Kuda canary (no PR, no flag change). No job was replayed, no production data was written, no flag was changed, nothing was deployed or merged.

Read-only verification attempt at 2026-09-18T20:59Z (new release, still no structured QA outcomes)

Deployment (GitHub deployment status only): release-prod is now e45afa48c1c887549fa2b374da8a7c6002a9fbff (#12521, from main d198f5f7), superseding c47e1113. Flightcontrol’s GitHub deployment status reports success in EU at 20:51:10Z and US at 20:51:38Z. AWS SSO remains expired, so task-definition commit SHAs, desired/running counts, migration startup logs and the first-slice error query for e45afa48 are unverified; the 16:06:07Z effects/gateway census and the 17:09Z c47e1113 worker observation remain the last measured baselines.

PostHog flags (project 63353), definitions unchanged, advisory last_called_at only: qa_sampling_recovery_native v9 (five exact slugs + 10% organization group) 20:15:01Z; conversation_transcription_state_native v2 Kuda + Qonto 18:28:50Z; conversation_upload_intake_native v1 Kuda never called; call_recording_attachment_native v1 Kuda 17:48:38Z; api2_conversation_upload_native_processing still absent (fail-closed).

Structured QA outcome evidence (independent read-only attempt): PostHog carries no qa_sampling_recovery_native cycle-summary event of any kind since 16:00Z (the only QA-adjacent event is four conversation.agent_evaluated), so the 20:00Z/20:15Z/20:30Z/20:45Z/21:00Z natural cycles cannot be reconciled from PostHog. $feature_flag_called rows 19:00Z-21:00Z show the scheduler evaluating true for kuda-pilot, rho-tnkw, easypay, nala, qonto and for the same zero-candidate percentage cohort seen at 16:15Z (intercom-demo, novo-quick-test-l5h6, synctera-demo, vestwell-demo); no new organization entered the 10% cohort. This is flag evaluation, not candidate/requested/failure counts. Both widening gates therefore remain unmet: no structured clean resumed-Qonto summary and no non-zero percentage-cohort outcome. No widening was recommended or made.

Advisory transcription signal: call.transcribed since 16:00Z is 38 posthog-node (API2) events for org:qonto through 20:43:08Z and 29 posthog-ruby (Rails) events for org:rho-tnkw through 20:52:30Z; the Qonto events do not carry the flag property, so they are advisory only and the DB row-reconciliation gate for conversation_transcription_state_native stays.

Sentry, e45afa48 first slice: RULEBASE-API2-F1  (Action Cable/SSE Redis subscriber-mode error) recurred at 20:52:22Z, immediately after the EU/US deployment success timestamps; the public-track session owns its triage. No QA sampling, QA stage, transcription, upload or call-recording issue appeared.

Decisions (20:59Z): all 19:12Z holds stand; no flag change, replay, deploy or merge.

Authoritative CloudWatch cycle evidence, 21:00Z natural cycle (post-e45afa48)

CloudWatch worker logs (authoritative, structured cycle summaries) show all five exact-slug QA canaries running owner=api2 at the 21:00Z natural cycle, the first after the e45afa48 task replacement, with zero candidate/transcription failures and unprocessed=0:

Qonto’s first attempt, begun before the task replacement, stopped cooperatively at 20:50Z with unprocessed=948 and stoppedEarly=shutdown; the post-deploy retry completed cleanly at 21:07Z. This is deployment interruption followed by natural recovery, not a canary failure, and it is the structured clean resumed-Qonto evidence the first widening gate asked for. Easypay’s zero-candidate cycle is API2-evaluated but adds no workload proof; its earlier non-zero cycles remain the useful evidence. Org IDs here follow the 21:18Z read-only production Rails query Organization.where(id: [19, 38]).pluck(:id, :slug) → [[19, "rho-tnkw"], [38, "easypay"]].

The completed Qonto summary at 21:07:08Z also carried ignored=1 and stoppedEarly=null.

Aborted flag advance (21:11:25Z → 21:16:22Z): the operator advanced qa_sampling_recovery_native (PostHog id 280844) from 10% to 25% (v10) at 21:11:25Z, citing a clean non-zero 20:15:06Z cycle for US org 38 (3 candidates / 1 requested / 2 skipped / 0 failures) as random-cohort proof. A read-only production Rails query at 21:18Z (Organization.where(id: [19, 38]).pluck(:id, :slug) → [[19, "rho-tnkw"], [38, "easypay"]], no data write) confirmed org 38 is Easypay and org 19 is Rho — both exact canaries — so that cycle was exact-canary evidence and the widening gate was not met. The flag was rolled back to v11 — five exact slugs at 100% + 10% group at 2026-09-18T21:16:22.308984Z (PostHog read 21:17Z). The 25% exposure lasted 4 m 57 s, covering only the 21:15Z cycle. US organizations selected around that interval: 16 rho-demo-2, 21 baserule, 22 frame-finance, 25 kuda-pilot, 26 intercom-demo, 38 easypay, 40 synctera-demo, 44 novo-quick-test-l5h6, 45 vestwell-demo, 48 nala, 50 alioune-sow. Every non-exact completion visible at the check had zero candidates and zero failures; no regression was observed. Record this as an aborted advance caused by incorrect regional-ID attribution, not an outcome regression. Current state: v11 / 10%. No further widening until a meaningful non-zero random-cohort cycle proves outcomes. No replay, deploy or merge.

Exact residual census at 2026-09-18T21:05:00Z / 21:06:56Z

Two fixed censuses were taken minutes apart; both are exact over their own window and are recorded as reported.

Cutoff 21:05:00Z: Rails /internal effects 24,963 / 24 h (US 7,225; EU 17,738) and 390,842 / 7 d (US 148,376; EU 242,466). QA sampling alone: 14,745 / 24 h (US 1,555; EU 13,190) and 144,877 / 7 d (US 35,620; EU 109,257).

Cutoff 21:06:56Z: 24,905 / 24 h (US 7,223; EU 17,682), down 8,570 (25.6%) from the 16:06:07Z 33,475. Families: QA sampling 14,694 (down 7,115 / 32.6%); transcription reconciliation 4,675; call recording 2,462; QA stage 1,050; direct upload processing 984; provider-sync completion 277; transcription context + write 422; Unit21 144. Fixed 7 d: 390,754 (US 148,373; EU 242,381). Families with zero current-24 h calls (pre-cutover residue only) total 184,129 — attachment purge 107,676; tools 42,646; image-caption context + update 33,755; small tail 52 — leaving 206,625 meaningful live/recent calls.

e45afa48 is live in both regions; public Action Cable has a separate blocker tracked in #12526 . Full Rails shutdown remains blocked.

Release boundary 06236881 and the 21:15Z partial cycle

origin/release-prod advanced to 06236881964c45ac32c9c8bc079ddcd484959182 (merge #12525; only the devalue 5.9.2 CVE bump since e45afa48). All four API2 services are COMPLETED on that SHA: web US/EU 1/1, logical production workers (physical fc-rulebase-api2-worker-pr-zy7q0 and -m9780) 5/5.

First post-deploy QA cycle at 21:15Z (snapshot taken before the heavy jobs finished): Rho org 19 completed 5 candidates / 3 requested / 2 skipped with zero candidate/transcription failures and unprocessed=0; Easypay org 38 and every percentage-cohort evaluation seen so far were zero-candidate / zero-failure; Kuda and Qonto had not yet finished. The percentage cohort therefore still has no meaningful non-zero workload proof. This was the only cycle under the aborted v10 25% window; the flag is v11 (five exact slugs + 10%) since 21:16:22Z and holds there until a clean natural percentage-cohort cycle with real candidates.

2026-09-19 evidence: eae24156 healthy, 17:47:25Z census, transcription canary past its 24 h gate

Runtime. Through the 17:47:25Z census the API2 task SHA was eae24156f80b5dc372a333e340a25f975ac01291 in both regions: web US/EU 1/1, logical production workers (physical fc-rulebase-api2-worker-pr-zy7q0 / -m9780) 5/5, pending 0, COMPLETED; migration advisory locks acquired EU 13:59:03Z, US 14:00:24Z. Intermediate release-prod UI/docs commits did not change the API2 task SHA. release-prod 2d02ce188799b8488bb2bfeb9ab26007867f2efc (#12541, human-merged) is now live and COMPLETED in both regions: US web TD 1650 1/1, US worker TD 1640 5/5; EU web TD 1668 1/1, EU worker TD 1657 5/5; all tasks on 2d02ce18, pending 0, failedTasks 0; migration locks acquired EU 17:58:13.243Z, US 17:59:21.806Z. No QA or transcription package error in the first slice. release-prod 279ae48c then completed in both regions: US web TD 1651 1/1, worker TD 1641 5/5; EU web TD 1669 1/1, worker TD 1658 5/5; all tasks on the exact commit, pending/failed 0, migration locks acquired and listeners up. All holds below, including the Rho transcription expansion, are unchanged by these deploys. Later: the API2 runtime commit on all four services is 4d5b1044; release-prod 610aebdb is docs-only and did not change the runtime. release-prod 5e3c514 is then fully deployed: US web TD 1653 1/1, US logical worker (physical -pr-zy7q0) TD 1643 5/5, EU web TD 1671 1/1, EU logical worker (physical -pr-m9780) TD 1660 5/5, all completed, zero pending; no worker level>=50 or package errors in the inspected 20-minute slice, and none through 19:10Z. Rails remains desired/running 3/3 in both regions — do not scale Rails down while the QA ladder is back at 10% and canaries are still being widened.

Fixed residual census ending 2026-09-19T17:47:25Z. Rails /internal effects 6,997 / 24 h (US 3,950; EU 3,047), down 17,908 (71.9%) from the 2026-09-18T21:06:56Z 24,905. Families: transcription reconciliation 3,921; call recording 1,834; direct upload 826; Unit21 144; counter cache 129; transcription context/write 64; expiry 48; QA stage 20; tiny tail 11; QA sampling and provider completion both zero. Seven-day effects 344,118 (US 136,068; EU 208,050), of which 288,199 is major zero-current-24 h residue (attachment purge 93,754; tools 37,778; captions 27,651; QA sampling 123,685; provider completion 5,331), leaving a 55,919 live/recent upper bound. Refreshed at cutoff 2026-09-19T18:09:40Z: effects 6,917 / 24 h (US 3,917; EU 3,000), down 72.2% from 24,905, led by transcription reconciliation 3,930, call-recording attachment 1,779 and direct upload 801; seven-day 343,395, of which 287,545 is pre-cutover / now-zero residue, leaving a 55,850 meaningful-live upper bound. Refreshed again at cutoff 2026-09-19T18:44:39Z: effects 6,860 / 24 h (US 3,938; EU 2,922), down 57 from 6,917. Combined methods: transcription reconciliation 3,912; call-recording create 1,708; direct upload process 762

• retry 1 + mark_failed 1; Unit21 144; QA execute_stage 95; counter cache 128; transcription context 25 + write 25; expiry 48; field values 4; report actions 6; Slack 1. Fixed seven-day 341,577 (US 135,048; EU 206,529); major now-zero / pre-cutover residue 285,878 (attachment purge 93,129; QA sampling 122,885; tools 37,263; captions 27,331; provider completion 5,270), leaving a 55,699 live/recent upper bound before tiny now-zero tails (55,657 including them).

Refreshed at cutoff 2026-09-19T19:11:23Z: effects 6,755 / 24 h (US 3,898; EU 2,857), down 105 — reconciliation 3,912 (empty heartbeats); recording 1,639 (all pre-cutover); direct upload 734; Unit21 144; QA stage 95; counter cache 128; expiry 48; transcription context/write 44; tiny tail 11. Seven-day 340,652 (US 134,791; EU 205,861), dominated by pre-cutover/now-zero residue. Rails stays 3/3 because uploads and Cable persist.

Refreshed at cutoff 2026-09-19T19:24:32Z: effects 6,718 / 24 h (US 3,898; EU 2,820) — reconciliation 3,906 (empty heartbeats); call recordings 1,603; direct uploads 741 (process + retry + mark_failed); Unit21 144; counters 128; QA stage 95; transcription context 21 + write 21; expiry 48; fields 4; reports 6; Slack 1. Seven-day 340,305; major now-zero/pre-cutover residue 284,645 (QA sampling 122,074; purge 93,129; tools 37,126; captions 27,074; provider completion 5,242), leaving 55,660 live/recent before tiny now-zero tails. At this cutoff PostHog read qa_sampling_recovery_native v16 (five exact + 50%, 19:22:47Z) and call_recording_attachment_native v4 (three exact + 10%, 19:23:51Z); because these were concurrent changes, no further widening of either flag until the first complete post-v16 19:30Z QA cycle and the first non-exact percentage recording outcome are both clean — document rollback on any regression. Runtime remains 5e3c514 in both regions; release-prod 8ec16 (ENG-3270) deployment has just started.

Refreshed at cutoff 2026-09-19T19:54:08Z: effects 6,610 / 24 h (US 3,871; EU 2,739), down 108 from 6,718 — reconciliation 3,882 (empty heartbeats); call recording 1,525; direct uploads 741; Unit21 144; counters 128; QA stage 95 (including one 500); transcription context 18 + write 18; expiry 48; tiny tail 10. Seven-day 338,760; major now-zero/pre-cutover residue 283,068 (QA sampling 121,370; purge 92,507; tools 37,074; captions 26,875; provider completion 5,242), leaving 55,692 live/recent before tiny tails. Release 8ec16b8d remains exact/COMPLETED on all four services (US web TD 1654 1/1, worker TD 1644 5/5; EU web TD 1672 1/1, worker TD 1661 5/5; zero pending/failed). call_recording_attachment_native remains v4 (exact Kuda + Qonto + Rho + 10%): every visible outcome since v4 is attached (Qonto through 19:51Z, Rho at 19:31Z) but there is still no non-exact percentage outcome — do not widen.

Refreshed at cutoff 2026-09-19T20:58:21Z: effects 6,262 / 24 h, down 456 (6.8%) from 6,718 — reconciliation 3,741 (empty heartbeats); call recording creates 1,347; direct uploads 741; Unit21 144; counters 128; QA stage 82; transcription context/write 22; expiry 48. Seven-day 336,636; major now-zero/pre-cutover residue 280,965, leaving 55,671 live/recent upper bound. qa_sampling_recovery_native has been at 100% (v21) since 20:08:04.328711Z; post-rollout native cycles US 108, EU 51. Qonto job ..._33_1988722 stopped only for shutdown during a deploy (5,000 candidates, 1,372 unprocessed, zero candidate/transcription failures) and the next natural cycle ..._33_1988723 completed at 20:52:08Z with 5,000 candidates / 4 requested / 732 skipped / 4,264 throttled, zero failures, zero unprocessed, no early stop. Decision: keep QA at 100%; the deploy shutdown is a known interruption class (see the 19:30Z bucket), not a regression, and the natural retry closed it without replay. The reversible zero-window clock for Internal::QASamplingRecoveriesController and Internal::ConversationTranscriptionReconciliationsController continues to run from 20:08:04.328711Z; retirement of those controllers is still a later, explicit decision.

Refreshed at cutoff 2026-09-19T21:10:29Z: Rails /internal effects 6,202 / 24 h (US 3,697; EU 2,505) and 336,317 / 7 d (US 133,669; EU 202,648). 24 h: reconciliation 3,700; call recording 1,332; direct upload 741; Unit21 144; counters 128; QA execute_stage 78; expiry 48; transcription context 22 + persist 22; tiny tail 11. Last-seen timestamps show the large families are now historical: reconciliation 20:00:03Z (before the QA v21 100% write at 20:08Z); call recording 18:41:47Z; direct upload 19:18:11Z; execute_stage 18:20:59Z; Rails transcription context/write 17:10:30Z. The 21:00Z natural all-organization QA cycle was clean: Kuda 1,706 candidates / 0 requested / 77 skipped / 1,627 throttled / 2 ignored; Qonto 5,000 / 11 / 97 / 4,892; Nala 237 / 21 / 55 / 161; Rho 4 / 1 / 3; zero candidate/transcription failures, zero unprocessed, no early stops. The 21:30Z all-organization cycle was also clean: Rho 3 / 3; Kuda 1,728 / 5 / 191 / 1,530 / 2 ignored; Nala 223 / 7 / 54 / 162; Qonto 5,000 / 6 / 85 / 4,909; zero failures, zero unprocessed. Later effects cutoff: 6,079 / 24 h and 334,979 / 7 d. Keep QA at 100%; the 24 h Rails-zero clock for the reconciliation controller runs from its last call at 20:00:03Z.

Public gateway at cutoff 21:10:29Z (24 h): 6,784 = US 4,795 (rejected 404 3,550 + blocked 404 495 + forwarded 404 22 + upload forwards 728) + EU 1,989 (rejected 404 1,868 + blocked 404 7 + forwarded 404 11 + WebSocket forwards 101 + WebSocket upstream errors 2). Classification: local blocked/scanner 5,920; Rails-forwarded 404 33; meaningful live 831 (728 uploads + 101 Cable forwards + 2 Cable upstream errors). Rails stays 3/3 for the gateway because uploads and Cable still persist.

Bounded backfill_run_agent fallback after the census (US only): the API2 worker emitted 25 qa_stage_rails_fallback events 21:17:19Z–21:20:45Z, all organization_id=25 (Kuda), reason=backfill_run_agent — 21 check_eligibility + 4 start_evaluation per the exact Logs Insights re-query (an earlier read reported 20 / 5) — each matched by a successful Rails Internal::QAEvaluationRequestStagesController#execute_stage; EU zero. This is the documented ENG-3237 fallback family (admin backfill-run requests carry qa_evaluation_backfill_run_agent_id; eligibility-stage.ts / evaluation-creation-stage.ts hand them to Rails because skip/fail paths enqueue the backfill replacement, auto-backdate and admin_forced sampling bypass live in QAEvaluationRequest). It is distinct from QA sampling recovery (still 100%, clean) and bounded by the run’s standby pool. Decision at evidence cutoff 21:20:45Z (last of the 25 events): no hot-fix; port the family natively. Implemented in #12573  (ENG-3280, in review, not merged): admin_forced requests are native in both stages regardless of qa_evaluation_backfill_run_agent_id (Rails sampling_required? / should_evaluate? already bypass sampling for them), and the native terminal-effect outbox dispatches the deterministic qa-eval-backfill-replacement-<runAgentId>-<requestId> job after commit (replacement re-evaluates auto-backdate). backfill_run_agent remains in native dispatch only for scheduled/ticket_closed rows carrying a run-agent id, a shape nothing produces; the other fallback reasons are unchanged. Rollback boundary: code revert — Rails keeps serving execute_stage, no flag was added. Until #12573 is deployed the Kuda family continues to fall back to Rails 1:1; this is not an irreversible retirement claim for QA stages.

Public gateway at cutoff 18:45:18Z (summary; package detail in the gateway runbook): 7,623 / 24 h = 6,730 local blocked/rejected + 40 forwarded 404 + 853 meaningful (755 uploads, 96 Cable forwards, 2 Cable upstream errors). Available structured seven-day 36,570 = 10,072 local + 15,761 forwarded 404 + 10,737 meaningful — not a full seven-day window.

QA sampling (qa_sampling_recovery_native, v11: five exact slugs + 10%). Exact cohorts stayed clean after eae24156. Kuda post-deploy 14:15Z cycle: 1,547 candidates / 66 requested / 99 skipped / 1,382 throttled / 37 counters lowered, zero candidate/transcription failures, unprocessed=0; later cycles also clean (17:32:55Z: 2,078 / 2 / 97 / 1,978, 38 counters, zero failures/unprocessed). Qonto completed repeated 5,000-candidate cycles after 14:30Z with zero failures and unprocessed=0; Nala, Rho and Easypay clean. The 18:00Z cycles were also clean: Kuda 2,171 / 84 requested / 127 skipped / 1,437 throttled / 39 counters; Qonto 5,000 / 9 / 118 / 1,203 / 256; Nala 235 / 0 / 58 / 177 / 15; Rho 1 candidate skipped no_agent_response; all zero candidate/transcription failures. Random-cohort orgs were again zero-candidate, so the percentage-cohort workload gate is still unproven: hold at five exact slugs + 10%; do not widen or replay.

Transcription state (conversation_transcription_state_native). The Qonto canary cleared its >24 h distribution/reconciliation gate. Since Qonto enablement (2026-09-18T16:12:41Z) structured logs contain exactly 196 API2 phase=context outcome=loaded and 196 phase=persist outcome=transcribed, with no retry, permanent, not_found or already_transcribed outcomes. Read-only EU DB reconciliation over the same window: exactly 196 Qonto rows, zero failures, zero duplicate credit usages, max attempt_count 0. Three zero-message / missing-duration rows match the prior Rails window shape (5/165); all 196 have no credit_usage, matching Qonto’s 0/165 Rails usage baseline (no active transcription credit). At about 17:59Z the operator expanded the flag progressively to exact kuda-pilot + qonto + rho-tnkw (no percentage); Rho had 30 Rails transcriptions in the preceding 24 h and zero failures. The first natural Rho native call.transcribed occurred at 18:10:12Z; read-only DB reconciliation of the post-canary window: one row, transcribed=1, failed=0, zero_message=0, 27 transcript messages, credited=0, max attempt_count 0, against a prior-24 h Rails baseline of 27/27 transcribed, 0 failed, 3 zero-message, 1,032 messages. US CloudWatch from the 18:10:12Z Rho boundary through 19:53Z shows conversation_transcription_state owner=api2 for rho-tnkw: 7 context loads, 5 persist|transcribed (fifth at 19:31:24.072Z) and, for the two 19:45:22.687Z / 19:45:23.202Z contexts, jobs that returned already_transcribed within ~1 ms with no persist transition — zero failure / retry / permanent / not_found telemetry. Advisory PostHog read-only reconciliation (about 19:55Z): exactly 5 API2-emitted (posthog-node) Rho call.transcribed events between 18:10:12.394Z and 19:31:24.311Z with 5 distinct call_detail_ids (A4o0BGnb…, BEWKkrew…, YD3a2kez…, 9AGXvjyb…, qAPynjmw…), no duplicates, zero Rails-emitted Rho events. Read-only production DB reconciliation (no writes) confirmed the five completed API2-owned Rho rows are clean: all transcribed_at set, transcription_failed_at / error null, attempt_count 0; transcript message counts 27, 16, 18, 28 and 0 — the zero-message row (qAPynjmw1vZCoayRbrWB6N5J) has transcribed_at 19:31:24.042Z and a clean failure state. The “already_transcribed” reading for the two 19:45Z calls was false. They decode to internal call details 138307 and 138306 whose stored public_ids are legacy short ids (wrkukptg880r / rrnqqpybfj5a); at 19:58:46Z both had transcribed_at null, zero transcript messages, transcription_failed_at / error null, attempt_count 0 and no worker error telemetry — i.e. they were never transcribed. Cause (code, not data): loadNativeTranscriptionContext returns call.id = row.public_id when present, while the queued job carries the prefixed conversation_part_call_detail_… id (Rails’ prefix_id, which is also what the Rails context endpoint returns); run.ts find(candidate.id === data.callDetailId) therefore misses on legacy-public_id rows and silently returns already_transcribed, so the completion job runs and the row is left untranscribed with no failure signal. The flag was rolled back at 2026-09-19T20:10:58.059946Z from v3 (exact Kuda + Qonto + Rho) to v4: exact kuda-pilot + qonto only; no replay or data write; the two Rho rows are left untouched pending a human recovery decision. Fix (id-form compatible lookup + explicit missing-call telemetry, regression tests for both id forms) is owned in a separate focused PR under ENG-3271 . Decision: hold at exact Kuda + Qonto; do not re-add Rho or widen until the fix is deployed and the two rows reach a terminal clean outcome.

Other holds. conversation_upload_intake_native exact Kuda: still no native traffic. call_recording_attachment_native: exact Kuda still has no native outcome; the Qonto canary produced 37 exact native intents from the 18:19Z boundary, all attached — max attempts 0, errors 0, 37 blobs / attachments / distinct call details. The flag was advanced at 18:43:12.702316Z to v3, exact kuda-pilot + qonto + rho-tnkw only (no percentage). Since v3, US Rho (org 19) produced its first native result at 18:59:35.125Z: outcome=attached, caller=provider_sync, transcriptionQueued=1, no error — one intent attached, one detail / blob / attachment, zero retry / error / orphan / duplicate; EU Qonto (org 33) produced 38 additional attached outcomes from 18:43:31Z through 19:11:08Z with zero error outcomes in the queried slice, and is 40/40 attached post-boundary. The initial decision was to hold at exact Kuda + Qonto + Rho for more natural Rho volume; after a re-read showing every verified 24 h cohort covered and clean (Rho 1/1, Qonto 40/40 post-boundary, Kuda prior clean; zero Rails recording creates after 18:43:12Z), the flag was advanced at 19:23:51.290614Z to v4: the same three exact 100% canaries plus a separate 10% organization rollout. Current state: v4. Next gate: hold v4 pending a natural percentage-cohort outcome plus continued API2 reconciliation and Rails-zero evidence; immediate rollback to exact-only on any regression. The v4 slice since 19:23:51Z has 15/15 attached log outcomes (Rho 1, Qonto 14), zero non-attached outcomes, and still no non-exact percentage-cohort outcome. Full Rails shutdown remains blocked.

Rho downstream outcome, 21:21Z–21:55Z (read-only): call detail 554172 attached natively at 21:21:10Z with transcriptionQueued=true, but no conversation_transcription_state telemetry ever ran for it. After 30 one-minute polls the call-summary job completed at 21:51:10Z with status=skipped, reason=transcript_missing_at_deadline; QA request 5985771 remained pending / waiting-for-transcription through at least 21:55Z. The shared transcription queue showed burst processing only through 21:22Z and carries the same 400/hour ceiling as Rails, so this is not yet attributable to the restored transcription-state handler (Rho was removed from conversation_transcription_state_native at 20:10:58Z, v4 exact Kuda + Qonto). Decision: no rollback, no replay, no data write; both Rho canaries (recording v4 slice and transcription) are held and widening is blocked until the queue-vs-handler cause is identified and 554172 / 5985771 reach a terminal outcome by human decision.

Rho 554172 queue evidence and transcription rollback to v6, 22:08:07Z (read-only): BullMQ job conversation_transcription_call_932d35dea92b5f31_conversation_part_call_detail_r4dBk5Lz0B2hLRl4bep6xZRv for call detail 554172 was in state waiting with attemptsMade=0, attemptsStarted=0, enqueued 21:21:10Z. The shared call-transcription queue had active 0, waiting 1,049, failed 1, completed 1,000, and the shared 400/hour limiter TTL was 566,961 ms. Backlog composition: 1,049 waiting = Kuda 1,044 + Rho 5; run sources 831 kuda-tx-yesterday, 197 kuda-tx-backfill, 20 live-hash, 1 provider-legacy-zendesk; oldest 19:51:52Z, newest 21:54:03Z. This is a queue backlog / global-limiter capacity failure, not a recurrence of the identifier lookup bug fixed in #12560 : the job was never attempted, so no handler ran. Its call-summary deadline expired and QA request 5985771 remained waiting. The coordinator rolled conversation_transcription_state_native from v5 exact Kuda + Qonto + Rho back to v6 exact Kuda + Qonto at 2026-09-19T22:08:07Z. Decisions: no replay, no queue or customer-row mutation; a human recovery decision was required for 554172 / 5985771 at the time (resolved naturally by 22:21:42Z, see below); the 1,049-job backlog behind the global limiter is a production blocker requiring a reviewed focused fix. #12572  / ENG-3279 (retry throttle-skipped calls) was reviewed on the QA track and extended so the retry sweep deduped candidates before its 100 cap and did not claim a new-runId duplicate while a call’s attach-time job was still waiting/active/delayed; its author closed it unmerged at 22:56:24Z in favour of a one-off enqueue script for the missed Kuda calls. The dedup findings are preserved on ENG-3279 and apply to that script too: exact-job-id dedup means a fresh runId for a still-waiting call (554172 included) adds a second job to the same limited backlog, and the 1,049 waiting jobs are already enqueued — the gap is limiter capacity, not missing enqueues. Live-vs-bulk precedence is the canonical fairness package #12577  / ENG-3283; the PGlite halfvec fixture fix #12570  is still open and not worked around. Nothing merged; human review required.

Rho 554172 drained naturally, ~23:20Z read-only check. The waiting job above was picked up once the limiter window turned: processedOn 22:19:58Z, finishedOn 22:20:08Z, attemptsMade/attemptsStarted 1, state completed. DB call detail 554172: transcribed_at 2026-09-19T22:20:08.161Z, 105 transcript messages, transcription_failed_at/error null, attempt_count 0. QA request 5985771 then completed normally at 22:21:42.701Z with no skip or error. No human recovery action remains for these two records; no replay or write occurred. This does not resolve the two earlier Rho rows 138306/138307 (legacy public_id, already_transcribed mis-match, fixed forward by #12560) — those stay on the human recovery list. Decision: conversation_transcription_state_native stays at v6 exact Kuda + Qonto (Rho off) pending a reviewed and deployed fairness package (#12577 / ENG-3283) and a fresh clean Rho canary. Since 22:13:05Z natural QA summaries are 180 US

• 85 EU API2-owned jobs with candidate failures 0, transcription failures 0, unprocessed 0, stoppedEarly 0; QA stays at 100%. The one-off Kuda repair Ronald proposed when closing #12572 is human-owned and is not run or facilitated from this track.

2026-09-20 post-release state (release-prod 0a6fd1e775faabc7ea0cbc8df404ab3841fb9e69). The release is healthy on US/EU API2 web and both logical worker services. Merged and deployed in it: #12569 (this plan’s QA 100% evidence), #12570 (ENG-3277 idempotent halfvec fixture — the shard-1 blocker), #12573 (ENG-3280 native backfill_run_agent stages) and #12577 (ENG-3283 live-over- bulk transcription precedence). qa_sampling_recovery_native is unchanged at v21 / all organizations; no flag change and no replay. Fresh Rails 24h read-only census, split into residue vs live dependency:

• Internal::QAEvaluationRequestStagesController#execute_stage: 102 US, 0 EU, last 21:20:46Z — that last call is the tail of the 25-call Kuda backfill_run_agent burst (21:17:25Z–21:20:46Z) recorded above, and that family is now native as of #12573 in 0a6fd1e7. No stage call has been observed since. Classification: pre-0a6fd1e7 residue; the gate is zero execute_stage calls over the next natural backfill runs.
• Internal::QASamplingRecoveriesController / Internal::ConversationTranscriptionReconciliationsController: 2,171 US + 1,002 EU, last 20:00:03Z, before the 20:08:04Z v21 cutover; all empty-heartbeat polls. Classification: pre-cutover residue, already ageing out of the 24h window (Rails-zero clock still runs from 20:00:03Z).
• Live dependencies (unchanged, tracked in their own sections): direct-upload proxy, call recording attach, Unit21, counters, expiry.

Decision: hold QA at 100%; keep the zero-window reversible (no controller removal until the 24h window is clean of both families and the next natural backfill run produces zero execute_stage calls). The post-release natural QA cycles must be read against the same gate — candidateFailures=0, transcriptionFailures=0, unprocessed=0, stoppedEarly=null — before the release is treated as steady for this track.

New Rho transcription blocker (2026-09-20 00:19Z): duplicate concurrent jobs. Call detail 554173 / conversation_PzJ7xaQR38vU8wNxWY3AwL48 ran four simultaneous conversation_transcription_call jobs with four different runIds at 00:19Z. The BullMQ job id is conversation_transcription_call_<runId>_<callDetailId>, so distinct runIds defeat the job-id dedup by construction. runId is derived as sha256(<callDetailPrefixId>:<updated_at µs>)[0,16] at every enqueue site that computes it (recording attach, the QA stage hand-off, the retry write path, which also bumps updated_at before deriving), and the reconciliation sweep mints a random 8-byte token per claim; Rails-originated enqueues through /internal/conversation_transcriptions/calls/enqueue pass their own runId. Any two of those sites (or the same site observing a fresh updated_at) therefore enqueue independent jobs for the same call. Which four sites fired here is not established from this track (no worker log access) — treat the per-site attribution as open. This is exactly the new-runId duplicate risk called out on ENG-3279 when #12572 was closed; the call-detail-keyed admission check from that PR (already_queued when any waiting/active/delayed/ prioritized job exists for the call, regardless of runId) was not merged and is not on 0a6fd1e7. Decision: conversation_transcription_state_native stays at v6 exact Kuda + Qonto with Rho off; Rho does not re-enter the canary until (a) worker logs attribute the four enqueues, (b) a reviewed call-keyed dedup lands across all enqueue sites (reopen the ENG-3279 scope or a successor, human decision), and (c) a fresh clean Rho canary follows. No replay, no write, no flag change from this track.

Reviewed call-keyed dedup for the 554173 blocker above — ENG-3286 / #12589  (gate (b)). Call detail 554173 (conversation_PzJ7xaQR38vU8wNxWY3AwL48 / conversation_part_call_detail_DGmPVlYzWn9ivZkxOg0de26J) was processed by four distinct runIds on the shared conversation-transcription-call queue at the same time: 12f2c6de4d598e48, provider-legacy-zendesk-38efa12e-0fc9-460c-8504-3a8648732b50, d4ca1de97276f6da, and provider-zendesk_reconciliation_ticket_19_3_285644_01a0bc1c-e8a7-73f6-b7c1-3ecaea393ddd. Rails logged four context loads and four persisted/transcribed outcomes. Read-only DB: one clean final row, 25 transcript messages, no failure/error, attempt_count 0, credit_usage_count 0, transcribed_at 00:19:26.343Z. This is redundant provider execution and cost, not duplicate stored messages; no replay or write occurred. Cause: the job id is deliberately per run (conversation_transcription_call_${runId}_${callDetailId}, because completed ids are retained 24 h and a stable id would swallow legitimate re-transcription), and the only cross-run guard was the transcript_available || transcription_failed read in run.ts, which four concurrent workers all passed before any of them persisted. Decision / fix: ENG-3286 — a bounded Redis claim per stable call (org slug + callDetailId, SET NX PX, 15-min TTL above the 10-min provider timeout, compare-and-delete release) acquired in runConversationTranscriptionCallJob before the context load and provider call for both owners; a job that finds the claim held throws a distinct error so the unchanged attempts: 3 / 60 s exponential retry re-runs it, at which point the ordinary already-transcribed check returns already_transcribed. Redis-claim errors fail open (provider parity over dedup). Single queue, the 400/hour aggregate limiter, live/bulk lanes, job ids, retry shape, and the existing backlog are unchanged; no DB lock spans the claim. Real-Redis/BullMQ concurrency coverage reproduces the four run ids above. Reducing the number of upstream enqueue sources is out of scope here. Transcription widening stays held (v6 exact Kuda + Qonto) pending review and deploy of this fix plus a fresh Rho canary; #12571 Unit21 stays separate and review-gated; nothing merged.

Direct-upload processing: canary re-enabled for exact Kuda, no post-fix proof yet (authoritative PostHog readback 2026-09-20). Flag 281904 api2_conversation_upload_native_processing is ACTIVE at v3, exact Kuda only, updated 2026-09-19T20:54:19.918961Z (a human re-enable after the parser fix #12555; this automation made no flag change). Its last_called_at is still 19:16:11.440Z, i.e. no natural direct upload has evaluated the flag since it was re-enabled, so there is no post-fix outcome — positive or negative — to cite. Decision: hold at exact Kuda, no widening, no replay, until a natural Kuda batch completes cleanly under v3. The paragraph below is the pre-20:54Z history and is kept for the record. Earlier text describing this flag as “not active” / “disabled” was stale as of 20:54:19Z.

api2_conversation_upload_native_processing (PostHog flag id 281904) was created for exact Kuda at 18:19:08Z. The first natural upload, 649840, failed terminally at 18:20:30Z with Invalid recorded_at, and the flag was disabled immediately (v2). Read-only evidence: the existing Kuda payload format 2026-09-19 19:20:22 (space separator) is accepted by the Rails parser but rejected by the API2 parser. Upload 649840 then recovered on its own through the normal scheduler after the flag rollback (read-only evidence): completed at 18:33:01Z, delivery 11387 completed, sync delivery 11388 completed, conversation 6890594, exactly one recording attachment — no human replay and no production write. Decisions: flag stays disabled pending review and deploy of the parser fix #12555  and a fresh clean natural Kuda batch afterwards. (Superseded at 20:54:19Z by the v3 re-enable recorded above; the hold on widening stands.)

Expired-evaluation sweep: native port prepared (2026-09-20). Fresh read-only evidence: Internal::ExpiredEvaluationsController#process_all is exactly 48 successful calls / 24h (US 24, EU 24), one per hour per region; post-release since 23:45Z exactly 4 calls (US 2, EU 2), latest 2026-09-20T01:05:00.180Z US / 01:05:00.309Z EU; at the fixed 01:18:53Z cutoff still 48/24h (24 US + 24 EU), both regions last 01:05Z. Rails internal effects at that cutoff: 5,007/24h (US 3,234 + EU 1,773) and 328,830/7d (US 132,073 + EU 196,757). 24h by family: 3,009 reconciliation + 794 recording + 741 direct upload + 144 Unit21 + 129 counter cache + 101 QA stage + 48 expiry + 32 transcription context/write + 9 tiny tail (field values 4 + report actions 4 + Slack 1) = 5,007; EU split reconciliation 950 + recording 743 + counters 56 + expiry 24 = 1,773. (Same cutoff, for the record: reconciliation 3,009 last 20:00:03Z/20:00:01Z and zero QA sampling — residue; execute_stage 101 US last 21:20:46Z — residue; QA v21 active at 100%, last_called_at 00:45:01Z. #12571 merged 01:19:47Z and dedup #12589 merged 01:20:05Z; release-prod 35c5dce7 contains both but is not yet proven live, so Rho transcription stays off at v6 exact Kuda + Qonto until it is, and a bounded fresh natural canary is explicitly advanced by a human.) Later read-only refresh: release-prod 017b86ef fully deployed/COMPLETED on US/EU API2 web 1/1 and workers 5/5. Rails /internal 4,749/24h and 327,254/7d; the post-release window 01:48:40Z → ~02:23Z holds exactly 10 calls: Unit21 4 US, expiry 1 US + 1 EU, field-value upsert 4 US. 24h composition: reconciliation 2,845 (last 20:00Z, empty-heartbeat residue), direct upload 741, Unit21 144, QA stage 101, counter cache 129, recording 696, expiry 48, transcription context/write 32, field values 8, reports 4, Slack 1. QA flag 280844 active v21 (five exact canaries + 100% group) since 20:08:04Z, last_called_at 01:45:02Z. Deletion claims for every family stay gated on the 24h/7d windows, not on this post-release slice. This is not residue: API2 already owns the hourly :05 BullMQ scheduler (mark-expired-evaluations-failed-hourly) and the Flipper kill switch pause_api2_expired_evaluation_cleanup, but runExpiredEvaluationCleanup still delegated the whole sweep to Rails via processExpiredEvaluationsInRails, so every hourly tick is one live Rails round-trip. Decision: port both markers natively in one focused package (Linear ENG-3289, PR below; review-required, unmerged, no deploy, no flag change, no data mutation):

• src/lib/expired-evaluations/native.ts reproduces ExpiredEvaluationsFailureMarker (snapshots running/pending with created_at < now() - 1 hour; processing_status = 'failed'; conversation evaluation_state = 'failed' only when the snapshot is still the conversation’s most recent) and ExpiredQAEvaluationRequestsFailureMarker (stale pending/ready by COALESCE(scheduled_for, created_at), stale processing by COALESCE(evaluation_started_at, updated_at); source-AI pending and ghost-completed agent evaluations failed first, parent score recomputed; then failNativeRequest with the Rails timeout message QA evaluation request timed out after 1 hour, which writes the terminal state, completed_at, api2_terminal_failure {retryable:false} and the durable qa_request_failure_effects outbox row whose post-commit delivery carries sampling/queue-slot release, qa_evaluated_at clearing, ActionCable broadcast, requester mail and backfill replacement). Batching is by primary key in find_each-sized (1,000) pages; every row fails in its own short transaction; per-row errors are logged and isolated; candidates/failed counts match the Rails shape. The request_completed/failure workflow event is emitted through the API2 PostHog workflow-event path (Sentry reporting is the documented difference: API2 logs a structured warning per timed-out request instead).
• src/lib/expired-evaluations/sweep-lock.ts takes the same advisory lock as ExpiredEvaluationSweepLock (expired_evaluation_failure_sweep, keyed [crc32(name) & 0x7fffffff, 0] exactly like with_advisory_lock), as a session-level pg_try_advisory_lock on a dedicated connection with no open transaction, released in finally. Contention throws ExpiredEvaluationSweepContentionError, and the BullMQ scheduler now retries 5 attempts, fixed 10 minutes to mirror the Rails jobs’ retry_on ContentionError, wait: 10.minutes, attempts: 5.
• Rollout is per organization on a new PostHog flag expired_evaluation_cleanup_native (off/fail-closed = Rails). Each hourly tick runs the native pass under the shared lock for owned organizations, then — only if stale rows for non-owned organizations exist — releases the lock and calls the Rails controller (which takes the same lock), so Rails stays a live, exercised fallback with no overlap. Rollback layers: unset the PostHog flag (native pass becomes a no-op, Rails handles everything on the same tick); or enable pause_api2_expired_evaluation_cleanup and re-enable the two retained Rails GoodJob crons.

Gates before any retirement claim for this family: the PR is human-reviewed and deployed; the flag is widened organization by organization with each natural :05 tick showing native snapshots/requests counts and deferredToRails = 0 for owned organizations; process_all drops from 48/24h toward zero as ownership widens, with no lock-contention retries exhausted; then a reversible zero-window (Rails controller and crons kept, flag at all organizations) before the Rails markers/controller are removed.

Cross-track state at 2026-09-20T04:06Z (release f990fc4999). The release was fully live at 03:44:40Z–03:44:42Z (US and EU web 1/1, physical workers 5/5). It carries the Unit21 accelerator, which has no natural webhook proof yet. The 04:05Z natural expired-evaluation sweep was clean in both regions — US 4/4 and EU 4/4 snapshot candidates handled natively, zero deferredToRails, zero Rails calls, zero errors, and an exact eight-row reconciliation against the Rails markers — so the expired-evaluation PostHog flag 282005 was advanced by this authorized automation at 04:06:20.993Z to v2: exact Easypay + Rho + NALA plus a 50% organization group; next gate is the 05:05Z tick. Counter-cache: the Flipper api2_counter_cache_native_repairs row is absent in both regions (native off, Rails still repairing) and the region-scoped PostHog gate is the focused #12615  / ENG-3299 . Legacy image edge package #12613  (head 401c2d9e0, fully green after the strict drift guards: apply is idempotent only against the exact managed origin/behavior; rollback refuses to delete anything it does not own) was open and unapplied at that point. Update (human merges, not automation): Chidi merged #12613 at 03:56:15Z (a00a3bf436) and #12615 at 03:56:26Z (8f84f67350), both with no recorded GitHub review despite Ronald being requested. Both are in release-prod 39f11065e: US already completed the intermediate 8cfa8d65dd containing both (web TD1669 1/1, worker TD1659 5/5); EU completed 39f11065e at 04:14:36Z (web TD1687 1/1, physical worker TD1676 5/5, one deployment each, rollout COMPLETED, zero pending). Activation status stays explicit: the CloudFront image behavior is unapplied in both regions and the counter-cache PostHog flag is not created, so both runtime paths remain on Rails; the required Ronald/human activation review is still outstanding before either cutover. The 03:30:00Z gateway and image census above is unchanged. The only automation mutation in this slice was the bounded PostHog flag 282005 v2 advance; no scheduler, CloudFront, data or deploy change.

QA ladder state. qa_sampling_recovery_native took the first ladder step at 2026-09-19T18:19:06Z: PostHog version 12, five exact slugs retained, organization group 10% → 50%. The first natural bucket at 50% (18:30Z) did not meet the complete-cycle gate: Kuda (org 25) completed at 18:35:02Z with 2,155 candidates / 77 requested / 304 skipped / 1,772 throttled / 2 ignored / 55 counters, zero candidate/transcription failures, unprocessed=0; Rho, Easypay, Nala and the percentage-selected organizations were clean (percentage-selected again zero-candidate); but Qonto (org 33) completed at 18:40:25Z with 5,000 candidates / 20 requested / 78 skipped / 3,634 throttled, zero failures, stoppedEarly=budget and 1,268 unprocessed. Per the ladder the flag was immediately restored to the last-known-good five exact slugs + 10% at 18:42:15.585896Z (PostHog version 13), with no replay. The budget-stopped run was Qonto job qa_sampling_recovery_33_1988714. The next natural cycle under v13 completed cleanly: Qonto job qa_sampling_recovery_33_1988715 finished at 18:52:58Z processing the full 5,000 candidates = 14 requested + 813 skipped + 4,173 throttled, 0 failures, no stoppedEarly, 255 counters lowered; Kuda job qa_sampling_recovery_25_1988715 finished at 18:47:13Z with 1,936 candidates / 41 requested / 219 skipped / 1,674 throttled / 2 ignored, 0 failures. The second post-rollback cycle was also clean and complete: Qonto job qa_sampling_recovery_33_1988716 finished at 19:06:55Z with the full 5,000 = 12 requested + 120 skipped + 4,868 throttled, 0 ignored, 0 candidate/transcription failures, no stoppedEarly, 254 counters; Kuda job qa_sampling_recovery_25_1988716 finished at 19:02:29Z with 1,810 / 39 / 153 / 1,617 / 1 ignored, 0 failures. On two consecutive complete Qonto cycles, PostHog flag 280844 was advanced conservatively at 19:07:38.696511Z to version 14: same five exact slugs + 25% group (not a direct retry of 50%), server-only active. The 19:15Z bucket under v14 passed: Qonto job qa_sampling_recovery_33_1988717 completed at 19:22:01.699Z with the full 5,000 = 5 requested + 86 skipped + 4,909 throttled, 0 ignored, 0 candidate/transcription failures, no stoppedEarly, 255 counters; Kuda job qa_sampling_recovery_25_1988717 completed at 19:18:22.061Z with 1,756 / 32 / 161 / 1,562 / 1 ignored, 60 counters, 0 failures, 0 unprocessed; Easypay 228 / 0 / 50 / 178 / 0 ignored, 15 counters, 0 failures, 0 unprocessed; all additional percentage-selected tenants completed with zero candidates and zero failures. The flag was then advanced to the same five exact slugs + 50%, server-only active — reported as version 15 at 19:22:39.140626Z and version 16 at 19:22:47Z (two writes eight seconds apart, same definition). The first 50% 19:30Z bucket was interrupted by a deployment, not a load failure: Qonto job qa_sampling_recovery_33_1988718 ended at 19:34:02.454Z with 5,000 candidates, 5 requested + 93 skipped + 2,776 throttled = 2,874 processed, 0 candidate/transcription failures, stoppedEarly=shutdown, 254 counters; native-run logged reason=shutdown, processed=2,874, unprocessed=2,126 at 19:33:59.918Z. Kuda job qa_sampling_recovery_25_1988718 had already completed clean at 19:33:03.858Z: 1,732 candidates, 18 requested, 126 skipped, 1,587 throttled, 1 ignored, 61 counters, zero failures/unprocessed; Easypay was clean (237 / 0 / 59 / 178, 15 counters, zero failures/unprocessed). Cause: #12557 and release sync #12558 merged at 19:26:32Z / 19:26:44Z, and the release-prod 8ec16b8d (8ec16b8d858d3047745e265e8829749b779baf73) rollout replaced the EU worker fleet mid-sweep (EU worker TD 1661 began 19:31:56Z, all five new tasks started 19:32:17Z, old tasks stopped 19:33:34Z / 19:33:46Z / 19:33:57Z). Per the ladder the flag was rolled back immediately to the known-good five exact slugs + 25%, server-only active, exact targeting preserved — reported as version 17 at 19:34:47.873628Z and version 18 at 19:36:30.489390Z (same definition). No replay. State after rollback: v18, five exact slugs + 25%. Services at that time: EU web TD 1672 completed 1/1; EU worker TD 1661 5/5 IN_PROGRESS; US web TD 1654 1/1 and worker TD 1644 5/5 IN_PROGRESS; zero pending/failed. Verified at 19:36:57Z: 8ec16b8d is exact and COMPLETED on all four services — US web TD 1654 1/1 (19:36:20Z), US logical worker (physical fc-rulebase-api2-worker-pr-zy7q0) TD 1644 5/5 (19:36:14Z), EU web TD 1672 1/1 (19:34:59Z), EU logical worker (physical fc-rulebase-api2-worker-pr-m9780) TD 1661 5/5 (19:35:42Z); all pending 0, failedTasks 0; no level>=50 first-slice errors. Treat this as a deployment interruption, not a package failure, but the explicit complete-cycle rollback gate stands: hold five exact slugs + 25% until a complete natural post-deploy cycle has candidateFailures=0, transcriptionFailures=0, unprocessed=0, stoppedEarly=null; do not widen during a rollout. That gate passed: the full 19:45Z post-deploy cycle (jobs ..._1988719) under five exact slugs + 25% completed clean — Kuda 1,709 candidates / 25 requested / 139 skipped / 1,544 throttled / 1 ignored / 61 counters; Rho 7 / 5 / 2; EU cohort org 41 (Nala) 221 / 0 / 43 / 178 / 15 counters; Qonto 5,000 / 7 / 761 / 4,232 / 254 counters; across all jobs zero candidate failures, zero transcription failures, zero unprocessed, no early stop. The flag was advanced at 19:52:38.132439Z to version 19 and reported again at 19:53:40.142567Z as version 20 — both the same five exact slugs + 50%, server-only active. The first complete natural 50% cycle (20:00Z, jobs ..._1988720) completed clean: Kuda 4,648 candidates / 16 requested / 157 skipped / 4,468 throttled / 6 ignored / 61 counters; Nala (org 41) 324 / 0 / 37 / 287 / 15 counters; Qonto 5,000 / 4 / 106 / 4,890 / 254 counters; and newly percentage-selected org 38 processed one candidate and requested one evaluation — the first non-exact percentage-cohort proof. Across all jobs: zero candidate failures, zero transcription failures, zero unprocessed, stoppedEarly=null. The flag was then advanced at 20:08:04.328711Z to all organizations (PostHog response version 21; likely a same-definition concurrent v20 write), server-only active. The first complete natural 100% cycle (20:15Z, jobs ..._1988721) completed clean at 20:23:02.661Z: Rho (org 19) 1 candidate / 1 requested; Nala (org 41) 214 / 0 / 35 / 179 / 15 counters; Kuda 1,693 / 7 / 105 / 1,580 / 1 ignored / 61 counters; Qonto 5,000 / 6 / 128 / 4,864 / 2 ignored / 253 counters. Across all jobs: zero candidate failures, zero transcription failures, zero unprocessed, stoppedEarly=null. A read-only Rails structured controller census (structured event time, not ingestion time) from the 20:08:04.328711Z 100% cutover through 20:24Z found exactly zero Internal::QASamplingRecoveriesController and zero Internal::ConversationTranscriptionReconciliationsController calls in both regions. Current state: v21, 100%, held; the reversible zero-window clock for these two controllers starts at 20:08:04.328711Z. This is not an irreversible retirement claim — the flag remains a one-write rollback to five exact slugs + 50% on any regression. No replay. The separate conversation_transcription_state_native v4 Rho rollback (see the transcription section) is unaffected.

Weekend fast-track decision (2026-09-19)

Evidence. In the 24 h before this decision every Rails-owned organization in both regions completed its QA sampling cycles with zero candidates, zero reconciled transcriptions and zero transcription handoffs. The 3,921 / day transcription-reconciliation residual in the 17:47:25Z census is empty 15-minute heartbeat traffic (each sweep finds nothing to reconcile), not meaningful work. The remaining Rails QA/transcription work is therefore already effectively idle; what is left is the operational cutover of the flags, not new proof of native correctness.

Recommended cutover ladder (operational, reversible). For qa_sampling_recovery_native and conversation_transcription_state_native, keep the exact canaries and advance the organization-group rollout 10% → 50% → 100%. Between steps:

1. wait for one complete natural 15-minute scheduler cycle at the new percentage (no manual replay);
2. explicitly check every completed cycle summary for candidateFailures=0, transcriptionFailures=0, unprocessed=0 and stoppedEarly null (or shutdown only across a deploy boundary), and for transcription phase=persist outcome=transcribed with no retry / permanent / not_found / already_transcribed outcomes;
3. on any regression, roll the flag back to the previous percentage immediately (PostHog flag edit, no deploy) and record the version, time and reason in the runbook ledger before re-attempting.

Weekend success criteria. Zero meaningful Rails production traffic on the QA/evaluation and transcription paths (heartbeat sweeps that find no work do not count), with Rails scaled to zero behind the flag rollback: the Rails services stay deployable and the flags can be returned to Rails within one cycle. This is a runtime state, not a deletion.

Still gated. Irreversible work — deleting Rails adapters, controllers, jobs or the API2 fallback branches — remains gated on the complete 7-day zero-call window, the effects-boundary evidence and human review described in Rails shutdown gate. Nothing in this decision widens a flag by itself; each step is an operator action recorded in the ledgers.

The same gateway cutover caused an MCP outage before #12391  restored it. The MCP feature router registered ALL /mcp, but the composed gateway did not recognize that registration as ownership of POST /mcp, so the wildcard fallback sent the request to Rails and returned Rails’ 404. Route-level tests had not exercised the composed application with the fallback enabled. Public-cutover work must therefore prove ownership at the final application boundary, not only inside a feature router.

Direct-upload canary regression at 2026-09-19T18:20:30Z (Kuda, rolled back)

api2_conversation_upload_native_processing (id 281904) was created as exact kuda-pilot at 18:19:08Z. The first natural Kuda job, conversation_upload 649840 (source xcally, direct, metadata recorded_at "2026-09-19 19:20:22"), failed terminally at 18:20:30Z with Invalid recorded_at; the flag was disabled at v2 immediately, so every later direct upload went back to Rails. Read-only row state: upload failed; delivery owner=api2, state=failed, attempts=1, replay_count=0, last_error=Invalid recorded_at.

Root cause: parseLocalDateTimeInZone in rulebase-web/rulebase-api2/src/lib/conversation-uploads/native-processing/xcally.ts accepted only YYYY-MM-DDTHH:MM[:SS] (optional T, optional trailing Z), while Rails (ConversationUpload::Processors::Xcally#recorded_at, ActiveSupport::TimeZone#parse after stripping Z) accepts the space-separated local time XCALLY actually sends. The fix makes the native parser accept T or whitespace as the separator (and surrounding whitespace), with the same organization-local-time semantics, covered by a unit test and by the production-shaped PGlite XCALLY test using the space-separated form.

No human replay or write was made. Upload 649840 nevertheless recovered automatically: the normal 18:30Z conversation-upload tick saw it as retryable and enqueued it after the flag rollback. The API2 worker recorded Rails 504s at 18:31:02Z (retry_direct) and 18:32:05Z (process_direct); Rails logged retry_direct 500, process_direct 500, then process_direct 200 at 18:33:01Z. Read-only state afterwards: upload completed at 18:33:01Z; process delivery 11387 owner=api2, state=completed, attempts=1, replay_count=0, last_error=null, failed_at=18:20:30Z, completed_at=18:33:01Z; sync-completion delivery 11388 completed; conversation 6890594 is the only conversation for the upload, with one part, one call detail, and one recording attachment. A read-only US census at 18:56Z for unique_id 1789841918.162363 confirms no duplicates: conversations=1, parts=1, call_details=1, recording_attachments=1, upload_links=1 (delivery row last updated 18:33:01.149Z, last_error=NULL). The earlier customer-recovery decision is therefore closed without any human replay or production write.

That recovery exposes a contract question rather than a data problem: a terminal native (owner=api2) failure was picked up by the periodic retry tick and handed to Rails once the flag was off. Whether this cross-owner retry needs a durable guard is tracked separately from the parser fix (see ENG-3267 ); the parser PR stays scoped to the whitespace separator.

The Kuda canary was re-enabled by a human at 2026-09-19T20:54:19.918961Z (flag 281904 v3, exact Kuda only) after the reviewed fix reached release-prod. As of the 2026-09-20 PostHog readback last_called_at is unchanged at 19:16:11.440Z, so no natural Kuda batch has yet exercised the fixed parser; the bounded single-organization gate (one clean natural batch, no widening, no replay) is still open. This automation did not change the flag.

Counter-cache repair goes native (2026-09-20, ENG-3288 )

Read-only census: the rolling 24 h ending 2026-09-20 holds exactly 129 successful Internal::CounterCachesController#fix calls (US 73, EU 56); the post-release window since 2026-09-19T23:45Z holds 17 (US 10, last 00:06:48.966Z; EU 7, last 00:00:29.298Z). The earlier 21:10Z / 21:53Z censuses reported the same family at 128. The fixed cutoff 2026-09-20T01:18:53Z confirmed it unchanged at 129 (US 73 / EU 56) inside a 24 h Rails-effects total of 5,007 (US 3,234 + EU 1,773; 7 d 328,830 = US 132,073 + EU 196,757), alongside Unit21 144 (last 01:10Z, now merged as #12571) and expiry 48 (last 01:05Z); the rest was pre-cutover residue. Full 24 h breakdown: 3,009 reconciliation + 794 recording + 741 direct upload + 144 Unit21 + 129 counter cache + 101 QA stage + 48 expiry + 32 transcription context/write + 9 tiny tail (field values 4, report actions 4, Slack 1). After Unit21 this is the largest verified live non-QA Rails effect. Reviewed release-prod 35c5dce7 contains #12571 and #12589 but was not yet proven live at that cutoff.

Mechanism: API2 already owns the 3-hourly counter-caches-tick scheduler, the fix_counter_caches session-level advisory lock, per-step resume (completedSteps) and the pause_api2_counter_cache_repairs Flipper kill switch, but runCounterCacheFix still posts each of the seven COUNTER_CACHE_STEPS to Rails via fixCounterCacheStep. Rails runs counter_culture_fix_counts for six models and EvaluationSamplingRuleCounter.fix_agent_counter_caches! for the seventh, all under ActsAsTenant.without_tenant.

Native replacement (rulebase-api2/src/lib/counter-caches/native-steps.ts and sampling-rule-counters.ts), behind the deployment-scoped PostHog gate api2_counter_cache_native_repairs (owner platform engineering, created 2026-09-20, review/remove by 2026-12-19; ENG-3299 ). API2 evaluates the gate with the deployment region (AWS_REGION, falling back to RULEBASE_REGION) as distinct id api2-deployment:<region> and a region person property, cached per process for 20 seconds; unknown, disabled, or unreadable flags keep every step on Rails. The pause_api2_counter_cache_repairs Flipper switch remains the independent rollback lever.

Gate blocker (read-only, 2026-09-20T03:38Z): no flipper_features row for api2_counter_cache_native_repairs in US or EU, so native repairs were disabled in both regions; the key moves to PostHog (ENG-3299 ) and is removed from Flipper use so no duplicate key exists. A per-organization canary was rejected because the sweep is global: org-filtered native steps would run alongside a Rails sweep that still covers every organization, doubling work without removing a single Rails call until 100%. No Flipper/production mutation by Devin.

Counter-cache regional rollout

Create the PostHog flag api2_counter_cache_native_repairs with a release condition for one deployment region (region = eu-central-1 or region = us-east-1) at 100%, with no percentage cohort. Wait for one three-hour tick; verify counter cache fix completed with zero Internal::CounterCachesController#fix calls in that region and unchanged counter parity. Then add the second region and repeat the same verification. Rollback by disabling the PostHog flag (the next run returns to Rails) or by enabling the Flipper pause_api2_counter_cache_repairs switch.

• Six counter_culture steps: one autocommit UPDATE target SET col = cnt FROM (SELECT id, COUNT(child.id) … GROUP BY id) WHERE col IS DISTINCT FROM cnt per counter (gem 3.11.2 reconciler semantics): discarded children excluded where the child model includes Discard (employees, sheet rows), unresolved-only for memberships.assigned_conversations_count, and GroupEmployee’s groups.employees_count (see the defect note below). No updated_at touch, matching Rails.
• evaluation_sampling_rule_counters: agent counters scanned in Rails’ updated_at DESC, id DESC order in batches of 1,000; only fixed_per_period + partition_by = agent rules; successful AI evaluations (active completed or pending) from ticket_closed/scheduled requests in the counter’s organization-local day/week/month period, with brand/group filters on the conversation and the team filter on qa_agent_evaluations.team_id; result clamped to rule.count. One deviation: the write is UPDATE … WHERE consumed_count = <loaded value> instead of Rails’ absolute update!, so a concurrent consume/release is left for the next sweep instead of being clobbered.
• No transaction, row lock, or external work spans statements; the advisory lock remains session-level on its own connection.

Validation: PGlite suites against the exact structure.sql tables cover parity, idempotency, discard/status filters, organization-timezone period bounds, caps, >1,000-counter batching, the optimistic guard, and two-tenant sweeps; run-loop tests cover gate off, gate on (zero Rails calls), gate read failure (Rails), and native-step failure with completedSteps resume.

Newly discovered defect: group_employees clobbers direct counts (2026-09-21, #12679 )

GroupEmployee declares counter_culture :group and counter_culture %i[group parent_group], both on groups.employees_count, so the incremental callbacks count each membership toward its group and that group’s parent. The reconciler (counter_culture_fix_counts, gem 3.11.2 Reconciler#update_count_for_batch) iterates the two definitions in order and writes column = count for each: the parent-group pass runs last and sets every leaf group to 0 and every parent to its child-group count only. The native step shipped in #12595 reproduced those two ordered writes exactly, so the defect exists on both sides of the boundary; Rails has had it for as long as the repair cron has existed, and API2 would inherit it on activation.

#12679 (found by Devin Review on an earlier revision, no recorded human review on #12615) replaces the two CounterCacheUpdates with one aggregate write per group, COUNT(ch.id) over memberships whose group_id is the group itself or one of its immediate children, matching the incremental definition rather than the Rails reconciler. native-steps.db.test.ts asserts a parent with 1 direct

• 2 child memberships ends at 3 and leaf groups with direct members keep their count. Consequences for rollout: on the first native group_employees tick, groups.employees_count will increase for leaf groups and parents with direct members relative to the Rails-repaired value — that is the correction, not a parity regression, and the per-step rows comparison in the regional runbook should expect a one-time non-zero delta on this step.

Rollout: follow the regional PostHog runbook above after review and deploy, compare the next tick’s per-step rows against the pre-cutover Rails cadence, then the other region; disabling the flag is an immediate Rails rollback. Expiry (~48/day) is a separate later package.

Exact residual census at 2026-09-19T21:10:29Z and the Unit21 sync decision

Rails /internal effects: 6,202 / 24 h (US 3,697, EU 2,505) and 336,317 / 7 d (US 133,669, EU 202,648). The large 24 h families are residue or idle rather than live: transcription reconciliation 3,700 (last seen 20:00:03Z, empty heartbeats; QA sampling became 100% at 20:08Z), call-recording attachment 1,332 (last 18:41:47Z), direct upload 741 (last 19:18:11Z), QA execute_stage 78 (last 18:20:59Z), transcription context/write 22 + 22 (last 17:10Z). The largest unquestionably live families were Unit21 sync 144 / 24 h (still firing through 21:10Z), counter-cache fix 128 / 24 h (through 21:05Z) and expired evaluations 48 / 24 h (through 21:05Z). Post-cutoff confirmation: Rails received two more successful sync_unit21 calls in US at 21:20:00Z and 21:30:00Z (none in EU). Recount around 21:40Z: 6,079 / 24 h (US 3,670, EU 2,409) and 334,979 / 7 d (US 133,365, EU 201,614); still-live recurring families unchanged (Unit21 144 / day through 21:30Z, counter-cache 128, expiry 48); no Kuda upload after 649843; Rho call detail 554172 attached natively at 21:21:10Z with transcriptionQueued=true, but no conversation-transcription telemetry ever ran; after 30 one-minute polls the call-summary job completed at 21:51:10Z as status=skipped, reason=transcript_missing_at_deadline, and QA request 5985771 stayed pending/waiting-for-transcription through at least 21:55Z. The shared transcription queue showed burst processing only through 21:22Z and carries the same 400/hour ceiling as Rails, so this is not yet attributable to the restored transcription-state handler; no rollback, replay or write was made, but both Rho recording and transcription canaries are held and widening is blocked pending attribution (outside this package). Recount at 2026-09-19T21:53:42Z: 6,006 / 24 h (US 3,639, EU 2,367) and 334,624 / available 7 d (US 133,251, EU 201,373): empty reconciliation heartbeats 3,583, recording pre-cutover residue 1,230, direct-upload residue 741 total (739 process_direct = 729 × 200 + 10 × 500, 1 retry_direct 500, 1 mark_failed 200), Unit21 144 (live through 21:50Z, four successful post-21:10:29Z calls), QA stage 103, counter-cache 128, expiry 48, transcription context/persist 20, tail 9. Holds unchanged: direct processing exact Kuda with zero Kuda uploads since 20:54:19Z; Rho transcription restore with zero telemetry since 21:00:53Z; recording v4 since 19:23:51Z exact-only (Rho 5 attached + 1 already-attached, Qonto 96 attached, zero percentage outcomes). Counter-cache (128 / day) is the next candidate package once this one is stable.

At 2026-09-19T22:08:07Z conversation_transcription_state_native was rolled back from v5 (exact Kuda + Qonto + Rho) to v6 (exact Kuda + Qonto). Read-only queue evidence for Rho call detail 554172 (job conversation_transcription_call_932d35dea92b5f31_conversation_part_call_detail_r4dBk5Lz0B2hLRl4bep6xZRv): state waiting, attemptsMade 0, attemptsStarted 0, enqueued 21:21:10Z; the shared transcription queue had active 0, waiting 1,049, failed 1, completed 1,000, and the shared 400/hour limiter TTL was 566,961 ms. Its call-summary deadline expired and QA request 5985771 remained waiting. The customer work was not replayed or mutated; recovery of 554172 / 5985771 is a human decision. The 1,049-job backlog behind the global limiter is treated as a production blocker requiring a separate reviewed, tested fix (outside this package). That fix is ENG-3283 / #12577: one queue, the unchanged aggregate 400/hour limiter, every new call job classified live (unprioritized) or bulk (BullMQ priority: 10, so live leads once the pre-existing unprioritized backlog drains), existing Redis jobs untouched.

Read-only follow-up at ~23:20Z: the Rho job for 554172 completed naturally at 22:20:08Z and QA request 5985771 completed cleanly at 22:21:42Z, with no replay or write. Queue reads during the drain: waiting 406 / active 20 / failed 1 / completed retained 1,000, then waiting 309, all 309 kuda-tx-yesterday jobs enqueued 21:53:57Z–21:54:03Z. Rails internal effects were 5,588 / 24 h (US 3,463, EU 2,125, down 418 from 6,006) and 332,199 / available 7 d (US 132,661, EU 199,538), still dominated by pre-cutover residue; live Rails calls after 22:13:05Z were US 7 Unit21, 5 transcription context + 5 writes, 1 expiry, EU 1 expiry, zero 5xx. Decision: #12577 remains necessary to prevent recurrence but is not a recovery prerequisite for 554172; transcription widening stays held until the fairness fix is reviewed and deployed and a fresh Rho canary is observed.

Unit21 was selected. Semantic inspection of Internal::OrganizationDataSourcesController#sync_unit21 / #sync_unit21_status, OrganizationDataSourceSyncRun and OrganizationDataSource::Sync::Unit21 shows a substantive, portable effect, not a no-op: API2 already owns scheduling (listUnit21SyncCandidates, one BullMQ job per source, pause_api2_unit21_sync Flipper switch) and only delegates the provider work to Rails, which claims a durable run (source-row lock, 30-minute lease, duplicate/in_progress/abandoned), then lists Unit21 cases and alerts (POST /v1/cases/list, POST /v1/alerts/list, page size 50, offset 1 advancing by response_count, created_after and dispositioned_after passes from the per-source incremental cursor minus a 5-minute buffer, 24-hour fallback), fetches missing rules (GET /v1/rules/:id), upserts compliance_cases, compliance_alerts, compliance_rules, compliance_events, compliance_narratives, case_rules, alert_rules, compliance_alert_cases with redacted content, and advances last_synced_at plus the two sync_data cursors.

The native package (ENG-3278 ) ports that whole flow into rulebase-web/rulebase-api2/src/lib/unit21-sync/native/ behind the organization-targeted PostHog flag api2_unit21_sync_native (owner platform, created 2026-09-19, review/remove by 2026-12-18, safe state off). runUnit21SourceSyncJob keeps the pause check, then resolves the source by exact (id, organization_id) and the flag; any lookup or flag failure, a missing source, or flag off keeps the existing Rails call, so Rails remains the rollback path with no code change. Native ownership reproduces the Rails claim (short transaction locking only the source row, run_key duplicate detection, 30-minute lease, stale-run abandonment, one retry on unique violation) and finishes the run as synced, skipped (missing_credentials, credentials decrypted from organization_data_sources.data in Rails order api_key, token, u21_key) or failed (error truncated to 500 characters). All provider and redaction calls happen outside database transactions and locks; persistence transactions per case/alert only run DB statements. Bounds: 30 s per provider request, 200 pages per list pass (non-retryable), BullMQ attempts: 3 unchanged; retryable provider (5xx/network/timeout) and redaction failures re-throw for retry, everything else is UnrecoverableError.

Intentional deviations from Rails, all fail-safe: (1) redaction failure fails closed and retries the attempt instead of persisting raw text; (2) the 200-page cap; (3) non-numeric action_time/created_at strings parse to null rather than Rails’ "abc".to_i epoch-0 1970-01-01. Verification: pnpm vitest run src/lib/unit21-sync (6 files, 44 tests incl. PGlite ownership, tenant isolation, pagination 50+3 at offsets 1/51, persistence and replay idempotency, empty-note redaction alignment, rule_id links, flag on/off/failure routing) and pnpm typecheck pass. No production write, replay, flag or provider change was made; the flag is not created. Expected removal when rolled out: about 144 Rails calls / day.

Native canary rollback at 2026-09-20T02:30Z (run-key collision, ENG-3293 )

#12571 shipped in reviewed release 017b86ef (API2 US/EU web 1/1, logical workers 5/5). PostHog api2_unit21_sync_native (ID 282001) was created at 2026-09-20T02:24:44Z for exact rho-tnkw only. The 02:30Z natural Rho tick selected engine=native and logged done, but it was a no-op: the worker’s durable key unit21-sync-source-53:1 collided with the already synced organization_data_source_sync_runs row id 1 from 2026-09-15T01:00:01Z, so the claim returned duplicate and no provider call, persistence or cursor advance happened (organization_data_sources 53 kept last_synced_at 2026-09-20T01:51:41Z; both native cursors stayed at 2026-09-15T01:00:01/02Z). The coordinator disabled the flag immediately (active=false, v2); the 02:40:00Z natural tick selected engine=rails and logged synced/done but added no run row (count still 1): the route selected Rails, yet it was a duplicate no-op and did not restore sync. Production mutations were limited to creating the exact-Rho PostHog canary and immediately disabling it on regression; no scheduler, Flipper, data, replay or provider change was made.

Root cause (not native-specific): since #12136 the worker derived the key as ${job.id}:${job.attemptsMade + 1} while the source job uses the deterministic id unit21-sync-source-<ods id> and addReplacingFinishedJob, so every scheduled replacement is attempt 1 with the identical key. Rails’ OrganizationDataSourceSyncRun.claim returns :duplicate for the same key, so the Rails-engine synced replies (including 02:40Z) were duplicate no-ops as well. Read-only production evidence (coordinator, 2026-09-20 ~02:45Z) confirms an active stale-sync incident for Rho source 53: compliance_cases 971 rows, max synced_at 2026-09-14T20:20:00.705Z (max updated_at 20:20:00.708Z, max provider_created_at 20:12:42Z); compliance_alerts 25,606 rows, max synced_at 2026-09-15T00:50:12.766Z (max updated_at 00:50:12.767Z, max provider_created_at 00:37:45Z); native cursors still Sep 15; exactly one durable run row. Unit21 data for this source has not advanced since #12136 shipped. The documented operational rollback (Flipper pause_api2_unit21_sync plus re-enabling the Rails sync_unit21 GoodJob cron) was deliberately not exercised: production scheduler/Flipper mutation is unauthorized and direct prod writes are prohibited; recovery is the reviewed/deployed fix, whose first tick claims a fresh run and syncs incrementally from the Sep 15 cursors. The fix (unit21SourceSyncRunKey, ${job.id}:${job.timestamp}:${attempt}, BullMQ’s creation timestamp being fixed per job instance and new per replacement) gives each scheduled replacement a distinct durable run for both engines while retries keep the per-attempt shape from #12136. Re-enable gate: fix reviewed and deployed, then one exact-Rho natural tick that adds a new run row and advances both cursors.

Recovery proof (read-only, 2026-09-20): #12606 was human-merged and release sync #12607 deployed b2f5094c fully across US/EU API2 web and workers (all desired, zero failed, exact SHA, no level>=50 first-slice logs). The 03:00Z tick with flag 282001 still off ran engine=rails under run key unit21-sync-source-53:1789873200232:1, created durable run id 2, hit the expected 504 then reconciled, and reached status synced at 03:02:10Z; compliance cases rose 971→984 and alerts 25,606→25,656. The operator then enabled flag 282001 v3 for exact Rho only. The 03:10Z tick ran engine=native, created run id 3 under unit21-sync-source-53:1789873800107:1, synced in 0.85s with no errors, and advanced both Unit21 cursors plus last_synced_at to 03:10:00.507Z. The stale-sync incident is resolved; the flag stays exact-Rho with no widening. Devin made no production mutation.

Webhook-triggered sync (ENG-3294)

Accepted Unit21 webhooks now enqueue a debounced source sync, coalescing bursts onto the deterministic per-source job and requesting an after-active successor when a delivery arrives after the active job fetched its data. Enqueue failures still return the provider-facing 200 and are surfaced through syncEnqueue=failed for alerting; the ten-minute scheduled tick remains the correctness backstop. PostHog flag 282001 is active v3 exact Rho with recovery proven (03:10Z run id 3); the 03:20Z exact-Rho native tick created distinct durable run id 4 under key unit21-sync-source-53:1789874400080:1, synced in 0.75s with no error, and advanced last_synced_at plus both cursors to 03:20:00.372Z. Rho is the sole non-demo Unit21 candidate, so exact-Rho is the complete meaningful production cohort; targeting remains unchanged with no percentage widening, and no production mutation was made by Devin. Live API2 remains b2f5094c in US/EU web 1/1 and workers 5/5, with zero pending or failed instances. Flag 282001 v3 exact Rho stayed active; the second consecutive natural native tick at 03:20Z completed in about 0.75s with no error. Since the 02:59:06Z deploy boundary, the only Unit21 Rails calls were the 03:00 recovery run (one sync_unit21 200 plus 15 status polls), with zero Unit21 Rails calls after flag enable at 03:02:52Z. Hold exact Rho with no widening; no production mutation was made by Devin. This package changes no flag or engine-selection behavior.

Deployment (2026-09-20): #12608  merged at ba712b982 and shipped through the human release sync #12612  as release-prod f990fc4999c47ccbffb60501b0b56d6b41c1f128; Flightcontrol marked the US and EU deployments successful at 03:44:40–42Z (API2 web TD1667/TD1685 1/1, workers TD1657/TD1674 5/5, exact FC_GIT_COMMIT_SHA=f990fc4999, all rollouts completed, zero pending). No natural native_unit21_webhook event has been observed yet, so the accelerator is not proven: the first accepted Rho webhook must show syncEnqueue followed by an immediate durable native run before any claim. Flag 282001 stays exact Rho; no merge, deploy trigger, or production mutation by Devin.

QA sampling 7 d Rails-zero gate check at 2026-09-21T20:10Z (not yet reachable)

Verdict: the 7 d zero-Rails-QA-sampling window cannot be complete yet. The last Rails Internal::ConversationTranscriptionReconciliationsController#create call recorded above is 2026-09-19T20:00:03Z (before the v21 100% write at 20:08:04Z), so the earliest instant a clean 7 d window can close is 2026-09-26T20:00:03Z. At this check about 48 h of the window have elapsed. Do not remove the Rails QA-sampling or reconciliation path before that date, and only after a fresh CloudWatch census confirms zero calls to both /internal/conversation_transcription_reconciliations and /internal/qa_sampling_recoveries in both regions across the full window.

What this session could and could not verify. No AWS credentials were available, so the CloudWatch census was not refreshed here; the 20:00:03Z last-call time is carried from the earlier authoritative census. As read-only corroboration, Sentry rulebase-api production http.server transactions (trailing 14 d) show the last sampled Internal::ConversationTranscriptionReconciliationsController#create at 2026-09-18T21:15Z and no sampled transaction since. Sentry traces are sampled and already missed the 2026-09-19 tail that CloudWatch recorded, so this is consistent with zero but is not gate evidence; the gate remains CloudWatch-only.

Correction of a false-positive dispatch. A brief measured 2,671 US / 1,234 EU reconciliation calls over a 72 h window and dispatched a native port of the endpoint as “the largest remaining /internal workload”. That window straddled the cutover and counted decaying pre-20:08Z volume; the package was already native and at zero. The port was stood down before any PR was opened; nothing merged. Note for the removal step: the rails ownership mode in rulebase-web/rulebase-api2/src/lib/qa-sampling-recovery/rails-run.ts still calls both Rails endpoints over HTTP, so a rollback of qa_sampling_recovery_native would resume Rails traffic and restart the 7 d clock. Removing the Rails path therefore also means removing that fallback mode, not only the Rails controllers.

QA/evaluation residual census at 2026-09-22T23:23Z and the churn-enum poison-row root cause

Fixed 24 h Rails /internal QA/evaluation residuals (CloudWatch, operator census ending 2026-09-22T23:23Z). Expired-evaluation process_all 24 US / 0 EU; transcription reconciliation 3 US / 1 EU; daily report backfill_preview/email 4 US / 1 EU; evaluation-check mark_failed 2 US (one HTTP 422); QA execute_stage 0. API2 release 05911a9b was live in both regions; Rails 3/3.

Classification (read-only; PostHog project 63353 flag inventory, Sentry rulebase-api errors, source on origin/main 3f6908c1).

• Expired evaluations, 24 US = exactly one call per hourly tick. runExpiredEvaluationCleanup only calls Rails when the native sweep deferred stale rows for an organization outside expired_evaluation_cleanup_native (100% for easypay, rho-tnkw, nala; 50% general). Rails-side QA evaluation request failed (...) events from Internal::ExpiredEvaluationsController#process_all stop at 2026-09-19T22:05Z (Sentry errors are unsampled), so the hourly Rails call has failed zero QA requests for 72 h (RULEBASE-API-5S4 last seen 22:05:00Z, RULEBASE-API-5WM last seen 21:05:00Z, both 2026-09-19). Inference, not yet confirmed by a row-level read: the native sweep is re-deferring the same stale snapshot rows each tick and Rails’ ExpiredEvaluationsFailureMarker rescues (Rails.logger.error, no Sentry event) the same RecordInvalid on them. Classified as poison-row residue, not live work, with the root cause below; a CloudWatch read of the hourly processed expired evaluations log line (native.deferredToRails, rails.snapshots) confirms or refutes it.
• Evaluation-check mark_failed, 2 US, one 422. The 422 is RULEBASE-API-5TJ : ActiveRecord::RecordInvalid: Churn risk level is not included in the list raised by ConversationCxRiskSnapshot#mark_as_failed! → update!(processing_status: :failed) for POST /internal/evaluation_check_runs/7040287/mark_failed at 2026-09-21T23:50:07Z (6 occurrences since 2026-09-07). The native native_evaluation_check_run_mark_failed flag does not exist in the PostHog project, so this family is 100% Rails by fail-closed default; the non-422 call is real fallback work, the 422 is the same poison row.
• Root cause (this package, ENG-3373 ): the native customer-issue check (rulebase-web/rulebase-api2/src/lib/evaluation-checks/customer-issue.ts) wrote conversation_cx_risk_snapshots.churn_risk_level with the conversation-risk enum (none 1, low 2, medium 3, high 4) instead of the Rails snapshot enum { low: 0, medium: 1, high: 2 } (validate: true). Every native customer-issue write therefore stored either the wrong level (low → 2 = high, none → 1 = medium) or an out-of-range value (medium → 3, high → 4). Completed snapshots are healed because conversation-cx-risk-completion.ts rewrites the column correctly, but a snapshot that stalls after the customer-issue check keeps the invalid value and then every Rails update! on it raises: Rails mark_failed returns 422, and the Rails expired-evaluation marker rescues and logs the same RecordInvalid each hour, so the row never leaves the stale scope. The fix writes the Rails enum (blank score → low, matching EvaluationCheck::Checks::CustomerIssue#calculate_churn_risk_level); the bodies.db.test.ts expectation that had codified 3 for a 0.6 score was corrected to 1. No data repair is performed by the PR.
• Transcription reconciliation, 3 US / 1 EU. qa_sampling_recovery_native is 100% for all organizations; rails-run.ts calls the reconciliation endpoint once per organization tick whenever the owner resolves to rails (PostHog unavailable/timeout or missing slug fails closed). These are fail-closed fallback residue, but they are Rails calls inside the 7 d window: the gate above is reset. The earliest clean close is now 7 d after the last of these calls, i.e. no earlier than 2026-09-28T23:23Z and the exact instant must be re-read from CloudWatch. Do not remove the Rails path or rails-run.ts before then.
• Daily report preview/email, 4 US / 1 EU. api2_daily_agent_evaluation_report_delivery does not exist in the PostHog project (only api2_daily_agent_evaluation_report_collect does), so backfill preview and email delivery are 100% Rails by fail-closed default. These are live fallback work, not residue; the native modules are complete but unrolled.
• execute_stage 0. Retired; no action.

Operational canaries that are now safe (operator action, not in the PR).

1. expired_evaluation_cleanup_native → 100% in US (EU already shows 0 Rails calls). The native sweep fails stale snapshots with raw SQL, so it drains the poisoned rows Rails cannot; expected effect is Rails process_all dropping from 24/day to 0 within one hourly tick. Rollback: unset the 100% condition (fail-closed returns to Rails) or enable pause_api2_expired_evaluation_cleanup; rollback criteria are any runExpiredEvaluationCleanupJob error, snapshots.failed/requests.failed diverging from candidates for owned organizations across two ticks, or any new Sentry rulebase-api2 error under expired-evaluations.
2. After this PR ships, a read-only census SELECT id, organization_id, processing_status, churn_risk_level FROM conversation_cx_risk_snapshots WHERE churn_risk_level NOT IN (0,1,2) bounds the remaining poison rows; a guarded repair mapping 3→1, 4→2, 2→0 (if customer_churn_risk_score < 0.4), 1→0 (if score IS NULL) is a separate, explicitly approved package.
3. Creating api2_daily_agent_evaluation_report_delivery in PostHog (off, then org-targeted) is the prerequisite for retiring the daily-report family; until then it is 100% Rails by design. The native_evaluation_check_run_mark_failed prerequisite is superseded: the ENG-3400  package below bakes the native transition in behind a Flipper kill switch instead.

Remaining-internal-effects census at 2026-09-22T23:23Z and the native mark_failed package (ENG-3372 )

The coordinator’s fresh 24 h census (ending 2026-09-22T23:23Z, API2 release-prod 05911a9b in both regions, Rails 3/3) left these non-QA API2-to-Rails internal effects: counter-cache fix 9 US / 7 EU, conversation-upload mark_failed 6 US, conversation-field-value upsert 4 US, Slack notification create 1 US, Unit21 sync 1 US. Read-only classification (PostHog $feature_flag_called, Sentry, code on origin/main 3f6908c1a; no AWS/CloudWatch or database access in this session):

Package chosen: native terminal failure marking for natively processed direct uploads (this is the only residual whose native owner already runs the attempt and whose remaining Rails call is a pure persistence step Rails itself performs with update!). Code (rulebase-web/rulebase-api2/src/lib/conversation-uploads/process.ts, native-processing/run.ts):

• markNativeDirectUploadFailed performs the exact ConversationUpload#mark_direct_upload_failed! write natively — status='failed', metadata.last_processing_error, metadata.last_processing_error_at (ISO8601), updated_at — scoped by id and organization_id, returning whether the row still exists. The periodic retry (isRetryableIncomplete, and Rails’ own retry scope) keys off status='failed' plus the retry counters, so natively marked rows are picked up exactly like Rails-marked ones.
• failUpload calls it only when the native processor ran the attempt (nativeEnabled for that upload); Rails-processed uploads still go through markConversationUploadFailed → mark_failed unchanged. A missing row on the native path drops the job like the Rails 404 path. The Rails endpoint, client adapter and controller are untouched (rollback = the existing organization flag api2_conversation_upload_native_processing off, which routes both processing and marking back to Rails immediately).
• Sentry annotation parity: Rails’ report_to_sentry: false branch only annotated the enclosing API2 event; API2’s reportTerminalFailure now receives source, organization_slug and conversation_upload_id from the native upload row so tags and fingerprint match the Rails-path events.
• classifyNativeProcessingError unwraps Drizzle’s cause so retryability uses the Postgres SQLSTATE (08xxx, 53xxx, 40001, 40P01, 57014, 57P0x, 55P03, 25P02 retry; constraint classes remain terminal) and the persisted/reported message names the real error (DatabaseError:<sqlstate> error class). The six RULEBASE-API2-FH events would have carried their SQLSTATE and message instead of Failed query.

Not in this package: the root cause of the conversation_parts lookup failure itself (needs the SQLSTATE the fix above will now surface, or a database read), the field-value flag creation, the Unit21 widening, the counter-cache Rails-path removal, and the Slack sender.

Gates and rollback for this package. Deploy with no flag change. After one full US day: Internal::ConversationUploadsController#mark_failed count for organizations enabled on api2_conversation_upload_native_processing must be 0, conversation_uploads rows failed natively must carry both last_processing_error* keys, and the periodic retry must still pick them up (periodic_retry_count increments). Roll back (disable the organization on api2_conversation_upload_native_processing, no deploy) if any natively processed failure lands with status <> 'failed' and no delivery-ledger failed row, if RULEBASE-API2-FH-class terminal events per day exceed the pre-package 6, or if a DatabaseError:* classified retryable retries past the BullMQ attempt budget for more than 3 uploads in a day. Separately and only as an operational canary (not code): a single-region counter-cache parity check — one 3 h tick, zero Internal::CounterCachesController calls in that region, native fixed/unchanged counters matching the Rails audit — is the next gate for removing the counter-cache Rails path; rollback for that canary is pause_api2_counter_cache_repairs on (Flipper) or the PostHog flag off, threshold any counter drift on the audited tables.

Remaining-internal-effects census at 2026-09-23T01:20:02Z and the Kuda expired-evaluation cutover (ENG-3376 )

Census (exact trailing 24 h ending 2026-09-23T01:20:02Z, supplied by the operator, not re-derived here). Rails /internal effects US 43 + EU 2 = 45. US: expired-evaluation sweep 24, upload mark_failed 6, field-value upsert 4, daily-report actions 4, reconciliation 3, Unit21 1, Slack 1. EU: reconciliation 1, report 1. Counter-cache repair and evaluation-check mark_failed have rolled out of the window. #12821 (native upload mark_failed) deployed in both regions at 00:56Z; the first post-deploy slice 00:57Z–01:20Z holds exactly one US expiry sweep and zero other internal calls. QA enum fix #12822 remains open (Ronald review pending) and is not part of this package.

Classification of the residual families.

• Expired-evaluation sweep (24 US, 53% of the total) — flag-targeting hold, not live Rails-only work. Flag 282005 expired_evaluation_cleanup_native had sat at v2 (exact easypay, rho-tnkw, nala + 50% organization group) since 2026-09-20T04:06:20.993Z with no further ladder step. PostHog $feature_flag_called for the flag over the trailing 48 h (read-only HogQL, 2026-09-23T01:2xZ): the only false evaluations are kuda-pilot, one per hourly tick where Kuda had stale rows (15 deduped rows from 2026-09-21T22Z through 2026-09-23T01Z), plus a single true for rho-tnkw at 2026-09-22T01Z; no other organization evaluated the flag at all. Because ownsOrganization is only consulted for organizations that actually have stale rows, every remaining US process_all call is a Kuda deferral (Kuda is outside the 50% bucket), and EU had zero sweep calls in the window. Outcome side: backend_workflow_stage qa_evaluation/request_completed/failure shows zero Rails- or API2-recorded Kuda request timeouts since 2026-09-19 (Rails recorded 209 + 724 Kuda timeouts on 09-18/09-19, before the flag existed) and zero API2 reason=timeout events anywhere in 7 days, so the hourly Kuda deferrals are stale conversation_cx_risk_snapshots rows and/or a row the Rails marker rescues without transitioning — not a stream of fresh request failures. Row identity is unverified without a production read.
• Upload mark_failed (6 US) — owned by #12821; its 24 h clock starts 00:56Z.
• Field-value upsert (4 US) — unchanged flag/configuration hold (api2_conversation_field_value_native_upsert still not created); smaller than the sweep and needs a human flag-creation decision, so not taken here.
• Reconciliation (3 US + 1 EU) and reports (4 US + 1 EU) — sub-daily tails with no new outcome census in this slice; left as is.
• Unit21 (1 US) — intentional fallback outside the exact canary. Slack (1 US) — unattributed, too small to package.

Bounded activation (this automation, reversible, no data write). Flag 282005 advanced to v3 at 2026-09-23T01:32:23.151665Z: exact easypay, rho-tnkw, nala, kuda-pilot at 100% plus the unchanged 50% organization group; server-only, active. Basis: the native snapshot marker was reconciled row-for-row on the 2026-09-20T04:05Z tick (US 4/4, EU 4/4), the native request marker shares failNativeRequest with the live native QA paths and is covered by the PGlite parity suite, and Kuda is the single organization whose deferral produces the whole remaining Rails sweep volume. Rollback: one PostHog write back to the v2 definition (remove kuda-pilot from the exact list); the same tick’s stale Kuda rows then defer to Rails again with no replay needed. Second layer: Flipper pause_api2_expired_evaluation_cleanup on plus re-enabling the two retained Rails GoodJob crons.

Focused code package (#12826 , review-required, unmerged). runNativeExpiredEvaluationSweep now returns deferredOrganizations: [{ slug, snapshots, requests }] (sorted by slug) alongside the aggregate deferredToRails, and the per-row error logs carry organizationId. The hourly processed expired evaluations and QA evaluation requests log line therefore names exactly which organizations forced the Rails round-trip, so the next census reads attribution from the job log instead of inferring it from deduplicated $feature_flag_called rows. No change to selection scopes, row transitions, lock handling or the Rails fallback decision.

Gates before any retirement claim. Next natural :05Z ticks: Kuda handled natively (native.snapshots/native.requests > 0 when stale rows exist), deferredToRails = {0, 0}, deferredOrganizations = [] once the PR is deployed, zero per-row Error marking expired … logs, and zero Internal::ExpiredEvaluationsController#process_all calls in US. Then the 24 h zero window from 01:32:23Z and the 7-day zero window in both regions before the Rails markers, controller and crons are removed. Widening the 50% organization group to all organizations is a separate, human-approved ladder step; this slice made no other flag, scheduler, deploy or data change.

Churn-enum poison rows: Rails failure-marking hardening and guarded repair task (ENG-3377 , follow-up to ENG-3373 / #12822)

State at 2026-09-23T01:50Z (operator-supplied, not re-derived). #12822 merged at 2026-09-23T01:49:31Z; release-prod d6607a6 was not yet deployed at that time. The operator later reported the d6607a6 API2 deployment live by ~02:08Z with one clean expiry cycle (see census below); that is the only post-deploy observation recorded here. #12825 and #12826 merged without a recorded Ronald review; nothing here depends on them and neither was flagged, deployed or bypassed.

Evidence (read-only).

• Sentry RULEBASE-API-5TJ (rulebase-api, US/Ashburn): ActiveRecord::RecordInvalid: Validation failed: Churn risk level is not included in the list, culprit Internal::EvaluationCheckRunsController#mark_failed, 6 occurrences, first seen 2026-09-07T02:35:44Z, last seen 2026-09-21T23:50:07Z. The latest event (ead16a3476634197be8847c235162098) is POST /internal/evaluation_check_runs/7040287/mark_failed; an earlier one at 2026-09-21T21:07:45Z is run 7038988. The stack ends in ConversationCxRiskSnapshot#mark_as_failed! → update!, i.e. the check itself was already flipped to failed and only the snapshot transition rejected. These are the API2→Rails mark_failed 422s counted in the 2026-09-22T23:23Z census above.

• Hourly ExpiredEvaluationsFailureMarker (process_all) hits the same update!, rescues per row and logs Error marking expired evaluation as failed with the snapshot prefix id, so a poisoned pending/running row stays in the stale scope and is re-attempted every tick until a native sweep (failSnapshot, raw SQL, enum-agnostic) owns the organization.

• Row-level US/EU census (operator, read-only via ECS Exec, 2026-09-23 ~02:08Z, after the d6607a6 API2 deployment; no writes or replay). conversation_cx_risk_snapshots WHERE churn_risk_level NOT IN (0,1,2):

  • US: 6 rows, all raw level 3 — org 38: 1676207, 1676266, 1676326; org 19: 1676121, 1800561; org 25: 1800129.
  • EU: 4 rows, all raw level 3 — org 41: 516999, 517000, 517001, 517002.
  • No raw level 4 row in either region. Under the pre-#12822 writer 3 was medium in the conversation-risk enum, so every poison row is a stalled medium-scored snapshot; the repair task’s score-based recomputation is expected to yield 1 (medium) for each, to be confirmed per row by the dry run.
  • processing_status, customer_churn_risk_score and the check correlation (query 2 below) were not part of the pasted census; run them before the repair decision and record them in ENG-3377.
  • Same operator read, 02:05Z natural expiry cycle on the deployed release: US one candidate / one native failure, EU zero candidates, deferredToRails = 0, deferredOrganizations = [], rails = null in both regions. One clean cycle — not a 24 h or 7 d zero-window claim.

  Census/correlation SQL for the follow-up reads (bin/aws/ecs-shell.sh api2-prod-{us,eu}, see .agents/skills/rulebase-prod-debug/references/prod-notes.md):

  -- 1. bound the poison set
  SELECT s.id, o.slug, s.processing_status, s.churn_risk_level,
         s.customer_churn_risk_score, s.created_at, s.updated_at
  FROM conversation_cx_risk_snapshots s
  JOIN organizations o ON o.id = s.organization_id
  WHERE s.churn_risk_level NOT IN (0, 1, 2)
  ORDER BY s.created_at;
   
  -- 2. correlate with stuck checks / the Sentry runs
  SELECT c.id, c.type, c.status, c.updated_at, c.conversation_cx_risk_snapshot_id
  FROM evaluation_checks c
  JOIN conversation_cx_risk_snapshots s ON s.id = c.conversation_cx_risk_snapshot_id
  WHERE s.churn_risk_level NOT IN (0, 1, 2) OR c.id IN (7040287, 7038988)
  ORDER BY c.id;

  Expected shape from the code path: rows with churn_risk_level IN (3, 4) whose processing_status is running/pending/failed (never completed, because completion rewrites the level), all created before the #12822 deploy.

Focused code package (review-required, no data write, no flag).

• ConversationCxRiskSnapshot#mark_as_failed! writes the terminal status with update_columns(processing_status: :failed, updated_at:) instead of update!. The transition no longer validates unrelated columns, so both EvaluationCheck#mark_as_failed! (the mark_failed 422 family) and ExpiredEvaluationsFailureMarker (the hourly retry loop) drain poisoned rows exactly like the native sweep does. The conversation evaluation_state update is unchanged.
• Maintenance::RepairInvalidSnapshotChurnRiskLevelsTask (maintenance_tasks UI): organization_slug (required), max_snapshots (default 500, fails instead of truncating), apply (default false = log-only). Collection is churn_risk_level NOT IN (0,1,2) for that organization; process takes a row lock, re-checks the stored value, recomputes the level from the row’s own customer_churn_risk_score with ConversationCxRiskSnapshot.churn_risk_level_for_score (≥0.7 → high, ≥0.4 → medium, else/NULL → low) and writes only churn_risk_level. It never touches processing_status, checks, requests or conversations. Running it in production is a separate, explicitly approved step per organization, dry-run first.
• Specs: poisoned-row mark_as_failed! on snapshot and check; task collection scoping/bounding, dry-run no-op, score-based repair, idempotence.

Rollback. Revert the PR (single Rails deploy). The model change is behaviour-preserving for valid rows; reverting only reintroduces the 422s. The task is inert unless an operator runs it with apply=true; a mistaken run is reversible per row from the task log (stored_level → repaired_level is logged for every row).

Gates.

• Outcome (first deploy tick): Internal::EvaluationCheckRunsController#mark_failed returns 200 for the runs Sentry listed (or they are already terminal); RULEBASE-API-5TJ receives no new event; the next :05Z Rails process_all tick logs zero Error marking expired evaluation as failed.
• 24 h: zero new RULEBASE-API-5TJ events in US and EU; census query 1 returns no pending/running rows (remaining poison rows, if any, are failed and inert); no new snapshot.failed divergence in the native sweep.
• 7 d: census query 1 returns zero rows in both regions after the approved per-organization repair, zero mark_failed 422s, and the Rails mark_failed family can then be retired on the ENG-3372 schedule. Until then the Rails endpoint stays.

Read-only ECS correlation at ~2026-09-23T02:24Z (coordinator-supplied; no repair run, no data write). All ten churn_risk_level = 3 snapshots are already processing_status = failed, so the invalid enum values are historical failed-row data quality, not currently stuck work, and are not a reason to run the repair task urgently. Any repair stays a separate, guarded operator decision.

Each snapshot has three evaluation_checks: 28 completed and two failed complaint checks — check 7038988 (Kuda snapshot 1800129) and 7040287 (Rho snapshot 1800561). No pending/running poison row exists in either region, which satisfies the 24 h gate’s “remaining poison rows are failed and inert” condition for this read.

Native evaluation-check body ran model/embedding calls inside the DB transaction and advisory lock: two-phase fix (ENG-3414 , follow-up to ENG-3400 / #12891)

Finding (static, confirmed on origin/main 801d6b7ac against the root AGENTS.md “Database lock safety” rule). rulebase-web/rulebase-api2/src/lib/evaluation-checks/index.ts tryRunNativeEvaluationCheck opened a single db.transaction, took pg_try_advisory_xact_lock on the Rails execution_lock_key twin, selected the check FOR UPDATE, set status = 'running', and then invoked bodies[checkType](tx, target) on that same transaction client. Every body awaits provider work on that client: customer-issue generateText and OpenAI embed, complaint detection and follow-up generateText, compliance generateText. The transaction, the row lock, and the transaction-scoped advisory lock were therefore held for the full duration of the model calls. Rails does not do this: EvaluationCheck#run_once wraps run in a session-level with_advisory_lock(execution_lock_key, timeout_seconds: 0) on a dedicated connection with no transaction open.

Production evidence (read-only, Sentry only). RULEBASE-API2-FA: Error: Failed query: rollback, culprit Object.tryRunNativeEvaluationCheck [as runNativeCheck], inner cause Client has encountered a connection error and is not queryable, worker evaluation-check-run, attempts_made 5 / max_attempts 5, 25 occurrences from 2026-09-21T13:34:55Z to 2026-09-23T13:39:27Z, release dd914087, us-east-1. The final event matches the 13:39Z Rails mark_failed call in the fixed 19:30Z census (ENG-3400). Attribution of the other fourteen mark_failed calls to this issue is inference, not per-event evidence. No CloudWatch read was attempted: both operator AWS SSO profiles expired at 2026-09-24T01:35Z, so no fresh traffic-zero claim is made here.

Coordinator census (exact, read-only, trailing 24h to 2026-09-24T07:40:00Z). US Rails /internal 19 successful calls, of which EvaluationCheckRuns#mark_failed 15 (latest at 2026-09-23T13:39Z, the same final event as the 19:30Z audit; no newer mark_failed call); EU Rails /internal 0. Both regions: zero successful API2-gateway forwards to Rails and zero successful direct /conversation_images/:signed_id GETs. These numbers were supplied by the coordinator’s read-only query; this track did not re-run CloudWatch. #12891 (ENG-3400) and #12912 (ENG-3414) were open, CI-green, labeled codex/codex-automation, Ronald requested, with no submitted Ronald approval; neither merged nor deployed.

Package (this section’s PR). Two-phase execution, keeping Rails serialization semantics:

1. withRailsCompatibleExecutionLock takes a session-level pg_try_advisory_lock on a dedicated pool connection with the same [crc32("evaluation-check-run-<id>") & 0x7fffffff, 0] key Rails and mark_as_failed_once use, and releases it in finally (same shape as expired-evaluations/sweep-lock.ts). Contention returns already-running without touching the row. The transaction-scoped acquireRailsCompatibleExecutionLock stays for the terminal-failure path. If the acquisition query itself throws (the Sentry failure mode is a non-queryable client), the connection is released with the error so pg-pool destroys it instead of handing it out again.
2. Claim phase: a short transaction re-reads the check FOR UPDATE, returns the terminal/not-applicable short-circuits unchanged, sets status = 'running', and commits. loadTarget, loadStatus, isApplicable and terminalPostCommitResult are SQL reads only (audited 2026-09-24); no provider, queue, Redis or HTTP work runs inside the claim.
3. Body phase: the body receives the autocommit db client, so no transaction, row lock, or transaction-scoped advisory lock is open while generateText/embed run. Each body’s write phase opens its own short transaction (customer-issue: issue insert + snapshot update + completion; complaint: create, then routeCreated in a separate transaction because the routing pass takes a pg_advisory_xact_lock (a no-op on an autocommit client), or follow-up link; compliance: results + completion), still tenant-scoped by organization_id. Known pre-existing debt, out of scope here: conversation-routing/notify.ts posts to Slack inside that routing transaction (shared with every other routing caller).
4. Post-commit effects (completion and customer-issue assignment enqueues) run after the lock is released, as before.

Behavior change to review. A body exception no longer rolls status back from running to pending; the check stays running (Rails parity) until the terminal-failure job (fail-evaluation-check-run, ENG-3400 path) applies failed. BullMQ retries of the same job take the session lock again and re-run the body from a fresh claim.

Tests (src/lib/evaluation-checks/index.db.test.ts, PGlite). The body observes txid_current_if_assigned() IS NULL, status = 'running', and the advisory lock held in pg_locks; the lock is released after success and after a body throw; a held lock yields already-running with the row untouched and the pooled connection released; a failing acquisition query rejects the run, leaves the row pending, and releases the connection with that error (fault-injection); the pre-existing flag-off, not-applicable and completion-enqueue cases are unchanged. Body DB tests in bodies.db.test.ts run unchanged on the transactional client.

Gates. Land after Ronald review only; no flag, provider, deploy, replay or production data change in the PR. Post-deploy read-only check: Sentry RULEBASE-API2-FA receives no new events across one full US business day and the evaluation-check-run exhaustion count returns to zero; the ENG-3400 mark_failed census then measures real terminal failures only. Rollback is revert of the carrying commit; there is no flag.

Native evaluation-check execution owned by default; PostHog fail-closed Rails run fallback retired (ENG-3453 , follow-up to ENG-3400 / ENG-3414 / ENG-3435)

Dependency. On the deployed code evaluation-checks/index.ts returned legacy — i.e. evaluation-check-run.ts posted to Rails POST /internal/evaluation_check_runs/:id/run (EvaluationCheck#run_once, body executed by Rails) — whenever (a) the per-type PostHog flag native_complaint_evaluation_check / native_customer_issue_evaluation_check / native_compliance_evaluation_check read anything other than true, including a PostHog error or null response (fail-closed), (b) the check row was missing for the organization, or (c) the type was outside the three native bodies. Per rollouts all three flags have been 100% since the gate passed on 2026-09-08 22:01Z, so (a) is the same latent shape as the QA sampling posthog_unavailable fallback fixed in ENG-3435 / #13003: a transient PostHog failure under a 100% flag silently selects Rails. No run call appeared in the 2026-09-22T23:23Z, 2026-09-24T07:40Z or 2026-09-25T02:15Z censuses already recorded here; this is a latent-dependency removal, not a live residual, and no new production evidence is claimed for it.

Decision and package. Bake the winning behavior in: native execution is the production default and the PostHog read is removed. Rails runs the check only while Flipper pause_native_evaluation_check_run is enabled or unreadable (logged ownerReason=paused|flipper_unavailable, fail-closed like pause_native_evaluation_check_run_mark_failed; owner Platform, created 2026-09-25, review by 2026-12-24, removed with the Rails endpoint). A missing check now returns missing natively — the Rails endpoint answered missing too — with no Rails call. A type Rails never creates (qa/dispute/sla; ConversationCxRiskSnapshot creates exactly complaint/compliance/ customer_issue) throws so the job fails the way Rails’ check_class constant lookup would, instead of round-tripping to Rails. rails-client.ts #runRailsEvaluationCheck and Internal::EvaluationCheckRunsController#run stay as the rollback path. The three PostHog flags are left for operator archival after deploy; no flag was created or changed.

Tests (index.db.test.ts, PGlite). Flipper switch read (off / boolean gate on / unreadable → "unavailable"); paused and unavailable both yield legacy with the row untouched; default run is native with one switch read and no PostHog; missing id and wrong organization yield missing without a body call; sla type rejects with the row left pending. Lock, fault injection, not-applicable and completion cases unchanged.

Rollback / gates. Rollback is enabling the Flipper switch (no deploy). Zero clock for evaluation_check_runs#run starts when a SHA containing this package is COMPLETED on all four API2 services; require exact regional 24 h then 7 d censuses with zero run calls, accounting for the remaining Rails selectors (switch on, Flipper unreadable). Land only after Ronald review and green required checks; codex + codex-automation labels.

QA/evaluation Rails internal-call closure matrix (2026-09-25 source audit, ENG-3465 )

Source audit of every remaining API2→Rails call in the QA/evaluation family on main at the #13098 merge. Production evidence is only what is already recorded above (2026-09-22T23:23Z, 2026-09-24T07:40Z / 15:40Z / 19:40Z, 2026-09-25T02:15Z censuses); no new counts and no post-rollout zero window are claimed. #13098 is merged but no census taken after a SHA containing it was COMPLETED on all four services is recorded here, so its deployed behavior is unverified.

Result/finalization package (this section). conversation-qa-result-import-finalization.ts returned legacy (Rails results) when native_conversation_qa_result_import read anything but true, including a PostHog error or null, and recorded completion_owner='rails' (Rails result_effects) when native_conversation_qa_completion did the same. Both flags have been 100% in both regions since 2026-09-08 22:01Z (rollouts), so this is the same latent fail-closed shape as ENG-3435 / ENG-3453. qa-evaluation-result-finalization.ts also still posted slug-less jobs to Rails: every producer has carried organizationSlug since #11219 (2026-09-06) and finalization jobs stop after 8 attempts with 30 s exponential backoff, so no slug-less job can remain queued. Decision: native import and completion are the production default; Rails takes a newly accepted run only while Flipper pause_native_conversation_qa_result_import is enabled or unreadable (ownerReason=paused|flipper_unavailable; Platform, created 2026-09-25, review by 2026-12-24, removed with the Rails routes). Existing completion_owner='rails' markers keep replaying to Rails result_effects; a slug-less payload fails the job instead of guessing ownership. postEvaluationResults / postEvaluationResultEffects and the Rails controller stay as rollback. The two PostHog flags are left for operator archival after deploy; no flag was created or changed.

Tests. conversation-qa-result-import-finalization.db.test.ts (PGlite): switch read off/on/unreadable; paused and unavailable yield legacy with the result claim untouched; default run imports, completes natively with one switch read, and replays durably; a persisted Rails marker replays as native-replay-rails-completion; terminal failure/skip, cross-tenant and conflicting runs, and rollback on import failure unchanged. qa-evaluation-result-finalization.test.ts: slug-less payload rejects before either writer; legacy still reaches the Rails writer after hash verification.

Remaining gates before the QA family can lose its Rails routes. (1) A SHA containing #13098 and this package COMPLETED on all four API2 services, then exact regional 24 h and 7 d censuses with zero calls on every row above, with the pause switches off and Flipper readable throughout. (2) One natural EU import → requests created natively → one recap delivery claim + Resend message. (3) execute_stage and finalize_failure compatibility branches resolved by their own reviewed packages. (4) Expired-evaluation cleanup poison-row repair and canary (ENG-3373). (5) public.* ownership G2–G7 remain independent shutdown blockers.

QA execute_stage / finalize_failure selector audit (ENG-3471 , child of ENG-3347 )

Status of the previous package. #13120 (ENG-3465) was merged to main (092227761) and synced to release-prod by a human merge with no submitted Ronald review. No census taken after a SHA containing it was COMPLETED on all four API2 services is recorded in this session, so its deployed behavior and any post-deploy zero window remain unverified. Nothing in this section adds production counts; the evidence column below repeats only what is already recorded above.

Re-scope after Rails-web-dark (#13143). main now contains #13143  (ENG-3347, Rails residual web paths API2-only) and the requester reports Rails configured at 0..0 tasks. That configuration and its deployment are unverified in this session (no ECS or census read is available here). With no Rails web target, any surviving API2 → Rails selector is a hard failure rather than a fallback, so this package no longer keeps any execute_stage reason on Rails: the first revision’s “keep on Rails” rows below were dropped, the direct execute_stage executor and rails-fallback.ts are deleted, and preparation-effects.ts leaves the RAILS_INTERNAL_BASE_URL source-gate allowlist. The web-dark gate retired-rails-web-routes.source-gate.test.ts gains the /execute_stage and /finalize_failure fragments and asserts the QA stage modules never reference the Rails origin, so neither selector can be restored without editing the gate.

Source audit of execute_stage on main. Rails Internal::QAEvaluationRequestStagesController#execute_stage accepted a call only while the live api2_stage_lease token and stage matched, then ran StageProcessor#execute_selected_stage! with transcription_owner: :api2, evaluation_owner: :api2 inside request.with_lock. API2 selected it through five RAILS_STAGE_FALLBACK_REASONS; the revised decision per reason:

Retired shapes are classified retired_request_shape by classifyNonRetryableQAEvaluationRequestError, so the job discards instead of burning retries and failNativeRequest records the terminal failure (api2_terminal_failure.retryable = false) the same way as provider_invalid_json.

Source audit of finalize_failure. Rails NativeFailureEffects.deliver! finalized failed agent rows, cleared qa_evaluated_at, released queue and sampling slots, mailed the requester (never for imports), enqueued backfill replacement/auto-backdate and broadcast. API2 failure-effects.ts already delivered every ordinary failed/skipped effect natively (same slot release, timestamp clearing, email, backfill dispatch, broadcast) and posted to Rails only when its native context query returned no row, i.e. when the request had evaluation_request_import_id set, was not yet failed/skipped, or its conversation/organization row was missing. Decision: imported requests are delivered natively with the requester email suppressed (Rails parity); a nonterminal or missing-row effect stays in the outbox and retries on the existing 5-minute schedule without a Rails call (Rails raised ArgumentError/404 for the same rows). The finalizeThroughRails client and the failure-effects.ts entry in the RAILS_INTERNAL_BASE_URL source gate are removed. Native no_agents_sampled skips write the same failure-effect row, so Rails mark_skipped! parity (slot release, workflow broadcast, requester notification, backfill replacement) comes from that outbox. Known pre-existing gap, unchanged by this package and out of its scope: neither owner path recomputed QAAgentEvaluation#finalize_failure! side effects for failed_agent_ids on natively delivered effects, because API2 never routed those rows to Rails.

Tests. eligibility-stage.db.test.ts / evaluation-creation-stage.db.test.ts: unsupported trigger, EasyPay historical and scheduled backfill-run-agent rows reject with RetiredQaRequestShapeError before any agent lookup; no live agents skips natively as no_agents_sampled with one failure-effect row both with and without ticket rules. preparation-effects.test.ts: retired shapes propagate with no fetch and no qa_stage_rails_fallback telemetry; workflow telemetry records outcome: "failure", error_class: "RetiredQaRequestShapeError". errors.test.ts: retired shape → retired_request_shape. failure-effects.db.test.ts: imported skip delivers natively with broadcast and no email; a nonterminal request leaves the effect pending with no broadcast, email or Rails call. Source gates: preparation-effects.ts removed from the RAILS_INTERNAL_BASE_URL allowlist; /execute_stage and /finalize_failure added to the retired web-route fragments.

Rollback. Code revert only (no flag: with Rails web-dark there is no Rails target to fall back to, and the removed branches were error or parity paths, not traffic). Rails execute_stage / finalize_failure routes are deletable with the QA family once gate (3) passes.

Gates. (1) Ronald review and green required checks on the package PR; no merge before both. (2) A SHA containing #13098, #13120, #13143 and this package COMPLETED on all four API2 services, recorded with its completion time — unverified. (3) From that time, exact regional 24 h then 7 d censuses (or, with Rails at 0..0, zero API2 RailsInternalApiError / connection failures for the QA family) with execute_stage = 0 and finalize_failure = 0 — unverified; the earliest 7-day Rails-zero gate the requester quoted is 2026-09-26T20:00:03Z and is not claimed here. (4) Any retired_request_shape terminal failure in production identifies a row this audit believed impossible and reopens the decision for that reason.

QA result-import terminal effects audit (ENG-3494 , child of ENG-3347 )

2026-10-01 status correction. #13220  merged 2026-09-28T20:53:46Z as 06c02fa1371d08077f5bf5edbbb762dcc4c58404; its final-head CI passed, but GitHub still reports REVIEW_REQUIRED and zero submitted reviews. This is a review exception, not Ronald approval. Deployment and natural QA failure-effects parity were not reverified in this correction, so the outcome gate below remains open.

Status of the previous package. #13132  (ENG-3471) was merged to main as 7c39ee573 at 2026-09-27T09:56:02Z by chidiwilliams with zero submitted GitHub reviews (verified against the GitHub API from this session). That merge is recorded as an external merge, not as Ronald approval; gate (1) of the ENG-3471 section was not met before merge. Gates (2) and (3) — a SHA containing #13098/#13120/#13132 COMPLETED on all four API2 services, and the regional 24 h / 7 d zero windows — remain unverified: this session has no ECS, CloudWatch or production DB read, and no census after the 2026-09-25T02:15Z one is recorded here. origin/main at audit time was 6087ec7dc.

Natural EU import → recap outcome. Still unproven. The only read-only evidence recorded in this plan is the requester’s 02:15Z investigation: zero new api2.evaluation_request_imports and zero new qa_evaluation_request_import_recap:% delivery claims since the 2026-09-24T17:45:18Z rollout. Sentry (the only observability source available to this session) shows absence of import/recap failures, which is not a positive outcome. No import was replayed or created to manufacture proof.

failed_agent_ids side-effect audit (source, main). Rails Internal::EvaluationsController#results ended a native result run through import_api2_results! → QAAgentEvaluation#mark_as_failed! → finalize_failure! (per-agent release_sampling_slot_after_failure, progress broadcast, QAEvaluation#calculate_overall_score!), which with no completed agents called fail_unscoreable_evaluation_request! → QAEvaluationRequest#mark_failed!; an API2 skip_reason went through request.mark_skipped!. Those request transitions release consumed sampling partitions and fixed-ticket queue slots, clear qa_evaluated_at, broadcast evaluation_request:stage failed, mail the requester and dispatch backfill replacement. Since #13120 made native result import unconditional, API2 qa-evaluation-request/completion.ts is the only owner of that ending, and its markEvaluationFailed / markRequestSkipped wrote only the processing_status = 'failed' and status = 'failed' | 'skipped' rows — no qa_request_failure_effects row, so none of the effects above ran for result runs ending in errorMessage, skipReason or a missing parent score. Every other native terminal path (failNativeRequest, eligibility / transcription / creation-stage skips) already writes that outbox row and failure-effects.ts delivers it. This is the concrete form of the “pre-existing gap” noted in the ENG-3471 section; the per-agent slot release Rails performed in finalize_failure! is covered by the request-level release of consumed_sampling_rule_partitions in prepareNativeTerminalState (same counters, same partitions), so no separate per-agent step is added.

Package. markEvaluationFailed and markRequestSkipped insert the outbox row (carrying the agent IDs failed in that call, or [] for skips) inside the same short import transaction, only when the request actually transitions out of a non-terminal status; enqueueNativeQaCompletionEffects calls deliverQaFailureEffect after the transaction commits, logging and leaving a failed delivery to the existing 5-minute outbox sweep. No Rails call, no flag, no network work inside the transaction.

Tests. completion.db.test.ts: model failure records one effect with failed_agent_ids = ["101"] and the error message; explicit skip records one effect with [] and the reason; an already-terminal request records no effect. completion.effects.test.ts: post-commit delivery is invoked once with the request key, a delivery failure is logged and does not block billing or summary enqueues, and evaluations without a request skip delivery.

Gates. (1) Ronald review and green required checks on the package PR; no merge before both. (2) A SHA containing the package COMPLETED on all four API2 services — unverified. (3) One natural production result run ending failed or skipped whose qa_request_failure_effects row is delivered (delivered_at set) with the matching broadcast — unverified; no count or outcome is assumed here.

Safety gates for every retirement package

Every implementation package must satisfy these gates before rollout:

• Flag-evaluation evidence: never use PostHog last_called_at to decide whether a gated native path has run. It does not distinguish a true evaluation from a false or null one, so an older flag-off evaluation can read as the latest activity and hide real native traffic. Query $feature_flag_called filtered to the flag key and $feature_flag_response = true, with timestamp after the enable or re-enable moment, then reconcile the rows those evaluations produced. A canary is “unexercised” only when that query returns nothing.
• Final-boundary routing: test the exact method/path matrix through the composed API2 application with the Rails fallback enabled. Assert that the request reaches the intended native handler and that the fallback is not invoked. Include OPTIONS, HEAD, streaming, and WebSocket behavior where the contract uses them.
• External canary: after deployment, exercise the public hostname in both regions and verify native structured telemetry before counting the route as migrated. A healthy service or feature-router unit test is insufficient.
• Mixed-version compatibility: prove that old and new tasks can overlap during a rolling deployment without route collisions, duplicate effects, or incompatible payloads. New behavior remains default-off when a mixed fleet is unsafe.
• Replay-safe ownership: use stable idempotency identities and distinguish definitive rejection from ambiguous delivery. Retries, timeouts after commit, rollback to Rails, and same-hour redelivery must not duplicate writes, object deletion, credit usage, or throttle consumption.
• Shared primitive parity: when Rails and API2 overlap, preserve the exact shared Redis keys, members, expiries, counters, storage service, signed-ID purpose, and regional host trust. Similar behavior is not sufficient; the QA member-counter defect is the current example.
• Short database critical sections: never hold a database transaction, row lock, or transaction-scoped advisory lock across model, provider, storage, Redis, queue, or HTTP work. Re-read and validate state in a short transaction before applying an externally computed result.
• Bounded work: cap discovery queries, fan-out, concurrency, retries, and per-tenant work. A queue move does not protect PostgreSQL from unbounded work or missing backpressure.
• Complete resource lifecycle: test database rows and external objects, including shared blobs, variants, foreign keys, orphan cleanup, and partial failure. Success-path row assertions alone do not establish parity.
• Tenant and origin isolation: prove cross-tenant negatives and use an exact regional host allowlist. Do not replace an exact trust boundary with a domain suffix or other broad match.
• Outcome reconciliation: compare accepted, rejected, not-found, duplicate, retry, and failure distributions before expanding a flag. Zero legacy calls without equivalent business outcomes is a regression, not completion.

Workstream 1: stop forwarding public calls to Rails

The canonical API2 gateway forwards a request only when API2 has no native route for its method and path. Explicit API2 proxy routes, such as conversation uploads, also count here when their implementation still calls the legacy Rails origin.

Track forwarded traffic in four groups:

• Unknown and scanner traffic: unmatched paths that Rails also answers with 404. Reject these at API2. They are not compatibility contracts.
• Compatibility HTTP: live provider callbacks, browser paths, redirects, uploads, and other public contracts still implemented by Rails.
• Explicit API2 proxies: API2-owned public routes whose handler deliberately invokes Rails, including the current conversation-upload implementation.
• Action Cable: /cable WebSocket traffic bridged through API2 to Rails.

Initial production baseline

The first post-cutover sample covered about 70 overnight minutes. It is useful for classification, not daily forecasting.

About 73% of forwarded HTTP in that sample was unknown-path or scanner noise. The meaningful tail was concentrated in:

• Zendesk sidebar requests;
• Active Storage redirects;
• Unit21 webhooks;
• conversation-upload requests and preflights;
• Slack callbacks and interactions;
• a small monitoring and legacy-route tail.

Replace this sample with complete rolling 24-hour and seven-day regional windows. Record method, normalized path family, status, caller class, request count, unique tenant count when available, and the owning migration package. Do not extrapolate a short overnight sample into a daily rate.

Current post-cutover observation

The most recent fixed trailing-24-hour gateway census (ending approximately 2026-09-18T10:16Z) is in Evidence as of 2026-09-18: 18,934 events with a clear live tail of 2,583 (Cable 1,415, uploads 944, Outreach 224). Structured coverage is still only about 30 hours, so no seven-day gateway figure is complete yet.

Structured gateway telemetry became available only after the September 17 cutover. The query ending at approximately 2026-09-17T17:11Z begins at 04:11:53Z in the US and 04:20:35Z in the EU, so it covers approximately 13 hours rather than a complete rolling 24-hour or seven-day window.

The meaningful compatibility tail contains 2,352 Active Storage redirects, 1,247 conversation-image requests, 850 conversation-upload requests or preflights, 422 Zendesk sidebar requests, 332 Unit21 requests, two Slack requests, 78 other requests, and 1,917 Action Cable attempts. After the media deployment completed at approximately 16:40Z, neither region forwarded a media request and the US served six conversation images natively. These counts establish the current path-family shape but must not be converted into a daily rate. The next evidence gate remains the first complete rolling 24-hour and seven-day regional census.

Execution sequence

1. Classify every non-404 path family as live, rollback-only, dormant, or dead.
2. Reject unknown and scanner paths at API2 instead of forwarding them.
3. Replace the wildcard fallback with an explicit compatibility allowlist.
4. Give every allowlisted family a native API2 owner or an approved retirement decision.
5. Replace the five active first-party Action Cable channel families with API2-owned server-sent events (SSE), then remove the /cable bridge.
6. Hold a complete production window with zero meaningful forwarded calls in both regions before removing the fallback and legacy public origins.

The wildcard fallback is removable only when a denied path cannot break a documented API, provider callback, browser workflow, media URL, or supported integration.

The first complete rolling 24-hour census (ending 2026-09-18T09:30Z) counted 18,888 gateway events: 526 blocked in API2, 12,066 forwarded requests Rails answered 404, and a meaningful tail of about 2,836 (Action Cable 1,689, conversation uploads 923, Outreach 224). Steps 2 and 3 are implemented as an explicit compatibility allowlist: the gateway forwards only paths whose first segment is a Rails route-family prefix (legacy-rails-gateway/rails-public-route-prefixes.json, kept equal to the Rails route table by spec/lint/api2_gateway_compatibility_prefixes_spec.rb) and answers everything else with Rails’ 404 body and outcome=rejected. LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward is the rollback gate; remove it once a complete regional window shows no rejected event for a documented path.

Retired prefix: /knowledge_base_documents (step 4, first native-owned retirement)

Evidence. A 24 h legacy_rails_gateway query found exactly four forwarded US GET /knowledge_base_documents/:id requests, all 200, all simultaneous at 2026-09-19T21:27:29Z; EU had zero. Read-only Rails mapping shows all four are Kuda (org 25) manual knowledge documents; process_action confirms KnowledgeBaseDocumentsController#show, 6 queries per request, 36–53 ms. The same IDs appear in Kuda QA/evaluation context and the burst followed the Kuda backfill fallback, so this is meaningful QA-linked traffic, not scanner residue. Caller: no code on main emits the path. The UI moved to the native API2 route GET /knowledge-library/documents/:id in #11781/#11940 (Sep 10/13) and the endpoint is session-authenticated, so the caller is a stale pre-#11940 UI bundle in an open Kuda tab — the same stale-client pattern as /cable.

Decision. API2 owns the boundary natively, so the legacy prefix is retired rather than kept as a compatibility route: /knowledge_base_documents is removed from rails-public-route-prefixes.json and listed in retired-rails-route-prefixes.json. The gateway answers every method under it (including HEAD, OPTIONS, nested paths, query strings and //-prefixed variants) with Rails’ JSON 404 locally and outcome=retired, never contacting Rails. /import_knowledge_base_documents_tasks is a distinct prefix and kept forwarding until batch 2 retired it. The Rails drift spec now checks that every retired prefix is still a Rails route and not already blocked, and that the allowlist equals the route table minus blocked minus retired. Expected effect: stale Kuda tabs get 404 until refreshed (accepted, as for /cable); current UI is unaffected.

Rollback. Revert the retiring PR (restores the prefix and forwarding), or set LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward on the API2 web service for an immediate hold — under forward the retired check is skipped and the path forwards again. The adapter and this config gate are retained. Post-cutover proof: outcome=retired for /knowledge_base_documents should decay to zero within the UI forced-refresh window; any retired event from a current UI build (or from a non-browser client) is a rollback trigger.

Coordinator census at 2026-09-23T01:20:02Z and the public.* ownership probe package (ENG-3378 , child of ENG-3345 )

Coordinator census ending 2026-09-23T01:20:02Z (API2 release-prod 3492e8a5 deployed steady in both regions at 00:56Z, Rails 3/3 in both). The two windows are kept separate on purpose; the 7 d figures are dominated by pre-cutover residue and do not establish a post-3492e8a5 zero, which needs its own clean window after 00:56Z:

This session had no AWS, CloudWatch or database access (aws sts get-caller-identity: no credentials), so the census is requester-supplied and no production probe of any kind was run.

Package chosen (Workstream 3): a read-only, TTY-free public.* live-diff probe plus the runbook gates, because it is the only step toward the zero-statement proof that needs no production DDL, no flag, and no deploy. What was done, all local against a structure.sql (05527e7c6) + baseline + 409 Drizzle migrations database and its _push clone:

• Reproduced the Phase 2 TTY drizzle-kit push --strict preview: 259 statements, public still exactly the 4 known idx_conversations_*nav_date drop/recreate statements; aborted at the apply prompt.
• Root-caused the churn in drizzle-kit 0.31.10 source: sql index columns are always serialized asc: true, nulls: "last" while introspection reads pg_index.indoption, so COALESCE(...) DESC can never round-trip and there is no declaration-side fix in this version. Options (upgrade, tablesFilter exclusion, or rewriting the indexes with production DDL) are recorded in the runbook and left for a human decision.
• Added pnpm db:probe-public-ownership (scripts/public-schema-diff-probe.ts + src/db/public-schema-probe.ts): drizzle-kit pull --schemaFilters public into a temp dir, generateMigration against the declaration, classification into known-expression-index-churn / strict-only-index-churn / public-drift / api2-drift / agent_api-drift, exit 0 only on public zero. It shares the loopback-and-*_push guard with drizzle.probe.config.ts and never calls an apply function. Local result: publicPushStatements: 4, public-drift: 0, api2-drift: 0, agent_api-drift: 0, strict-only-index-churn: 48 (24 pairs that differ only in introspected opclass/WHERE text — tool artefacts, documented). 12 unit tests cover the guard, the push-equivalence squash and the classifier.
• The strict pass exposed two real declaration bugs push-mode hides: drizzle-kit pull truncates expressions at an embedded '', so index_pg_search_documents_on_content and index_pg_search_documents_on_content_unaccent_tsv were declared as COALESCE(content, '. Restored from pg_get_indexdef. Consequence recorded in the runbook: a push-mode zero is necessary but not sufficient.

Decisions. No production DDL, data write, flag change, deploy, replay or regional repair was performed or proposed for automation; regional index reconciliation and the ten invalid indexes remain a human-owned repair PR. drizzle.config.ts is unchanged (["api2", "agent_api"]). The runbook now carries operator gates G1–G7 (regional reconciliation → per-region regeneration → two-region probe exit 0 with the strict list read → expression-index strategy → non-declarable owners → baseline/ordering → Rails migration-freeze CI) and the rollback statement (nothing runtime to roll back while gates are open; api2 image redeploy if schemaFilter is ever widened).

Next gates. G1 repair PR (human); then restore a schema-only dump of each region into a local *_push database and run the probe per region, expecting exit 0 in both; only then Phases 3–5.

Coordinator read-only audit at 2026-09-23T02:20:00Z (epoch 1790130000)

#12831  was human-merged at 02:19:54Z under user review authorization, without a submitted Ronald review (not Ronald approval); API2 release sync #12833  merged at 02:20:07Z. The probe, runbook gates and declaration fixes above are on main; no deploy of them had been observed by the cutoff. The coordinator then supplied a second read-only CloudWatch audit with a fixed cutoff of 2026-09-23T02:20:00Z. This session still had no AWS, CloudWatch or database access, so the figures are requester-supplied and no production probe or DDL was run. The schema scope of this package is unchanged: G1–G7 stay blocked on the human-owned repair PR.

Deployment attribution at the cutoff. API2 release-prod advanced to 89d1b45a at 02:11:23Z via sync #12832  for the merged QA PR #12830 , but as of ~02:22Z ECS US and EU web/worker services all still ran d6607a6d (every deployment COMPLETED); Rails 3/3 in both regions. All “post-deploy” reads below are therefore attributed to d6607a6d, not 89d1b45a. #12830  was merged by a human under user review authorization without a submitted Ronald review; this is recorded as such and not as Ronald approval.

Trailing 24 h ending 02:20:00Z (mixed pre/post-d6607a6d; not a post-deploy zero measure):

Exact trailing 7 d ending 02:20:00Z (almost entirely pre-cutover residue; does not establish any post-deploy zero):

First fully post-d6607a6d slice, 02:04:56Z → 02:20:00Z (~15 min). Non-health Rails controller calls US and EU: zero (/up health checks were intentionally excluded from the query). Gateway: US zero events; EU 15 websocket_bridge_removed and 10 rejected 404, zero forwarded. The natural 02:05Z expiry tick ran US 1 candidate / 1 failed and EU 0, both with deferredToRails = 0 and rails = null (no Rails deferral). No natural Kuda external_api_key_auth or conversation_upload_native_intake event occurred through the cutoff, so those native paths have no post-deploy natural sample yet. A 15-minute clean slice is evidence of direction only; the 24 h and 7 d zero gates remain open.

Gate position after this audit. Workstream 1/2 24 h and 7 d zero windows: open. Workstream 3 schema gates G1–G7: blocked on the human-owned G1 repair PR; no production probe, DDL, data write, flag change, deploy or replay was performed. Rollback position unchanged from the section above.

Field-upsert attribution and bounded activation (ENG-3379 ; coordinator-performed, recorded here — not performed by this package). The four US “field upsert” calls in the 24 h table were PUT /internal/conversations/7054000/field_values ×2 and PUT /internal/conversations/7158696/field_values ×2; read-only ECS pg at ~02:34Z resolved both conversations to organization_id = 19, slug rho-tnkw. PostHog project 63353 already has api2_conversation_relationships (flag 273643) active at 100%. The coordinator created the server-only flag api2_conversation_field_value_native_upsert (id 285250, v1) at 2026-09-23T02:35:13.538613Z with a single condition — exact $group_key rho-tnkw at 100%, no other group or percentage. Owner Platform Engineering; review/remove by 2026-12-21. Rollback: disable the flag or remove the condition (≤ 20 s SDK cache). No synthetic write, replay or data mutation was performed. Hold at exact Rho until a natural field-upsert produces an outcome with DB row parity and error evidence; roll the flag back immediately on any regression. The 24 h/7 d zero gates are unaffected by this activation.

Later deployment addendum (read-only ECS at ~02:29Z; distinct from the fixed 02:20:00Z window, which stays attributed to d6607a6d). API2 b905bc20 (FC_GIT_COMMIT_SHA=b905bc20cd3b7eec68c0d888205bce9b3f57a693 on all four new task definitions) finished rolling out in both regions:

Rails remains 3/3 per region. No post-b905bc20 traffic outcome is claimed; the next census must open a fresh window starting no earlier than 02:28:57Z and attribute it to b905bc20.

Early smoke slice only (read-only CloudWatch, 02:28:58Z → 02:31:07Z, 129 s; cannot close any 24 h / 7 d gate). Non-health Rails controller calls US and EU: zero. Gateway US: two rejected GET 404 (/, /favicon.ico). Gateway EU: three websocket_bridge_removed (/cable) and two rejected GET 404 (/wp-json/wp/v2/posts, /wp-login.php). Zero forwarded in both regions.

Coordinator read-only audit at 2026-09-23T03:20:00Z (epoch 1790133600)

Operator-supplied CloudWatch/ECS evidence; no production probe, DDL, data write, flag change, deploy, replay or runtime-code change accompanies this record. Historical windows above are preserved unchanged.

Trailing 24 h ending 03:20:00Z (mixed pre/post-cutover; not a post-deploy zero measure):

Exact trailing 7 d ending 03:20:00Z (mostly pre-cutover residue; not a zero-window claim):

Post-canary slice, 02:35:13Z → 03:20:00Z (~45 min, on b905bc20). Non-health Rails controller calls US and EU: zero. Gateway: US 5 events, all rejected scanner/unknown GET 404; EU 77 events — 43 websocket_bridge_removed (GET /cable, removed at the gateway and never reaching Rails) and 34 rejected scanner/unknown GET 404. Zero forwarded events in both regions, so none of this gateway activity is live Rails traffic. The natural 03:05Z expiry ticks ran in both regions with candidates = 0, deferredToRails = 0, rails = null — the absence of candidates means no outcome transition occurred, so this tick is not a native-path sample.

Rho field-upsert canary (ENG-3379 ). Flag 285250 v1 (api2_conversation_field_value_native_upsert, exact rho-tnkw 100%) has been active since 02:35:13.538613Z; no native Rho upsert and no matching error had been observed by 03:20Z. HOLD — no widening until a natural outcome with DB row parity is observed.

Gate position after this audit. Workstream 1/2 24 h and 7 d zero windows: open. Workstream 3 schema gates G1–G7: blocked on the human-owned G1 repair PR. Rollback position unchanged.

Coordinator read-only audit at 2026-09-23T04:20:00Z

Operator-supplied CloudWatch/ECS evidence; no production probe, DDL, data write, replay, deploy or runtime-code change accompanies this record (the flag activation below was coordinator-performed and is recorded, not performed, here). Historical windows above are preserved unchanged.

Trailing 24 h ending 04:20:00Z (mixed pre/post-cutover; not a post-deploy zero measure):

Exact trailing 7 d ending 04:20:00Z (pre-cutover dominated; not a zero-window claim): Rails internal US 58,892 / EU 92,052 = 150,944; API2 gateway US 39,792 / EU 25,845 = 65,637; direct Rails-origin conversation-image GET successes US 2,594 / EU 2,220 = 4,814 (all pre-cutover; 24 h zero).

Post-canary slice, 02:35:13Z → 04:20:00Z (~105 min, on b905bc20). Non-health Rails controller calls: US exactly one — POST /internal/daily_agent_evaluation_reports/combined_email 200 at 04:00Z, the natural daily-report tick; EU zero. Gateway forwards zero in both regions; US 16 rejected GET 404, EU 44 websocket_bridge_removed (/cable, never reaching Rails) + 50 rejected GET 404. The natural 04:00Z report tick ran US 38 organizations (2 emailed, no failures) and EU 16 organizations (0 emailed, no failures); US report rows were rho-tnkw 3 and kuda-pilot 1. The natural 04:05Z expiry ticks completed in both regions with candidates = 0, failed = 0, deferredToRails snapshots/requests = 0, rails = null — no positive candidate transition, so not a native-path outcome sample.

Daily-report delivery canary (coordinator-performed). PostHog server-only flag api2_daily_agent_evaluation_report_delivery (id 285273, v1) was activated at 04:24:03Z with exact rho-tnkw only; no post-activation natural outcome yet. Scope, per internal code review: the daily report cron enqueues per-organization jobs with sendEmail: false, so the natural 20:00Z/04:00Z ticks can only prove native backfill_preview (queued backfill preview plan with source = native, zero Rails backfill_preview for organization 19). Native per-organization email status = sent is unreachable from the cron and is not a hold criterion; it needs a later natural backfill reconcile. The Rails combined_email call remains expected on every tick while the Kuda contributor stays off — this flag does not eliminate the next combined_email, and all other report contributors keep their preview/email on Rails. HOLD — no widening until a natural tick shows the native Rho backfill_preview outcome above; roll back on any regression. The Rho field-upsert flag 285250 (exact Rho) remains held with no natural positive outcome yet.

Related. The EU empty-id GET /conversations//... prefetch has a UI-only fix in #12838  (ENG-3381 ; green, awaiting Ronald review, not merged); its plan edit lives in a separate ENG-3381 subsection of this document, not in this one.

Gate position after this audit. Workstream 1/2 24 h and 7 d zero windows: open (the 04:00Z US combined-email call is live Rails traffic and resets any post-canary zero slice). Workstream 3 schema gates G1–G7: blocked on the human-owned G1 repair PR. Rollback position unchanged.

Status note, 2026-09-23 (after 05:21Z)

The 04:20:00Z census above remains the latest verified exact 24 h / 7 d census: the coordinator’s AWS SSO session expired at 05:21Z, so no later CloudWatch/ECS read has been taken. Docs-only #12834  (this plan) and UI-only #12838  (ENG-3381 ) both have Ronald review requested; #12838’s final head was CI-green, and #12834’s prior head 432805067 was CI-green while the head carrying this note has checks pending at the time of writing. Neither has a submitted Ronald review and neither is merged. Rho flags 285250 (field-value native upsert) and 285273 (daily-report delivery) remain held at exact rho-tnkw pending natural positive outcomes. G1 schema repair remains blocked and human-owned. No counts above are revised by this note.

Status note, 2026-09-23 06:23Z

AWS SSO token refresh still fails for both configured coordinator profiles, so 04:20:00Z remains the latest verified exact cutoff: 24 h Rails internal 41, successful Rails gateway forwards 25, direct image GET successes 0; 7 d Rails internal 150,944, gateway 65,637, direct image GET successes 4,814 (pre-cutover); post-canary slice one US combined_email Rails call and zero forwarded requests in both regions. Elapsed time since 04:20Z is not a measured window and must not be read as a zero window. Open focused PRs, CI-green on their latest heads (for #12834, the head preceding this note; the head carrying it has checks pending at the time of writing), labeled codex + codex-automation, Ronald requested, no submitted Ronald approval on any, none merged: #12834  (this plan, ENG-3378), #12838  (ENG-3381 empty-id prefetch), #12839  (ENG-3382  QA fallback attribution) and #12840  (ENG-3383  object-field prefix retirement; its plan edit is the separate “Legacy object-field prefixes retired” section, not this one). Rho flags 285250/285273 still held; G1 schema repair still blocked and human-owned (G1–G7 unchanged).

Coordinator read-only audit at 2026-09-23T07:25:00Z

Fixed CloudWatch cutoff 07:25:00Z (epoch 1790148300); AWS SSO recovered, so this supersedes 04:20:00Z as the latest verified exact census. Read-only; no production DDL, data write, flag change, replay or deploy.

Trailing 24 h ending 07:25:00Z (mixed pre/post-cutover; not a post-deploy zero measure):

Exact trailing 7 d ending 07:25:00Z (pre-cutover dominated; not a zero-window claim): Rails internal US 57,795 / EU 87,117 = 144,912; API2 gateway US 59,304 / EU 50,722 = 110,026; direct Rails-origin conversation-image GET 200 US 2,498 / EU 2,200 = 4,698.

Strictly post-04:20:00Z slice, 04:20:00Z → 07:25:00Z (~185 min). Rails /internal/* calls zero in both regions; successful forwards zero in both regions; forwarded GET 404 US 291 / EU 318, scanner only. This is a ~3 h measured slice on top of the earlier post-canary slice (which contained the one US 04:00Z combined_email call) and does not constitute a 24 h or 7 d zero window; the Workstream 1/2 zero gates remain open.

Merges and release state. #12840  (ENG-3383 , object-field prefix retirement) was human-merged by chidiwilliams at 06:40:54Z as 84199d09 under user review authorization, without a submitted Ronald review; its plan edit is the “Legacy object-field prefixes retired” section below. Release syncs: #12841  merged 06:41:08Z (release-prod 1c5886a5, includes #12840) and #12843  merged 07:24:29Z (release-prod tip c463d408). Read-only ECS, observed in successive reads from ~07:3xZ: US API2 web TD1727 2/2 COMPLETED on 1c5886a5; EU API2 web TD1745 for c463d408 went IN_PROGRESS (desired 0 / running 0, TD1743 b905bc20 1/1 still active) → running 1/1 → the final read COMPLETED, desired/running 1/1, pending 0. Rails US 3/3 (TD2912) and EU 3/3 (TD2008), both COMPLETED. So #12840 is now deployed on API2 web in both regions (worker services not read in this pass). The 07:25:00Z traffic cutoff predates EU completion, and a 06:45Z–07:25Z retired-prefix query returned 0 requests in both regions — no natural retired-prefix request has been observed post-#12840. Docs-only #12834  (the 02:20Z–06:23Z plan updates above) was human-merged by chidiwilliams at 07:30:36Z as a1291ec7, and UI-only #12838  (ENG-3381) at 07:28:53Z as 3ac4f850, both under user review authorization, without a submitted Ronald review; this 07:25Z audit therefore lands via a fresh docs-only follow-up PR on the same ENG-3378 scope. #12839  (ENG-3382) remains open, CI-green on its prior head, labeled codex + codex-automation, Ronald requested, no submitted Ronald approval, not merged.

Decisions. Rho flags 285250 (field-value native upsert) and 285273 (daily-report delivery) stay held at exact rho-tnkw: the 24 h US PUT field-value upsert ×4 and report backfill_preview ×2 above are all pre-04:20Z Rails calls, and no natural positive native outcome has been observed for either flag. G1 schema repair remains blocked and human-owned; G1–G7 unchanged. No counts in earlier sections are revised by this audit.

Coordinator read-only audit at 2026-09-23T13:30:00Z

Fixed CloudWatch cutoff 13:30:00Z (epoch 1790170200), complete read-only regional queries; supersedes 07:25:00Z as the latest verified exact census. Read-only; no production DDL, data write, flag change, replay or deploy.

Trailing 24 h ending 13:30:00Z (mixed pre/post-cutover; not a post-deploy zero measure):

Exact trailing 7 d ending 13:30:00Z (pre-cutover dominated; not a zero-window claim): Rails internal US 50,375 / EU 72,598 = 122,973; API2 gateway US 59,414 / EU 51,524 = 110,938; direct Rails-origin conversation-image GET 200 US 2,200 / EU 1,829 = 4,029.

Post-04:15:00Z slice, 04:15Z → 13:30Z. Rails /internal/*: US 14 (the 13 evaluation-check-run mark_failed calls and the 1 Slack call all fall in this slice), EU 0. Successful gateway forwards zero in both regions. The US mark_failed calls are live Rails effects, not residue, so the Workstream 1/2 24 h and 7 d zero gates remain open; no elapsed time is a zero window.

Release attribution. release-prod branch advanced to 2fd3b757 at 13:33:50Z; the running ECS SHA per service had not been read at this cutoff, so no deployment claim is made for 2fd3b757 (last verified ECS state is the ~07:3xZ read above: US API2 web 1c5886a5, EU API2 web c463d408). #12839 , #12845  (this audit’s docs-only PR) and #12847  remain open, CI-green, with no submitted Ronald approval; none merged. (Superseded at 19:30Z below: #12845 was human-merged at 19:27Z, and #12847 is CI-red.)

Decisions. Rho flags 285250 and 285273 stay held at exact rho-tnkw (no natural positive native outcome observed; the 24 h PUT upsert ×4 and backfill_preview ×2 remain Rails calls). No scale cut. G1 schema repair remains blocked and human-owned; G1–G7 unchanged. No counts in earlier sections are revised by this audit.

Coordinator read-only audit at 2026-09-23T19:30:00Z

Fixed CloudWatch cutoff 19:30:00Z, complete read-only regional queries; supersedes 13:30:00Z as the latest verified exact census. Read-only; no production DDL, data write, flag change, replay or deploy.

Evaluation-check-run mark_failed timing. Reconciling every US /internal path row in this window gives 15 calls: 1 at 08:07Z, 6 at 09:07–09:08Z, 8 at 13:07–13:39Z. The 13:30Z audit above counted a different 24 h window and is left as written.

Trailing 24 h ending 19:30:00Z (mixed pre/post-cutover; not a post-deploy zero measure):

Direct-upload mark_failed, PUT field-value upsert, Unit21 and the US reconciliation calls have all aged out of the 24 h window with no recurrence; the only US Rails internal effects since 04:15Z are the evaluation-check mark_failed family and the single Rho Slack call.

Exact trailing 7 d ending 19:30:00Z (pre-cutover dominated; not a zero-window claim): Rails internal US 40,689 / EU 59,984 = 100,673; API2 gateway US 59,627 / EU 52,239 = 111,866; successful direct Rails-origin conversation-image GET US 1,798 / EU 1,152 = 2,950.

Slack family classification (read-only, code on main). The 13:00:44Z call is createSlackTools → resolveNativeSlackDelivery returning null because api2_slack_notification_native does not exist in PostHog project 63353 (found:false at 13:4xZ), so the agent slack-send-<channel> tool fell through to sendSlackNotification (Rails) by fail-closed design. The native path (#12719 , ENG-3352 ) is deployed: sendNativeSlackNotification posts chat.postMessage with the OrganizationDataSource token and relays Slack refusals as { ok:false, error } exactly as Internal::SlackNotificationsController#create does; 53/53 focused tests (agent-slack-notifications, slack-client, agent-slack-tools) pass on main. The gate is read with organizationSlug: organizations.rulebaseId, which Rails resolves via Organization.friendly.find, i.e. the slug — so an exact rho-tnkw $group_key condition matches the one natural caller. Bounded Slack canary activated (operator, user-authorized): api2_slack_notification_native flag 286616 v1 created 2026-09-23T19:44:52.277399Z in PostHog project 63353, active, server-only, exact organization $group_key=rho-tnkw 100%, no other cohort or percentage; owner Platform Engineering, review/remove by 2026-12-22. Held pending the next natural Rho agent Slack send: success = native chat.postMessage ok:true with a ts, zero Rho POST /internal/slack_notifications and POST /internal/conversations/:id/slack_notifications, no Sentry exception; no widening, replay or manual send. Rollback = disable the flag (≤20 s SDK cache). The 13:00:44Z Rails call above predates the flag. Evaluation-check mark_failed (15/24 h, the largest live family) is owned by the QA track; its gate native_evaluation_check_run_mark_failed is likewise absent in PostHog.

Release and deployment attribution. release-prod branch is 290c0ca4 at 19:27Z from the human-merged docs-only #12845  (Chidi approval). ECS new task definitions are rolling: API2 web and worker IN_PROGRESS in both regions at the cutoff, Rails 3/3 in each region. No traffic in this audit is attributed to 290c0ca4. #12839  (5342b77) is CI-green with REVIEW_REQUIRED and no submitted Ronald review; it stays open. #12847  was red on API2 test shard 4 / aggregate at this cutoff and had no Ronald review; its 13:30Z census text is superseded by this section (see the 01:35Z update below for its current state). Rho flags 285250 and 285273 are held at exact rho-tnkw with no natural positive native outcome yet (the 20:00Z report tick is the first gate for 285273). No scale cut. The 24 h and 7 d zero gates for every Workstream 1/2 family remain open.

Evaluation-check-run mark_failed attribution at the 19:30:00Z audit (ENG-3400 )

Owned by the QA track; complements the coordinator 19:30:00Z census above (same fixed cutoff, same 15-call figure). Read-only; no production write, flag change, replay or deploy.

Attribution of the 15 mark_failed calls. They are not residue. The only caller of POST /internal/evaluation_check_runs/:id/mark_failed is API2’s fail-evaluation-check-run BullMQ job (failEvaluationCheckRunTerminally in rulebase-web/rulebase-api2/src/lib/evaluation-check-run.ts), enqueued when an evaluation-check-run job exhausts its 5 attempts. The native port tryMarkNativeEvaluationCheckFailed existed but was gated on PostHog native_evaluation_check_run_mark_failed, which was never created (23:23Z PostHog inventory above), so every terminal failure fell back to Rails by design. The exhaustions themselves are API2-side: Sentry RULEBASE-API2-FA  (Error: Failed query: rollback caused by Client has encountered a connection error and is not queryable, culprit tryRunNativeEvaluationCheck, 25 production events, first 2026-09-21T13:34:55Z, last 2026-09-23T13:39:27Z with attempts_made 5 / max_attempts 5, job evaluation-check-run-7063997-…, US worker booted 13:13:43Z) lines up exactly with the last call of the 13:07–13:39Z cluster. Per-call attribution of the 08:07Z and 09:07–09:08Z calls to the same issue is inference from the shared job path, not a verified per-event match. The connection loss occurs while the native check body holds the execution advisory lock and transaction across the model call, which violates the database lock-safety rule; that is a separate follow-up and is not changed by this package.

Decision (this PR). Bake the native transition in: the PostHog opt-in flag is removed from code and tryMarkNativeEvaluationCheckFailed owns mark_failed unconditionally, with a Flipper kill switch pause_native_evaluation_check_run_mark_failed (safe state “on” → Flipper per AGENTS.md; owner Platform, created 2026-09-23, removed with the Rails endpoint, review by 2026-12-22) that hands the transition back to Rails while that endpoint still exists. An unreadable switch defers to Rails. Rails parity is unchanged: with_advisory_lock(timeout: 0), terminal statuses returned as-is, already_running re-thrown so the failure job retries, check

• snapshot + (most-recent-snapshot) conversation failed in one transaction. No flag, deploy, replay or data change happens in the PR; the deploy that carries it is the operational canary.

Gates and rollback. After the carrying deploy, one full US and EU day must show Internal::EvaluationCheckRunsController#mark_failed at 0 in Rails logs while fail-evaluation-check-run completions continue and no new rulebase-api2 Sentry issue appears under evaluation-checks/mark-failed. Rollback is enabling pause_native_evaluation_check_run_mark_failed in Flipper (both regions; the fallback is immediate, no deploy) or reverting the PR. Rollback criteria: any fail-evaluation-check-run job error, an already_running deferral that survives the job’s retry budget, or any snapshot/conversation left running/in_progress after its check reached failed. The remaining internal families (expiry 6, reports 4, Slack 1, EU reconciliation 1) are unchanged by this package and keep their own gates.

Coordinator status update at 2026-09-24T01:35Z (no new traffic evidence)

Both coordinator AWS SSO profiles are expired, so no CloudWatch read was possible; the latest exact traffic cutoff remains 2026-09-23T19:30:00Z above and no zero-window credit is taken for the elapsed 19:30Z → 01:35Z interval. Rails 3/3 per region is the last verified ECS state; the 290c0ca4 API2 rollout (four API2 services) was last verified completed at 19:45Z.

Release branch vs. running SHA. The release-prod branch advanced to e66de8cb via #12909  (sync main into release-prod, merged 2026-09-23T23:50:24Z). This is branch state only from GitHub: the ECS running SHA per service is unverified because both AWS SSO profiles fail token refresh, so no deployment or traffic attribution is made for e66de8cb.

PR state. #12847  is now CI-green at c5e53fbf (the shard-4 failure noted above is resolved) but still has no Ronald review; it stays open. #12891  (QA track, ENG-3400 : native evaluation-check mark_failed behind a Flipper kill switch) is CI-green and unapproved. #12839  (5342b77) is CI-green with no Ronald review. This audit’s docs-only #12890  was CI-green (required ci passing) at prior head 05fc322e; checks on the head carrying this section were still pending when it was written. Ronald is absent. Nothing merged.

Decisions. The exact-Rho Slack canary (flag 286616) stays held with no natural Rho send observed (no telemetry read); Rho flags 285250 and 285273 unchanged and held; no widening, replay, manual send, data write, deploy or scale cut. The evaluation-check mark_failed family remains the largest live Rails effect (15 in the 19:30Z 24 h window) and is owned by the QA track via #12891. All Workstream 1/2 24 h and 7 d zero gates remain open.

Coordinator read-only audit at 2026-09-24T07:40:00Z

AWS read-only access recovered; fixed CloudWatch cutoff 07:40:00Z, complete regional queries. Supersedes 19:30:00Z as the latest verified exact census. Read-only; no DDL, data write, flag change, replay or deploy.

Trailing 24 h ending 07:40:00Z (mixed; not a post-deploy zero measure):

Exact trailing 7 d ending 07:40:00Z (historical, pre-cutover dominated; not a zero-window claim): Rails internal US 27,953 / EU 43,346 = 71,299; gateway events US 59,470 / EU 52,383 = 111,853; direct successful image GET US 1,627 / EU 964 = 2,591.

Deployment. API2 running SHA afabdfcd (release sync #12927 , commit 2026-09-24T06:19:44Z) on all four regional web/worker services; rollout completion is gated by US API2 web at 07:25:58Z (US worker 06:28:35Z, EU web 06:27:39Z, EU worker 06:27:40Z), so this census has only ~14 minutes after the complete afabdfcd rollout and takes no post-release 24 h gate credit. Rails 3/3 per region. The release-prod branch (e29ef0df) is ahead of the running SHA — branch state and running SHA are not equated.

Report family (still live, code on main). The two combined_email calls are the 04:00Z/20:00Z cron ticks: combinedEmailGoesNative requires every contributing report to carry native_delivery:true, so with flag 285273 exact-Rho the cross-org combined email stays on Rails by design until every reportable organization with rows is gated on. The single backfill_preview call is one per-org job whose nativeDeliveryEnabled gate returned false (a non-Rho org with rows, or an unavailable PostHog read — the gate does not distinguish these; see the attribution package below). Rho flags 285250 (field-value upsert) and 285273 (report delivery) remain held: a positive natural native outcome is not yet verified — this is a Rails-only census and no API2 native log was queried, so it proves neither presence nor absence of a native Rho outcome.

Slack family. The single Rails Slack call is the Rho POST /internal/slack_notifications at 2026-09-23T13:00:44.953Z, conclusively before flag 286616’s activation at 19:44:52Z; there is no post-flag Rails Slack call in the fixed 24 h. Flag 286616 stays exact-Rho, held, pending a natural Rho send with ok:true + ts (native outcome likewise not yet verified).

Decisions. No widening, replay, manual send, data write, deploy or merge. Next focused packages: (a) attribute which organizations keep the combined email on Rails and why a per-org gate returned false, from the report tick/org job logs; (b) operator-owned widening of 285273 only after Rho native backfill_preview evidence; (c) Slack canary outcome from the next natural Rho send; (d) source attribution of the two /agent/* forwarded 404s (public track). All internal-effects and gateway 24 h and 7 d post-cutover zero gates remain open; no credit taken.

Coordinator status update at 2026-09-24T13:37Z (no new traffic evidence)

Evidence access boundary. The coordinator’s two configured AWS SSO profiles (default and AdministratorAccess-879381255132) both returned Token has expired and refresh failed on sts get-caller-identity, and the automation (Devin) box has no AWS credentials at all, so no newer CloudWatch census was taken. 07:40:00Z remains the latest exact cutoff: US 19 internal calls / 24 h (mark_failed 15, combined_email 2, backfill_preview 1, Slack 1), EU 0; 7 d 71,299 (historical). The 04:00Z/20:00Z report ticks and any natural Rho field-upsert, report or Slack outcome that occurred between 07:40Z and 13:37Z are unobserved, not absent; no zero-window credit accrues from the gap.

Deployment uncertainty. afabdfcd (all four API2 services, complete 07:25:58Z) is the last verified running SHA. Coordinator git ls-remote origin refs/heads/release-prod at 13:38Z returned c3a17f2d66d309c8eb6b3707e28c90b3a2d579eb — branch state only; whether ECS has rolled past afabdfcd is unknown without AWS/ECS reads. No deployment claim is made from the branch.

PR state. This docs PR (#12890 , head 143e8339) and the report attribution package (#12932 , head 2db22176, ENG-3419 ) are open and green but REVIEW_REQUIRED with no Ronald review; neither is merged, so combinedEmail.railsGatedOrganizations attribution is not yet deployed — the next ticks’ combined-email path still does not name the gated-off organizations. #12839  (ownerReason) is in the same state.

Decisions. Rho flags 285250, 285273 and 286616 stay exact-Rho and held; no widening, replay, manual send, data write, deploy or merge. Next gate: a fresh regional 24 h / 7 d census with per-family method/path and the pre-cutover-vs-live split, once AWS read access returns; until then all 24 h / 7 d post-cutover zero gates remain open with no credit.

Coordinator audit at 2026-09-24T15:40Z and the exact-Rho Slack canary rollback (ENG-3428 )

Exact 24 h Rails /internal census (CloudWatch, both regions, cutoff 15:40:00Z).

US total 4, EU total 5. Seven-day internal US 16,229 + EU 26,915 = 43,144, historical / pre-cutover dominated. mark_failed (both the upload and evaluation-check-run families) has rolled out of the 24 h window for the first time. No /internal call in the 15:34–15:40Z slice, but the API2 a6b5071 rollout completed only ~15:33Z, so that slice is a ~7 min post-rollout observation, not 24 h credit.

Slack canary regression. Flag 286616 (api2_slack_notification_native, exact rho-tnkw) was evaluated true at 13:00:01Z. API2 then logged two warnings Active Rails Slack data source 11 not found and resolveNativeSlackDelivery returned null, so the notification went to Rails (201 at 13:00:34.969Z). Fail-closed behaviour held: no send was lost and no double-send is possible on this path. The coordinator disabled the flag at ~15:47Z (inactive, version 2). Native { ok: true, ts } has never been observed for Rho; the canary produced negative evidence only.

Root cause boundary (code, read-only). API2 loadSlackAccessTokenForRailsDataSource selects organization_data_sources with id = 11 AND organization_id = 19 AND data_sources.key = 'slack' AND discarded_at IS NULL. Rails Internal::SlackNotificationsController#create resolves OrganizationDataSource.find_by_prefix_id! under the Rho tenant — (id, organization_id) only, with no kept scope and no data-source-key check — and #slack_client posts using data['token'] regardless. Because Rails located and used ODS 11 for Rho, the API2 miss is one of two production-data states: ODS 11 is soft-discarded (Rails still posts through discarded connections) or its data_sources.key is not slack. Prefix decoding (connection) and org resolution are correct. Distinguishing the two needs a read-only operator query (SELECT id, organization_id, data_source_id, discarded_at FROM organization_data_sources WHERE id = 11); the automation has no production DB or AWS access and performed no production read or write.

Package. #12976  (ENG-3428 ) keeps the strict native lookup (loosening it would replicate Rails’ post-through-discarded behaviour rather than parity with the intended one) and makes the miss attributable: SlackDataSourceNotFoundError carries reason ∈ {missing, other_organization, discarded, wrong_data_source} and the actual key, logged as missReason / dataSourceKey on the existing falling back to Rails warning. No routing change; 11/11 focused tests; user-merged 17:08:54Z without a submitted Ronald review (see 17:13Z).

Decisions. 286616 stays disabled; re-enable only after (a) the operator query identifies the ODS 11 state and, if the row is discarded or non-Slack, a human decides whether Rails’ behaviour or API2’s is the intended one, and (b) a natural Rho send then yields native ok:true + ts, zero Rho Rails Slack calls and no Sentry error. 285250 and 285273 remain exact-Rho and held with no natural positive outcome verified. No widening, replay, manual send, data write, deploy or merge. All 24 h / 7 d post-cutover zero gates remain open; the Slack family’s window restarts from any future re-enable, not from 15:47Z.

Coordinator status update at 2026-09-24T17:13–17:17Z (merges, G1 completion, deployment boundary)

Merges. #12976  (ENG-3428 ) was user-merged at 17:08:54Z without a submitted Ronald review. It is attribution-only (miss-reason on the fail-closed Slack fallback); it changes no routing and does not by itself justify re-enabling 286616, which remains disabled. This docs PR (#12977 ) is still open.

Deployment boundary. At 17:13Z ECS API2 was still running 28cd9ee8 and Rails its prior task definitions. By ~17:17Z all US/EU Rails and API2 web+worker services had new tasks serving e504aad3 (release sync #12985 , commit 17:08:42Z), rollout IN_PROGRESS draining old tasks. e504aad3 predates the #12976 merge (17:08:54Z), so the Slack miss-reason attribution is not deployed by this rollout; a Rho Slack attempt would still log the unclassified not found warning. Later syncs #12987  / #12988  carry the merge commits but had no observed ECS rollout. Branch state is not runtime state.

G1 regional index reconciliation. Coordinator read-only preflight before the migration: US 2,398 public indexes, 4 invalid, 3 _cc leftovers, all six composites, unsourced index absent; EU 2,395, 6 invalid, 4 _cc leftovers, only the compliance composite, unsourced index present. The Rails G1 migration (schema_migrations marker 20260924155417) then completed in both regions: 2,395 public indexes each, identical name+definition md5 8d73d977a51edc8baaf2f21fff92606c, zero invalid indexes, zero leftovers. G1 is therefore satisfied; G2–G7 (per-region regeneration, two-region probe exit 0, expression-index strategy, non-declarable owners, baseline/ordering, migration-freeze CI) remain open. Figures are coordinator-supplied; the automation ran no production query or DDL.

Traffic. 15:40–17:10Z: zero Rails /internal and zero direct conversation-image calls in both regions. This is a 90-minute slice with scheduled work (20:00Z / 04:00Z report ticks, QA recaps) still ahead of it, so it is an observation only — no 24 h / 7 d credit.

Next. Combined-email/report and Slack are pursued as one grouped internal-effects package: report side needs the 20:00Z tick after railsGatedOrganizations is live to name the Rails-gated organizations; Slack side needs the operator ODS 11 read (discarded vs non-Slack key) and a human parity decision before any exact-Rho re-enable. Held flags unchanged.

Coordinator read-only investigation at 2026-09-25T02:15Z: 17 Rails /internal calls, ODS 11 resolved, Kuda named (ENG-3379 , ENG-3428 , ENG-3419 )

Coordinator-supplied evidence (CloudWatch plus read-only US/EU production DB reads; the automation ran no production query). Exact trailing 24 h ending 2026-09-25T02:15Z: 17 Rails /internal calls, all preceding the 315a8404 API2 rollout that completed in both regions at ~02:21Z, so all 17 are historical relative to the running SHA.

Post-rollout slice. 02:21–02:46:20Z zero Rails /internal and zero direct image calls in both regions — ~25 minutes, observation only, no 24 h / 7 d credit. Rails stays 3/3 per region.

QA sampling (#13003 deployed). The running SHA removes PostHog from the production owner decision and defaults to API2 (ownerReason=native_default) with Flipper pause_qa_sampling_recovery_native as rollback. 02:30Z cycle: US 39 API2-owned org runs / 57 candidates, EU 16 / 145; 02:45Z: US 39/57, EU 16/143; zero candidate, transcription, or unprocessed failures either region. Remaining Rails selectors: Flipper read unavailable, pause switch on, or an environment override of rails — each must be accounted for before any scale-to-zero claim.

Daily reports. Rho rows were collected natively (1 row at 04:00Z, 0 at 20:00Z) but no native Rho preview/email delivery claim has been found since Sep 23; collection is not delivery proof. Combined email remains Rails whenever any contributing report lacks native_delivery:true, and the gate fails closed to Rails on a PostHog error. Do not widen 285273 to Kuda on this evidence. Order: natural Rho preview/email success with ledger proof → bounded Kuda canary (preview run + email outcome, zero Rails calls, no regression) → reviewed removal of the PostHog fail-closed Rails dependency → re-observe 04:00Z/20:00Z ticks.

Slack root cause — resolved read-only. Rho Rails organization_data_sources.id=11 is a Slack source discarded 2026-06-30; Rho’s only other Rails Slack source (id 55) is also discarded. Rails’ notification controller looks the source up without an active-source condition and still posts; API2’s native resolver requires discarded_at IS NULL and therefore rejects it — the discarded branch of the #12976 classifier (and of the #12998  flag-off readiness probe, still open). API2-side: two Slack connections with legacy rulebase_id are active but hold no encrypted API2 token; a third modern active connection has an encrypted token and external account but no legacy rulebase_id (last_used_at=2026-09-25T01:01:07Z, no last_error_at, channel sync present — readiness context, not a notification outcome). Re-enabling 286616 now would repeat the miss. Required before any re-enable: a human parity decision (bind the agent Slack tool to the modern API2 connection vs intentionally honour the legacy discarded-source semantics), a reviewed implementation with focused tests for discarded/stale mappings and the delivery response, deploy, then one natural Rho chat.postMessage ok:true with ts, zero Rails fallback, no error.

Slack parity decision (2026-09-25, Chidi): bind to the modern API2 connection. The native path must not mirror Rails’ discarded-source semantics. Implementation #13023  (ENG-3428 ; codex / codex-automation, Ronald requested, open — no merge before Ronald approval and green required checks): after the 286616 flag gate, prefix decode and Rails-org resolution, resolveNativeSlackDelivery looks up the organization’s connections row with provider='slack', auth_type='oauth2', status='active' and a nonempty encrypted token; exactly one match → decrypt and post with it (tokenSource=api2_connection); zero or ≥2 matches → the existing strict Rails ODS token path (tokenSource=rails_data_source, which for Rho still misses on the discarded row and falls back to Rails); any lookup/decrypt error → null (Rails, fail-closed). Legacy legacy_rails_passthrough rows are excluded by auth type. Focused tests: owner unit tests (preference, ambiguity, lookup failure, flag-off, no token in logs) plus a pglite predicate test; typecheck clean; required checks green at head e60afb0c6 after merging main (the earlier shard-5 red was main’s scorecard_version_id drift, fixed by #13024 ). Tokens are never logged. No flag, data, deploy or provider change was made.

Slack gate (ordered). (1) Ronald approval + green required checks → human merge of #13023; (2) release-prod sync and verified ECS running SHA containing #13023 on all four API2 services (branch state is not runtime state); (3) operator re-enables 286616 exact rho-tnkw only (automation does not touch the flag); (4) one natural Rho agent Slack send logging tokenSource=api2_connection, chat.postMessage ok:true with nonempty ts, zero Rails POST /internal/slack_notifications for Rho, no Sentry error; (5) 24 h then 7 d zero-Rails-Slack windows counted from the re-enable, not from the merge; wider rollout is a separate human step. Rollback at any step = disable 286616 (Rails path unchanged). Interim: flag stays disabled; #12998  (flag-off readiness probe) remains open and would log missReason=discarded for ODS 11 once deployed.

Report-delivery gate (ordered). (1) natural Rho 04:00Z/20:00Z tick with native preview and email delivery proven in the delivery ledger (collection alone does not count; none found since Sep 23); (2) bounded exact-Kuda 285273 canary — operator action — requiring one preview run + one email outcome, railsGatedOrganizations=[], zero Rails combined_email / backfill_preview calls, no regression; (3) reviewed removal of the PostHog fail-closed Rails dependency; (4) re-observe both daily ticks for 24 h / 7 d. Rollback = flag back to exact Rho (Rails combined email resumes). Do not widen 285273 on the current evidence.

Report-delivery package for gate step (3) — prepared, unmerged. #13097  (ENG-3419 ; both codex labels, Ronald requested, no submitted review) moves the delivery-owner decision out of PostHog into daily-agent-evaluation-report/delivery-owner.ts, the same shape as the QA sampling owner in #13003: env_override → non_production → Flipper kill switch pause_daily_agent_evaluation_report_native_delivery (paused / flipper_unavailable → Rails) → native_default (API2). The boolean gate pauses every organization; an Organization;<id> actor holds one tenant (e.g. Kuda) on Rails while the rest run natively, so rollback no longer depends on PostHog. The PostHog collection gate in rollout-gate.ts is unchanged; railsGatedOrganizations / source attribution on the combined email is unchanged and each org now also logs owner + ownerReason. Rails endpoints and client functions are retained. Merge precondition: an absent Flipper feature means native for every reportable org, so merging cuts Kuda’s delivery over — it must follow gate steps (1)–(2) above, or an operator must enable the pause gate (boolean or exact-Kuda actor) in Flipper before the deploy so runtime behaviour is unchanged at rollout and orgs are un-paused deliberately. Evidence basis is only the 02:15Z investigation already recorded here; no fresh production outcome is claimed and no flag, data, deploy or provider change was made. Once shipped, 285273 is dead for delivery (still read for collection) and is removed with the Rails endpoints after the 24 h / 7 d zero-Rails windows.

EU import/recap. Deployed import code calls native bulkCreateUserRequestedQaEvaluationRequests and sendBulkRecapEmail; the former Rails client functions are removed. Read-only US+EU counts since 17:45:18Z: zero new api2.evaluation_request_imports and zero qa_evaluation_request_import_recap:% delivery claims — no natural native import→recap outcome yet. Latent dependency in the running SHA: evaluation-request-imports.ts still calls ensureImportedConversation (Rails POST /internal/conversation_imports/ensure, which may import or sync provider data) when a CSV row’s conversation is absent or unsynced; it produced no call in this window but blocks claiming the import journey is API2-owned. Required: reviewed native ownership (or proven safe removal) of the ensure branch, tests for absent/unsynced conversations and invalid membership, a natural EU import creating requests plus native recap claim/provider outcome, zero Rails import-family calls through full windows. No customer job replay.

Decision. Rails 3/3 per region; Kuda report delivery and Rho Slack stay off (Slack re-enable only after #13023 is reviewed, merged and verified running); 24 h / 7 d post-cutover zero windows remain open; public.* G2–G7 is an independent shutdown blocker.

Coordinator census at 2026-09-24T15:40Z and the native import bulk-create package (ENG-3429 )

Evidence. Coordinator CloudWatch, fixed cutoff 2026-09-24T15:40Z, trailing 24 h: Rails /internal US 4, EU 5. Every EU call is Internal::QAEvaluationRequestsController: bulk_create 200 ×3 (latest 13:10:22.595Z), bulk_create 404 ×1 (09:45:58.263Z), send_bulk_recap 200 ×1 (10:06:26.513Z). API2 a6b5071 completed on all four services at about 15:33Z, so all five calls precede the running SHA; no post-rollout zero window is claimed from this cutoff.

Classification: meaningful live work, not residue. Static trace on origin/main 57e04fd27:

• API2 evaluation-request-imports.ts submitEvaluationRequestRows (CSV and agent-selection evaluation-request imports) posts every submit-ready row to Rails bulk_create and every already-queued row to tag_bulk_import; syncEvaluationRequestImportRows posts send_bulk_recap once no tracked row is open.
• Rails bulk_create runs QAEvaluationRequest::UserRequestedCreator (conversation with_lock, supersede scheduled requests, save!, process_later → API2 orchestration) and answers per conversation with created, already_queued (tagging the active request with the import), not_found or failed. The whole-batch 404 fires only when Organization.friendly.find or Membership.find_by_prefix_id! misses, so the 09:45Z 404 is one import whose organization/membership lookup failed in Rails; per-conversation misses are 200 responses with not_found rows.
• send_bulk_recap runs BulkImportRecapNotifier → QAEvaluationRequestMailer.bulk_completed.deliver_later: a real customer email through GoodJob.

API2 already owned the single-conversation twin (createUserRequestedQaEvaluationRequest behind POST /conversations/:id/qa_evaluation_requests); the bulk import path was the last reason evaluation-request imports still went through Rails.

Package. qa-evaluation-request/bulk-user-requested.ts: bulkCreateUserRequestedQaEvaluationRequests reproduces the controller’s per-row contract on the native creator (tenant-scoped conversation lookup → not_found; membership of another organization → failed; missing membership → throws, the 404 twin; QaEvaluationRequestConflictError → already_queued plus import tagging), and tagQaEvaluationRequestsForImport reproduces BulkImportRequestTagger (only untagged rows of the tenant). submitEvaluationRequestRows uses both; the Rails client functions bulkCreateQaEvaluationRequests and tagBulkImportQaEvaluationRequests are deleted. Lock safety is unchanged from the creator: SQL-only transaction, BullMQ enqueue and Action Cable broadcast after commit. Covered by bulk-user-requested.db.test.ts (created / not_found across tenants / already_queued tagging / cross-tenant membership / missing membership / tagger scoping).

Out of scope. send_bulk_recap stayed on Rails in this package; the native port is ENG-3431 below. The Rails bulk_create / tag_bulk_import routes stay deployed as rollback until a complete post-deploy window shows zero calls.

Gates and rollback. Canary is the carrying deploy. Watch bulk-user-requested-qa-evaluation-requests warn logs, imports failing with Membership not found (the former Rails 404), and import-created qa_evaluation_requests rows carrying stage_progress.processing_owner=api2 with api2_enqueue.outcome=accepted. Rollback is a revert of the PR; the Rails endpoints remain. No flag, deploy, replay or production write in this package. Next evidence: a regional 24 h census after the carrying deploy with QAEvaluationRequestsController split by action; expected shape is bulk_create 0, tag_bulk_import 0, send_bulk_recap ≥ 0.

Native bulk_completed recap email (ENG-3431 )

Rollout state (user-reported, 2026-09-24). #12978 (ENG-3429) was merged at 17:09:27Z without a submitted Ronald review and is carried by release-prod syncs #12987/#12988. ECS was serving 28cd9ee8 at 17:13Z and e504aad3 (sync #12985, 17:08:42Z, includes G1 #12980) at ~17:17Z with old tasks still draining; both SHAs predate the #12978 merge, so #12978 is not deployed and no native import outcome or bulk_create-zero window may be claimed until a SHA that contains it is confirmed running on all four API2 services. The 15:40Z census above therefore remains pre-rollout evidence for both packages.

Evidence. The single remaining QA-family residual is the EU send_bulk_recap 200 at 2026-09-24T10:06:26.513Z. It is live customer work, not residue: maybeSendBulkRecapNotification fires once every tracked row of a submitted import is terminal and Rails then emails the requesting member (QAEvaluationRequestMailer#bulk_completed, deliver_later).

Decision. Port the mailer natively (rulebase-api2/src/lib/qa-evaluation-request/bulk-recap-email.ts + mailer-chrome.ts, a literal port of _chrome.html.erb). Eligibility follows BulkImportRecapNotifier#call (membership account with an email, ≥ 1 request tagged with the import in the same organization, all of them completed|skipped|failed|cancelled); counts, created_at order, the 10-ticket cap, (overall_score * 100).round, conversation and /home/tasks links, both subject lines and the %b %d, %Y at %l:%M %p %Z footer in the organization timezone match the Rails templates. Transport is Resend via sendAgentEmail from Rulebase <notifications@rulebase.co>. At-most-once delivery uses the existing public.outbound_delivery_claims table with key qa_evaluation_request_import_recap:<import_id>, also passed as the Resend idempotency key; the claim is marked delivered only after Resend accepts, then recapNotificationSentAt is set. A Resend refusal releases the claim and throws so the BullMQ sync retries (parity with deliver_later); a lost answer keeps the claim and later attempts report uncertain without resending. No transaction or lock spans the provider call. sendBulkEvaluationRecapNotification is removed from rails-client.ts, leaving zero API2→Rails calls in the QA evaluation-request family. Covered by bulk-recap-email.db.test.ts (send once + counts/copy/links, open request blocks, no requests / cross-tenant membership, refusal retry, lost answer, 10-ticket cap and import scoping, timestamp format) with outbound HTTP guarded.

Gates and rollback. Canary is the carrying deploy; the Rails send_bulk_recap route stays deployed as rollback (revert the PR, no flag). Evidence required before the route is deleted: (1) a SHA containing this package confirmed on all four API2 services, (2) one natural import completion producing exactly one outbound_delivery_claims row with the recap key and delivered_at set plus one Resend message to the requester, (3) a regional 24 h and then 7 d census with QAEvaluationRequestsController split by action reading bulk_create 0, tag_bulk_import 0, send_bulk_recap 0. Roll back on any qa-evaluation-request-bulk-recap uncertain warning without a matching Resend message, or on an import completing without a recap claim. No deploy, flag, replay or production write in this package.

Rollout evidence at 2026-09-24T17:45–17:55Z (gate 1 met, gates 2–3 open). #12991  was human-merged with no submitted Ronald review (main 8e8823d36). The coordinator read at 17:45Z shows all four API2 web/worker services COMPLETED on a62a5f66, which contains #12978 and #12991; the last EU worker completed at 17:45:18Z. That is the earliest instant any post-release zero clock for bulk_create / tag_bulk_import / send_bulk_recap may start. The coordinator’s “zero Rails /internal calls since 17:20Z in both regions” slice is in-rollout (e504aad3 and a62a5f66 tasks overlapping) and earns no credit; the 24 h and 7 d windows run from 17:45:18Z.

Read-only checks from the automation (Sentry only; no AWS, database or Resend access from this session): no new rulebase-api2 issue in the 3 h to 17:55Z touches the import or recap path, and no bulk recap email not sent, failed to resolve import or qa-evaluation-request-bulk-recap log line exists after 17:45:18Z. That is absence-of-failure only; gate 2 (one natural import completing with one delivered qa_evaluation_request_import_recap:<import_id> claim and one Resend message) needs an operator outbound_delivery_claims / Resend read and is unproven. Pre-rollout note for the same family: import 01a0d315-c148-7a58-92bd-c0787dcf6fd4 logged failed to resolve agent selection import three times at 13:12:28Z, 13:15:59Z, 13:19:50Z (Sentry traces 67f25a88…, 017eb79f…, e7840cdc…), on the SHA that still posted to Rails bulk_create; whether it reached a terminal state or a recap is unknown and is a coordinator read, not a gate for this package. No replay, synthetic import or production write was performed.

Coordinator census fixed at 2026-09-24T19:40:00Z: post-release Rails reconciliation call, zero clock reset (ENG-3435 , #13003 ). Trailing 24 h: US Rails POST /internal/conversation_transcription_reconciliations HTTP 200 ×3 at 18:45:06Z, EU 0. That is after a62a5f66 completed at 17:45:18Z, so the post-cutover Rails-zero clock for the QA sampling family (/internal/conversation_transcription_reconciliations and /internal/qa_sampling_recoveries) is reset; no window starting at 17:45:18Z may be credited. The send_bulk_recap / bulk_create / tag_bulk_import family recorded no call in the same census, but the natural import-to-recap outcome (gate 2 above) remains unproven.

Attribution (read-only PostHog $feature_flag_called rows, no CloudWatch access from the automation): the three sweeps for aircall-demo (18:45:06.103Z), test-aircall (.105Z) and rho (.107Z) evaluated qa_sampling_recovery_native with $feature_flag_error = unknown_error and a null response (posthog-node 5.37.0); the eight other organizations in the same tick evaluated true (flag v21, “Matched condition set”). A non-boolean SDK result becomes posthog_unavailable in readOrganizationPosthogFlag, resolveQASamplingRecoveryOwnership maps it to owner: rails, and rails-run.ts calls the reconciliation endpoint once per Rails-owned sweep regardless of candidates. The 2026-09-22 census shows the same signature: exactly 4 unknown_error evaluations (17:00:05Z … 23:15:03Z) for the 3 US / 1 EU reconciliation calls recorded then. These are meaningful live dependencies: each call runs the Rails reconciliation and enqueues its transcriptions, triggered by transient PostHog errors under a flag that has been 100% since 2026-09-19. Per-request CloudWatch↔PostHog pairing is by timestamp coincidence within one second and is marked as inference.

Decision: #13003 bakes native ownership into code (root AGENTS.md: finished rollout → remove the flag and fallback branch; Flipper for gates whose safe state is “on”). owner.ts stops reading PostHog; production owner is api2 (native_default) unless Flipper pause_qa_sampling_recovery_native is enabled (paused) or unreadable (flipper_unavailable). rails-run.ts and both Rails controllers stay as the rollback path; the PostHog flag is left for the operator to archive after the deploy. Gates: canary is the carrying deploy; rollback is enabling the Flipper switch (no deploy) on native sweep errors or an admission drop across two ticks; the reconciliation / qa-sampling zero clock restarts when a SHA containing #13003 is COMPLETED on all four API2 services, and deletion still needs the exact regional 24 h and 7 d zero census. No merge before Ronald review; no deploy, flag change, replay or production write.

Retired prefixes, batch 2 (step 4, 2026-09-21, ENG-3344 )

Evidence. Operator 24 h legacy_rails_gateway census ending 2026-09-21: 1,131 forwarded US requests, prefixes conversations 1,076 (essentially all POST /conversations/upload), admin 25, pages 12, agent 9, clients 1; everything but the uploads was scanner probing (/admin/.env, /agent/vicidial.php, /pages/index.astro.mjs.map). A clean 44 h window after the 2026-09-20 gateway deploy, measured on Rails’ own request logs (so it also covers direct API2 fetches), shows Rails served only conversations (2,002 US), conversation_images (758 US / 275 EU), internal (232 US / 108 EU, blocked at the gateway) and integrations (10 US). Static classification of the remaining 155 allowlisted prefixes against the composed API2 router and every first-party client is in the gateway prefix retirement runbook. The 120 h window was not used: it straddles the reject cutover and the MCP/Zendesk/webhook native cutovers and is contaminated by pre-cutover behaviour. CloudWatch was not reachable from the authoring session; the figures above were supplied by the operator.

Decision. 85 prefixes move from rails-public-route-prefixes.json to retired-rails-route-prefixes.json (159 → 74): 25 whose every Rails method/path has a method-specific API2 handler that does not proxy Rails (each of the 48 routes gets an exact PUBLIC_ROUTE_OWNERSHIP_CONTRACTS entry, so app.route-ownership.test.ts fails if a handler is removed or downgraded to ALL), and 60 with no Rails-served traffic and no URL built by any first-party client. /conversations, /conversation_images, /rails and /integrations are left to their own packages. /admin and /agent stay because Rails still serves authenticated routes with live callers under them despite scanner-only census traffic; /zendesk-app stays because the native sidebar handler still proxies Rails. Retiring a prefix changes only the gateway’s handling of inbound public requests; API2’s direct RAILS_INTERNAL_BASE_URL fetches are unaffected and are audited separately before the Rails shutdown gate.

Rollback. As for /knowledge_base_documents: LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward on the API2 web service is the immediate hold; reverting the PR is the permanent rollback. Post-cutover proof: outcome=retired events in both regions should be scanner paths only; any retired event for a path Rails previously answered with a non-404 is a rollback trigger.

Empty-segment /conversations paths redirected to API2 (step 4, 2026-09-23, ENG-3371 )

Evidence. Operator 24 h legacy_rails_gateway census ending 2026-09-22T23:23Z, after the Cable removal and the #12686 prefix retirement (API2 release-prod 05911a9b): 25 forwarded requests Rails answered non-404 — EU GET /conversations/ 23, EU GET /conversation_fields 1, US POST /conversations/upload 1 — and 394 forwarded 404s, mostly empty-id /conversations//... paths plus scanner probes. Rails’ own logs additionally show three US ConversationImagesController#show 200 direct reads (EU: the synthetic not-a-signed-id probes only). CloudWatch was not reachable from the authoring session; the figures were supplied by the operator.

Classification.

Decision. The gateway now applies Rails’ path normalization before giving up ownership: when the request path differs from its normalized form (repeated slashes squeezed, trailing slash dropped) and a method-specific API2 handler owns the normalized path, the gateway answers 308 to that path (query preserved) and logs outcome=normalized with path and normalizedPath. 308 keeps method and body, so the client’s retry is served by the same route Rails would have resolved, but by API2 and behind API2 auth. Preflights for a normalizable native path stay in API2. Paths whose normalized form Rails still owns (/conversation_fields/ at the time; /locations/ is the contract example since /conversation_fields was retired) keep forwarding, so no compatibility contract changes. UNOWNED_PUBLIC_REQUESTS gains normalized contracts for the four census shapes, checked against the composed router. No flag, no data effect, no operator step.

Rollback. Revert the PR. Triggers: any outcome=normalized event whose normalizedPath Rails answered non-404 in the pre-deploy window but API2 answers with a 5xx, or a first-party client surfacing redirect failures.

Post-deploy gate. One 24 h window in both regions with outcome=forwarded for GET /conversations/ and /conversations//* at zero, every normalized event carrying an expected normalizedPath; then locate the UI call sites that emit the empty id (request ids are on the event) and fix them at source.

Operator audit at 2026-09-23T03:20Z and the empty-id source fix (ENG-3381 )

Evidence (operator-supplied, trailing 24 h ending 03:20Z). Gateway events US 1,216 / EU 4,171. Successful Rails forwards 25, all inside the window overlapping the #12820 deploy at 00:56Z: US POST /conversations/upload 201 ×1, EU GET /conversations/ 200 ×23, EU GET /conversation_fields 200 ×1. EU forwarded 404s: 384 empty-id /conversations//* plus 2 scanner probes. Removed /cable traffic US 1 / EU 1,611. From 02:35Z through this 03:20Z cutoff, no non-health Rails controller call in either region (see the 04:20Z refresh below for the first later call). These are the same 25 successes recorded at 01:20:02Z and 02:20Z ageing through the window, not new forwards; the first fully post-deploy slices remain at zero meaningful forwards. The Kuda native upload canary is still unproven — no natural ciphertext_fallback, response=true or outcome=created event has been observed — and absence of forwards is not evidence for it.

04:20Z operator refresh (trailing 24 h ending 04:20Z; same shapes, one hour later). Gateway events US 1,244 / EU 4,195. The 25 successful forwards are unchanged and all pre-cutover residue. Forwarded 404s: US 8 scanner; EU 384 empty-id plus 2 scanner. Removed /cable US 1 / EU 1,611. Exact 7 d gateway US 39,792 / EU 25,845, pre-cutover dominated, not a post-deploy zero. Fully post-canary slice 02:35:13Z → 04:20Z: zero outcome=forwarded in both regions (US 16 rejected 404; EU 44 Cable removed + 50 rejected 404). Non-health Rails controller calls in the slice: one US POST report combined_email 200 at 04:00Z, EU zero — a direct Rails call on the report path, not a gateway forward, so it does not alter the zero-forwarded reading. 04:05Z expiry ticks in both regions: candidates=0, deferredToRails=0, rails=null. Direct Rails image successful GETs: 24 h zero in both regions; exact 7 d US 2,594 / EU 2,220 = 4,814, all pre-cutover. Kuda native intake still unproven; flag not widened. The ENG-3378 coordinator audits at the same cutoffs live in #12834  and are the authority for internal-effects, schema-gate and Rho report-delivery (flag 285273) status; this subsection records only the empty-id / public gateway view.

Source. The audited request set (GET /conversations/{id}, /{id}/thread, /{id}/navigation, /{id}/applicable_scorecards) is the fan-out of prefetchConversation() / prefetchCriticalConversation() in rulebase-ui/app/features/conversations/components/prefetch-conversations.ts, called from list-row, drilldown-row and thread-item hover handlers and from evaluation prev/next navigation. Neither function guarded conversationId, and openapi-fetch substitutes "" verbatim, yielding /conversations/ and /conversations//…. The specific caller passing an empty id cannot be pinned from source alone (ids are typed string, several are nullable at runtime); the fix therefore sits on the shared prefetch entry points.

Change. Both prefetch functions return a resolved promise without issuing a request for an empty or whitespace id, with focused vitest coverage. UI-only; no API2, flag, data, or schema change. The #12820 308 normalized backstop stays in place.

Gate. One complete 24 h window per region after the UI deploy with legacy_rails_gateway outcome=normalized for /conversations/ and /conversations//* at zero and outcome=forwarded for the same shapes at zero. Rollback: revert (no data effect).

Bounded operator activation still outstanding (not performed here). The #12613 CloudFront /conversation_images/* behavior remains the only remaining route for the three direct US media reads; its activation and rollback criteria are unchanged in the direct media section.

Legacy API-key digest gate closed in API2 (2026-09-23, ENG-3375 )

Evidence. Exact 24 h legacy_rails_gateway census ending 2026-09-23T01:20:02Z (supplied by the operator; CloudWatch not reachable from the authoring session). Forwards Rails answered non-404: US POST /conversations/upload 201 ×1, EU GET /conversations/ ×23, EU GET /conversation_fields ×1 — every one timestamped before or overlapping the #12820  deployment at 00:56Z, so they are pre-cutover residue of the ENG-3371 normalization package, not post-deploy behaviour. First fully post-deploy slice 00:57Z–01:20Z: zero meaningful forwards; four US scanner 404s only. The 7 d window was not refreshed at this cutoff; the last exact 7 d figures remain those recorded above and still contain pre-cutover upload 201s (~700–900/day), which must be kept separate from the 24 h count.

Residual chosen. With Cable deleted, normalization deployed and the media edge change awaiting Ronald, the only supported public family still forwarding is POST /conversations/upload, and its native owner has been blocked since 2026-09-18 for one reason: nativeDecision() authenticates before evaluating conversation_upload_intake_native, API2 resolved keys by key_digest only, and Kuda’s key has no digest. Rails authenticates that key via ApiKey.active.find_by(key: token) — an exact-equality match on the encrypts :key, deterministic: true ciphertext.

Decision. authenticateExternalApiContext performs the same lookup Rails does, in the same order: digest first; on a miss, and only when API2 has ENCRYPTION_DETERMINISTIC_KEY + ENCRYPTION_KEY_DERIVATION_SALT (the same secrets API2 already uses to mint keys in api-key-writes), compare the deterministic ciphertext of the presented token to api_keys.key with active = true. No decryption, no plaintext at rest, no widening of who can authenticate: the accepted set is exactly Rails’ accepted set. A ciphertext match writes key_digest/key_prefix via COALESCE (never overwrites) and logs event=external_api_key_auth outcome=ciphertext_fallback api_key_id=…, which is the lazy, per-key form of Maintenance::BackfillApiKeyDigestsTask. Without the env the code path is inert and behaviour is byte-identical to today. pglite coverage: ciphertext auth + backfill then digest-only re-auth, COALESCE non-overwrite, env-absent 401 with no write, inactive / wrong-token / wrong-key-material 401s. No flag, schema, provider, production-data or replay change by automation; the maintenance task stays documented as the bulk alternative.

Canary gates (natural Kuda traffic only). (1) exactly one ciphertext_fallback for Kuda’s api_key_id, then none; (2) first natural $feature_flag_called response=true for kuda-pilot after that timestamp; (3) conversation_upload_native_intake outcome=created ≈ Kuda’s daily volume, Kuda unauthenticated_fallback → 0, no invalid/storage_failed/insert_failed, handoff=accepted ≥ 99.9 %; (4) 1:1 upload/blob/attachment/delivery/conversation reconciliation over one 24 h window; then Kuda outcome=forwarded path=/conversations/upload at zero for a complete window before the proxy-removal PR.

Rollback. Flag to 0 % (proxy resumes within 20 s; Rails accepts the key either way). Auth fallback: revert the PR, or unset the env on API2 to make it inert. A backfilled digest needs no rollback — it is the value the Rails task would have written. Triggers: any ciphertext_fallback for an api_key_id that does not authenticate on Rails in the same window, any rise in public 401s for previously-201 clients, or native intake outcomes failing gate 3.

Legacy object-field prefixes retired (step 4, 2026-09-23, ENG-3383 )

Evidence (no fresh telemetry). The coordinator’s AWS SSO expired at 05:21Z, so this package makes no new production claim; it works from the exact censuses already recorded above. Across the operator-fixed 24 h windows ending 01:20Z, 03:20Z and 04:20Z, the only forwarded request under /conversation_fields, /customer_fields or /employee_fields was one EU GET /conversation_fields 200, timestamped before the 00:56Z #12820 cutover and already classified as stale-client traffic in the ENG-3371 table; the fully post-canary slice 02:35:13Z → 04:20Z had zero outcome=forwarded in both regions. Static verification on main (7861d24f4): no file in rulebase-ui, rulebase-admin, rulebase-beam-web, rulebase-zendesk-app or api2 builds any of the three legacy URLs (the 0/0/0/0/1 api2 count in the 2026-09-21 classification was doc comments in routes/object-field-reads.ts, routes/object-field-writes.ts and lib/employee-payload.ts), no RAILS_INTERNAL_BASE_URL caller touches them, and every first-party read/write already goes to the API2-native /conversation-fields, /customer-fields and /employee-fields routes (object-field-reads.ts, object-field-writes.ts). /company_fields was retired in batch 2 for the same reason.

Decision. Same shape as /knowledge_base_documents: API2 owns the boundary natively under a different path, so the three legacy prefixes move from rails-public-route-prefixes.json to retired-rails-route-prefixes.json (allowlist 74 → 71, retired 86 → 89) and are answered with Rails’ JSON 404 and outcome=retired without contacting Rails. Rails’ show route has no first-party caller and no successor. PUBLIC_ROUTE_OWNERSHIP_CONTRACTS gains the twelve native successor routes (object-fields family, anonymous 401), and UNOWNED_PUBLIC_REQUESTS replaces the ENG-3371 forwarded contract for /conversation_fields/ with retired entries for the legacy shapes (the retired check runs before normalization). Classification detail is in the prefix retirement runbook. No flag, no data effect, no operator step; Rails routes unchanged.

Rollback. Revert the PR, or LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward on the API2 web service for an immediate hold. Trigger: any outcome=retired event under these prefixes from a current UI build or a non-browser client.

Post-deploy gate. One 24 h window per region with outcome=retired for the three prefixes at zero or stale-tab residue only, and no forwarded event for them (the gateway no longer can). Kuda native intake remains unproven and its flag unchanged; this package does not touch upload, media or report paths.

Merge record. #12840  was merged by Chidi at 2026-09-23T06:40:54Z as 84199d09 with no submitted Ronald review (human merge; not a retroactive approval). ECS at read time: US API2 web TD1727 running release-prod 1c5886a (sync #12841, includes #12840) 2/2; EU TD1745 c463d40 still IN_PROGRESS with 0 running behind active TD1743 b905bc2; Rails 3/3 in both regions. The post-deploy gate is therefore not credited: it needs EU rollout plus natural outcome=retired telemetry per region, not the merge. #12838  (ENG-3381) was likewise merged by Chidi at 07:28:53Z as 3ac4f8509 with no submitted Ronald review.

Scanner-shaped paths under live prefixes rejected locally (2026-09-23, ENG-3385 )

Evidence (operator-fixed CloudWatch cutoff 07:25Z; no query by this session). 24 h gateway events US 20,731 / EU 28,648. Successful Rails forwards 25, unchanged and all pre-cutover (US POST /conversations/upload 201 ×1; EU GET /conversations/ 200 ×23, GET /conversation_fields 200 ×1). Forwarded 404s US 299 / EU 704 (EU = stale empty-id residue plus scanners); removed Cable US 1 / EU 1,588. The slice 04:20Z → 07:25Z has zero successful forwards in both regions and only scanner 404s (US 291 / EU 318). Seven-day gateway US 59,304 / EU 50,722, pre-cutover dominated. These windows are distinct from the 01:20Z/03:20Z/04:20Z censuses above and scanner traffic is not counted as a meaningful residual anywhere in this plan; the remaining meaningful post-cutover forward count is zero, the Kuda native-intake canary is still unproven, and nothing here infers otherwise.

Source classification. The post-cutover forwards are dotfile and PHP-style probes under live prefixes (/admin/.env, /admin/config.php, /agent/vicidial.php). /admin (68 Rails routes, rulebase-admin Admin::BaseController, 38 admin-app callers) and /agent (Agent::BaseController) are live Rails families and stay on the allowlist; retiring the prefixes is not justified by scanner traffic. The source proof has two strengths. Dot-leading segments are unroutable: Rails’ default dynamic-segment pattern [^\/.?]+ never matches a segment beginning with ., routes.rb declares no custom segment constraints and no literal dotted path, and the only glob route (*filename under /rails/active_storage) is excluded — every such forward is a guaranteed Rails 404. PHP/ASP/JSP/CGI suffixes are traffic-proven, not routing-proven: none is a registered Rails MIME extension (config/initializers/mime_types.rb is absent; the suffix set is disjoint from json/csv/xml/html/pdf/zip/… by focused test), routes.rb declares no such path and no first-party client emits one; where the suffix lands on an :id(.:format) position (e.g. /accounts/index.asp) Rails would route it, authenticate and fail the lookup, so no legitimate request is lost, but the response is not structurally guaranteed to be 404. Both forward classes spend a round trip and the internal credential for nothing.

Decision. isScannerShapedPath() in legacy-rails-gateway/compatibility-allowlist.ts: when the unknown-path policy is reject, a path under an allowlisted prefix whose any segment starts with . or whose last segment ends in .php|.phtml|.asp|.aspx|.jsp|.jspx|.cgi is answered with Rails’ JSON 404 and outcome=rejected without contacting Rails; /rails/* is exempt. Allowlist and retired lists are unchanged (71/89). UNOWNED_PUBLIC_REQUESTS flips /admin/.env and /agent/vicidial.php from forwarded to rejected and adds forwarded contracts for real routes under the same prefixes (/admin/feedback_statistics, /agent/statistics) plus an Active Storage dotfile that must still forward. No flag, no data effect, no Rails change.

Rollback. Revert the PR, or LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward (which already bypasses this check together with the allowlist). Trigger: a rejected event for a path a first-party client or Rails log shows as a real route.

Post-deploy gate. One 24 h window per region in which every gateway forwarded event resolves to a Rails route (no 404 forwards), and rejected absorbs the dotfile/PHP probes. This changes the shape of the scanner column only; the meaningful-forward gate, the Kuda gate and the object-field gate above are unaffected.

Refresh (operator-fixed CloudWatch cutoff 2026-09-23T13:30:00Z, epoch 1790170200; read-only regional queries, none by this session; distinct from the 07:25Z census). 24 h gateway US 20,601 / EU 28,508 = 49,109. Successful Rails forwards 23, all old pre-cutover (US POST /conversations/upload 201 ×1; EU GET /conversations/ 200 ×22 — the /conversation_fields success has aged out of the window). Forwarded 404 US 297 / EU 672 = 969 (scanner plus stale empty-id); EU removed Cable 1,437; direct Rails image successful GET 0. Rails /internal 24 h US 44 / EU 1 = 45 (US: evaluation check_runs mark_failed 13, expiry 12, upload mark_failed 6, field PUT 4, reconciliation 3, reports backfill 2 + combined 2, Unit21 1, Slack 1; EU reconciliation 1); all 13 evaluation mark_failed and the Slack call fall in the 04:15Z → 13:30Z slice, EU 0 there. Post-04:15Z: 14 US Rails internal calls, zero EU, and zero successful gateway forwards in both regions. Seven-day: Rails internal US 50,375 / EU 72,598 = 122,973; gateway US 59,414 / EU 51,524 = 110,938; direct image GET 200 US 2,200 / EU 1,829 = 4,029, pre-cutover dominated. The 24 h and 7 d zero gates therefore remain open (pre-cutover residue still inside both windows). Rollout: release-prod moved to 2fd3b757 at 13:33:50Z; the running ECS SHA was not verified, so neither the #12838 nor the #12840 deployed gate is credited. At that cutoff #12839, #12845 and #12847 were open with green checks and no Ronald approval (state superseded below); Rho flags held, no scale cut. Correction from the 19:30Z per-row reconciliation: the 24 h evaluation check_runs mark_failed count is 15, not 13 (1 at 08:07Z, 6 at 09:07–09:08Z, 8 at 13:07–13:39Z).

Refresh (operator-fixed cutoff 2026-09-23T19:30Z; none queried by this session; distinct from 07:25Z/13:30Z). 24 h gateway US 20,732 / EU 27,784. Zero successful Rails forwards observed in both regions — an observation only: the window starts 2026-09-22T19:30Z, before the #12838 UI source fix merged and deployed and before the post-cutover rollout completed, so it is not an all-post-cutover 24 h window and does not meet the 24 h gate. Forwarded 404 618 (US 298 across 91 distinct paths / EU 320 across 89), all status 404 and probe-like, but not all match the #12847  predicate: e.g. POST /clients/MyCRL and POST /rails/actions (2 each per region) have no dotfile segment or PHP/ASP/JSP/CGI suffix and will keep forwarding. Only the dotfile/PHP-like subset moves to rejected after deploy; the remainder stays in the separate 24 h forwarded-404 observation gate, which this package does not close. EU removed Cable 1,431 (7 d gateway US 59,627 / EU 52,239, pre-cutover dominated). The canonical 19:30Z figures — Rails /internal 24 h/7 d breakdown (US mark_failed 15), direct image GET 7 d, Slack classification and the exact-Rho Slack canary (flag 286616), release 290c0ca4 attribution — live in the coordinator audit section “Coordinator read-only audit at 2026-09-23T19:30:00Z” (#12890 ) and are not restated here. Both the 24 h post-cutover zero gate and the 7 d zero gate remain open; deployed-SHA verification, the Kuda canary and the object-field/empty-id post-deploy gates remain uncredited and no Rho flag or scale change follows from this package. #12847 was red at 5e9790d6c on API2 shard 4 (instruct-tool-ui-coverage missing the create_agent_monitoring_monitor label, introduced on main by #12846 and fixed on main since); it was re-merged with main for green checks and still has no Ronald review. Both operator AWS SSO profiles expired at 2026-09-24T01:35Z; the next exact census is the operator-supplied 07:40Z read below.

Refresh — exact fixed CloudWatch cutoff 2026-09-24T07:40:00Z, scanner scope only. 24 h gateway US 1,124 / EU 2,260 events; zero successful Rails forwards in both regions (observation only, no 24 h post-cutover or 7 d zero-window credit). Forwarded Rails 404 US 10 / EU 2 = 12, classified per path against the #12847 predicate as source-shipped (not deployed):

So 8 of 12 match the predicate; 4 keep forwarding after deploy and stay in the separate forwarded-404 observation gate. /agent/auth and /agent/identity are not counted as scanners: they are unroutable in Rails by source, but the external caller is unknown and needs attribution (user agent / source IP) before dismissal. EU websocket_bridge_removed 703 are stale/blocked upgrade attempts, not supported live Cable. Direct Rails successful ConversationImages GET 0 / 24 h both regions. 7 d gateway US 59,470 / EU 52,383 = 111,853 and successful direct images US 1,627 / EU 964 = 2,591, pre-cutover dominated. API2 running SHA afabdfcd complete across the four web/worker services; Rails 3/3 each; #12847 is not in that SHA, so no outcome=rejected telemetry exists yet and no deployed gate is credited.

Refresh — exact fixed CloudWatch cutoff 2026-09-24T15:40:00Z, operator read (session has no AWS access). 24 h gateway by outcome: US rejected 2,220 / forwarded Rails 404 16 / retired 12 / blocked 8; EU rejected 1,610 / forwarded Rails 404 2 / retired 2 / websocket_bridge_removed 224. Zero successful Rails forwards in both regions — observation only, no 24 h post-cutover or 7 d zero-window credit. #12847 merged to main as af5b2a4f7 at 2026-09-24T15:17Z (human merge, no submitted Ronald review — no retroactive approval implied). The latest API2 rollout a6b5071 completed ~15:33Z; whether that SHA includes af5b2a4f7 is unverified from this session, so the rejected counts above are attributed to the pre-existing unknown-path (non-allowlisted) rejection, and the scanner branch is credited no deployed telemetry. Post-a6b5071 slice 15:34Z–15:40Z: US no gateway events, EU four rejected 404 only. Exact 24 h forwarded 404 method/paths (all GET), classified against the predicate:

17 of 18 match the predicate; bare /admin stays forwarded. No /agent/* forwards in this window, so /agent/auth / /agent/identity remain caller-unattributed from the 07:40Z window only. All 18 are treated as pre-#12847-deployment residue ageing through the 24 h window; the deployed gate needs a full 24 h per region with outcome=rejected observed for these shapes and forwarded 404s limited to non-predicate paths. Direct Rails successful ConversationImages GET 0 / 24 h; 1,506 / 7 d, pre-cutover.

Seven resources prefixes retired; wildcard-middleware ownership shadow fixed (step 4, 2026-09-25, ENG-3463 )

Evidence (no fresh telemetry). This package makes no new production claim; the latest censuses remain the 07:40Z, 15:40Z and 17:45Z windows above (zero successful Rails forwards in both regions, every forwarded 404 under /admin or /agent, none of these prefixes present). The trigger was code evidence: an exact-match ownership check against bin/rails routes showed routes/org-directory-mutations.ts guarding /teams/*, /partners/*, /memberships/* and /invitations/* with a method-scoped wildcard middleware, which the gateway’s matchedRoutes predicate counts as native ownership. Unported Rails routes under those shapes — POST /invitations (InvitationsController#create) and PUT /partners/:id — were being answered 404 locally with no legacy_rails_gateway event, so no census could have surfaced them. This is a live Rails dependency the gateway was hiding, not scanner cleanup.

Change. Middleware attached per route; native PUT aliases for the Rails resources update verb on /assignment_queues, /compliance_agents, /controls, /memberships, /partners, /teams, /work_schedules and PATCH /roles/:id; the seven prefixes with full exact-method parity moved to the retired list (allowlist 71 → 64). /memberships is retired with two action-less Rails routes (POST /memberships, POST /memberships/:id/deactivate) whose Rails answer is already 404. /partners and /invitations stay live. UNOWNED_PUBLIC_REQUESTS pins POST /invitations and POST /partners/123/nope as forwarded. Details and the step 2 addendum in the prefix-retirement package. No Rails, flag, data, schema or deploy change. Gate: one 24 h window per region post-deploy with outcome=retired for the seven prefixes at zero or stale-tab residue; outcome=forwarded for POST /invitations appearing is expected and becomes the next port candidate.

Six compliance-program resources prefixes retired on PUT alias parity (step 4, 2026-09-26, ENG-3469 )

Evidence (no fresh telemetry; source audit only). This package makes no new production claim. No AWS credentials were available to the authoring session, so the latest censuses remain the recorded 2026-09-24 windows above and the last recorded gateway window with zero successful Rails forwards; none of them lists these six prefixes as forwarded, and no new zero window is asserted. The trigger was an exact-match ownership audit on main 53c53de5d (ENG-3463 step 2 addendum: bin/rails routes against the composed api2 router with wildcard registrations excluded). For /agent_test_columns, /compliance_criterion_groups, /program_risk_factors, /program_sampling_rules, /program_test_questions and /test_risk_scenarios the only Rails method/path pair without an exact api2 handler was PUT /:id, the Rails resources alias of update; every first-party caller (rulebase-ui, admin, beam, core, api2 RAILS_INTERNAL_BASE_URL clients) uses rulebaseApi2.PATCH/GET/POST/DELETE and none sends PUT or targets Rails for these prefixes (0 callers each). Classification: no meaningful live traffic can reach Rails under these prefixes except an unknown-client PUT, which is now answered natively.

Change. Native PUT /:id alias registered beside PATCH /:id with the same route config and handler (the ENG-3463 pattern); six prefixes moved to the retired list (allowlist 64 → 58, retired 96 → 102); six session-owned ownership families (23 exact method/path pairs, anonymous 401) and retired pins for never-routed paths under each prefix. Details in the prefix-retirement package. No Rails, flag, data, provider, job, scale, CloudFront or deploy change; no operator activation performed. Gate: Ronald review and green required checks, then one clean 24 h window per region post-deploy with outcome=retired under the six prefixes at zero (non-zero = stale tab or unknown client to trace before the next batch). Rollback: temporary LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward; permanent revert of the PR.

Remaining public-route gaps after this batch (same audit). Live prefixes with unported Rails routes: /invitations (POST /invitations create, GET /invitations/download_template, GET /invitations/import/bulk/download_template, plus action-less PATCH/PUT/DELETE /invitations/:id), /partners (GET /partners/:partner_id/employees), /conversations (23 of 66 pairs), /accounts (7 of 30), /admin (61 of 68), /agent (4 of 4). POST /invitations remains the next port candidate. Direct /conversation_images/:signed_id Rails-origin media is unchanged: the ENG-3297  CloudFront behavior is still unapplied in both regions pending human/Ronald activation review.

/cable bridge removed; conversation-upload proxy removal gates

/cable WebSocket bridge — deleted (2026-09-20). The bridge is no longer a gated compatibility path awaiting a seven-day zero window. Gate 1 was already met: #12426  (6970f3248, 2026-09-17) removed Action Cable from rulebase-ui, so no shipped build opens /cable, and every first-party channel family has a deployed SSE owner. The 017b86ef cutoff (boundary 2026-09-20T01:48:40Z) then drove websocket_forwarded to zero in both regions; the only residue was EU websocket_bridge_disabled events from stale tabs.

Chidi decided not to hold the adapter for the remaining window: “we should completely delete cable window. it’s fine, let clients fail and refresh and switch over.” Accordingly legacy-rails-gateway/cable.ts, its test, the LEGACY_RAILS_CABLE_BRIDGE_ENABLED environment gate and the http-proxy dependency are deleted. legacy-rails-gateway/websocket-upgrades.ts replaces them with a handler that answers every WebSocket upgrade with 404 and logs /cable attempts as outcome=websocket_bridge_removed, so the census keeps its stale-tab evidence without any Rails coupling.

Consequences, accepted. Stale tabs stop receiving live updates and get a 404 on upgrade until refreshed. Rollback is no longer config-only: restoring the bridge means reverting the deleting PR and redeploying. Rolling the UI back to a pre-#12426 build is therefore no longer supported, because that build would re-create Cable demand with no upstream. This closes the /cable line of the Rails shutdown gate — the bridge has no supported consumer and is removed — and removes the last WebSocket dependency on Rails.

Legacy GET /knowledge_base_documents/:id (observed 2026-09-19T21:27:29Z). The 24 h gateway window also holds exactly four forwarded US GET /knowledge_base_documents/:id requests, all 200, all in the same second, EU zero. Rails maps them to Kuda (org 25) manual knowledge documents served by KnowledgeBaseDocumentsController#show (6 queries, 36–53 ms); the same IDs appear in Kuda QA/evaluation context and the burst followed Kuda backfill-fallback activity, so this is classified as meaningful QA-linked traffic, not scanner residue, until disproven. Caller: no first-party code on main emits that path — the UI’s Knowledge Library boundary moved to API2 GET /knowledge-library/documents/:id in #11781 (f626438b2, 2026-09-10) and #11940 (c92f3bd2b, 2026-09-13), Rails builds no knowledge_base_document_url, and the endpoint needs a session (200s rule out scanners). The strongest explanation is the same stale-client pattern as /cable: a pre-#11940 rulebase-ui bundle still open in a Kuda tab, loading a cited document plus its neighbours. API2 already owns the native route, so no new code is required for ownership; the legacy path is served only through the compatibility allowlist (/knowledge_base_documents prefix in rails-public-route-prefixes.json). Retirement package, when the evidence gate passes (zero forwarded /knowledge_base_documents for 24 h on a release newer than the stale build, or an explicit human decision as for Cable): drop the prefix from the allowlist so API2 answers 404 locally, reversible by re-adding the prefix. Not done in this PR; until then these 200s count as live and keep Rails at 3/3 alongside uploads and Cable.

Direct Rails GET /conversation_images/:id (observed 2026-09-19T21:12:53Z to 21:29:20Z). Outside the gateway census, US Rails process_action shows 11 successful ConversationImagesController#show requests in the window from the 21:10:29Z cutoff through 21:34:51Z (EU zero), all on signed /conversation_images/:id paths — mostly an Anti-Scam Email Footer.png signature image plus a few image/IMG uploads. The API2 legacy_rails_gateway stream has zero /conversation_images events over 24 h while native_public_media independently served 85 US + 42 EU conversation images in the same 24 h, so these 11 did not pass through the API2 gateway and are not evidence that the native route failed: they reach Rails by a separate direct/legacy-origin path. Candidate emitters in code: Conversation::Sync::Base/Jira rewrite inline img[src] to conversation_image_url(signed_id) on the Rails default_url_options host (RulebaseConfig#api_base_url), so stored conversation HTML carries absolute Rails-origin image URLs that any consumer (browser rendering an email footer, Jira/Zendesk preview, API2 caption fetch via defaultImageOrigins) loads directly; ImageUrlConverter rewrites such URLs to Active Storage blob URLs only where it is applied. The controller is unauthenticated (signed blob ID only), so 200s do not identify the caller. Classification: meaningful direct Rails dependency until the origin host/consumer is identified; it is outside the gateway retirement gates and does not change the 3/3 hold on its own, but it must be attributed (host, referer family, consumer) before Rails public ingress can be closed. Any code for this (e.g. emitting API2 media URLs from sync, or a redirect) is a separately scoped follow-up, not part of the Cable or knowledge-document packages.

Full-window census (both regions, fixed at 2026-09-19 ~21:45Z). The 11-call slice above understated the dependency. Direct Rails ConversationImagesController#show served 194 successful requests in the trailing 24 h (US 184, EU 10) and 7,221 successful requests over the full trailing 7 days (US 3,633, EU 3,588), plus one EU 404 in the 7-day window; latest US 21:43:17Z, latest EU 15:02:10Z. All bypass legacy_rails_gateway telemetry. In the same 24 h API2 native_public_media independently served 98 US + 42 EU conversation images and one EU Active Storage redirect, so the native route is healthy while legacy-host image URLs remain live. Reading: treat 194/day as a meaningful live direct dependency (not stale-tab residue — volume is steady in both regions) and the 7,221/7d as pre-cutover plus live residual from absolute Rails-origin URLs embedded in stored conversation HTML. Rails stays 3/3. Next step is exact attribution of the old-origin emitter (host, referer family, consumer); any fix is its own focused package after that, not mixed into #12564 or #12574.

Emitter attributed (read-only source trace). The live emitter is API2 itself, not only stale rows: provider-conversation-sync/embedded-images.ts conversationImageUrl() builds every newly imported provider-conversation image URL from RAILS_INTERNAL_BASE_URL, even though API2 already owns GET /conversation_images/:signed_id natively (native_public_media) and the repo has the regional getRulebaseApi2BaseUrl() helper (RULEBASE_API2_BASE_URL is set per region in Flightcontrol). Every new import therefore mints a Rails-origin URL, which explains steady direct Rails hits alongside healthy native traffic. Decision: a separate focused package switches only newly minted URLs to the regional public API2 origin (signed IDs and the /conversation_images/ replay guard unchanged; API2 caption defaultImageOrigins accepts the API2 host), with production-shaped US/EU and replay tests. Existing stored rows are not rewritten, so the old-URL tail keeps Rails at 3/3 until it ages out or a later, separately evidenced package handles it. No deploy from this track.

Status at 2026-09-19T21:53:42Z. Gateway 24 h 6,786 = 5,918 locally rejected/blocked + 33 Rails-forwarded 404 + 835 meaningful (728 US upload 201s, 101 EU /cable forwards, 2 old Cable upstream errors, 4 US GET /knowledge_base_documents reads). Available structured 7 d 36,752 (not a full seven days): 25,988 local/scanner + 4 client aborts + 10,760 meaningful. Cable live through 21:49:04Z, so the cutoff is not deployable and Rails cannot scale to zero. Direct (non-gateway) Rails calls since 21:10:29Z: 48 US, 0 EU = 25 QA-stage + 15 ConversationImagesController#show (through 21:43:17Z) + 4 knowledge-document reads + 4 Unit21 calls (through 21:50Z). #12570 fully green awaiting human review; #12564 review-required with only the known shard-1 halfvec failure; #12574 owns the knowledge-document prefix retirement separately.

Status at about 2026-09-19T23:20Z. Rolling 24 h selected gateway events 6,429 = 5,593 blocked/rejected/forwarded-404 scanner traffic + 836 meaningful/residual: US 728 upload forwards (201, latest 19:15:23Z) + 4 GET /knowledge_base_documents forwards (200, latest 21:27:29Z) + EU 102 /cable forwards (latest 22:51:34Z) + 2 old Cable upstream errors (latest 02:04Z). No US Cable forward in 24 h; EU Cable remains live, so the post-cutover zero clock has not started. Available structured 7 d selected events 36,803 (US 27,736; EU 9,067), still not a full seven-day window. #12564, #12574 and #12575 remain human-review blockers, unmerged; the transcription priority package #12576 / ENG-3284 was closed as superseded by ENG-3283 / #12577 and is no longer tracked here. No deploy, flag, data or replay change.

Status after release-prod 0a6fd1e775faabc7ea0cbc8df404ab3841fb9e69 (healthy; #12570, #12574 and #12575 human-merged and deployed). Rolling 24 h gateway census 6,503 = 5,663 rejected/blocked/forwarded-404 scanner traffic plus 840 meaningful (728 US conversation uploads, 4 knowledge-document calls (the 21:27:29Z burst, still inside the rolling window), 106 EU /cable websocket forwards, 2 old EU websocket upstream errors). Available structured 7 d 36,933, pre-cutover-heavy and not a post-cutover zero window. EU Cable remains live after the deploy: websocket_forwarded at 23:45:02Z, 23:45:55Z and 2026-09-20T00:13:31Z, with Rails registering the connection and streaming ConversationChannel at 00:13:31Z, so the cutoff in this PR is still undeployed and its zero clock has not started. Two direct Rails ConversationImagesController#show reads at 00:17Z are the expected tail of already-stored Rails-origin URLs (new URLs mint on the api2 origin since #12575); no backfill is planned. #12564 is green and review-gated. No deploy, flag, data or replay change.

Fixed cutoff 2026-09-20T01:18:53Z (#12571 human-merged 01:19:47Z, #12589 01:20:05Z; release-prod 35c5dce7 includes both but is not yet proven live). Gateway 24 h 6,272 = US 4,194 + EU 2,078: scanner/local 5,434 (4,910 rejected + 499 blocked + 25 forwarded 404) plus 838 meaningful or recent = 728 US POST /conversations/upload 201 (last 19:15:23Z, pre-cutover residue), 4 US GET /knowledge_base_documents (last 21:27:29Z, pre-retirement residue), 105 EU /cable websocket forwards (last 01:16:02Z) and 1 EU Cable upstream error (last 02:04Z). Structured nominal 7 d 37,092 (US 27,907; EU 9,185; coverage began Sep 17 so not a full seven days) = 15,764 forwarded 404, 9,064 rejected, 1,484 blocked, 2,753 Cable forwards + 2 errors, 2,691 upload 201s, and older Zendesk/Unit21/media/Outreach residue. Cable is the only meaningful gateway path still live at the cutoff; the cutoff in this PR remains undeployed, its zero clock has not started, the adapter stays reversible, and Rails stays 3/3. #12564 fully green, labels present, Chidi and Ronald requested, unmerged. No deploy, flag, data or replay change.

Rails internal effects at the same 01:18:53Z cutoff (query-result sums, corrected): 5,007 / 24 h (US 3,234 + EU 1,773) and 328,830 / 7 d (US 132,073 + EU 196,757). 24 h by family: reconciliation 3,009; call recording 794; direct upload 741; Unit21 144; counter cache 129; QA execute_stage 101; expiry 48; transcription context/write 32; tiny tail 9 (field values 4, report actions 4, Slack 1). EU 1,773 = reconciliation 950 + recording 743 + counters 56 + expiry 24. These are internal Rails-owned effects, separate from the public gateway census above, and do not change the Cable gate.

Post-cutoff release 017b86ef fully deployed (release boundary 2026-09-20T01:48:40Z; census through 02:23:02Z). Trailing 24 h gateway total 6,359 = US 4,187 + EU 2,172. Meaningful/recent old tail: 728 US POST /conversations/upload forwards (last 19:15:23Z), 4 US GET /knowledge_base_documents 200 forwards (last 21:27:29Z) and 102 EU /cable websocket forwards (last 01:28:02Z, before the boundary). Since 01:48:40Z there are zero forwarded or websocket_forwarded events in either region; EU shows 39 websocket_bridge_disabled /cable events through 02:23:02Z (expected stale tabs, rejected locally), US has no non-blocked/non-rejected gateway events. The Cable 24 h / 7 d zero clocks run from 01:48:40Z; the adapter stays reversible and Rails stays 3/3. No deploy, flag, data, replay or CloudFront change from automation.

Fixed cutoff 2026-09-20T03:30:00Z (first census with a >90-minute post-cutoff window). Trailing 24 h gateway total 5,736 = US 3,431 + EU 2,305. Meaningful pre-cutover residue: 728 US POST /conversations/upload 201 (last 19:15:23Z), 4 US knowledge-document GET 200 (last 21:27:29Z) and 97 EU websocket_forwarded (last 01:28:02Z). Since the Cable cutoff at 01:48:40Z there are zero meaningful forwarded or websocket_forwarded events; the only post-cutoff forward is one US scanner GET /admin/config.php 404 at 03:16:27Z, and EU logged 122 websocket_bridge_disabled stale-tab attempts through 03:29Z. Available structured 7 d is 37,406 = US 27,967 + EU 9,439 (not a full post-cutover window). The gateway is therefore quiet; the largest verified live public dependency is now outside gateway telemetry — direct Rails ConversationImagesController#show on the legacy-rails.* origins, see the section below (194 successful reads in the same fixed 24 h).

Conversation-upload proxy (lib/conversation-uploads/proxy.ts). The native intake and status routes exist behind the kuda-pilot conversation_upload_intake_native canary created on 2026-09-18T09:36Z; use that canary’s evidence rather than creating another. Remove the proxy only when:

1. the canary shows multipart, presign, status, and idempotency parity with the Rails responses for the pilot’s full request mix, with no durable handoff losses in reconciliation;
2. the native path is default-on for every tenant in both regions and a complete 24-hour window shows zero legacy_rails_gateway outcome=forwarded events for /conversations/upload* and /v1/conversation_uploads* other than stale-client retries;
3. a mixed-version rollback (API2 native on, Rails route retained) has been exercised once so the Rails route can still absorb traffic if the native flag is turned off.

Both removals ship as separate PRs that delete code only after their window completes; this package records the gates so they are not deleted blindly.

Superseded (2026-09-27). Gates 1–3 above assumed a live Rails upstream to compare against and to absorb rolled-back traffic. With Rails configured at zero replicas in both regions (#13164 ) neither the parity comparison nor the mixed-version rollback can be exercised, and the fallback itself can only fail. The proxy was therefore removed under ENG-3488  with a code-revert rollback and a post-deploy gate expressed on API2’s own terminal outcomes; see the ENG-3488 section.

Conversation-upload Rails proxy fallback and PostHog gate removed (2026-09-27, ENG-3488 )

Evidence available to the authoring session (no fresh production census; CloudWatch/PostHog event queries were not reachable).

• Source audit on origin/main c6c6557bd: four proxyLegacyRailsRequest call sites remain in API2 — lib/conversation-uploads/proxy.ts, routes/public-media.ts, routes/zendesk-app-sidebar.ts, routes/outreach-webhooks.ts. The upload proxy was the only one on a contract that produced a successful public Rails forward in any recorded census (US POST /conversations/upload 201 ×1 at 2026-09-23 01:20Z; ≈700–900/day before the canary).
• PostHog flag conversation_upload_intake_native (id 280845, read 2026-09-26): active, v1, exact kuda-pilot at 100 %, last_called_at 2026-09-26T16:13:31Z, created 2026-09-18T09:36:04Z, review by 2026-12-16. A fresh last_called_at is advisory only (see the intake runbook); it does not prove native volume, and no conversation_upload_native_intake log query was possible from here.
• Rails capacity: flightcontrol.cue on main sets both production Rails services to minInstances: 0 / maxInstances: 0 (#13164) and #13167 made the file a production watch path. Whether the zero-replica hold has reconciled in Flightcontrol, and whether any upload forward has since produced upstream_error/timeout, is unverified from this session.
• #13143  (Rails-web-dark) merged as c3cc5473d; its 24 h window ending 2026-09-26T16:18:57Z is the published figure (six US internal requests, EU zero), not re-queried here.

Change. nativeDecision() in lib/conversation-uploads/public-intake-shared.ts no longer evaluates PostHog and never proxies. It returns the terminal response API2 owes the caller: 401 {"error":"Unauthorized"} (the Rails ApiKeyAuth body) with outcome=unauthorized when the bearer token resolves by neither digest nor deterministic ciphertext; 503 {"error":"Service Unavailable"} with outcome=storage_unconfigured when the regional storage resolver has no credentials. lib/conversation-uploads/proxy.ts is deleted and removed from the RAILS_INTERNAL_BASE_URL source-gate allowlist; CONVERSATION_UPLOAD_INTAKE_NATIVE is removed from posthog.ts. The five public upload contracts keep their exact-route ownership pins and deprecated-alias headers. No Rails route, flag, provider, data or deploy change; the Rails ConversationUploadsController retirement is a later package.

Tests. public-intake.db.test.ts (PGlite composed app): 503/401 terminals, per-request conversation_upload_native_intake outcomes, bearer token absent from every log line, proxyLegacyRailsRequest never called. rails-internal-base-url.source-gate.test.ts fails if the module or allowlist entry returns; app.route-ownership.test.ts unchanged (232 tests across the three files green locally; pnpm typecheck clean).

Rollback. Revert the PR and redeploy API2. A revert only restores the attempt to forward; a working Rails destination additionally requires minInstances: 3 / maxInstances: 10 on rulebase-api-prod and rulebase-api-prod-eu (#13164’s stated rollback) and a deploy. The PostHog flag is not touched by the package; the operator archives it after the revert window.

Gate (post-deploy; unverified until a census is run). One complete 24 h window per region on the deployed release with conversation_upload_native_intake outcome=unauthorized and outcome=storage_unconfigured both zero for every organization that produced Rails upload 201s in the last live census (Kuda), outcome=created at Kuda’s daily volume with handoff=accepted ≥ 99.9 %, and the 1:1 upload/blob/attachment/delivery/conversation reconciliation from the intake runbook §2c clean. Rollback trigger: any unauthorized for a known customer key (means ENCRYPTION_DETERMINISTIC_KEY/ENCRYPTION_KEY_DERIVATION_SALT are missing from the API2 task, or the key is undecryptable and must be rotated with the customer), or storage_unconfigured above zero.

Merge record (review exception). #13196 was merged externally at 2026-09-27T09:56:19Z as df9b31c2ab by Chidi with no submitted Ronald review (review had been requested). The package is on main; whether it is in the running API2 release, and the post-deploy gate above, are unverified from this session. The exception is also recorded on ENG-3488.

Not changed by this package. routes/public-media.ts, routes/zendesk-app-sidebar.ts and routes/outreach-webhooks.ts kept their explicit Rails fallbacks at the time; they are retired by the ENG-3495 section below. Public Rails-only routes still forwarded by the allowlist (POST /invitations, invitation template downloads, GET /partners/:id/employees, parts of /conversations, /accounts, /admin, /agent) remain the next gateway candidates once a census can rank them.

Last three route-level Rails fallbacks removed (2026-09-27, ENG-3495 )

2026-10-01 status correction. #13221  merged 2026-09-28T15:42:33Z as 9486119ea087daebbe399efeeeee3548831086f5; its final-head edge-routing and aggregate CI passed, but GitHub still reports REVIEW_REQUIRED and zero submitted reviews. This is a review exception, not Ronald approval. The historical authoring-session and ledger entries below retain their original timestamps; post-deploy native outcome/error parity remains unverified here.

Evidence available to the authoring session (source audit of origin/main 6087ec7dcf; no CloudWatch, PostHog event or Flightcontrol access, so no production window in this section is fresh).

• After #13196, exactly three proxyLegacyRailsRequest call sites remained outside the wildcard gateway (app.ts), each guarded by LEGACY_RAILS_FALLBACK_ENABLED:

  Route	Fallback trigger	Terminal response without fallback
  routes/public-media.ts (/conversation_images/:signed_id, Active Storage redirect)	Flipper pause_api2_public_media_routes on, or the blob’s service has no API2 storage client	empty 404
  routes/zendesk-app-sidebar.ts	Flipper pause_api2_zendesk_sidebar on, or RULEBASE_BEAM_BASE_URL / ZENDESK_MARKETPLACE_APP_ID missing	503 Service Unavailable
  routes/outreach-webhooks.ts	OUTREACH_WEBHOOK_SECRET unset	500 {"status":500,"error":"Internal Server Error"}

• With both production Rails services at minInstances: 0 / maxInstances: 0 on main (#13164), each branch is a guaranteed transport failure (upstream_error/timeout), not a rollback path: pausing the media route would return a gateway error instead of Rails bytes. Whether the zero-replica hold has reconciled in Flightcontrol is unverified here.

• Their published activation prerequisites are already met: OUTREACH_WEBHOOK_SECRET provisioned in Infisical prod/production-eu at 2026-09-18T15:24Z with unsigned canaries 403 in both regions (section above); RULEBASE_BEAM_BASE_URL=https://beam.rulebase.co in both environments (ENG-3290 note in the sidebar runbook). Both are published facts from earlier sessions, not re-queried now.

• ENG-3297 CloudFront /conversation_images/* behavior: not changed by this session, but edge probes run from the authoring box at 2026-09-27T15:43Z show it is now applied in both regions (by someone else): GET https://legacy-rails.rulebase.co/conversation_images/invalid and the EU equivalent return 404 with API2 response headers (x-request-id, access-control-allow-origin: https://app.rulebase.co / https://eu.app.rulebase.co, access-control-expose-headers: Content-Disposition,Deprecation,Sunset,Link), while every other path on the same hosts (/up, /zendesk-app/sidebar) returns 503 from server: awselb/2.0 (an ALB with no healthy targets, consistent with Rails at 0..0 having reconciled). This is inferred from response headers; the CloudFront distribution config was not read, no signed-id 200 path was probed, and no Rails-origin request count exists to close the direct-media gate. The repo-level edge-routing check failed on its /up still Rails assertion for the same reason (Rails answers nothing); check.sh now also accepts a 5xx from the Rails ALB (server: awselb/2.0) on /up as “still on the Rails origin” and fails on anything else, so a repoint of more than the image path is still caught while Rails is at 0..0. The outcome=loop / 508 guard the README listed as a post-apply check no longer exists because there is no fallback to loop.

• #13130 (ENG-3469) was refreshed against main (b979211b52, clean merge, no conflict) and the previously skipped Rails prefix lint spec/lint/api2_gateway_compatibility_prefixes_spec.rb was run locally against a throwaway pgvector/pgvector:pg15 test database: 2 examples, 0 failures. #13130 stays open with Ronald review as the merge gate.

Change. All three routers drop their fallback / fallbackEnabled dependencies and end locally with the terminal response in the table. Media logs legacy_rails_media_fallback with outcome=paused_not_found, forwardedToRails:false (event name kept for query continuity; it now means “left the native path and terminated locally”); the gateway-hop 508 loop response is removed. Zendesk outcomes paused_fallback / misconfigured_fallback are deleted; paused_unavailable / misconfigured_unavailable remain. Outreach outcome forwarded_secret_unconfigured is deleted; misconfigured remains. The three modules leave the RAILS_INTERNAL_BASE_URL source-gate allowlist, so app.ts (wildcard gateway, still governed by the prefix allowlist and LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS) is the only gateway-fallback entry left. The Flipper pause flags stay as incident switches and are now fail-closed. No Rails route, flag, env, provider, CloudFront, data or deploy change.

Tests. public-media.test.ts, public-media.gateway.test.ts, zendesk-app-sidebar.test.ts, zendesk-app-sidebar.gateway.test.ts, rails-internal-base-url.source-gate.test.ts, app.route-ownership.test.ts (247 tests) and outreach-webhooks.db.test.ts (13, PGlite) green locally; pnpm typecheck clean.

Rollback. Revert the PR and redeploy API2. As with ENG-3488, the revert only restores the attempt to forward; a working Rails destination also needs 3..10 on both Rails services (#13164) and a deploy.

Gate (post-deploy; unverified until a census is run). One complete 24 h window per region on the deployed SHA with legacy_rails_media_fallback reason=paused at zero, native_zendesk_sidebar outcome in (paused_unavailable, misconfigured_unavailable) at zero, native_outreach_webhook outcome=misconfigured at zero, and legacy_rails_gateway outcome=forwarded at zero for /webhooks/outreach, /zendesk-app/sidebar and /conversation_images/*. Rollback trigger: any misconfigured* outcome above zero for a real tenant (an env var was lost on the API2 task; fix the task, the revert would not help). Direct Rails-origin /conversation_images reads are unaffected by this package and closed only when the ENG-3297 apply is confirmed from the CloudFront config (not just the probes above) and a fixed 24 h window shows zero Rails ConversationImagesController#show requests.

Direct /conversation_images/:signed_id reads on the Rails origin

This dependency does not appear in legacy_rails_gateway telemetry: the requests hit the Rails origin directly, so the gateway census undercounts Rails’ live public surface. Read-only census (trailing windows ending 2026-09-19T21:43Z):

Classification: the native route is healthy, so this is not a native-media failure. 194/day is a meaningful live dependency; 7,221/7 d is pre-cutover volume plus the live residual of absolute Rails-origin URLs that are embedded in stored conversation HTML.

Emitter. Read-only source tracing found the live emitter is API2 itself, not only stale rows: provider-conversation-sync/embedded-images.ts conversationImageUrl() built every newly imported provider-conversation image URL from RAILS_INTERNAL_BASE_URL, even though API2 already owns GET /conversation_images/:signed_id natively (routes/public-media.ts) and the regional getRulebaseApi2BaseUrl() helper exists (RULEBASE_API2_BASE_URL is set per region in Flightcontrol: https://api2.rulebase.co / https://eu.api2.rulebase.co). Every new import therefore minted a Rails-origin URL, which explains steady direct Rails hits alongside healthy native traffic.

Package (ENG-3282). Newly minted URLs use the regional public API2 origin; the Rails ActiveStorage signed id and the /conversation_images/ replay guard are unchanged, so /conversation_images/:signed_id verifies identically on API2 and a replay of a rewritten part still downloads nothing and writes no second blob. API2 image-caption defaultImageOrigins() accepts the API2 host alongside the Rails origins. Stored rows are not rewritten and no backfill runs: URLs already persisted on the Rails origin keep resolving through Rails (and are still skipped by the replay guard), so the old-URL tail keeps Rails at 3/3 until it ages out or a later, separately evidenced package handles it. Post-deploy gate: the direct Rails ConversationImagesController#show rate should fall from the ~194/day mint-driven baseline toward only old-URL reads; a flat rate means another emitter exists. Rollback is a revert (new rows would resume minting Rails-origin URLs; API2-origin rows already written stay valid because API2 owns the route). No deploy, provider, or production-data change is part of the package.

Post-deploy census (fixed 24 h ending 2026-09-20T03:30:00Z; #12575 deployed in 0a6fd1e7). Direct Rails ConversationImagesController#show served 194 successful reads (US 177, latest 03:18:00Z; EU 17, latest 01:18:29Z); available 7 d is 7,228 successful (US 3,633 + EU 3,595) plus one EU 404. The rate is flat against the pre-fix 194/day, so the emitter fix did not move the live number: the reads are the stored Rails-origin URLs, and at 194/day they are now the largest live Rails public dependency (larger than counter cache 129/day or expiry 48/day) and the one blocker to scaling Rails web to zero over a weekend.

Exact path. These URLs were minted by API2 from RAILS_INTERNAL_BASE_URL, which in production has been https://legacy-rails.rulebase.co / https://eu.legacy-rails.rulebase.co since #12326 (merged 2026-09-17) repointed it from the canonical api.* hosts (rows minted before that carry api.* and already resolve natively on API2). A read is therefore browser → Rails CloudFront distribution (alias legacy-rails.*) → Rails ALB → ConversationImagesController#show (find_signed + send_data), never touching the API2 gateway. The two origins are distinguishable on the wire: Rails replies carry x-runtime and set session cookies; the API2 native handler does neither. The decision in ENG-3282 to leave stored rows untouched stands (no backfill or row rewrite); the reversible owner has to sit at the edge: a path-scoped ordered behavior /conversation_images/* on each Rails distribution whose origin is the regional API2 ALB, so the same signed id verifies on the already-deployed native handler (routes/public-media.ts) with method, status and header behavior preserved and regional routing unchanged. Rollback is deleting that one behavior. Flightcontrol’s CUE schema cannot express a per-path origin, so — as with the Zendesk sidebar exception — it ships as a guarded operator runbook plus an external probe; API2 additionally refuses to fall back to Rails for a /conversation_images/* request that already carries the gateway hop header, so pausing the native media route while the edge behavior exists cannot loop. Tracked as the separate focused package ENG-3297  / #12613 : head 401c2d9e0 is fully green with fake-aws fixture tests proving apply is a no-op only against the exact managed origin/behavior and that rollback refuses foreign or drifted state. Chidi human-merged it at 03:56:15Z as a00a3bf436 with no recorded GitHub review (Ronald requested); it ships in release-prod 39f11065e. Activation status: the CloudFront /conversation_images/* behavior is unapplied in both regions — merging only lands the runbook, probes and API2 loop guard; a human/Ronald activation review is still required before APPLY=1 is run. No CloudFront, data or deploy change from automation.

Gate test (requester-supplied, clean 44 h window ending approximately 2026-09-21T19:00Z; not independently re-queried — no CloudWatch access from the session). Rails served 758 US (724 200 + 34 404) and 275 EU /conversation_images requests while API2 served 704 US + 773 EU natively with near-zero fallbacks. Roughly 563/day is above the 194/day baseline, not below it. Re-reading the gate: the “flat means another emitter” rule assumed reads track mints. They do not — a stored URL is read every time the conversation is opened (agent view, QA review, evaluation drill-down; Rails send_data replies Cache-Control: private, must-revalidate, so every render revalidates against the origin), and review lags import by days. The Rails-origin cohort is bounded by source: URLs were minted from RAILS_INTERNAL_BASE_URL only between #12326 (a9c91b566, 2026-09-17T03:21Z) and the 0a6fd1e7 deploy of #12575 (merged 2026-09-19T23:30Z); rows before that carry api.* (Cloudflare → API2 native) and rows after carry api2.*. A rising read rate over a still-fresh 2.8-day cohort is therefore consistent with the old-URL tail and is not by itself evidence of a second emitter. Read-only source tracing of main found no second emitter: the only /conversation_images/ minter in API2 is embedded-images.ts conversationImageUrl() (now getRulebaseApi2BaseUrl(), with RULEBASE_API2_BASE_URL set on both api2 web and worker services in both regions); the Rails minters (Conversation::Sync::Base#import_and_update_src, Conversation::Sync::Jira) use routes.default_url_options[:host] = RULEBASE_API_BASE_URL, which is https://eu.api.rulebase.co in the EU CUE and is not set in flightcontrol.cue for rulebase-api-prod (US) — its value lives outside the repo and should be confirmed as https://api.rulebase.co (a legacy-rails.* value there would be a live emitter). ImageUrlConverter rewrites /conversation_images/ to rails_blob_url on the same host, and the API2 QA-feedback renderer (qa-feedback-generation/render/parts.ts) presigns R2 directly.

The decisive discriminator is the blob id, not the rate: the signed id is base64(JSON) -- signature with plaintext {"_rails":{"data":<blob_id>,"pur":"blob_id"}}, so every Rails-served path decodes to a blob id without any secret. Revised gate: SELECT id, created_at FROM active_storage_blobs for the decoded ids — every id must have created_at inside the #12326→0a6fd1e7 cohort window. Any Rails-served blob created after the 0a6fd1e7 deploy proves a second emitter; none proves the tail, which only the ENG-3297 edge behavior retires. Logs Insights on the Rails groups: filter path like /^\/conversation_images\// | parse path "/conversation_images/*--" as b64 | stats count(*) by b64, then decode b64 offline. #12685  removes the offline step: Rails ConversationImagesController and every Active Storage controller log event=legacy_rails_media_request with blob_id, service_name, host, via_api2_gateway, referer_host and user_agent, so blob age, edge host and reader are one query; API2 logs event=legacy_rails_media_fallback with reason and serviceName on every remaining fallback, and serves email_assets / conversation_uploads blobs natively instead of falling back for non-cloudflare services.

Outreach webhook cutover: OUTREACH_WEBHOOK_SECRET provisioning

#12474  is deployed in API2 build cmu70 (US web task definition 1638, EU 1656). The first natural slice after deploy (2026-09-18T13:54Z) proved the allowlist is active: US recorded 35 forwarded 201 conversation uploads, 7 blocked 404, and 3 forwarded 400; EU recorded 6 Cable forwards and 1 rejected 404, with no gateway or Outreach errors. The three US 400s were POST /webhooks/outreach and native_outreach_webhook logged outcome=forwarded_secret_unconfigured status=0 for each: OUTREACH_WEBHOOK_SECRET was not configured for API2 in either region, so Outreach ingress stayed Rails-owned. Deployment alone does not change ownership; the route only validates natively once the secret is present.

Configuration is complete as of 2026-09-18T15:24Z. The existing non-empty SecureString from SSM /rulebase/prod/OUTREACH_WEBHOOK_SECRET was copied into Infisical project 848b62cf-7704-49b0-9f39-70ae0b4c3f19 environments prod and production-eu (both keys existed but were empty; both now match the SSM value exactly; staging was intentionally left unchanged). The API2 web services were force-cycled with no code or task definition change: US ecs-svc/4785599704546838455 on TD 1638 complete at 2/2, EU ecs-svc/2513181737475495035 on TD 1656 complete at 1/1, zero pending, migration locks acquired cleanly. Unsigned POST /webhooks/outreach canaries at 15:28Z returned 403 in every attempt (US 4/4, EU 2/2) and native_outreach_webhook recorded outcome=forbidden_signature status=403 in both regions; no forwarded_secret_unconfigured remains in the canary slice. No provider change was made and no signed or manual delivery was sent. The next gate is natural signed traffic: step 3 below.

API2 does not read application secrets from flightcontrol.cue. Each API2 service starts through infisical run --projectId=$INFISICAL_PROJECT_ID --env=$INFISICAL_ENV (project 848b62cf-7704-49b0-9f39-70ae0b4c3f19; INFISICAL_ENV is prod for api2.rulebase.co, production-eu for eu.api2.rulebase.co, and staging for the dev deployment), so the reviewed configuration path is the same one used for ZENDESK_WEBHOOK_BEARER_TOKEN (see rollouts):

1. (Done 15:24Z.) A human with Infisical and AWS access provisions OUTREACH_WEBHOOK_SECRET into the API2 Infisical prod and production-eu environments (and staging for parity) with the value of the Parameter Store key /rulebase/prod/OUTREACH_WEBHOOK_SECRET, which is what Rails’ OutreachConfig#webhook_secret reads in both regions (the EU Rails service references the same /rulebase/prod/... key in flightcontrol.cue). No flightcontrol.cue change and no Outreach provider change is required. infisical run injects secrets at process start, so after provisioning cycle the API2 web service tasks in the Flightcontrol Production and Production EU environments (a no-code redeploy is sufficient) and confirm the new task definition revisions before verifying.
2. (Done 15:28Z.) Verify per region with the unsigned canary from the ownership contract: POST /webhooks/outreach returns 403 and native_outreach_webhook logs outcome=forbidden_signature (not forwarded_secret_unconfigured).
3. (Open.) Require real signed deliveries to log outcome=accepted with action=created|existing, then reconcile conversations and provider-sync enqueues against the Rails baseline for a complete 24-hour window with zero legacy_rails_gateway outcome=forwarded for /webhooks/outreach. Do not send synthetic signed deliveries and do not change flags, ownership manifests, or the Outreach provider registration before that window closes.

Rollback is unsetting the secret in Infisical and cycling the task: the route returns to explicit Rails forwarding without a code change.

Workstream 2: stop calling Rails for effects

API2 still calls authenticated Rails /internal/* controllers for behavior it does not own natively. “Effect” is intentionally broad: it includes writes, callbacks, context construction, model-bound reads, file lifecycle operations, notifications, bookkeeping, and orchestration.

Measure this boundary at the API2 Rails-client call site and at the Rails controller. Group calls by semantic family, not only by URL. A single generic controller can hide many unrelated tools; several endpoints can implement one workflow.

Initial production baseline

The first complete classification found 432,931 API2-to-Rails calls over seven days, averaging about 61,850 calls per day. Approximately 22 Rails controllers and 30 actions were live. Four families accounted for 375,443 calls, or 86.7% of the total:

This is the preferred implementation order, not descending-volume order. It front-loads bounded primitives, gathers the missing tool distribution before porting generic behavior, and leaves the largest semantic migration until its native building blocks are ready.

Current production baseline

The fixed trailing windows ending approximately 2026-09-18T10:16Z are summarized in Evidence as of 2026-09-18: 47,230 calls in 24 hours, of which 40,268 are meaningful live calls, and 417,996 over seven days, of which 218,426 remain after removing shipped-family residue. The earlier census below is retained for the per-family trend.

The exact rolling windows ending at approximately 2026-09-17T17:08Z show that the effects boundary remains stable while the deployed families begin to roll out of the shorter window. The seven-day total is 433,186 calls: 165,197 in the US and 267,989 in the EU. The rolling 24-hour total is 68,563 calls: 30,013 in the US and 38,550 in the EU.

These four families account for 377,192 calls, or 87.1% of the current seven-day total. None has materially redistributed since the prior baseline. From 2026-09-17T13:08Z through this census, both regions recorded zero legacy attachment-purge and zero legacy image-caption calls. That four-hour signal is longer than the initial smoke slice but does not replace a complete production window or business-outcome reconciliation.

Package 1: attachment purge

API2 already tombstones the business attachment row, then makes one Rails call per attachment to purge Active Storage state.

The native package must:

• detach the active_storage_attachments row transactionally;
• delete the blob row only when it is orphaned;
• remove tracked Active Storage variant records and enqueue deletion of their generated variant objects before deleting the source blob;
• persist the object key and storage service to a durable purge outbox inside the transaction;
• delete the R2 object after commit and retry failed outbox deliveries;
• preserve shared blobs;
• serve retention and manual attachment deletion through the same primitive.

Completion requires no growing purge outbox, no attachment metadata for tombstoned rows, preserved shared blobs, and zero Rails purge calls for a complete production window.

Package 2: image-caption context and writeback

The model call and BullMQ lifecycle already run in API2. Rails still builds the caption context and persists the result.

The native package must:

• resolve and tenant-check the organization, conversation, and part;
• preserve the revision or run ID used for stale-write rejection;
• select up to five supported embedded or attached images within size limits;
• generate direct signed R2 URLs instead of Rails blob redirects;
• build prompt context from the preceding and following five parts;
• persist conditionally as completed, already_captioned, or stale;
• replace both the ordinary caption worker and QA caption-stage calls.

Completion requires comparable captioned, no-image, and stale ratios; continuing caption population; no unredacted output or tenant mismatch; and zero Rails caption context or update calls.

Package 3: generic Rails tool executor

/internal/tools/call combines multiple tool IDs behind one controller. Total request volume is not enough to choose an implementation order.

The first package must record tool_id, region, outcome, and duration at the API2 Rails-client boundary without logging tool arguments. Observe at least one representative business day, then:

• port live read tools to existing API2 query, search, and analytics engines;
• port AI tools to API2’s AI runtime;
• make live mutators typed and idempotent;
• delete unused dispatcher IDs instead of porting dead code;
• replace the generic Rails fallback with an exhaustive native registry.

Completion requires a native handler or a proved-dead disposition for every observed tool ID, contract parity for returned citations and errors, idempotent mutators, and zero /internal/tools/call traffic.

Package 4: QA sampling recovery

API2 owns the schedule and candidate query but still sends selected candidates to Rails in batches. Rails applies credits, eligibility, sampling, audit, counter, transcription, and request-creation behavior.

Do not translate the Rails job line by line. Reuse API2’s native eligibility, scorecard, agent, percentage-sampling, fixed-period-sampling, and evaluation creation implementations. Port only the missing behavior:

• evaluation-credit admission;
• customer-response and untranscribed-call cases;
• work-item completeness;
• skipped-request audit records;
• sampling-counter reconciliation;
• outstanding-transcription reconciliation and handoff;
• deterministic enqueueing and one active request per conversation.

Roll out per organization with decision and skip-reason parity metrics. Completion requires stable ticket-closed request creation, comparable skip distributions, no sampling-counter drift, no duplicate or overdue active requests, successful transcription handoffs, and zero Rails sampling or reconciliation calls.

Direct RAILS_INTERNAL_BASE_URL call sites outside the Rails client (2026-09-21)

lib/rails-client.ts is not the only API2 code that reaches the Rails origin. A source census of RAILS_INTERNAL_BASE_URL (rulebase-web/rulebase-api2/src, non-test files) found 18 references. They bypass createLegacyRailsGateway, so they emit no legacy_rails_gateway event and are invisible on the gateway dashboards; only the Rails request logs see them. Reconciled against this ledger: