Canonical API2 gateway

The gateway milestone establishes public ownership. The remaining forwarding and API2-to-Rails dependencies are tracked in the Rails runtime retirement plan.

Goal 1 of the Rails to API2 migration makes api.rulebase.co and eu.api.rulebase.co resolve to API2 while Rails keeps serving its remaining contracts behind regional legacy-rails.* origins. The milestone shipped in both production regions on September 17. This document records the compatibility fallback in rulebase-api2, the paths it refuses to forward, the deployed edge configuration, and the temporary /cable WebSocket bridge.

Milestone boundary

This is deliberately separate from full Rails retirement:

1. Canonical gateway: api.* and api2.* serve the same API2 application. API2 owns native routes and forwards unmatched legacy HTTP contracts to the regional Rails origin. Existing provider, OAuth, and public API URLs remain valid, so the cutover does not require a blanket callback reconfiguration.
2. Rails retirement: API2 no longer calls Rails, every compatibility route has a native owner or is retired, Action Cable and GoodJob have no required traffic, and the Rails services can be deleted.

Forwarded traffic is compatibility coverage, not API2 ownership credit. A rolling 24-hour Rails web census on September 16 found the remaining public traffic concentrated in provider webhooks, media delivery, conversation uploads, the Zendesk sidebar, and Action Cable. The gateway must retain at least these live contracts in both regions:

• legacy Jira, Outreach, Notion, Slack, Unit21, and Resend webhook paths;
• zendesk-app/sidebar;
• conversations/upload, its presign and status routes, and their historical payload and idempotency behavior;
• conversation_images/* and Rails Active Storage redirects;
• first-party Rails API calls that have not moved natively;
• trmnl/data;
• /cable through its temporary WebSocket route.

Intercom and Zendesk already have native API2 routes. Rails must continue generating externally usable URLs on api.*; do not change its external RULEBASE_API_BASE_URL when changing API2’s internal origin.

Fallback middleware

rulebase-web/rulebase-api2/src/lib/legacy-rails-gateway/ registers a wildcard middleware directly after request-id and request logging in src/app.ts. For every request it decides ownership with Hono’s route match:

• If any method-specific native handler matched, API2 owns the request and the middleware calls next(); CORS, authentication, and the route run as before. Wildcard app.use middleware does not count as ownership.
• CORS preflights are native when the Access-Control-Request-Method has a native handler for the path, so preflights for Rails-only routes reach Rails and its rack-cors configuration.
• Otherwise, when LEGACY_RAILS_FALLBACK_ENABLED=true and the first path segment is a Rails route-family prefix listed in legacy-rails-gateway/rails-public-route-prefixes.json, the request is proxied to RAILS_INTERNAL_BASE_URL before API2’s CORS or auth middleware run, so Rails keeps answering its own CORS, CSRF, and session semantics.
• Any other path (scanner probes, /, unknown /v1/..., PUT /mcp, typos) is answered by API2 with Rails’ public 404 body ({"status":404,"error":"Not Found"}) and outcome=rejected, without a Rails round trip. This applies to every method including HEAD and OPTIONS. LEGACY_RAILS_FALLBACK_UNKNOWN_PATHS=forward restores the old wildcard behavior as a rollback gate while the allowlist bakes; blocked prefixes stay blocked under either policy.
• The allowlist is the Rails route table’s top-level prefixes minus the blocked prefixes below. rulebase-api/spec/lint/api2_gateway_compatibility_prefixes_spec.rb fails when a Rails route family is added or removed without updating the checked-in list, so a new Rails route cannot silently become unreachable behind API2.
• When the flag is off, the middleware is inert and unmatched routes behave as before the change. It remains off in development and is enabled in both production regions.

Forwarding semantics:

• Method, path, and query string are preserved verbatim; the request body is streamed (duplex: "half"), so multipart uploads and chunked bodies are not buffered.
• Hop-by-hop headers and anything named in Connection are dropped in both directions. Host, Expect, and Accept-Encoding are regenerated; the upstream is asked for identity encoding so response bytes pass through unchanged.
• Authorization, API-key, X-CSRF-Token, Idempotency-Key, Cookie, Origin, X-Request-Id, and Sentry trace headers pass through unchanged. A client-supplied X-Internal-Api-Key is discarded and replaced with RAILS_INTERNAL_API_KEY.
• X-Forwarded-Host/X-Forwarded-Proto carry the public host and scheme; X-Forwarded-For is appended with the connection peer; Via gains 1.1 rulebase-api2.
• Every Set-Cookie value is re-appended individually. Absolute Location headers on the upstream origin are rewritten to the public origin; relative and third-party locations (for example R2 presigned URLs) are left alone.
• Requests carrying X-Rulebase-Gateway-Hop are answered 508 so a misconfigured RAILS_INTERNAL_BASE_URL pointing back at API2 cannot loop.
• Response headers must arrive within LEGACY_RAILS_FALLBACK_TIMEOUT_MS (default 60s) or the client receives 504; connection failures return 502; a client disconnect aborts the upstream request.
• Each decision logs one structured legacy_rails_gateway event with outcome (forwarded, rejected, blocked, loop, timeout, client_abort, upstream_error), method, path (no query string), status, duration, request id, and upstream origin.

Never forwarded

These prefixes return 404 from API2 without contacting Rails, even when the fallback is enabled. They stay reachable only on the access-restricted legacy origin:

/rails/active_storage/* is deliberately not blocked: Active Storage redirect URLs are a live public contract. API2 natively owns /rails/active_storage/blobs/redirect/:signed_id/*filename and /conversation_images/:signed_id; other Active Storage paths retain their existing compatibility behavior.

/cable uses a WebSocket bridge until SSE retirement

The fetch-based HTTP fallback cannot carry WebSocket upgrades, so API2’s Node server handles only /cable upgrades with a dedicated streaming proxy to the regional legacy Rails origin. It preserves the browser session and Action Cable subprotocol, adds the internal gateway credential and loop marker, and rejects non-Cable upgrades. Plain HTTP /cable requests still receive 404. The bridge was deleted on 2026-09-20 (see Cable cutoff decision). cable.ts, the LEGACY_RAILS_CABLE_BRIDGE_ENABLED gate and the http-proxy dependency are gone; legacy-rails-gateway/websocket-upgrades.ts answers every WebSocket upgrade with 404 Not Found and logs /cable attempts as legacy_rails_gateway outcome=websocket_bridge_removed. Nothing can reach Rails Action Cable, and re-enabling is a revert plus redeploy rather than an environment flip. The paragraph below describes the superseded gated bridge. This keeps API2 as the public ingress without making the canonical cutover wait for every live channel family to move to SSE.

Action Cable is not bidirectional in the current first-party app: there are no active channel perform calls. The five subscribed event families are:

API2 already has authenticated SSE patterns for conversations, sheets, tasks, and companies. Reuse their authorization, heartbeat, reconnect, and Redis fan-out behavior. Rails and API2 publish compatible envelopes through the shared Redis backend, so API2 can bridge existing streams to SSE before every producer moves. Each endpoint must authorize the resource before subscribing and refetch authoritative state after reconnect because Redis pub/sub is not replayable.

UserChannel, MarketingReviewChannel, SimulationReportChannel, and SimulationSetupRunChannel have no first-party subscriber in current web source. Verify external consumers, then delete them instead of porting them. A September 16 CloudWatch search found 28 US and three EU duplicate-subscription warnings in 24 hours, all for ConversationChannel; successful subscriptions are not logged, so this is a lower-bound signal rather than a volume census.

Repository configuration

flightcontrol.cue defines legacyRailsOriginDev, legacyRailsOriginProd, and legacyRailsOriginProdEu, used for every API2 web and worker RAILS_INTERNAL_BASE_URL. The production values use the regional legacy origins; the development value remains canonical. The HTTP fallback is enabled for both production API2 web services and remains disabled in development.

As of September 17, the deployed edge state is:

• The US API2 CloudFront distribution serves api.rulebase.co and api2.rulebase.co. The EU distribution serves eu.api.rulebase.co and eu.api2.rulebase.co.
• The Rails distributions serve only legacy-rails.rulebase.co and eu.legacy-rails.rulebase.co.
• Cloudflare points the canonical records at API2. Rails continues to generate canonical api.* URLs, and the UI uses canonical api.* HTTP and Cable URLs.
• Native API2 health and public API authentication boundaries, Rails-only uploads and Zendesk routes, CORS preflights, OAuth callbacks, and /cable were probed through the canonical hosts in both regions. Structured legacy_rails_gateway events confirmed the forwarded requests and WebSocket upgrades.

The legacy origins remain publicly routable for operational compatibility, but they are not published in customer docs, generated URLs, or UI configuration. This is a deliberate Goal 1 tradeoff, not an access control. Restricting those origins can be added as separate hardening and is not a gate for the canonical gateway milestone.

Post-allowlist evidence (build cmu70, release-prod 677ba560)

#12474  replaced the wildcard fallback with the explicit compatibility allowlist and added the native POST /webhooks/outreach route. It is live in API2 build cmu70 (release-prod 677ba560, complete and steady in both web regions and both logical production workers; the physical worker names containing -pr- are production, not preview).

Fixed 24 h census ending 2026-09-18T15:07:25Z: 14,605 events (US 12,317; EU 2,288), down 1,082 from the prior 15,687: 529 API2-blocked; 10,940 forwarded or rejected 404 scanner/stale traffic; 3,134 non-404/Cable upper bound; 2 client aborts. The nominal seven-day query still covers only structured telemetry since September 17, not seven full days: 27,006 events; 761 blocked; 16,604 forwarded/rejected 404; 9,637 non-404/Cable; 4 client aborts.

Residue versus live dependencies. The rolling 24 h still contains 644 Zendesk 200s and 267 Unit21 200s, but neither family appears after cmu70 became live; both are pre-cutover residue, not current forwarding. The exact post-cmu70 live slice 2026-09-18T13:54:00Z..15:07:45Z contains only three families:

• US: 81 POST /conversations/upload forwarded 201 (last 15:06:59.715Z);
• US: 9 POST /webhooks/outreach forwarded 400 (last 14:45:12.003Z), each logged by native_outreach_webhook as outcome=forwarded_secret_unconfigured;
• EU: 11 GET /cable WebSocket forwards (last 15:05:54.023Z).

No other non-404/Cable family and no gateway 5xx or upstream error occurred in that slice. The Outreach forwards in that slice were the expected explicit fallback while OUTREACH_WEBHOOK_SECRET was still empty in the API2 Infisical prod and production-eu environments. That was resolved at 15:24Z (SSM value copied into both environments, web tasks force-cycled on the same TDs 1638/1656), and unsigned canaries at 15:28Z returned 403 with native_outreach_webhook outcome=forbidden_signature in both regions (US 4/4, EU 2/2). Native ownership now waits on natural signed deliveries logging outcome=accepted plus a reconciled zero-forwarding window; the steps are in the retirement plan. Do not change providers or flags before that window closes.

Cable cutoff decision (2026-09-19)

Attribution. Current rulebase-ui no longer uses Action Cable: human-merged #12426  (6970f3248, 2026-09-17) removed ActionCableProvider, @rails/actioncable, useActionCable/useChannel and moved every remaining UI family to API2 SSE. The gateway still forwarded 103 EU-only GET /cable upgrades in the 24 h to ~20:36Z (zero US, often near-simultaneous pairs/triples). The matching EU Rails log window contains 99 authenticated ActionCable -- Registered connection events (two organizations, five accounts: org 41 / account 383 = 50; org 33 / accounts 460 = 17, 368 = 17, 373 = 10, 370 = 5), 37 accepted ConversationChannel stream starts and 36 stops on three conversations (4008035 23/23, 4163841 9/9, 4436003 5/4); latest stream start 19:49:06Z, latest registered connection 20:13:02Z. Nearly every forwarded upgrade therefore authenticated — this is not scanner traffic and not unattributed; it is an authenticated stale-client dependency (pre-#12426 browser tabs / cached bundles), since no shipped UI build opens /cable.

Human decision. Chidi, 2026-09-19: “lets just remove it it’s fine let those stale clients fail if needed. the tabs will be refreshed anyway.” This authorizes cutting /cable for those stale clients. The narrowest reversible implementation is the code default flip in env.ts (LEGACY_RAILS_CABLE_BRIDGE_ENABLED true → false); production sets no explicit value for this variable in Flightcontrol, so the code default is what runs. The adapter, its tests and the websocket_forwarded telemetry are retained unchanged for rollback; nothing is deleted.

Expected effect. After the release containing the flip completes, a stale tab’s Action Cable client receives 404 on upgrade and enters its own reconnect backoff; those tabs show no live QA/sheet updates until refreshed, which is the accepted outcome. The gateway keeps counting them as websocket_bridge_disabled, so the drain is observable without Rails. Uploads remain the only live HTTP dependency, so Rails stays 3/3 in both regions for uploads — the Cable cutoff alone does not authorize scaling Rails.

Rollback trigger (superseded 2026-09-20 — the adapter is deleted, so rollback is now a revert of the deleting PR plus a redeploy). Escalate if any of: a customer report of missing live updates on a current UI build; websocket_bridge_disabled from a User-Agent/origin that cannot be a pre-#12426 client; any first-party consumer found to still open /cable; or an SSE regression that requires falling back to Rails Action Cable. Rollback is config-only and needs no code revert.

Zero-window start. The mandatory post-cutover windows begin at the timestamp the flip release is COMPLETED in both regions, not at merge:

1. 24 h window: websocket_forwarded = 0 in both regions (structural, since the bridge is off); websocket_bridge_disabled declining toward zero; Rails Registered connection = 0 for the same period; no rollback trigger.
2. 7 d window: the same, sustained; websocket_bridge_disabled at zero or only from user agents older than the window. Only after this window may the adapter be deleted (gate 2 in the retirement plan); until then the code stays for rollback.

Pre-cutoff baseline (census cutoff 2026-09-19T20:58:21Z). The cutoff is not live and the zero clock has not started: three fresh EU websocket_forwarded GET /cable events after the 8ec16b8d release and zero US. Fixed 24 h gateway census: 6,783 total = 5,920 blocked/rejected locally + 33 forwarded scanner 404s + 830 meaningful (728 successful uploads + 100 Cable forwards + the same 2 old pre-fix Cable errors); total down from 7,602 but meaningful nearly flat against 832. Available-coverage seven-day total (not a full seven days): 36,714 = 10,197 local + 15,762 scanner + 10,755 meaningful. Re-measured at 2026-09-19T21:10:29Z on release-prod 76b9a98a (exact, COMPLETED on all four API2 services): 6,784 = US 4,795 (3,550 rejected 404 + 495 blocked 404 + 22 forwarded 404 + 728 upload forwards) + EU 1,989 (1,868 rejected 404 + 7 blocked 404 + 11 forwarded 404 + 101 websocket forwards + 2 websocket upstream errors); i.e. 5,920 local blocked/scanner + 33 Rails-forwarded 404 + 831 meaningful live. Latest upload 201 19:15:23Z; latest Cable forward 20:59:46Z. Cutoff undeployed; clocks not started.

Post-cutover evidence (release-prod 279ae48c, census cutoff 2026-09-19T18:15:08Z)

Current runtime. release-prod 279ae48c is complete in both regions with the exact commit everywhere: US web TD 1651 1/1, worker TD 1641 5/5; EU web TD 1669 1/1, worker TD 1658 5/5; pending/failed 0; migration startup clean. It follows 2d02ce18 (see Fix deployed below); both contain the d69a1b41d Cable subscriber fix.

Fixed 24 h census ending 2026-09-19T18:15:08Z (coordinator-measured): 8,586 events (US 5,496; EU 3,090), down 21.2% from 10,899. Exact split: 7,652 blocked/rejected locally (never reached Rails); 50 forwarded stale/scanner 404s; 884 meaningful non-404/Cable upper bound = 790 conversation-upload successes + 92 Cable forwards + the same 2 old pre-fix Cable upstream errors. Available-coverage seven-day total — exact over the structured telemetry available since 2026-09-17, not a full seven days: 36,507 (US 27,491; EU 9,016) = 10,015 blocked/rejected + 15,761 forwarded 404 + 10,731 meaningful upper bound; meaningful families over that coverage are Cable 2,714, uploads 2,687, Zendesk 911, Unit21 529, Outreach 232, retired media 3,599, other 59.

Cable decision at 18:15:08Z. Post-fix task restarts and natural EU/US SSE reconnects on 2d02ce18 and 279ae48c are clean with no new handshake race; Sentry RULEBASE-API2-F1 remains count 2 / users 2, last 2026-09-18T20:52Z on e45afa48, with no event on 2d02ce18 or 279ae48c. The first restart proof is therefore clean, but retain the /cable bridge pending broader natural reconnect/cutover proof and the remaining channel families; the Sunday criteria below govern when LEGACY_RAILS_CABLE_BRIDGE_ENABLED may be flipped.

Newer fixed 24 h census ending 2026-09-19T18:45:18Z (coordinator-measured, recorded after the processing rollback below): 7,623 events, down 963 (11.2%) from 8,586. Exact split: 6,730 blocked/rejected locally (never reached Rails); 40 forwarded Rails 404/scanner; 853 meaningful (down 31 from 884) = US 755 POST /conversations/upload 201 + EU /cable 96 websocket_forwarded + the same 2 old upstream_errors. Current-24 h live dependencies are only uploads and Cable. Available-coverage seven-day total (still not a full seven days): 36,570 = 10,072 blocked/rejected + 15,761 forwarded 404 + 10,737 meaningful upper bound.

Current release (release-prod 8ec16b8d858d3047745e265e8829749b779baf73, release sync #12558 human-merged 2026-09-19T19:26:44Z). Fully healthy/exact in both regions: US web TD 1654 1/1, logical prod worker (physical zy7q0) TD 1644 5/5; EU web TD 1672 1/1, logical prod worker (physical m9780) TD 1661 5/5; pending/failed 0; every task definition on the exact SHA; no level ≥ 50 first-slice web/worker errors. The release replaced workers as expected and caused no gateway regression. Coordinator re-verification at 19:36:57Z confirmed the same exact SHA and COMPLETED state on every task above with pending 0 / failedTasks 0. It supersedes 5e3c514 below. This is a deployment boundary only: no new runtime-ownership claim is made until fresh natural gateway evidence on 8ec16b8d is recorded, and Rails stays US/EU 3/3 because uploads and EU Cable remain meaningful dependencies.

Fixed 24 h census ending 2026-09-19T19:54:08Z (release 8ec16b8d exact and fully completed in both regions): 7,603 events (US 5,059; EU 2,544). Exact split: 6,729 blocked/rejected locally; 40 forwarded scanner 404; 834 meaningful = 728 successful POST /conversations/upload + 104 websocket forwards + the same 2 old websocket upstream errors. Available-coverage seven-day total (coverage still begins 2026-09-17, not a full seven days): 36,618 = 10,106 blocked/rejected + 15,761 forwarded scanner 404 + 10,751 meaningful. Uploads and Cable remain live, so Rails stays 3/3 in both regions and scale-to-zero is blocked.

First natural slice on 8ec16b8d (from 19:36:00Z to the coordinator check). US emitted 4 locally blocked events and zero meaningful forwards; EU emitted 3 locally rejected events plus one natural websocket_forwarded GET /cable at 2026-09-19T19:40:24.909Z with no upstream_error. No upload event arrived in this short slice. This proves the Cable bridge is still live on the exact current release; Rails remains 3/3 in both regions.

Fixed 24 h census ending 2026-09-19T19:24:32Z (runtime 5e3c514 in both regions at measurement time; the 8ec16b8d deployment had just started): 7,602 events (US 5,060; EU 2,542), down 21 from 7,623. Exact split: 6,730 blocked/rejected locally; 40 forwarded scanner 404; 832 meaningful = 728 successful US POST /conversations/upload + 102 EU websocket forwards + the 2 old EU websocket ECONNRESET errors (00:56Z/02:04Z); no new Cable error. Available-coverage seven-day total (coverage still begins 2026-09-17): 36,599 = 10,091 blocked/rejected + 15,761 forwarded 404 + 10,747 meaningful (8,017 non-404 HTTP + 2,724 Cable forwards + 2 websocket errors + 4 client aborts). Rails remains 3/3 because uploads and Cable are live.

Fixed 24 h census ending 2026-09-19T19:11:23Z (runtime release-prod 5e3c5140b0aac8acafa7e8379ae9bbc14a8c7c7, live/healthy in both regions): 7,597 events (US 5,060; EU 2,537), down 26 from 7,623. Exact split: 6,729 blocked/rejected locally; 40 forwarded scanner/stale 404; 828 meaningful = 727 successful US POST /conversations/upload 201 + 99 EU /cable websocket_forwarded + the same 2 old upstream errors (EU read ECONNRESET at 00:56:00Z and 02:04:00Z, pre-fix; no new post-fix Cable error). Available-coverage seven-day total (instrumentation is still not a full seven days): 36,597 (US 27,565; EU 9,032) = 10,089 blocked/rejected + 15,761 forwarded 404 + 10,743 meaningful + 4 client aborts. Uploads and Cable remain the only live gateway dependencies, so Rails stays 3/3 in both regions.

Post-5e3c514 20 min slice. After release-prod 5e3c514 completed, the US gateway emitted only 7 blocked GET /up and 2 rejected GET / 404s; the EU gateway emitted 7 websocket_forwarded GET /cable and no HTTP forwards. The Cable bridge is therefore still a live dependency and Rails must stay at 3/3 in both regions.

Prior fixed 24 h census ending 2026-09-19T17:47:25Z (runtime eae24156, web 1/1, -pr- workers zy7q0 / m9780 5/5, migration locks clean): 8,568 events (US 5,526; EU 3,042), down 2,331 (21.4%) from 10,899. Exact split: 7,609 blocked/rejected locally (never reached Rails); 50 forwarded stale/scanner 404s; 909 meaningful non-404/Cable upper bound = 822 conversation-upload 201s + 85 Cable forwards + 2 Cable upstream errors. Both upstream errors were EU read ECONNRESET at 00:56:00Z and 02:04:00Z, on an older pre-eae24156 task. The current 24 h contains no Outreach, Zendesk or Unit21 forwarding. Seven-day total 36,439 = 9,956 blocked/rejected, plus 15,761 forwarded 404, plus 10,722 meaningful upper bound; retired media is exactly 3,599/7 d (US 912, EU 2,687), last seen 2026-09-17T14:46:10Z, zero in the current 24 h.

Cable status at this cutoff. No subscriber-mode / ready-check race (RULEBASE-API2-F1) was found in the current 24 h or during the eae24156 startup, but eae24156 did not contain the fix, so that alone was not a clean-slice pass. The two EU ECONNRESETs are recorded as Cable upstream-read failures on the old task, separate from the Redis race.

Fix deployed (release-prod 2d02ce18). The subscriber-handshake fix #12526  was human-merged at 2026-09-19T17:44:22Z (d69a1b41d) and shipped via release #12541 as release-prod 2d02ce188799b8488bb2bfeb9ab26007867f2efc, COMPLETED in both regions: US web TD 1650 1/1, worker TD 1640 5/5; EU web TD 1668 1/1, worker TD 1657 5/5; pending 0, failedTasks 0; migration advisory locks acquired EU 17:58:13.243Z, US 17:59:21.806Z. The web first slice has zero error/fatal rows and zero subscriber-mode / ready-check errors. Three natural EU Cable forwards completed cleanly on the new tasks at 18:00:21.303Z, 18:00:21.358Z and 18:01:01.911Z with no upstream error — the first natural post-fix reconnect proof. Cable retirement remains gated, now on a longer clean natural window rather than on review: keep the /cable bridge and LEGACY_RAILS_CABLE_BRIDGE_ENABLED until a sustained post-2d02ce18 slice shows zero RULEBASE-API2-F1 events across at least one web-task restart per region and continued clean natural Cable/SSE traffic.

Conversation uploads: native intake awaits post-release natural traffic

Read-only evidence at 2026-09-19 (weekend fast-track): every conversation_upload row in the last 24 h is Kuda direct/XCALLY (794 rows); gateway logs show 790 legacy POST /conversations/upload forwarded 201s with the last batch around 17:07Z; and the current observed window has zero native-intake outcomes even though the exact-Kuda conversation_upload_intake_native flag evaluates true. Read this as missing post-release natural traffic, not a failed evaluation: the last Kuda batch predates the 2d02ce18 tasks, so no matched request has yet reached the native path.

Processing blocker (2026-09-19T18:20:30Z) — distinguish intake from processing. Two separate flags are in play: conversation_upload_intake_native (exact-Kuda; API2 accepts the POST /conversations/upload request) remains on, and the exact-Kuda direct-processing flag (PostHog flag 281904; API2 runs the XCALLY processing pipeline instead of handing off to Rails) was activated at 18:19:08Z and immediately rolled back / disabled (flag v2) after the first natural upload (conversation_upload 649840) failed terminally with Invalid recorded_at at 18:20:30Z. The existing Kuda metadata value was 2026-09-19 19:20:22 — a space separator between date and time — which Rails accepts but the API2 native-processing parser (native-processing/xcally.ts parseLocalDateTimeInZone, regex ^(\d{4})-(\d{2})-(\d{2})T?(\d{2}):(\d{2})…$, T optional but no space) rejects as NaN, raising the non-retryable ConversationUploadProcessingError("Invalid recorded_at", InvalidMetadata). No human replay or data write was performed; the scheduler automatically retried 649840 after the rollback and it completed through Rails at 18:33:01Z with no duplicate: exactly 1 conversation / part / call detail / recording attachment, delivery completed, replay_count 0. Flag 281904 remains disabled. Upload retirement is not ready: it requires a reviewed and deployed parser fix that accepts the space-separated Rails-compatible shape (with a regression test on 2026-09-19 19:20:22; tracked separately as ENG-3267 / PR #12555 — not in this runbook track), then a fresh natural Kuda batch through the proof plan below with the processing flag re-enabled. Until then, processing stays Rails-owned even though intake is native.

Next-batch proof plan. On the next natural Kuda/XCALLY batch, all of the following must hold before native intake counts as proven:

1. conversation_upload_native_intake telemetry records outcome=accepted for every request in the batch, with organization and request IDs matching the corresponding conversation_upload rows.
2. The legacy gateway family POST /conversations/upload records zero forwarded 201s for the same window (the batch did not fall through to Rails).
3. The resulting conversation_upload rows and downstream conversation / transcription outcomes are indistinguishable from the previous Rails-served batches (same counts, no new RULEBASE-API* upload issues, no Api2ConversationUploadEnqueuer timeouts).
4. No conversation-upload 5xx, ambiguous-delivery or duplicate-row signal in either region.

If any step fails, keep the flag scoped as-is and the Rails proxy live; do not widen the flag over the weekend.

Sunday operational-retirement criteria

The weekend target is zero meaningful Rails gateway traffic and scale-to-zero behind rollback — not irreversible code deletion, which stays behind the 7-day proof in the retirement plan. Rails web can be scaled to zero (with the Rails distributions and the legacy aliases still deployable for rollback) once, for a full observed day ending Sunday:

• The meaningful non-404/Cable upper bound is 0: no forwarded POST /conversations/upload, no Outreach, Zendesk, Unit21, KB-document or /conversations/ forwards. Only locally blocked/rejected traffic and forwarded stale/scanner 404s remain (those may then be answered natively when the wildcard is closed).
• Native conversation-upload intake has passed the next-batch proof plan above for at least one full natural batch in each region that receives uploads, and the recorded_at processing blocker above has a reviewed, deployed fix with the direct-processing flag re-enabled and a fresh clean natural batch.
• Cable: /cable forwards are 0 for the day and the post-2d02ce18 slice shows sustained natural reconnect/restart proof — zero RULEBASE-API2-F1 events across at least one web-task restart per region, with natural SSE opens succeeding on the restarted tasks. Only then may LEGACY_RAILS_CABLE_BRIDGE_ENABLED be flipped off; the flag and bridge code stay in place until the 7-day proof.
• No new gateway 5xx/upstream error, and Sentry RULEBASE-API2-F1 / RULEBASE-API-5WT show no new events in the window.
• Rollback remains a one-step operation: re-scale Rails web and, if needed, re-point the CloudFront aliases; no migration, schema or provider change is part of the retirement step.

Any single criterion failing keeps Rails web running at its current count and resets the day-long window.

Prior evidence (e45afa48 / 06236881, census cutoff 2026-09-18T21:05:00Z)

release-prod e45afa48 (e45afa48c1c887549fa2b374da8a7c6002a9fbff, #12521, main d198f5f7): Flightcontrol GitHub deployment status Success in EU at 20:51:10Z and US at 20:51:38Z. A later production deployment, 06236881, was then the deployment boundary with all services otherwise healthy. Both first slices were not clean for Cable; see Cable first-slice error below. The prior release c47e1113 (c47e11134b535abbef717e66a3a6a731664c95da, #12520) was Success in both Flightcontrol regions, API2 web at 1/1 and API2 workers at 5/5 in each region. The EU worker rollout took 9m54s at rev 1651; its first visible runtime slice at 17:09Z shows natural production processing with no error/fatal row. Earlier deployments on the same day, all of which reached steady with zero level>=50 web errors and no gateway- or Outreach-specific error: 623cf7a7 (#12516; EU 16:35:56Z, US 16:36:16Z; US web TD 1643 / worker fc-rulebase-api2-worker-pr-zy7q0 TD 1633, EU web TD 1661 / worker fc-rulebase-api2-worker-pr-m9780 TD 1650), cf0374d (~16:10Z; US web TD 1641 / worker TD 1631, EU web TD 1659 / worker TD 1648) and c7d66aaa (US web TD 1640, EU TD 1658).

e45afa48 was live and steady: US and EU web 1/1, workers 5/5, zero pending, completed rollouts, both migrations acquired their advisory locks, /health returns 200.

Prior fixed 24 h census ending 2026-09-18T21:05:00Z (coordinator-measured; this track has no AWS/Flightcontrol/PostHog access of its own): 10,903 events (US 7,635; EU 3,268), down from 13,930. Exact split: 9,008 status-404 / blocked / rejected / scanner events; live dependencies — conversation uploads 977, Cable 562, Outreach 132, KB documents 4, /conversations/ 6; pre-cutover residue — Zendesk sidebar 174, Unit21 22; the remaining 18 are robots/probes/client-abort/error miscellany. Cable was last observed in EU at 20:53:00Z. Structured seven-day partial coverage: 29,934 (US 22,872; EU 7,062). A near-simultaneous re-run at cutoff 21:06:56Z gave the same picture with a different classification cut: 10,899/24 h (US 7,630; EU 3,269), 3,910 locally blocked/rejected (never reached Rails), 5,098 forwarded scanner/stale 404s, 1,891 meaningful non-404/Cable upper bound including 2 client aborts; uploads 977, Cable 563, Zendesk 172, Outreach 132, Unit21 21; 7 d 29,935 with 4,280 blocked/rejected, 15,729 scanner 404, 9,926 meaningful upper bound, of which retired-media pre-cutover residue is exactly 3,599 (zero in the current 24 h), leaving 6,327 live/recent upper bound. Sentry at the same cutoff: RULEBASE-API2-F1 unresolved, 2 events / 2 users, first 13:46:31Z, last 20:52:22Z; RULEBASE-API-5WT unresolved, 2 events / 0 users, first 09:52:53Z, last 17:55:40Z.

Measurement boundary. Between the 16:06:07Z census below and the 21:05:00Z/21:06:56Z censuses above, this track could not measure independently: at the c47e1113 evidence the AWS SSO session had expired and PostHog held zero ingested app logs; refresh attempts at 19:12Z and 21:05Z found no AWS or Flightcontrol credentials, PostHog logs-count over -7d = 0, and no rows for gateway/Outreach/Rails event names over 24 h, so Sentry was the only fresh natural evidence reachable. Per-request outcomes inside the 21:05:00Z families (signed Outreach outcome, upload forward status, Cable age) are not broken out and remain unverified until recorded here.

Cable first-slice error (Sentry RULEBASE-API2-F1)

Sentry rulebase-api2 issue RULEBASE-API2-F1 (id 147920053, unresolved, 2 events / 2 users at the 21:05Z census, with a third natural occurrence at 21:12:41.980Z after the 06236881 deployment) is Error: Connection in subscriber mode, only subscriber commands may be used, thrown from the ioredis ready check (event_handler.js → Redis.info). Occurrences:

• 2026-09-18T21:12:41.980Z, release 06236881, US, GET /conversations/<conversation_id>/channel/stream on the freshly restarted web task — action-cable-sse:subscriber ReplyError: Cannot execute INFO while in subscriber mode. Third natural occurrence, on the first stream after a deploy restart, exactly the predicted window.
• 2026-09-18T20:52:22Z, release e45afa48, production, GET /sheets/<sheet_id>/channel/stream — 44 s after the US deployment status turned Success.
• 2026-09-18T13:46:31Z, release 938db2af, a conversation channel stream.

Root cause (from source, reproduced locally against Redis 7). actionCableSubscriberHub() lazily creates one shared ioredis subscriber connection and issues SUBSCRIBE for the first stream immediately. ioredis flags SUBSCRIBE as valid during its connect handshake status, so that command reaches the socket before the ready check’s INFO; Redis rejects the INFO with the error above, ioredis fails every queued subscribe and reconnects. The window is the first SSE request after a web task starts or after any Redis reconnect, which fits the 20:52:22Z event landing 44 s after the US deploy and explains why the issue predates e45afa48 rather than being introduced by it.

Impact. streamActionCableChannel awaits the subscribe before opening the SSE response, so the affected request fails as a 500 with no stream; once ioredis has reconnected the connection is ready and later stream opens succeed. Observed blast radius is one failed stream open per fresh subscriber connection (2 events / 2 users today), not a sustained outage; whether the client recovers without a page reload depends on its EventSource retry handling on a 500 and was not verified here. It is not a Rails-fallback, gateway or /cable bridge fault, and it does not involve the WebSocket path.

Decision. The native SSE replacement for Cable has a live, now three-times-reproduced first-slice error that recurs on every deploy restart, so #12526 is a hard Cable rollout blocker and Cable retirement stays blocked: keep the /cable bridge and LEGACY_RAILS_CABLE_BRIDGE_ENABLED rollback gate as-is until the fix in #12526  (hold subscribes until the handshake settles; real-Redis reproduction and regression test) is human-reviewed, merged, deployed, and a clean natural post-fix slice shows zero new RULEBASE-API2-F1 events across at least one web-task restart per region. No provider, flag or production data change was made for this triage. Update 2026-09-19: #12526 is merged and deployed in release-prod 2d02ce18; the gate is now the sustained clean natural window described under Fix deployed above.

Adjacent upload evidence. Sentry rulebase-api issue RULEBASE-API-5WT (Api2ConversationUploadEnqueuer::AmbiguousDeliveryError: Net::ReadTimeout, ConversationUploadsController#create) has 2 events at 09:52:53Z and 17:55:40Z, 0 users. That is the Rails-side upload intake timing out on its internal enqueue call to API2 after accepting the forwarded upload; it does not measure gateway forwarding but is the only post-17:09Z upload signal visible to this track and belongs in the upload package’s natural-outcome gate review.

OUTREACH_WEBHOOK_SECRET remains loaded in both regions: unsigned canaries still return 403 with outcome=forbidden_signature, no new Outreach delivery has arrived after 16:10Z, and signed Outreach still has no accepted natural delivery.

Prior fixed 24 h census ending 16:06:07Z: 13,930 events, down 675 from 14,605: 528 API2-blocked; 10,529 forwarded or rejected 404 scanner/stale traffic; 2,871 non-404/Cable upper bound; 2 client aborts. The meaningful live families are conversation uploads (989) and Cable (825). Outreach (210), Zendesk (~580) and Unit21 (239) in the rolling window are all pre-cutover residue: from 15:28Z through the cutoff the gateway forwarded 27 US POST /conversations/upload and 11 EU GET /cable upgrades and zero /webhooks/outreach requests. The nominal seven-day structured query is still partial coverage since September 17: 27,332 events (US 21,327; EU 6,005); 780 blocked; 16,650 forwarded/rejected 404; 9,898 non-404/Cable; 4 aborts.

Outreach: native ownership is live, acceptance is on hold

Three natural signed US deliveries at 15:43:18Z, 15:43:19Z and 15:43:50Z passed signature validation and resolved sourceId=15, then returned outcome=malformed_event status=500. No outcome=accepted event exists yet. Treat this as parity/error evidence and a hold, not a proven regression:

• malformed_event is emitted when parseOutreachWebhookEvent cannot read meta.eventName (or data for a call event) from the parsed body. It deliberately mirrors OutreachWebhook#process_event, where event['meta']['eventName'] raises NoMethodError (500) when meta is absent.
• The Rails baseline for the same family is not 200 either: every natural delivery forwarded to Rails in the 13:54Z..15:07:45Z slice came back 400, which on this controller can only come from the JSON ParseError or ActionController::BadRequest path before process_event runs. So neither runtime has produced an accepted natural Outreach delivery in the observed window, and the reconciliation gate in the retirement plan cannot be evaluated until one does.
• Payloads are not logged and must not be pulled to diagnose this. The telemetry attached to malformed_event is sourceId only.

One contract gap is provable from code alone, without any payload: when meta is present but eventName is missing or not a string, Rails evaluates case nil (or case <non-string>), matches no branch, and renders {"message":"OK"} with 200; API2 returns malformed_event 500. The same holds when meta is a JSON string (String#[] returns nil in Ruby). It is unknown whether the three 15:43Z deliveries hit that shape, so the fix is a proposal, not a deploy: make parseOutreachWebhookEvent return an ignored result for a present-but-non-string eventName, keep 500 only for a missing or non-indexable meta, and add payload-free shape telemetry to the malformed_event log line (media type, body length, meta present/type, eventName type, data present) so the next natural delivery is diagnosable. Rollback stays unset-secret-plus-cycle; do not send synthetic signed deliveries, and do not change flags, manifests or the provider registration while the hold is open.

Acceptance and rollback

The milestone is complete in both regions: api.* and api2.* serve the same API2 release, native routes take precedence, representative Rails-only HTTP contracts pass through the fallback, /internal/* and operational paths are blocked at the gateway, and /cable upgrades through API2. Gateway logs expose forwarded method/path families, status, latency, request IDs, and outcomes.

Rollback moves each canonical CloudFront alias back to its Rails distribution and restores the previous Cloudflare target; it does not change provider URLs. Keep Rails and the legacy aliases live through the observation window. Rails retirement remains blocked until compatibility traffic, API2-to-Rails calls, Action Cable, and GoodJob usage reach zero.