Native conversation-upload intake

API2 owns every public conversation-upload contract:

• GET /v1/conversation_uploads/:id
• GET /conversations/upload/:id (deprecated compatibility alias)
• POST /v1/conversation_uploads/presign / POST /conversations/upload/presign
• POST /v1/conversation_uploads / POST /conversations/upload

Cutover: Rails proxy fallback removed (2026-09-27, ENG-3488 )

Why now. The remaining Rails destination for presign/create was the proxyConversationUpload fallback taken when API-key authentication failed, the conversation_upload_intake_native flag was off for the organization, or regional storage was unresolved. Rails production is configured at minInstances: 0 / maxInstances: 0 in both regions since #13164  (watch-path fix #13167 ), so on the configured topology that fallback no longer has a healthy upstream: a forwarded upload would surface as a gateway upstream_error/timeout, not a Rails 201. Keeping the branch therefore only preserved a failure mode; the flag-ring evidence gates written for a live Rails upstream (Kuda ciphertext_fallback → natural $feature_flag_called → 1:1 reconciliation) cannot be satisfied against a zero-replica origin and are recorded below as historical. Whether the 0..0 configuration has actually reconciled in Flightcontrol was not verified from the authoring session.

Change. nativeDecision() no longer reads PostHog and no longer proxies. It returns the terminal response itself:

lib/conversation-uploads/proxy.ts is deleted and dropped from the RAILS_INTERNAL_BASE_URL source-gate allowlist, so a reintroduced upload proxy fails CI. Exact-route ownership pins for the five contracts are unchanged. The deprecated /conversations/upload* aliases still carry the Deprecation/Link headers from annotateLegacyPublicApiResponse.

Tests. public-intake.db.test.ts (PGlite) asserts the 503/401 terminals, the per-request intake log lines, that the bearer token never reaches a log, and that proxyLegacyRailsRequest is never invoked; rails-internal-base-url.source-gate.test.ts fails if the allowlist entry or the module returns. app.route-ownership.test.ts still pins native ownership of the five contracts.

Rollback. Revert the package PR and redeploy API2 (the fallback was code, not configuration). Restoring a working Rails destination additionally requires minInstances: 3 / maxInstances: 10 for rulebase-api-prod and rulebase-api-prod-eu per #13164; without that, the revert only restores the forwarding attempt. The PostHog flag (id 280845, still exact kuda-pilot at 100 % per the 2026-09-26 read) is left for the operator to archive after the revert window; nothing reads it.

Gate (post-deploy, not yet observable). One complete 24 h window per region with conversation_upload_native_intake outcome=unauthorized and outcome=storage_unconfigured at zero for organizations that produced Rails 201s in the last live census (Kuda ≈ 700–900/day), outcome=created at Kuda’s daily volume, and legacy_rails_gateway outcome=forwarded path=/conversations/upload* absent by construction. A non-zero unauthorized count for a known customer key is the rollback trigger: it means ENCRYPTION_DETERMINISTIC_KEY/ENCRYPTION_KEY_DERIVATION_SALT are missing from the API2 task or the key is undecryptable (rotate with the customer). No production census was available to the authoring session; the sections below are the pre-cutover history.

Pre-cutover history (flag-gated rollout, 2026-09-18 → 2026-09-26)

Presigning and multipart direct creation were behind the default-off organization PostHog flag conversation_upload_intake_native. API2 evaluated the flag only after API-key authentication and only when the regional conversation-upload storage resolver had credentials. Otherwise the existing Rails proxy remained responsible for the request.

Current gate: natural matched traffic, not configuration

A read-only presence check of the deployed API2 task configuration at 2026-09-18T15:15:21Z (build cmu70; variable presence only, no values read) showed the storage resolver is configured in both regions: the US task has AWS_REGION and AWS_CONVERSATION_UPLOADS_BUCKET; the EU task has CONVERSATION_UPLOADS_USE_R2, CLOUDFLARE_CONVERSATION_UPLOADS_ENDPOINT, CLOUDFLARE_CONVERSATION_UPLOADS_BUCKET, CLOUDFLARE_S3_ACCESS_KEY_ID, and CLOUDFLARE_S3_SECRET_ACCESS_KEY. The earlier blanket statement that API2 tasks had no conversation-upload configuration or credentials is stale.

The conversation_upload_intake_native flag is still version 1, scoped to the single pilot organization (kuda-pilot), with last_called_at null (advisory only — confirm with a $feature_flag_called query filtered to $feature_flag_response = true, per the flag-evaluation evidence gate), and there are zero conversation_upload_native_intake events since cmu70 went live. In the post-deploy gateway slice 13:54:00Z..15:07:45Z, all 81 POST /conversations/upload requests were still forwarded to Rails (201). The gate for expanding the flag is therefore natural matched traffic from the pilot organization and a native-versus-Rails outcome comparison, not provisioning. Do not replay or synthesize uploads to force that evidence.

Why the pilot never produced native events (2026-09-21 → 2026-09-23)

nativeDecision() authenticates the bearer token before evaluating the flag, and until ENG-3375  API2 resolved API keys by key_digest only. Kuda’s key predates that column, so Rails authenticated it through its deterministic-encryption key equality fallback while API2 returned unauthenticated_fallback and proxied every upload to Rails without ever evaluating the flag. The $feature_flag_called history (three probe events at 2026-09-19T18:16Z) and last_called_at therefore say nothing about the canary.

ENG-3375 gives authenticateExternalApiContext the same exact-equality lookup Rails performs: digest first; on a miss, and only when API2 has ENCRYPTION_DETERMINISTIC_KEY and ENCRYPTION_KEY_DERIVATION_SALT, the deterministic ciphertext of the presented token is compared to the stored key (still active = true; no decryption, no plaintext at rest). A match backfills key_digest/key_prefix with COALESCE and logs event=external_api_key_auth outcome=ciphertext_fallback once per legacy key, after which the digest path serves it. Without the env the behaviour is unchanged, and Maintenance::BackfillApiKeyDigestsTask remains the bulk alternative. No flag, schema, provider or production-data change is required.

Fresh evidence (exact 24 h ending 2026-09-23T01:20:02Z). Gateway forwards Rails answered non-404: US POST /conversations/upload 201 ×1, EU GET /conversations/ ×23, EU GET /conversation_fields ×1 — all pre/overlapping the #12820 deployment at 00:56Z. First fully post-deploy slice 00:57Z–01:20Z: zero meaningful forwards, four US scanner 404s. The upload is the only remaining supported public residual; the exact 7 d window is reported separately in the retirement plan and still carries pre-cutover upload residue (≈700–900 201s/day before the canary can go live).

Canary gates after ENG-3375 deploys (natural traffic only, no replay).

1. Exactly one outcome=ciphertext_fallback for Kuda’s api_key_id, then none (key_digest populated; read-only SQL check in the runbook 1a).
2. First natural $feature_flag_called response=true for kuda-pilot after that timestamp.
3. conversation_upload_native_intake outcome=created ≈ Kuda’s daily volume; unauthenticated_fallback for Kuda → 0; no invalid/storage_failed/ insert_failed; handoff=accepted ≥ 99.9 %.
4. 1:1 upload/blob/attachment/delivery/conversation reconciliation for one 24 h window before any widening.

Rollback. Set conversation_upload_intake_native to 0 % (or disable); every API2 instance resumes forwarding within 20 s and Rails still accepts the same key (digest or ciphertext). The auth fallback itself is inert without the env and reverts by PR; a backfilled digest is exactly what the Rails maintenance task would have written, so no data rollback is needed.

The resolver requires these values per region (Rails uses R2 via CONVERSATION_UPLOADS_USE_R2 in development and EU, and AWS S3 through the task role in US); re-verify them before enabling the flag for any additional organization:

• R2: CONVERSATION_UPLOADS_USE_R2=true, CLOUDFLARE_CONVERSATION_UPLOADS_ENDPOINT, CLOUDFLARE_CONVERSATION_UPLOADS_BUCKET, CLOUDFLARE_S3_ACCESS_KEY_ID, and CLOUDFLARE_S3_SECRET_ACCESS_KEY.
• US S3: AWS_REGION and AWS_CONVERSATION_UPLOADS_BUCKET, with an IAM policy allowing the API2 task role to put, delete, and presign objects.

The conversation-upload service is deliberately separate from the existing cloudflare Active Storage service (CLOUDFLARE_R2_*). No storage credential should be exposed to a task until the regional bucket and IAM policy have been verified.

Rollout

1. Deploy API2 with the storage resolver disabled or unconfigured.
2. Verify native status reads, tenant isolation, and deprecated response headers in each region.
3. Provision and verify the regional bucket, credentials, and (in US) IAM policy.
4. Exercise presign and direct-create in a non-production organization while the flag remains off, then enable the flag for that organization.
5. Compare native and Rails status, duplicate, storage, and delivery-ledger outcomes before expanding the rollout.
6. Keep the Rails processor unchanged during the migration. API2 writes the same Rails-owned upload, Active Storage, and delivery-ledger tables and hands processing to the existing API2 queue.

The flag must remain off for every organization other than the pilot until the pilot produces natural native intake events whose status, duplicate, storage, and delivery-ledger outcomes match the Rails baseline.