Native conversation transcription state

ENG-3235  moves the three Rails round-trips that the conversation-transcription call job still made — GET /internal/conversations/:id/transcription/context, POST …/transcription and POST …/transcription/failure — into API2. API2 already owned the provider call and the post-transcription continuation; this package makes it the writer of record for transcript rows, the audio-attempt counter, permanent failure state, credit usage and the callback effects Rails ran on those writes. The Rails endpoints stay deployed as the rollback path.

Baseline

Rolling 24 hours ending 2026-09-17T23:18Z: 688 successful Rails conversation transcription context/writeback calls. Seven days: 9,539. Both numbers count the internal endpoints above across both regions. Re-measure per region before enabling the flag anywhere; do not extrapolate from these values.

Ownership contract

• Owner is resolved once per call job by resolveConversationTranscriptionStateOwner(organizationSlug): CONVERSATION_TRANSCRIPTION_STATE_OWNER=api2|rails overrides everything; outside production API2 owns; in production the organization-scoped PostHog flag conversation_transcription_state_native decides and fails closed to Rails. The flag does not exist yet; this package ships default-off.
• Context (native-context.ts) reads the conversation, organization, ordered call details, transcript presence, recording attachment and key phrase sources without a transaction. The conversation must belong to the job’s organization slug; a mismatch is not_found with no writes.
• Recording URLs are presigned only after every read completes, only for blobs on the Rails production Active Storage service (cloudflare), through the same getStorageBlobSignedUrl helper the native blob routes already use, with the six-hour expiry Rails passes to the provider. Any other service_name throws instead of recording a failure.
• Key phrases mirror ConversationPartCallDetail::Transcription.key_phrases_for per call (organization greeting, per-call employee first name, agents, customers, Zendesk brand, company) and are capped at 100 entries. Expected languages follow multilingual_support_enabled and organization.languages.
• Persistence (native-writes.ts) runs one short transaction: lock the call detail FOR UPDATE scoped to organization and conversation, re-check transcript_available?, replace transcript rows using the Rails sequence, map customer→client/agent→agent/other→unknown, default confidence to 0.0, resolve duration as Rails’ update_duration_from_transcription!, recompute conversations.handling_time and transcription_confidence_score, insert the idempotent transcription credit_usages row, set transcribed_at/language, and apply the transcript-message callback effects (conversation_parts.body/ redacted_body with .presence semantics, first agent/customer response timestamps for type = 'call' parts, excluded employees skipped).
• Failure (native-writes.ts) increments transcription_attempt_count under the same lock; non-permanent errors return retry until three attempts, then set transcription_failed_at and a 500-character truncated transcription_error. Provider-confirmed permanent input failures record immediately. Sentry capture happens after commit and only for non-permanent terminal failures, as in Rails.
• Telemetry, call.transcribed PostHog capture (org:<slug> distinct id, organization group) and Sentry all run after commit. No provider, storage, Redis or HTTP work happens while a lock is held.
• Queue payloads, the stable runId, the completion fan-in and the Rails analysis/translation_fallback endpoints are unchanged, so any mix of API2 revisions and either owner is safe during rolling deploys.

Telemetry

emitConversationTranscriptionTelemetry logs one structured line per phase with owner (api2/rails), phase (context/persist/failure), outcome (loaded, not_found, transcribed, already_transcribed, retry, permanent), region, environment, organization slug, conversation and call-detail prefix IDs, and attempt counts. No text, URLs or names are logged.

Deploy, reconcile, expand

1. Deploy in both regions with the flag absent. Confirm the Rails path is still serving: the three internal endpoints keep tracking the baseline and the owner=rails telemetry appears.
2. Create the PostHog flag off. Enable it for one organization with recent call volume in one region.
3. For that cohort, compare per region and per outcome against the retained Rails path and the baseline:
   • owner=api2 transcribed + already_transcribed + retry + permanent + not_found counts versus the Rails endpoint volume the cohort used to produce;
   • transcript row counts per call, transcribed_at, language, duration_ms, handling_time, transcription_confidence_score, conversation_parts.body, first-response timestamps and credit_usages for API2-owned calls versus Rails-owned calls of similar shape;
   • transcription_attempt_count distributions and permanent-failure rate;
   • duplicate transcript rows or duplicate credit_usages (must be zero);
   • not_found and tenant-mismatch warnings (must be zero for real jobs);
   • Sentry events from the native path;
   • signed URL provider errors (403/404 on the recording URL) versus the Rails cohort, which proves signing/host parity.
4. Zero Rails calls with fewer business outcomes is a regression. Expand only when the distributions match for a complete 24-hour window, then repeat per region before organization-wide enablement.
5. Rails endpoint retirement requires a complete zero-call 24-hour window in both regions with clean reconciliation.

Rollback

Set CONVERSATION_TRANSCRIPTION_STATE_OWNER=rails (immediate, region-wide) or disable the PostHog flag (per organization, ≤20 s cache). In-flight jobs retry through Rails with the same payload; Rails re-checks transcript_available? under its own lock and the attempt counter is shared, so replay after rollback does not duplicate rows, credits or attempts. Do not edit transcript rows, attempt counts or queue payloads during rollback.

Owner: Platform. Flag conversation_transcription_state_native is proposed (not created) on 2026-09-17; review or remove by 2026-12-16.