Kuda evaluation delay and conversation-thread merge stall

Incident date: July 22-23, 2026

Environment: US production

Status: Resolved; permanent corrective actions pending

Severity: High

Summary

Kuda did not reach its QA evaluation target on July 22. During recovery, a CompleteConversationSyncJob for Kuda conversation 3218432 held a PostgreSQL transaction open for approximately 109 minutes while merging related conversation threads. The transaction accumulated enough subtransaction state to leave approximately 100 other database sessions waiting on SubtransSLRU or SubtransBuffer, which slowed the shared GoodJob worker fleet and delayed evaluation work.

We paused CompleteConversationSyncJob, terminated the single guarded PostgreSQL backend, force-discarded the affected GoodJob record, and resumed the job class. The database waiters fell to zero immediately. A manual QA backfill recreated the missed Kuda evaluation requests; after recovery, API2 completed 84 Kuda evaluations in 10 minutes with no new failures.

The recent removal of the 16-item conversation-thread cap is a strong contributing factor because it made automatic thread merges unbounded. The affected thread currently has 103 items. We did not retain enough lock history to prove whether the final stall was caused by a row-lock collision or by the large nested transaction itself.

Moving Kuda’s Intercom provider sync to the existing API2 BullMQ path is recommended, but it is not the complete fix. The API2 effects callback still enqueues the same Rails CompleteConversationSyncJob, so the thread-merge path must also be bounded and isolated.

Customer impact

Kuda missed its QA evaluation target for July 22.
Automatic evaluation work was delayed behind the US GoodJob backlog and database contention.
The shared US worker database accumulated approximately 100 sessions waiting on PostgreSQL subtransaction state.
Missed Kuda evaluations required a manual backfill.

No permanent evaluation data loss was identified. The manual backfill recreated the missed work, and API2 resumed completing Kuda evaluations without new failures.

Telemetry

The investigation used the US Rails GoodJob database, PostgreSQL activity, ECS service state, and the API2 BullMQ queue.

Stalled job

GoodJob ID: e00feff3-66c0-4bed-9eb2-10dfc79ed583
Job class: CompleteConversationSyncJob
Queue: latency_30s
Conversation: 3218432
Organization: Kuda pilot
Started: July 23 at 5:04 AM Eastern Time
PostgreSQL backend: 11977
Transaction age at termination: 6,545 seconds
Last observed statement:


UPDATE "conversations"
SET "updated_at" = $1,
    "conversation_thread_id" = $2
WHERE "conversations"."id" = $3

Database contention

Approximately 100 sessions waited on SubtransSLRU or SubtransBuffer.
The worker service was already at its configured maximum of 10 tasks.
The Rails backlog was approximately 28,000 jobs during recovery, dominated by redaction and Intercom fan-out.
After terminating the backend, both the guarded long-running sync backend count and Subtrans waiter count were zero.

Evaluation recovery

The API2 conversation-evaluation worker ran at its full concurrency of 20.
Kuda occupied all 20 active evaluation slots during recovery.
One recovery snapshot showed 84 Kuda completions in 10 minutes and zero Kuda failures.
The manual backfill, not termination of the sync job, recreated the missed evaluation requests.

Thread-merge amplification

Commit 9e0ea29527 removed MAX_THREAD_ITEMS = 16 on July 22.
The affected Kuda thread currently contains 103 items.
A read-only reconstruction found 2 currently related conversations: 1 customer match and 1 phone-attribute match.
Thread merging copies every item from the previous thread using find_or_create_by! inside a transaction.

Detection

The incident was detected through Kuda’s missed evaluation target rather than an automated error alert. The stalled job did not raise an exception, so it did not reach the final-retry Sentry capture path. PostgreSQL continued to execute or wait inside an open transaction, which also meant there was no application error for Sentry to report.

The investigation then identified:

A large GoodJob backlog and delayed evaluation launchers.
A single CompleteConversationSyncJob with a transaction older than 100 minutes.
The conversation-thread update issued by that backend.
Cluster-wide Subtrans waits that disappeared after the backend was terminated.

Timeline

All times are Eastern Time.

July 22, 3:44 AM: Commit 9e0ea29527, which removed the 16-item conversation-thread cap, was merged.
July 22: Kuda did not reach its QA evaluation target.
July 23, 5:04 AM: CompleteConversationSyncJob e00feff3-66c0-4bed-9eb2-10dfc79ed583 began processing Kuda conversation 3218432.
July 23, approximately 6:45 AM: Investigation connected the evaluation delay to the shared GoodJob backlog and PostgreSQL Subtrans contention.
July 23, 6:52 AM: CompleteConversationSyncJob was paused to prevent the target job from being reacquired.
July 23, 6:53 AM: PostgreSQL backend 11977 was terminated and the exact GoodJob record was force-discarded.
July 23, 6:54 AM: The job class was resumed. Verification showed zero matching long-running sync backends and zero Subtrans waiters.
During recovery: A manual Kuda backfill recreated missed evaluation requests.
After recovery: API2 completed 84 Kuda evaluations in a 10-minute window with no new failures.

Root cause

The incident had 3 layers: an unbounded thread merge stalled database work, shared GoodJob infrastructure amplified its effect, and the evaluation pipeline lacked automatic recovery for missed work.

Automatic relationship detection performed an unbounded thread merge

CompleteConversationSyncJob calls Conversation#complete_conversation_sync, which runs relationship detection before transcription, translation, evaluation preparation, and redaction scheduling.

Relationship detection finds conversations with matching customers, requesters, or phone attributes. For every match, it merges the related conversation’s thread into a base thread. The merge:

creates a thread item;
updates the related conversation’s conversation_thread_id;
copies every item from the previous thread with find_or_create_by!;
reparents the previous thread’s remaining conversations;
destroys the previous thread.

The July 22 change removed the check that stopped a merge when the combined thread would reach 16 items. The affected thread now has 103 items, so merge cost is no longer bounded. Repeated find_or_create_by! calls inside the merge transaction can also create nested savepoints and PostgreSQL subtransactions.

The last observed statement was a single-conversation thread reassignment. We did not capture the blocking process graph before termination, so we cannot distinguish conclusively between:

a row-lock collision during that reassignment; and
a large transaction that had already accumulated enough nested work to stall.

Both paths are unsafe without a transaction, lock, or job-runtime limit.

Shared GoodJob infrastructure amplified one pathological job

CompleteConversationSyncJob runs on latency_30s, alongside provider webhooks, imports, evaluation checks, translations, and redaction handoffs. GoodJob stores queue state and executes business writes in the same PostgreSQL database used by customer traffic.

The long-running transaction therefore affected more than one worker slot. PostgreSQL needed subtransaction visibility information from the old transaction, and approximately 100 unrelated sessions began waiting on Subtrans state. Adding Rails workers would have added more database sessions without removing the blocking transaction.

This is another instance of the queue-control-plane problem described in the GoodJob backlog and PostgreSQL saturation postmortem.

Missed evaluations required a manual repair

Terminating the sync job restored worker and database health, but it did not create the evaluations Kuda had missed. The manual backfill generated replacement evaluation requests and fed them through the Rails preparation pipeline into the API2 conversation-evaluation BullMQ queue.

The recovery therefore had 2 independent actions:

terminate the pathological sync transaction to restore throughput;
backfill Kuda’s missing evaluation requests.

What went well

PostgreSQL activity exposed the exact long-running backend and statement.
The target was guarded by job ID, class, query shape, application name, and transaction age before termination.
Pausing only CompleteConversationSyncJob prevented immediate reacquisition.
Terminating one backend removed all observed Subtrans waiters.
The exact GoodJob record was force-discarded, preventing a retry of the same pathological execution.
The class was resumed immediately after verification.
The manual backfill restored Kuda’s evaluation work.
API2 evaluation processing remained healthy, with no new Kuda failures during recovery.

What did not go well

The thread-item cap was removed without an alternative work or transaction bound.
One post-sync job could hold a database transaction for more than 100 minutes.
No alert covered long-running GoodJob executions or old PostgreSQL transactions by job class.
Sentry only observed exceptions, not jobs that remained stuck without failing.
Provider sync, post-sync relationship work, evaluation preparation, and other fan-out shared the same Rails worker fleet and database-backed queue.
Evaluation coverage required a manual backfill after the pipeline recovered.

Proposed fix

The fix should be split into independently revertable changes. Moving provider sync to API2 reduces GoodJob pressure, but the thread-merge safety change must ship first because the API2 callback still reaches the same Rails completion path.

PR 1 — Bound automatic conversation-thread merges

Tracked in ENG-1938 .

Restore a safety envelope specifically for automatically detected conversation-to-conversation merges:

Cap the number of items eligible for an automatic merge. ENG-1938 restores a limit of 50; the previous limit was 16.
Do not apply that cap to manually linked work items, Jira issues, or Slack messages. Those item types were part of the reason to allow larger threads, but they should not make automatic conversation merging unbounded.
Skip and log a structured reason when a merge exceeds the cap.
Add a short lock timeout and statement timeout around the merge transaction.
Add a maximum runtime for CompleteConversationSyncJob so a stuck execution fails and retries instead of holding a transaction indefinitely.

The post-deploy signal is that no automatic merge exceeds the configured size or remains active for more than the runtime budget.

PR 2 — Make thread merging proportional and idempotent

Tracked in ENG-1946 .

Replace per-item transactional copying with bounded bulk operations:

Lock both thread IDs in deterministic order.
Insert missing thread items in one INSERT ... ON CONFLICT DO NOTHING operation.
Reparent conversations in one bounded update.
Keep the transaction short and avoid find_or_create_by! savepoint creation per item.
Record item counts, transaction duration, lock-wait duration, and outcome.

If threads must grow beyond the automatic-merge cap, process the merge as a resumable plan with a durable cursor rather than one large transaction.

PR 3 — Move Kuda Intercom provider sync to API2

Tracked in ENG-1889 .

Enable api2_provider_conversation_sync for Kuda using the Nala EU rollout as the model.

The existing API2 path provides:

a dedicated BullMQ provider-conversation-sync queue;
Redis-backed queue coordination;
a global concurrency limit;
a provider request performed without holding a database connection;
deterministic job IDs and retry behavior;
an outbox and idempotent Rails effects receipt.

Roll out to Kuda as a canary, then expand to other Intercom tenants after duplicate-part and concurrent-writer protections are verified.

This PR reduces provider-sync pressure on GoodJob but does not remove CompleteConversationSyncJob: ProviderConversationSync::EffectsApplier still schedules it after applying API2 effects.

PR 4 — Isolate post-sync completion from shared Rails workers

Tracked in ENG-1949 .

Split complete_conversation_sync into independently budgeted stages:

spam detection;
relationship detection and thread merging;
transcription or translation scheduling;
SLA calculation;
redaction scheduling.

Move relationship detection and merging to a dedicated BullMQ queue, or to a dedicated Rails worker service as an interim step, with a hard regional concurrency budget. Do not let general worker autoscaling increase this workload’s database concurrency.

Keep the API2 outbox pattern so a committed provider sync cannot lose its post-sync work. Every stage must be idempotent and safe to replay.

PR 5 — Add detection and automatic recovery

Tracked in ENG-1947 and the broader backlog alert ENG-1937 .

Alert on GoodJob executions older than 5 minutes, grouped by job class.
Alert on PostgreSQL transactions older than 5 minutes from worker applications.
Alert on sustained SubtransSLRU and SubtransBuffer waiters.
Add BullMQ and GoodJob oldest-job-age dashboards by organization and class.
Add evaluation-target alerts before the reporting day closes.
Add a guarded evaluation backfill runbook and consider automatically creating replacement requests after a resolved pipeline outage.

What we will not do

Move only provider fetching to API2: This leaves the unsafe Rails completion and thread-merge path unchanged.
Scale Rails workers above 10: More workers cannot resolve a blocking transaction and can increase database contention.
Raise the thread cap globally: A larger finite cap postpones the same failure and still couples unrelated item types to automatic conversation merging.
Remove Nala work-item linking: Work-item links can remain unbounded if automatic conversation merges have their own safety budget.
Treat a larger database as the fix: More headroom reduces symptoms but does not bound transaction work or prevent a 100-minute job.

Verification

Thread-merge safety

No CompleteConversationSyncJob runs longer than 5 minutes.
No worker transaction remains open longer than 5 minutes.
Automatic merges above the configured conversation-item cap are skipped with a structured metric.
SubtransSLRU and SubtransBuffer waiters remain at zero during Intercom bursts.

API2 provider sync rollout

Kuda provider-sync queue age remains below 1 minute.
Provider-sync global concurrency stays at its configured regional limit.
No increase occurs in duplicate conversation parts, concurrent-writer errors, or effects-delivery retries.
Rails SyncConversationJob volume declines for Kuda without reducing imported-conversation throughput.

Evaluation coverage

Kuda reaches its daily evaluation target without a manual backfill.
Evaluation requests progress from pending to completed without unexplained gaps.
API2 conversation-evaluation failures remain at zero or within the normal baseline.
An incident replay confirms that terminating a stuck sync does not require manual database cleanup.

Follow-up status

Update this document with links to each corrective PR, the Kuda API2 feature-flag rollout time, and 24-hour production verification after the thread-merge guard ships.