US GoodJob backlog from relationship-detection fan-out

Incident date: August 4, 2026

Environment: US production

Status: Resolved; recovery backlog draining

Severity: High

Summary

US production’s shared GoodJob workers stopped draining a pre-existing backlog of approximately 189,000 runnable jobs after Kuda conversation-completion jobs exhausted the worker pool. We found 79 concurrent CompleteConversationSyncJob executions across 10 Rails worker processes. The jobs were not blocked on one long PostgreSQL statement. They were each performing unbounded relationship detection for delayed Kuda Intercom conversations created approximately 12 hours earlier.

The affected conversations shared a customer identity attached to 75,365 conversations. For one representative conversation, the existing time-overlap filters still produced 74,318 customer matches. Relationship detection loaded every matching ID and attempted to merge the conversations one at a time. Individual database statements remained below the 60-second statement timeout, but the full application-level loop could run for hours.

We paused CompleteConversationSyncJob, scaled the worker service down, and restarted its tasks to clear the affected executions. PR #9204 , deployed to production through PR #9205 , adds a hard bound before candidate materialization: each relationship source reads at most 51 IDs, skips the source if it exceeds the existing 50-item thread limit, and continues evaluating other bounded sources. The merge loop also stops when the surviving thread reaches 50 items.

After deployment, we resumed the job class. Completion jobs returned to approximately 1-second turnover with no renewed stalls. The total due queue fell by 583 jobs over 30 seconds, a net drain rate of approximately 1,166 jobs per minute. The incident path is resolved while the pre-existing queue backlog continues to drain.

Moving provider sync to API2 did not prevent this incident. The API2 provider-sync effects callback still schedules the Rails completion job, and relationship detection remains in that Rails path.

This incident follows the July 23 Kuda thread-merge stall. The 50-item guard added after that incident prevented oversized completed threads, but it did not cap the candidate discovery performed before a merge.

Customer impact

US background work stopped making useful progress while the shared worker slots were occupied.
Conversation completion, imports, evaluations, translations, and other jobs sharing the worker fleet were delayed.
Queue wait reached approximately 44,900 seconds for some work.
Delayed Kuda Intercom completion jobs triggered the pathological relationship searches.
A concurrent deployment migration repeatedly hit its lock timeout while the production database was under load.

We found no evidence of permanent customer-data loss. The primary impact was delayed asynchronous processing.

Detection

The incident was detected during a US deployment. The AddOrganizationCreatedAtIndexToEvaluationContests migration repeatedly hit its lock timeout while adding a concurrent index. Inspection of GoodJob then showed a large runnable backlog and no effective drain.

The jobs did not raise exceptions, so normal error reporting did not identify the failure. Better Stack also rejected OpenTelemetry spans because the trace source was paused; those exporter errors were unrelated to the worker exhaustion but reduced trace visibility during the response.

Telemetry

Worker and queue state

Runnable GoodJob backlog: approximately 189,000 jobs
Rails worker processes: 10
Configured shared slots: approximately 80
Running CompleteConversationSyncJob executions: 79 at the first snapshot
Queue: latency_30s
Organization: Kuda, organization ID 25
Organization data source: Intercom, ID 19
Oldest affected starts: approximately 1:36 PM Eastern Time
Observed queue wait: approximately 44,900 seconds

Healthy executions of the same job normally completed in approximately 82 ms during the preceding 15 minutes. The pathological executions therefore differed by orders of magnitude from the normal path.

Relationship fan-out

The affected conversations shared client ID 1287450.
That client was attached to 75,365 conversations.
Representative conversation 4055419 produced 74,318 customer matches after time-overlap filters.
The 79 affected conversations were created between 1:41 AM and 1:54 AM Eastern Time.
Only 19 of the 79 later had requester_id populated; the other 60 remained null.
Those 19 requester fields were updated between 1:47 PM and 1:57 PM, after the worker-exhaustion wave had begun.
A post-incident reconstruction found 5,668 same-requester candidates, but this does not prove the running jobs evaluated that source after the requester updates.
A second client attached to an affected conversation appeared on 10,307 conversations.
The representative conversation contained only 7-8 parts, ruling out transcript size as the main cause.

Timeline

All times are Eastern Time.

1:41-1:54 AM: The 79 affected Kuda conversations were created and their completion work entered the existing queue backlog.
12:08 PM: The Intercom requester change in PR #9168 merged into release-prod.
12:28-12:33 PM: The first US Rails and API2 worker deployments containing PR #9168 started tasks and reached steady state.
Approximately 1:36 PM: The delayed Kuda CompleteConversationSyncJob executions began occupying shared Rails worker slots.
1:47-1:57 PM: requester_id was populated on 19 of the 79 affected conversations; 60 remained null.
2:11 PM: The US deployment migration began adding index_evaluation_contests_on_organization_id_and_created_at concurrently and encountered repeated lock timeouts.
Approximately 2:12 PM: Investigation identified the approximately 189,000-job backlog and the cluster of long-running completion jobs.
During response: CompleteConversationSyncJob was paused. The Rails worker service was reduced to 2 tasks and restarted to release all live locks held by the affected executions.
2:51 PM: ENG-2240 was created for the candidate-fan-out fix.
2:54 PM: PR #9204 was opened with the bounded-query implementation and regression tests.
2:58 PM: PR #9204 merged into main.
3:01 PM: Production deployment PR #9205 merged into release-prod.
3:12 PM: US Rails worker task definition :2466, built from production commit 938681ed, was registered with the candidate-fan-out fix.
3:12-3:17 PM: The worker service was restarted. The old tasks stopped, stale completion-job locks began clearing, and replacement tasks started on task definition :2466.
3:21-3:22 PM: With completion jobs still paused, due jobs fell from 185,201 to 184,845 in 30 seconds, approximately 712 jobs per minute net.
Approximately 3:25 PM: CompleteConversationSyncJob was resumed.
3:25-3:26 PM: Due jobs fell from 183,216 to 182,633 in 30 seconds, approximately 1,166 jobs per minute net. The 11 active completion jobs were all approximately 1 second old.

Root cause

Relationship detection bounded the size of a completed conversation thread, but it did not bound the work required to discover candidates for that thread.

CompleteConversationSyncJob runs relationship detection after a provider conversation is synchronized. Detection queries 3 independent sources:

conversations with the same customer;
conversations with the same requester;
conversations with a matching phone attribute.

Before this incident, each query used pluck(:id) without a limit. The code combined every returned ID, loaded matching conversations in batches, and attempted a thread merge for each conversation. ConversationThread::MAX_THREAD_ITEMS = 50 prevented an oversized final merge, but only after candidate discovery and repeated application work had already occurred.

For Kuda client 1287450, one source returned more than 74,000 eligible IDs. The candidate query, batched loads, thread creation, locking, and individual merge attempts multiplied across many delayed conversations. The shared worker fleet admitted 79 copies of the same pathological workload, consuming almost every execution slot.

Intercom requester sync was not the origin of the backlog

PR #9168 began populating conversations.requester_id from the first Intercom contact. The release reached US Rails and API2 workers approximately 1 hour before the completion-job pileup, but it arrived long after the affected conversations and the broader queue backlog already existed.

The requester change is not required to explain the incident:

All 79 affected conversations shared the high-cardinality customer through conversation_customers.
The representative customer source alone produced 74,318 overlap-qualified candidates.
Only 19 affected conversations received a requester, and those updates happened after the pileup began.
The other 60 affected conversations had no requester and still participated in the pathological workload.

Requester matching may have added another high-cardinality source for jobs that evaluated it after the later updates, but the available timestamps do not prove that occurred in the already-running executions. We therefore treat requester sync as a possible amplifier, not the root cause or the start of the backlog.

Why the database timeout did not stop the jobs

PostgreSQL’s statement_timeout applies to each SQL statement, not to the total duration of a Rails job. Relationship detection issued many statements that individually completed in less than 60 seconds. The timeout therefore reset for each statement while the overall Ruby loop continued for hours.

This was not one query holding every worker. It was many similar jobs performing bounded individual queries inside an unbounded application-level workflow.

Why moving sync to API2 did not prevent the incident

The provider fetch and persistence path had moved to API2, but the API2 effects callback still enqueues CompleteConversationSyncJob in Rails. That job owns relationship detection and thread merging. Moving the provider request reduced GoodJob ingestion work, but it did not remove or isolate the unbounded post-sync stage.

Moving relationship detection to BullMQ could isolate queue capacity, but it would not make the processor safe by itself. The query and merge work must be bounded regardless of the queue backend.

Contributing factors

A high-cardinality customer identity linked tens of thousands of historical conversations.
Relationship queries materialized every candidate ID before the existing thread-size guard could apply.
Each Rails worker process consumed both latency_30s and wildcard queues, so the pathological class occupied shared capacity needed by unrelated work.
No maximum job runtime stopped a completion job that continued making individually valid database calls.
No alert identified a sudden cluster of long-running jobs by class and organization.
The paused OpenTelemetry source removed traces that could have shortened diagnosis.

What went well

GoodJob execution state showed that worker processes were alive rather than crashed.
Grouping running jobs by class, organization, and data source quickly isolated Kuda’s completion jobs.
A representative conversation reproduced the high-cardinality source and quantified the 74,318-candidate fan-out.
Existing structured boundaries made it possible to fix candidate discovery without changing normal matching semantics.
Focused regression coverage passed with 35 examples and no failures.

What did not go well

The existing 50-item thread limit created a data-size bound but not a work bound.
A valid high-cardinality customer identity could consume hours of worker time without raising an error.
Shared worker pools allowed one pathological job class to starve unrelated queues.
Statement timeouts were treated as a broader safety net than they provide.
Provider-sync migration and post-sync completion ownership were easy to conflate during diagnosis.

Resolution

PR #9204 , released to production by PR #9205 , implements the immediate safety bound:

Use ConversationThread::MAX_THREAD_ITEMS as the per-source candidate limit: 50.
Fetch at most 51 IDs from each source, using the 51st row as an overflow sentinel.
Skip an overflowing source entirely instead of arbitrarily linking its first 50 conversations.
Continue processing other relationship sources that remain within the limit.
Stop the merge loop as soon as the surviving thread reaches 50 items.
Refresh the surviving thread after each merge so the limit remains accurate when an older thread wins.
Log source overflow and thread-cap exits as structured events.

The sentinel approach avoids an unbounded COUNT(*) and lets PostgreSQL stop after proving that a source is unsafe. Skipping the whole source prioritizes avoiding false-positive mass linkage over choosing an arbitrary subset.

Production recovery

We used a controlled pause and restart so the unsafe jobs could not be reacquired before the fix was available:

Pause CompleteConversationSyncJob in US production.
Scale the Rails worker service from 10 tasks to 2, clearing 8 tasks immediately.
Stop and replace the remaining 2 tasks so no live process retained an affected execution.
Deploy task definition :2466 containing production commit 938681ed from PR #9205.
Scale the replacement worker service back up.
Resume CompleteConversationSyncJob after confirming the deployed artifact contained the 51-row bound.

The stale completion-job lock rows fell from 80 to 64 during task replacement, with zero remaining locks attached to a live GoodJob process. GoodJob continued cleaning up the stale rows while replacement workers processed other classes.

Corrective actions

Immediate

Interrupt the affected completion-job executions to restore worker capacity.
Add a per-source candidate limit and thread-cap exit in ENG-2240 .
Add regression coverage for source overflow and merge-loop termination.
Deploy PR #9204 to US production through PR #9205 .
Resume CompleteConversationSyncJob without new long-running executions.
Confirm the total runnable GoodJob backlog resumes declining after deployment.
Confirm the oldest runnable GoodJob timestamp advances as higher-priority incident work clears.
Verify that skipped_excessive_source_matches appears for Kuda in production telemetry.

Queue and runtime isolation

Add a hard runtime budget for CompleteConversationSyncJob or its relationship-detection stage.
Isolate post-sync completion from shared GoodJob workers under ENG-1949 .
Keep a hard regional concurrency limit if relationship detection moves to API2/BullMQ.
Make thread merges bounded, bulk, and idempotent under ENG-1946 .

Detection and observability

Alert on GoodJob executions older than 5 minutes, grouped by job class and organization, under ENG-1947 .
Alert when one job class consumes most shared worker slots.
Restore the paused Better Stack OpenTelemetry trace source or remove the exporter configuration.
Add metrics for relationship-source candidate overflow and thread-cap exits.

Production verification

Immediate post-deployment verification showed:

The US Rails worker service ran production task definition :2466 with the PR #9205 release commit.
After resuming completion jobs, total due work declined by 583 jobs in 30 seconds.
Net drain increased to approximately 1,166 jobs per minute.
11 concurrently locked CompleteConversationSyncJob executions were all approximately 1 second old.
Shared worker slots remained available to Jira, Intercom, imports, translations, uploads, and SLA work.

Continue monitoring the following conditions while the recovery backlog drains:

No CompleteConversationSyncJob runs longer than 5 minutes.
No relationship source materializes more than 51 candidate IDs.
Overflowing customer matches are skipped while bounded requester or phone matches still link correctly.
No conversation thread exceeds the existing 50-item limit through automatic matching.
Shared latency_30s and wildcard worker slots remain available to unrelated jobs.
Oldest runnable job age and total runnable backlog decline continuously.

Lessons

A data-size limit is not automatically a work limit. Safety checks must run before large result sets are materialized and before application loops begin.

Database statement timeouts bound individual statements only. Background jobs also need end-to-end runtime limits, bounded candidate discovery, and queue-level concurrency isolation.

Moving a provider sync to a new queue does not move all downstream work. Queue migrations must trace the complete callback chain and explicitly account for every stage that remains on the original worker fleet.