Skip to Content
Internal docs are powered by Nextra Docs Theme.
Incidents2026GoodJob backlog and database saturation

GoodJob backlog and PostgreSQL saturation during worker autoscaling

Incident date: July 20-21, 2026

Environments: US production, with related GoodJob pressure in EU production

Status: Recovering; US backlog continues to drain under a temporary worker cap

Severity: High

Summary

US production experienced customer-facing timeouts followed by a long-lived GoodJob backlog after a burst of Intercom webhook work caused the shared Rails worker fleet to autoscale. Each added worker task also added database-heavy conversation-sync concurrency. PostgreSQL reached approximately 99% CPU, connection counts spiked, and many sync jobs contended on hot rows.

We reduced the worker fleet and disabled AutoscaleWorkersJob to contain the timeout incident. The pause did not expire and had no alert, so US remained at 2 workers for approximately 23 hours while queued work accumulated. When capacity returned, the backlog exposed additional database bottlenecks: per-event updates to organization_data_sources.last_synced_at, GoodJob candidate-selection queries, advisory-lock churn, and downstream fan-out into imports, completion jobs, translations, evaluations, and redaction handoffs.

The incident shows that GoodJob was a reasonable original choice for low-volume, transactionally enqueued Rails work, but it is no longer the right shared queue for Rulebase’s high-volume ingestion and fan-out paths. Queue coordination, application traffic, analytics, worker writes, cleanup, and autovacuum now compete for the same 4-vCPU PostgreSQL instances.

This incident is separate from the EU redaction discovery incident, although both incidents demonstrate the need for explicit database concurrency budgets.

Customer impact

  • US API requests, including /organization/statistics, returned 504 responses or exceeded PostgreSQL’s statement timeout.
  • Background work waited for as long as approximately 23 hours.
  • Conversation imports, completion work, translations, evaluations, and redaction scheduling were delayed.
  • Recovery traffic repeatedly pushed PostgreSQL to 97-99% CPU.
  • EU also accumulated a smaller GoodJob backlog and reached approximately 99% database CPU while draining it.

No evidence indicates permanent customer-data loss. Some background jobs remain delayed while the queues drain.

Detection

The incident was first detected through customer-visible slowness and a 504 from /organization/statistics. Sentry showed a broader burst of PostgreSQL statement timeouts across Rails endpoints and jobs.

AWS and PostgreSQL inspection then showed:

  • US PostgreSQL at approximately 99.4% CPU.
  • Average database load near 382 active sessions on a 4-vCPU instance.
  • Connections averaging approximately 589 and peaking at 1,156.
  • 113 sessions waiting in GoodJob polling.
  • 38 sessions contending while updating employees.conversation_agents_count.
  • Only 3 sessions running the statistics sentiment aggregate, confirming that the endpoint was a victim rather than the initiating load.

Timeline

All times are Eastern Time.

  • July 20, 12:53 PM: Customer-facing slowness and 504 responses were reported from /organization/statistics.
  • Approximately 1:00 PM: AWS and PostgreSQL inspection showed that the US database was saturated by worker-driven load. The shared Rails worker fleet had scaled from 12 tasks toward its maximum of 30.
  • Approximately 1:07 PM: The worker fleet was manually reduced to 2 tasks. The autoscaler immediately began restoring capacity because queue latency remained above its threshold.
  • Approximately 1:12 PM: AutoscaleWorkersJob was disabled as an emergency containment action.
  • Approximately 5:34 PM: US had 107,579 runnable GoodJobs and only 2 worker tasks. EU had 111,812 runnable jobs, largely at priorities excluded from autoscaling.
  • Overnight: The US autoscaler remained disabled without an expiry, heartbeat alert, or minimum-capacity fallback. The oldest queued work eventually reached approximately 23 hours.
  • July 21, approximately 8:35 AM: US had 178,946 ready jobs. The fleet began scaling back up.
  • During recovery: PostgreSQL returned to 98-99% CPU. A live snapshot showed 107 connections contending on organization_data_sources.last_synced_at and another 56 contending in GoodJob candidate selection.
  • After deploying PR #8442 : The last_synced_at row-lock convoy disappeared. Net drain improved from approximately 225 to 650 jobs per minute before downstream fan-out and renewed worker growth reduced the net rate.
  • Later July 21: Autoscaling returned the fleet to 24 workers. PostgreSQL again reached approximately 98% CPU, with 94 BufferMapping waiters and approximately 445 connections. Reducing the fleet to 10 removed the persistent BufferMapping waiters and reduced connections to approximately 300 while the primary Intercom webhook backlog continued to decline.

Root cause

The incident had 2 root-cause layers: an unsafe scaling model caused the initial database saturation, and an unguarded containment pause caused the long-lived backlog.

Shared autoscaling multiplied database-heavy concurrency

An Intercom webhook burst placed large numbers of ForwardIntercomWebhookJob records on the default queue. Normal volume was approximately 200-270 jobs per minute, then increased to 824, 1,167, 893, and 970 jobs across 4 consecutive minutes.

AutoscaleWorkersJob watches queue latency for default and latency_30s. It responded correctly to the latency signal by adding 2 worker tasks every 2 minutes. However, each worker task consumes all configured GoodJob pools:

sync_conversation: 4 threads
latency_30s: 4 threads
other queues: 4 threads

Scaling toward 30 tasks therefore allowed up to 120 concurrent sync_conversation jobs even though conversation-sync latency was not the scaling signal. The additional workers increased PostgreSQL polling, connections, row updates, and lock contention faster than they increased useful throughput.

Emergency containment persisted without safeguards

Reducing worker capacity and disabling autoscaling was the correct immediate response to customer-facing timeouts. The containment became a separate failure because it had:

  • no automatic expiry;
  • no alert that the autoscaler heartbeat was missing;
  • no queue-age or backlog-growth alert;
  • no minimum-capacity fallback;
  • no recovery checklist requiring the pause to be revisited.

The worker fleet therefore remained at 2 tasks for approximately 23 hours while new work continued arriving.

Recovery amplifiers

Several conditions made recovery slower after worker capacity returned:

  • Hot parent-row updates: Every provider webhook event updated organization_data_sources.last_synced_at, causing many transactions to serialize on a small number of rows. PR #8442 now coalesces those updates to at most once per minute while preserving every connection log.
  • GoodJob acquisition pressure: Every worker thread selects, locks, and updates jobs through PostgreSQL. Large backlogs caused many concurrent candidate-selection and advisory-lock queries.
  • Downstream fan-out: Processing one webhook can create imports, completion jobs, translations, evaluations, redaction handoffs, and other follow-up work. Gross completions were much higher than the visible net decline in ready jobs.
  • Shared database: Interactive API queries, analytics, queue coordination, job writes, cleanup, and autovacuum all used the same PostgreSQL instance.
  • Priority mismatch: Important large backlogs at priorities 25 and 50 did not contribute to the autoscaler’s priority-10 latency signal.
  • Preserved queue history: High GoodJob record churn increased table and index maintenance and required sustained autovacuum work.

Architectural assessment of GoodJob

GoodJob was not inherently a bad decision. It provided a low-operations, transactionally consistent job system when Rulebase had lower volume, fewer workers, shorter Rails-local jobs, and more PostgreSQL headroom.

The current use is a bad architectural fit for high-volume ingestion and fan-out. GoodJob places the queue control plane on the same database as the application data plane:

GoodJob
  queue polling + locking + retries + scheduling + history -> PostgreSQL
  business reads and writes                              -> PostgreSQL

BullMQ
  queue polling + locking + retries + scheduling         -> Redis
  business reads and writes                               -> PostgreSQL

This difference explains why BullMQ has not shown the same queue-acquisition failure mode at comparable job counts. Redis absorbs rapid queue-state mutation without competing with customer queries or PostgreSQL autovacuum.

BullMQ does not protect PostgreSQL from an inefficient processor. The separate EU redaction incident ran through BullMQ and still saturated PostgreSQL with an unbounded discovery query. A queue migration must therefore include hard global concurrency limits, efficient bounded queries, and database-aware backpressure.

What went well

  • AWS Performance Insights and PostgreSQL activity identified the active wait classes and statements.
  • Manually reducing workers protected customer-facing traffic during the first incident.
  • Removing the unused employee counter-cache writes eliminated the initial hot-row update path.
  • Coalescing last_synced_at updates removed the main recovery lock convoy.
  • Reducing the recovery fleet from 24 to 10 immediately reduced database connections and persistent BufferMapping contention.
  • The primary Intercom webhook backlog continued to decline even when total queue depth was obscured by downstream fan-out.

What did not go well

  • Autoscaling treated every Rails worker task as interchangeable even though each task multiplied database-heavy sync concurrency.
  • The autoscaler used queue latency without considering database CPU, connections, active sessions, or lock waits.
  • The manual pause had no expiry, alert, or minimum-capacity fallback.
  • We lacked queue-age and per-job-class growth alerts.
  • GoodJob queue history and polling competed directly with customer-facing PostgreSQL work.
  • Total queue depth obscured progress because upstream processing generated large downstream queues.
  • Recovery initially focused on adding workers rather than maximizing useful throughput per database connection.

Resolution and recovery

The current recovery controls are:

  • US Rails workers are capped at 10 while the backlog drains.
  • The unused employees.conversation_agents_count maintenance path has been removed.
  • Webhook event logging updates organization_data_sources.last_synced_at at most once per minute per data source.
  • Queue depth, dominant job classes, database load, connections, and wait events are being checked with read-only production queries.

The incident remains in recovery until the oldest incident backlog has cleared and both regions sustain normal database and request latency for at least 30 minutes.

Corrective actions

Highest priority

  • Split sync_conversation into a dedicated worker service with a hard regional concurrency budget that shared worker autoscaling cannot change.
  • Stop general workers from consuming sync_conversation.
  • Add database-aware scaling guards. Do not increase worker concurrency when PostgreSQL CPU, connections, active sessions, or lock waits exceed defined thresholds.
  • Add an expiring autoscaler pause with a visible owner, reason, expiry time, and automatic alert before expiry.
  • Add a minimum-capacity fallback when the autoscaler heartbeat is missing.

Queue architecture

  • Move high-volume webhooks, provider sync, imports, translations, evaluations, redaction, and similar fan-out pipelines to dedicated BullMQ queues.
  • Keep GoodJob for low-volume transactional Rails work, cron, and lightweight relay or outbox delivery.
  • Use deterministic BullMQ job IDs and idempotent processors.
  • Use a Rails outbox or retryable relay when a database commit and BullMQ enqueue must not diverge.
  • Configure global concurrency per database-heavy BullMQ queue rather than relying only on per-process concurrency.
  • Reduce GoodJob record retention and verify cleanup and autovacuum behavior under production-shaped churn.

Operational safeguards

  • Alert on oldest runnable job age, total ready jobs, and growth by job class in each region.
  • Alert when AutoscaleWorkersJob is disabled or has not completed successfully within its expected interval.
  • Add dashboards for useful throughput: arrivals, gross completions, net ready-job change, and downstream fan-out.
  • Define a recovery runbook that specifies worker caps, database thresholds, nonessential queues to pause, and safe verification queries.
  • Separate maintenance jobs such as QA eligibility sweeps, counter reconciliation, risk calculations, and backfills into a concurrency-1 service.

Lessons

Queue latency is not a sufficient autoscaling signal for database-heavy workers. The system must scale each workload independently and enforce a database concurrency budget across all worker processes.

GoodJob remains useful where transactional enqueueing and Rails locality matter. It should not remain the shared ingestion backbone now that queue coordination itself materially competes with customer traffic. BullMQ removes that queue-control-plane pressure from PostgreSQL, but every processor still needs bounded database work.

Emergency controls are part of the production system. A pause without an expiry, alert, owner, and recovery path can turn a successful mitigation into the next incident.

Last updated on
Qonto stale QA score aggregates postmortemEU PostgreSQL saturation