GoodJob traffic baseline and retirement priorities

GoodJob traffic is the most direct runtime measure of how much asynchronous application work still passes through Rails. It is not, by itself, proof that Rails still owns the business logic: several retained jobs are delivery, debounce, fallback, or compatibility bridges into API2. Use this page to separate those bridges from Rails-owned work and to order their removal.

September 16 embedded-execution decision

The retained production window contained no unfinished jobs and was dominated by low-volume Rails residue: Active Storage analysis, CloudWatch metrics, and a small number of QA backfill jobs. That workload no longer justifies dedicated Flightcontrol worker services.

Rails web processes now execute the durable GoodJob queue in-process with one GoodJob thread per Falcon child. Falcon preloads Rails before forking, so the deployment keeps GoodJob in external mode during preload and starts embedded execution only from each child’s post-fork lifecycle hook. Postgres remains the source of durability, retries, cron state, and dashboard history. Each child also drains GoodJob for up to 20 seconds during graceful shutdown; Falcon’s child interrupt window is 25 seconds so rolling deploys allow that drain to complete before escalating termination.

The deployment definition no longer declares the development, US production, or EU production Rails worker services. The worker autoscaler and its internal admin controls were retired with that topology. Deleting the existing live Flightcontrol services is a separate, explicitly approved operational step after the web deployment is healthy and embedded job execution is verified.

Embedded execution evidence (September 17 logging repair)

After the embedded cutover both regions ran healthy Falcon GoodJob processes and the minute metrics cron kept executing, yet the Rails web CloudWatch groups contained zero ActiveJob start records. Semantic Logger writes through one appender thread, and that thread does not survive Falcon’s fork: the ActiveJob subscribers were still attached in every child, but their records were queued in memory and never flushed to stdout. FalconGoodJobLifecycle.start now reopens Semantic Logger in the child before starting GoodJob, and the perform_start.active_job record is emitted by ActiveJobLogSubscriber with job_class, active_job_id (identical to job_id) and executions.

• Authoritative evidence is the structured JSON in the CloudWatch log groups of the current Rails web ECS services: cluster fc-rulebase-api-prod-xv109mm (US, us-east-1) and fc-rulebase-api-prod-eu-u56f166n (EU, eu-central-1), the same targets as bin/aws/ecs-shell.sh api-prod-us|api-prod-eu. Resolve the exact group name from the running task definition’s awslogs-group option rather than from memory.
• Cross-cutover boundary. Records before the embedded deploy live only in the retired Rails worker log groups, which have since been deleted. Any window that spans the cutover therefore cannot be reconstructed from CloudWatch; start certification windows at the first web-group start record after this repair deploys.
• Not evidence. The good_jobs table and dashboard keep finished rows for 12 hours and are not a substitute for the log stream. Process heartbeats and the PublishCloudWatchMetricsJob metric series prove liveness, not per-job execution.
• Count one perform_start.active_job record per execution attempt; count distinct active_job_id values for unique jobs. Retries reuse the same active_job_id with executions incremented, so the two series diverge exactly by the retry volume.

fields @timestamp, payload.job_class, payload.active_job_id, payload.executions
| filter payload.event_name = "perform_start.active_job"
| stats count(*) as attempts, count_distinct(payload.active_job_id) as unique_jobs by payload.job_class

September 11 QA evaluation backfill snapshot

Complete US worker logs after the earlier ownership release recorded normal-path starts from 2026-09-11T20:00:03Z through 20:07:19.776Z for the five QAEvaluationBackfill classes: 11 attempts / 11 unique jobs from two automatic runs (run 171, organization 19, which created request 5899294 and replacement run-agent 1512; run 172, organization 25). EU recorded none. These are not maintenance rows: they reset the zero-GoodJob clock and block the milestone.

After deploy and per-organization gate enablement, expect zero new rows for the five classes for gated organizations; the zero-GoodJob query and outcome queries are in the QA evaluation backfill runbook.

September 10 live residue census

The exact two-hour census ending around 17:15 UTC contained 382 US and 578 EU executions including Rails maintenance. Excluding maintenance, the largest US families were SummarizeConversationJob (46 attempts / 45 unique), ProcessIntercomWebhookEventJob (29 / 29), the transcription cluster (8), and the two expiry sweeps (2 each). EU had 44 summary jobs, 5 ForwardRulebaseWebhookJob rows, and 2 executions of each expiry sweep.

The matching rolling 24-hour census contained 3,810 US and 3,475 EU executions. Most were operational residue: CloudWatch metrics, Active Storage analysis, autoscaling and application maintenance. The normal-path families relevant to this package were summaries (48 US attempts / 47 unique; 56 EU), expiry sweeps (17 of each class in each region), report collection (70 US / 34 EU), and relationship creation (33 combined rows per region). Intercom’s final US GoodJob row was created at 17:08 UTC while the regional deploy rolled; all API2 and Rails services were steady on the new release around 17:10 UTC.

ENG-3000 removes summaries and expiry sweeps from the normal GoodJob path. Conversation summary producers now hand off directly to API2 and retain SummarizeConversationJob only for a proven delivery failure or already queued work. API2 registers one hourly :05 BullMQ maintenance job that invokes the same Rails model-bound expiry implementations synchronously; both GoodJob crons are disabled by default. Production credit still requires a post-deploy regional window showing the BullMQ sweep succeeding and no new normal-path rows for the three retained job classes.

Measurement rules

• The production GoodJob database and dashboard retain completed jobs for only 12 hours. A dashboard count labelled “today” is therefore a retained rolling window, not a complete calendar day and not a valid historical baseline.
• Historical totals below come from structured Rails worker logs in the US and EU CloudWatch log groups. They count Performed records in UTC calendar-day windows. Those worker groups were deleted with the worker services; current evidence comes only from the Rails web log groups described above.
• An execution is one performed attempt. Retries count more than once. A unique job is one distinct ActiveJob job_id, calculated with a two-stage exact aggregation for the selected comparison days.
• Do not compare a retained GoodJob count directly with a full-day CloudWatch count. Compare like-for-like windows, or use the full UTC-day series below.
• ActiveStorage::AnalyzeJob, PublishCloudWatchMetricsJob, and heartbeat jobs are Rails operational residue. They do not require a dedicated worker service and do not determine which application workload should move to API2 next.

AWS production access was restored and used for read-only regional database and worker-log queries. The historical series and September 8 retained-traffic snapshot were collected on September 8, 2026 at approximately 22:55 UTC. The September 9 flag baselines were collected on September 9, and the live residue censuses above were collected on September 10 at approximately 17:15 UTC.

Historical reduction

The comparable full-day execution series shows a material reduction in Rails worker traffic.

Average daily executions fell 87.3% between the two seven-day windows; the median fell 91.1%. The median is the better description of the normal September level because a Zendesk import retry storm inflated EU executions on September 7.

An exact unique-job comparison confirms that the reduction is not only fewer retries:

Unique jobs fell 92.8% from August 4 to September 7. Executions fell 79.2%, with the smaller execution reduction explained by the September 7 retry amplification.

The August 4 baseline included 389,610 SLA-metric executions, 242,520 Intercom webhook/import executions, 128,901 translation executions, 123,290 conversation completion executions, and 105,219 general conversation-sync executions. Those families have disappeared or contracted sharply. For example, CompleteConversationSyncJob fell to 3,182 combined executions on September 7, a 97.4% reduction. This is production evidence that the migration has already removed most normal Rails queue volume.

The September 7 EU outlier included 236,299 ImportZendeskConversationJob executions. Treat that as retry amplification until its underlying cause and unique-job count are reconciled, not as 236,299 new conversations.

September 8 retained traffic snapshot

The user-visible dashboard totals of roughly 34,000 EU and 16,000 US completed jobs are expected to be non-zero because several high-volume ingress and completion bridges still enqueue GoodJob. The class-level snapshot is more useful than the aggregate:

The hourly dispute scheduler also produced 24 retained Disputes::CheckDisputeAcknowledgementsJob fan-out executions, 12 per region. Low-volume maintenance, reporting, heartbeat, and fallback classes remain in the complete inventory even when they do not affect the immediate ordering.

September 9 Knowledge Library sync snapshot

A rolling 12-hour EU+US window ending September 9 at ~10:13 UTC showed the Knowledge Library pipeline as the largest remaining retry amplifier outside Zendesk ingress:

The 16.6 executions-per-unique-job ratio is the storm signature this baseline warns about: GoodJob’s throttle treats an over-limit attempt as a failed execution, so 158 documents produced 2,600+ rows. After deploy, expect zero new normal-path rows for all six classes in both regions; use the queries in the Knowledge Library sync runbook.

September 9 XCALLY completion snapshot

A rolling 12-hour window on September 9 showed CompleteConversationSyncJob still executing on the normal path for XCALLY uploads: roughly 765 executions in the US from organization 25 / data source xcally, and zero in the EU. Every fresh XCALLY completion callback belonged to a GoodJob::Batch whose sole child was ActiveStorage::AnalyzeJob. The direct producer was ConversationUpload::Processors::Xcally#process, where attach_recording ran inside GoodJob::Batch.enqueue(on_success: CompleteConversationSyncJob). Two Jira callbacks existed before the latest Rails deploy, with none afterward, so Jira is not part of this package.

The batch’s wait on ActiveStorage::AnalyzeJob was implicit capture, not a required ordering: the batch was introduced to move the recording attach outside the database transaction, and nothing in complete_conversation_sync reads analyzed blob metadata — transcription discovery only needs the committed attachment. ActiveStorage analysis stays in Rails; it is only decoupled from completion.

ENG-2948  hands XCALLY completion to the API2 provider-conversation-sync-completion BullMQ queue through POST /internal/provider_conversation_sync_completions/enqueue behind the PostHog flag api2_conversation_upload_sync_completion. Dispatch is claimed durably under outbound_delivery_claims (conversation-upload/sync-completion/<id>) so the API2 hand-off and the GoodJob fallback share one single-execution identity: a proven API2 rejection takes the fallback, an ambiguous delivery leaves the claim unresolved for manual reconciliation, and a retried upload never dispatches twice. The GoodJob batch on_success additionally remains the flag-off rollback path; there is no normal-path CompleteConversationSyncJob enqueue from this producer once the flag is on.

September 9 Rulebase webhook forwarding snapshot

A rolling-hour window on 2026-09-09 showed ForwardRulebaseWebhookJob running 3 times in the US and 0 times in the EU, all successful — a low-volume pure GoodJob bridge whose only live producer is the conversation_assigned event from ConversationAssignment. ENG-2958 removes it from the normal path: after the outermost Rails transaction commits, Api2RulebaseWebhookEventEnqueuer POSTs {eventId, event} to API2’s POST /internal/rulebase_webhook_events/enqueue, which durably enqueues a rulebase-webhook-events BullMQ job (jobId = rulebase_webhook_<event_id>, deduplicating ambiguous retries) before answering 202. The PostHog flag api2_rulebase_webhook_events (safe state off) restores the GoodJob producer; a proven pre-enqueue rejection falls back to ForwardRulebaseWebhookJob, and an ambiguous delivery retries through EnqueueApi2RulebaseWebhookEventJob with the same event_id. After deploy, expect zero new normal-path ForwardRulebaseWebhookJob rows in both regions; the verification window is 24 hours in the US and EU per the rollout gate.

September 9 report row collection snapshot

The rolling hour 2026-09-09T19:56:47Z–20:56:50Z (Active Storage and CloudWatch excluded) showed QAEvaluationReport::CollectRowsJob executed 70 times in the US and 34 in the EU — 104 total, all completed with zero errors. This is the scheduled 20:00 UTC burst for the daily agent evaluation report, not steady traffic; the class exists only because synchronous Rails collection can exceed CloudFront’s ~60s origin timeout, while Rails permits a 5-minute statement timeout.

The collector is now ported into API2 inside the existing daily-agent-evaluation-report BullMQ org job, selected per organization by the PostHog flag api2_daily_agent_evaluation_report_collect (safe state off). The burst is bounded by the report worker’s concurrency: 3 and the same 5-minute statement_timeout inside the collection transaction; the Rails /collect endpoint and job class remain the flag-off rollback until bake. The zero-new-rows assertion for this class is part of the rollout gate in rollouts.

September 9 SyncConversationJob residue

The reported rolling-hour baseline on September 9, 2026 was four successful top-level SyncConversationJob executions in US and zero in EU. This is a baseline observation, not a serialized GoodJob classification: AWS production access was unavailable during the audit, so the four conversation/source argument pairs could not be inspected and should not be attributed wholesale to the known legacy Zendesk source.

The durable producer guard now sends only source-matched, configured supported providers to API2. A source with a missing Zendesk/Freshdesk subdomain or a conversation/source mismatch uses the retained Rails fallback directly, avoiding bounded API2 retry amplification. API2 enqueue/delivery errors still fall back once, and unsupported providers remain on Rails. After deployment, repeat the regional query below for a clean US/EU verification window before removing the retained fallback.

SELECT
  serialized_params->>'job_class' AS job_class,
  count(*) AS executions,
  count(DISTINCT active_job_id) AS unique_jobs,
  count(*) FILTER (WHERE performed_at IS NOT NULL) AS performed,
  count(*) FILTER (WHERE finished_at IS NOT NULL) AS finished
FROM good_jobs
WHERE serialized_params->>'job_class' = 'SyncConversationJob'
  AND created_at >= now() - interval '1 hour'
GROUP BY job_class;

For the known configuration gap, use this read-only tenant-scoped candidate query before any repair. Replace :organization_id only after the owning tenant and source have been confirmed from the serialized GoodJob arguments:

SELECT
  ods.organization_id,
  ods.id AS organization_data_source_id,
  ds.key AS provider,
  ods.user_data->>'subdomain' AS subdomain,
  count(c.id) AS conversation_count
FROM organization_data_sources AS ods
JOIN data_sources AS ds ON ds.id = ods.data_source_id
LEFT JOIN conversations AS c
  ON c.organization_data_source_id = ods.id
WHERE ods.organization_id = :organization_id
  AND ods.discarded_at IS NULL
  AND ds.key IN ('zendesk', 'freshdesk')
  AND NULLIF(trim(ods.user_data->>'subdomain'), '') IS NULL
GROUP BY ods.organization_id, ods.id, ds.key, ods.user_data->>'subdomain';

The guarded repair is a runbook only; this change did not mutate production:

BEGIN;
SELECT id, organization_id, data_source_id, user_data
FROM organization_data_sources
WHERE organization_id = :organization_id
  AND id = :organization_data_source_id
  AND discarded_at IS NULL
FOR UPDATE;
-- Confirm exactly one row, the expected provider, and the approved subdomain.
-- UPDATE organization_data_sources
-- SET user_data = jsonb_set(user_data, '{subdomain}', to_jsonb(:subdomain), true),
--     updated_at = now()
-- WHERE organization_id = :organization_id
--   AND id = :organization_data_source_id
--   AND discarded_at IS NULL
--   AND NULLIF(trim(user_data->>'subdomain'), '') IS NULL;
ROLLBACK;

Why Zendesk and Jira still use GoodJob

Zendesk

Rails still receives the provider webhook. Coalescible events are recorded in zendesk_webhook_ticket_syncs, then a 30-second GoodJob debounce runs ProcessZendeskWebhookTicketSyncJob. The normal job path already forwards to API2, and all sampled September 8 ticket-sync jobs used that default API2 handoff rather than the Rails business fallback.

GoodJob nevertheless remains on the normal path because it provides the first debounce/delivery step, and API2’s follow-up path can call Rails to schedule another delayed GoodJob when events arrive during processing. That makes the current path a loop:

Zendesk → Rails ingress → GoodJob debounce → API2 BullMQ → Rails follow-up → GoodJob

The temporary cutover should instead be:

Zendesk → existing Rails URL → synchronous authenticated API2 accept → BullMQ

Rails should forward the raw body and required routing context without applying business effects, and acknowledge Zendesk only after API2 durably accepts the event. API2 must own signature-compatible validation, connection lookup, standard and reduced event semantics, coalescing, delay, retries, ticket import, provider-sync enqueue, and events that arrive during processing. After a clean regional bake, change the provider destination to API2 directly and remove the Rails relay.

Jira

The current Rails Jira route validates the provider JWT, resolves cloudId connections, and enqueues ProcessJiraWebhookEventJob. Rails then owns issue import/upsert plus comment create, update, and delete semantics. API2’s existing /webhooks/jira/events route is for Jira Assistant and is not a compatible destination for this ingestion contract.

Use the same temporary relay shape as Zendesk: keep the configured provider URL on Rails, synchronously forward the raw event and routing context to a new authenticated API2 ingestion boundary, and acknowledge only after durable BullMQ acceptance. Move all Jira business behavior behind that new boundary. Once observed, point Jira directly at the API2 provider route and remove the relay. Do not reuse or change the Assistant route for this package.

Ordered retirement work

1. Zendesk and Jira ingress: remove about 36,700 retained normal-path jobs and ensure API2 owns the business behavior before changing provider URLs.
2. Dispute acknowledgement: replace the hourly GoodJob fan-out with a bounded API2 scheduler and BullMQ worker. The scheduler should select only unacknowledged disputes whose conversation parts or relationships changed since last_acknowledgement_checked_at, while preserving the current prompt, model settings, selected-part timestamp, tenant isolation, and retry safety.
3. Conversation completion residue: retire remaining CompleteConversationSyncJob producers and fallbacks after the webhook work lands. Most of this family has already moved; the last live producer (the XCALLY upload batch) is retired by ENG-2948 behind api2_conversation_upload_sync_completion, pending flag rollout and the zero-new-rows bake.
4. CX risk: move the remaining CalculateCxRiskForConversationJob entry point and its review/evaluation state effects. This is a wider state-machine package than its raw volume suggests and should follow stable completion ownership.

Conversation completion and CX-risk ownership

The normal path is API2-owned after this migration:

• conversation-cx-risk-start accepts tenant-scoped initiation requests, applies eligibility and deduplication, refreshes the transcript, creates the snapshot, and fans out evaluation checks.
• evaluation-check-run owns check retries and terminal failure transitions.
• conversation-cx-risk-completion owns snapshot fan-in and idempotent completion.
• Provider sync completion and transcription fan-in remain on their existing deterministic BullMQ queues.

The Rails jobs CompleteConversationSyncJob and CalculateCxRiskForConversationJob are rollback-only compatibility paths for ambiguous API2 delivery or rolling deploys. They must not be normal producers. The rollout validation is a zero-increase counter for new normal-path enqueues of either class, checked per region and compared with the API2 queue depth and failure metrics. ownership. 5. Rails operations: replace or delete Active Storage analysis, Rails-only CloudWatch publication, autoscaling, heartbeats, and other worker lifecycle jobs as part of infrastructure retirement, not application-priority ranking.

Two implementation handoffs were active when this snapshot was written:

• Zendesk and Jira autoforwarding and API2 business ownership 
• Dispute acknowledgement migration 

These links record work in progress, not merged or production-complete credit. Verify their PRs, deployment, and observation gates before changing this page’s ownership state.

Runtime completion gate

The September 8 retained-contract milestone is an implementation-readiness gate, not the GoodJob-zero gate. Runtime queue retirement is complete only when:

1. No migrated class receives new normal-path GoodJob enqueues in either region for a comparable 24-hour window.
2. BullMQ accepts and drains the corresponding real provider and scheduled work without duplicate effects, unbounded retries, or retained failure.
3. Delivery-fallback and API2-to-Rails callback counts remain zero or match an explicitly approved rollback-only contract.
4. Provider URLs have moved to API2 where required and the temporary Rails relays have completed their bake windows.
5. Rails-operational jobs have replacements or are no longer required, GoodJob is empty and disabled, and Rails workers receive no traffic.

The next program milestones remain regional cutover by September 11, at least 72 hours of clean final operation during September 12–14, and approved Rails and GoodJob retirement on September 15.