2026-05-14 — The `ticket_updated` trigger was silently broken since launch

Date: 2026-05-14 Author: Chidi (with Cursor) Scope: Postmortem of the silent-failure bug in WorkflowTriggerEvent#enqueue_api2_forward_if_needed that prevented every ticket_updated event from reaching api2 since the feature shipped.

Headline

Rho operators reported that no ticket_updated-triggered agent runs were happening on their org, despite the plan describing the path as “end-to-end plumbed.” A prod-console debug session traced the issue to a one-character method-name bug masked by a rescue StandardError that only logged to Rails.logger.error. Zero ForwardRulebaseWebhookJob rows had ever been enqueued for any org in production. Fixed in PR #6540.

The bug existed for the entire lifetime of the feature. It was invisible because:

The rescue swallowed the exception.
The only signal was a Rails.logger.error line not routed to Sentry.
The api2 side responds 202 { accepted: true, matchedAgents: 0 } to every webhook (because none arrived), which looks indistinguishable from “the webhook arrived but no agents are subscribed” — and the only agents subscribed in Rho’s org were paused [WIP] clones, so even a healthy path would have produced the same operator-visible behavior.

What broke and why

The bug

rulebase-api/app/models/workflow_trigger_event.rb, in enqueue_api2_forward_if_needed:


ForwardRulebaseWebhookJob.perform_later(
  event_type: type.to_s,
  organization_id: organization.prefix_id,   # ← raises NoMethodError
  conversation: conversation_payload_for_api2
)
rescue StandardError => e
  Rails.logger.error('Failed to enqueue API2 webhook forward', { event_type: type, error: e.message })

Organization is friendly_id-slugged (via friendly_id :slug_candidates, use: :slugged), not has_prefix_id-prefixed. organization.prefix_id raises NoMethodError: undefined method 'prefix_id' for an instance of Organization. The rescue StandardError caught it and only logged.

api2 stores the Rails-side org identifier in organizations.rulebase_id and looks it up at routes/webhooks.ts and lib/upsert-session.ts — both of which use the Rails slug. Every other Rails-side caller that talks to api2 (RunConversationQAEvaluationJob, QAEvaluationRequest.post_eligibility_check_to_api2) already uses organization.slug. The webhook-forward call was the outlier.

The fix

PR #6540 in two parts:

organization.prefix_id → organization.slug so the call succeeds.
The rescue now also calls Sentry.capture_exception(e) so a future regression here surfaces in error tracking, not just buried Rails.logger.error output.

Plus a regression spec at spec/models/workflow_trigger_event_spec.rb covering both ticket_updated (asserts organization_id: organization.slug) and sales_call_updated (asserts the forward is skipped, since api2 only consumes ticket_updated).

Debug walk-through (for the next person who runs into something similar)

The session was a top-down ladder through the call chain. Each step is one Rails-console command and the question it answered.

Step 0 — Rule out the silent killers

The first two failure modes both produce “no webhooks arrive” without leaving a trace:

Org out of evaluation credits. Conversation#schedule_transcription_or_translation early-returns on transcribe_after_sync?, which is gated by evaluation_credits_available?. No credits = no trigger fires.
No active agent has ticket_updated. Rails fires, api2 returns 202 { matchedAgents: 0 }, no run is enqueued. From the operator’s side this looks identical to “Rails never sent the webhook.”


org = Organization.find_by!(slug: "rho-tnkw")
ActsAsTenant.with_tenant(org) do
  puts "evaluation_credits_available? = #{org.evaluation_credits_available?}"
  credit = org.credits.evaluation.active.first
  puts "credit: quantity=#{credit&.quantity} used=#{credit&.credit_usages_count}" if credit
end

Rho returned true with 1,627 evaluations remaining — credits were fine, so the silent-killer #1 was ruled out. (Silent-killer #2 was confirmed later in step 3 but was secondary.)

Step 1 — Did Rails attempt to enqueue at all?


GoodJob::Job
  .where(job_class: "ForwardRulebaseWebhookJob")
  .where("created_at > ?", 1.hour.ago)
  .order(created_at: :desc)
  .limit(10)
  .each { |j| puts [j.created_at, j.finished_at, j.error&.truncate(180) || "ok"].join(" | ") }

ForwardRulebaseWebhookJob rows in the last hour: 0. So Rails wasn’t even getting as far as enqueuing. Two possibilities:

perform_post_sync_analysis isn’t being reached on recent syncs.
perform_post_sync_analysis is reached and fire_conversation_updated_trigger is silently raising.

Step 2 — Is the sync pipeline reaching Rho at all?


ActsAsTenant.with_tenant(org) do
  rho_recent = Conversation.where("updated_at > ?", 1.hour.ago).order(updated_at: :desc)
  puts "Rho conversations updated in last hour: #{rho_recent.count}"
  rho_recent.limit(5).each do |c|
    puts [c.updated_at, c.prefix_id, c.interaction_type, c.transcription_state].join(" | ")
  end
end

51 conversations updated in the last hour, all interaction=ticket, all transcription=completed. So the sync was firing. complete_conversation_sync → schedule_transcription_or_translation → perform_post_sync_analysis → fire_conversation_updated_trigger should have been reaching all 51. Something in the trigger was silently failing.

Step 3 — Manually fire the trigger and watch the delta


ActsAsTenant.with_tenant(org) do
  c = Conversation.find_by_prefix_id!("conversation_qWpgQ92YV4vCazoZB8kVN4xj")
  before = GoodJob::Job.where(job_class: "ForwardRulebaseWebhookJob").count
  c.fire_conversation_updated_trigger
  after = GoodJob::Job.where(job_class: "ForwardRulebaseWebhookJob").count
  puts "delta=#{after - before}"
end

before=0 after=0 delta=0. Manually calling the method did not enqueue a job. The rescue was eating the error. The aggregate before=0 was the smoking gun — across the entire history of the good_jobs table, this job class had never been enqueued, period.

(Side discovery in this step: find_by(prefix_id: ...) raised PG::UndefinedColumn: column conversations.prefix_id does not exist. The conversations model uses has_prefix_id which provides find_by_prefix_id! as a class method, and the prefix_id value is derived from the integer id rather than stored as a column. The local db/structure.sql still references the column — that’s the schema-drift follow-up.)

Step 4 — Bypass the rescue

fire_conversation_updated_trigger has an outer rescue, and the enqueue_api2_forward_if_needed it calls has an inner rescue. Bypass both by building the trigger payload manually and calling the perform_later directly:


ActsAsTenant.with_tenant(org) do
  c = Conversation.find_by_prefix_id!("conversation_qWpgQ92YV4vCazoZB8kVN4xj")
  ForwardRulebaseWebhookJob.perform_later(
    event_type: "ticket_updated",
    organization_id: c.organization.prefix_id,    # ← reproduces the bug
    conversation: { id: c.id, prefixId: c.prefix_id, transcript: c.transcript, interactionType: c.interaction_type }
  )
end


(rulebase-api):175:in `block in <main>': undefined method `prefix_id' for an instance of Organization (NoMethodError)
Did you mean?  _prefix_id
               _prefix_id?

There it is. Replacing c.organization.prefix_id with c.organization.slug enqueued cleanly.

Lessons + follow-ups

1. `rescue StandardError → Rails.logger.error` without Sentry capture is invisible failure mode

This bug existed for the entire lifetime of the feature and produced zero alerts. The rescue was load-bearing — without it a single sync failure would have taken down the post-sync analysis path for the conversation — but the silent failure was indistinguishable from “everything is fine.” Every existing rescue site of this shape in rulebase-api is a latent equivalent of this bug.

Action item (this plan’s parent index): sweep rg "rescue StandardError" across rulebase-api and audit each site. Each should either re-raise, capture to Sentry explicitly, or be observable from an alerting pipeline. Tracked in the engineering follow-ups section of the parent plan.

2. Schema drift between local and prod

rulebase-api/db/structure.sql line 10518 has prefix_id text on conversations, but the column doesn’t exist in production. No migration removes it; the divergence has been there for a while and nobody noticed because nothing queries prefix_id as a column (everyone uses find_by_prefix_id! which decodes from id).

Action item: sanity-check the conversations table on prod (Conversation.column_names.include?("prefix_id")), refresh structure.sql either way, and scan other models for the same pattern.

3. The webhook-handler safety improvements that landed alongside the fix

PR #6538 (merged the same day) added two things that make the now-functional webhook path more honest:

Paused-agent filter — webhook handler in routes/webhooks.ts filters on agents.status = 'active'. The previous behavior (no filter) meant that the cloned [WIP] Stale Ticket Auditor (copy) agents in Rho’s UI — all paused — would have also fired from the now-fixed webhook path. With the filter, paused agents stay quiet from both the scheduler tick (worker.ts already had this) and from real-time events.
Trigger envelope in agent_runs.input — every webhook-triggered run now sees a structured trigger_event: block describing the conversation that fired it. Documented in the system prompt via TRIGGER_INPUT_CONTRACT so per-agent instructions can rely on it. Lets Ownership Watcher and future real-time agents scope to one conversation cheaply instead of doing a global org scan on every fire.

4. Sentry log volume tightened

Separate to the bug but uncovered while looking at error tracking, PR #6541 narrowed Sentry.consoleLoggingIntegration in api2 to forward only warn and error (was log, info, warn, error). Third-party console.log / console.info chatter from SDK boot banners and drizzle breadcrumbs was high volume / low signal in Sentry Logs and made it harder to spot real issues — exactly the kind of noise that contributed to nobody noticing the silent rescue’s log line.

Verification

After PR #6540 deployed:

GoodJob::Job.where(job_class: "ForwardRulebaseWebhookJob").count should start climbing on every Rho ticket sync.
api2 logs should show POST /webhooks/rulebase 202 responses.
Once an Ownership Watcher agent is activated with the ticket_updated trigger, agent_runs rows with source = 'webhook' should appear in api2’s DB and each row’s input should contain the trigger_event: envelope.

2026-05-14 — The ticket_updated trigger was silently broken since launch