Skip to Content
Internal docs are powered by Nextra Docs Theme.
SystemsPlatformAPI2Consensys account-escalation audit

Consensys account-escalation audit

Last updated: 2026-08-23

Reference: Rho / Consensys, Rulebase conversations 4250201, 4250314, 4251299, 4251432, 4251540, and 4251556.

Six related customer contacts produced no successful Slack escalation. This audit records why, what changed in production, and what remains before Consensys becomes a passing regression case.

Incident

The customer contacted Rho six times in 3 hours and 22 minutes:

UTCTicketConversationEvent
Aug 14, 20:382731954250201Greg Brenner requested recipient-bank information. Rho correctly required an authorized user.
Aug 14, 20:582732034250314Greg returned with bookkeeper Wendy Yu and asked whether recurring Congit-related charges shared a recipient. Rho could not provide the underlying merchant or bank details.
Aug 14, 23:082732324251299Greg expanded the issue into suspected fraud spanning many transactions and years. Rho requested an email because he lacked permissions.
Aug 14, 23:342732374251432The email reported suspicious transactions across nine merchant descriptors and became FRAUDTM-4046.
Aug 14, 23:562732434251540A follow-up call reached dead air and was merged into the email ticket.
Aug 15, 00:002732464251556A second voicemail had no transcript and was merged into the email ticket.

The first three calls were marked solved within approximately 7-16 minutes; the account incident remained unresolved until August 20-21. The main email eventually involved nine agents, but only three frontline agents received ticket-level quality evaluations.

The calls scored 90%, 100%, and 70%. Three evaluations on the email each scored 100%. They correctly rewarded safe communication, privacy, and escalation, but did not ask whether the account was progressing toward resolution.

Every touch can be handled correctly while the account is still being failed.

Contact surge and missed promise

The early and late warning signals were different:

SignalEvidence
Contact surgeSix contacts across calls, email, and voicemail in 3 hours and 22 minutes, with changing agents, fragmented identity, and no account-level resolution.
Missed promiseRho said on Aug 14 at 23:08 UTC that a full response would likely arrive Monday, while also suggesting a reply that night. At Aug 15, 01:48 UTC it committed to end of day Monday, Aug 17. After another Aug 17 check-in, Rho repeated the end-of-day commitment. At Aug 18, 12:21 UTC, Greg reported that no update arrived the previous day.

Contact velocity should create the incident. Unresolved follow-up and missed promise should show that it is worsening.

Failure and remediation ledger

#FailureStatusRemediationRemaining work
1No successful Slack alertRouting remediatedCS Escalations v54 points all 19 Slack nodes to #cs-quality-rulebase-alerts (C0B2YHBCFDK).Replay the incident through the live replacement signals.
2Identity fragmentation hid five contactsCompleted for this incidentA five-record production canary made all six records resolve under CONSENSYS SOFTWARE INC through agent_api.company_conversations. The enqueue timestamp fix merged in PR #10556  under ENG-2553 .Verify automatic enqueueing on a new Rho conversation before broad backfill.
3Complaint detection did not triggerExplained; no taxonomy change plannedBraintrust and PostHog showed no complaint creation or notification. Four evaluated conversations reasonably passed as non-complaints; two voicemails were not evaluated. Fraud and disputes are not automatically complaints.Keep service-drift detection in escalation workflows.
4Reassignment detection posted unsuccessfully, then stamped dedupeCompleted for this workflowDisabled the noisy, low-relevance detector and moved every Slack destination to the working channel.Add platform-level delivery-success gating.
5The strongest P1 output had a disabled notification loopReplacement signals live; replay pendingCS Escalations v55-v57 added unresolved-follow-up and account-repeat coverage. The original no-channel-switch loop remains disabled.Replay Consensys, then retain, replace, or remove no-channel-switch intentionally.
6Ticket-level QA masked account driftWorkflow mitigation liveAccount-repeat detection now evaluates related conversations together.Add durable incident progress and ownership if the workflow proves useful.
7Solved tickets misrepresented incident resolutionWorkflow mitigation liveAccount-repeat detection includes new, open, pending, hold, solved, and closed conversations.Separate ticket closure from durable incident resolution.
8Voicemails lacked quality and risk signalsLinking improvedCompany linking can place weakly attributed records in account history; empty voicemail is context, not a qualifying contact.Measure transcription and evaluation coverage.

Supporting evidence

Complaint classification

No target conversation emitted complaint.created or complaint.notified, although both events worked for other Rho conversations in the same period. Treating every fraud dispute as a complaint would create false positives. The account-level combination of repeated contact, fragmented handling, a serious topic, and missed commitments was the useful signal.

Company identity

Five records had null canonical company_id; only the email was directly linked to Consensys. The production canary repaired the incident through company_conversations.

The relationship store may contain multiple extracted entities, including Rho, merchant names, and a duplicate Consensys record. This is acceptable for v1, but alert consumers must verify which company is the supported account. See Conversation-to-company linking plan.

Slack delivery

On August 18, the reassignment Slack action returned:

{ "posted": false, "error": "not_in_channel" }

The workflow still wrote ticket_reassignment_guideline_alerted_at. This happened after 274 reassignment failures from July 5 through August 22. The required state model is:

detected -> attempted -> delivered -> stamped -> failed -> retryable

Disabled P1 delivery

On August 19 at 20:30 and 21:30 UTC, no-channel-switch classified the email as P1, citing four client-CS exchanges over four business days, the fraud report, and the urgent request for a call. Its agent ran, but no_channel_switch_loop was disabled, so neither Slack action executed. This was recovery evidence, not the earliest intervention.

Jira context

The output FRAUDTM-4046 remains new exposed missing context: it omitted when the issue was created, what Rho requested, and what progress occurred. ENG-2555  and PR #10563  add get_jira_issue_context, backed by organization-scoped synced Jira fields and comments. Synced data keeps workflow reads fast, replayable, and independent of live Jira availability.

Live CS Escalations configuration

Production is live at v57:

NodeStateDecision
#9 Find multi-ticket companiesDisabledKeep disabled. Two open tickets, especially unrelated ones, do not establish escalation.
#10 Find account repeat-contact patternsEnabled hourlyPrimary account-level detector.
#11 Find wire repeat contactsEffectively disabled: schedule disabledAudit historical precision before folding its useful coverage into #10 or removing it.
#12 Find txn repeat contactsDisabledAudit historical precision before folding or removing it.
#13 Find unresolved follow-upsEnabled hourlyKeep separate because it evaluates message meaning, not contact volume.
#24 Find no-channel-switch casesAgent enabled; notification loop disabledDecide after replaying #10 and #13.

Account repeat-contact thresholds

#10 returns one strongest subtype per incident:

SubtypeThreshold
high_risk_surge2 distinct conversations in 6 hours about one high-risk issue, excluding an orderly callback requested by Rho
acute_surge3 distinct conversations in 6 hours about one unresolved issue
sustained_recurrence4 distinct conversations in 7 days about one unresolved issue
chronic_recurrence5 distinct conversations in 45 days about one unresolved issue

It unions conversations.company_id with company_conversations.company_id, counts distinct conversations, includes solved and closed tickets, excludes non-customer and empty-voicemail activity, verifies the supported account and same incident, then posts to #cs-quality-rulebase-alerts before stamping chronic_repeat_contact_alerted_at.

The v57 field writer still receives only anchor_conversation_id, not all related_conversation_ids. This can allow a later related conversation to alert again.

Unresolved follow-up subtypes

#13 no longer requires emotional frustration. It treats calm follow-up such as “We did not receive any update yesterday” as evidence and returns one of:

  • missed_promise
  • no_response
  • repeat_request
  • no_meaningful_progress
  • re_explanation

Remaining work

  1. Replay Consensys against v57.
  2. Verify #10 alerts by the third substantive call without treating the requested authorization callback as a failure.
  3. Verify #13 classifies the August 18 message as missed_promise.
  4. Stamp all related_conversation_ids, not only the anchor.
  5. Verify failed Slack delivery does not stamp dedupe.
  6. Audit historical precision for #9, #11, and #12 before consolidation.
  7. Decide whether no-channel-switch adds independent value.
  8. Verify automatic production enqueueing before broad Rho backfill.

The regression passes when account identity resolves, the third substantive contact produces one Slack alert, the missed promise is classified correctly, and failed delivery remains retryable.

Last updated on