Consensys account-escalation audit
Last updated: 2026-08-23
Reference: Rho / Consensys, Rulebase conversations 4250201, 4250314,
4251299, 4251432, 4251540, and 4251556.
Six related customer contacts produced no successful Slack escalation. This audit records why, what changed in production, and what remains before Consensys becomes a passing regression case.
Incident
The customer contacted Rho six times in 3 hours and 22 minutes:
| UTC | Ticket | Conversation | Event |
|---|---|---|---|
| Aug 14, 20:38 | 273195 | 4250201 | Greg Brenner requested recipient-bank information. Rho correctly required an authorized user. |
| Aug 14, 20:58 | 273203 | 4250314 | Greg returned with bookkeeper Wendy Yu and asked whether recurring Congit-related charges shared a recipient. Rho could not provide the underlying merchant or bank details. |
| Aug 14, 23:08 | 273232 | 4251299 | Greg expanded the issue into suspected fraud spanning many transactions and years. Rho requested an email because he lacked permissions. |
| Aug 14, 23:34 | 273237 | 4251432 | The email reported suspicious transactions across nine merchant descriptors and became FRAUDTM-4046. |
| Aug 14, 23:56 | 273243 | 4251540 | A follow-up call reached dead air and was merged into the email ticket. |
| Aug 15, 00:00 | 273246 | 4251556 | A second voicemail had no transcript and was merged into the email ticket. |
The first three calls were marked solved within approximately 7-16 minutes; the account incident remained unresolved until August 20-21. The main email eventually involved nine agents, but only three frontline agents received ticket-level quality evaluations.
The calls scored 90%, 100%, and 70%. Three evaluations on the email each scored 100%. They correctly rewarded safe communication, privacy, and escalation, but did not ask whether the account was progressing toward resolution.
Every touch can be handled correctly while the account is still being failed.
Contact surge and missed promise
The early and late warning signals were different:
| Signal | Evidence |
|---|---|
| Contact surge | Six contacts across calls, email, and voicemail in 3 hours and 22 minutes, with changing agents, fragmented identity, and no account-level resolution. |
| Missed promise | Rho said on Aug 14 at 23:08 UTC that a full response would likely arrive Monday, while also suggesting a reply that night. At Aug 15, 01:48 UTC it committed to end of day Monday, Aug 17. After another Aug 17 check-in, Rho repeated the end-of-day commitment. At Aug 18, 12:21 UTC, Greg reported that no update arrived the previous day. |
Contact velocity should create the incident. Unresolved follow-up and missed promise should show that it is worsening.
Failure and remediation ledger
| # | Failure | Status | Remediation | Remaining work |
|---|---|---|---|---|
| 1 | No successful Slack alert | Routing remediated | CS Escalations v54 points all 19 Slack nodes to #cs-quality-rulebase-alerts (C0B2YHBCFDK). | Replay the incident through the live replacement signals. |
| 2 | Identity fragmentation hid five contacts | Completed for this incident | A five-record production canary made all six records resolve under CONSENSYS SOFTWARE INC through agent_api.company_conversations. The enqueue timestamp fix merged in PR #10556 under ENG-2553 . | Verify automatic enqueueing on a new Rho conversation before broad backfill. |
| 3 | Complaint detection did not trigger | Explained; no taxonomy change planned | Braintrust and PostHog showed no complaint creation or notification. Four evaluated conversations reasonably passed as non-complaints; two voicemails were not evaluated. Fraud and disputes are not automatically complaints. | Keep service-drift detection in escalation workflows. |
| 4 | Reassignment detection posted unsuccessfully, then stamped dedupe | Completed for this workflow | Disabled the noisy, low-relevance detector and moved every Slack destination to the working channel. | Add platform-level delivery-success gating. |
| 5 | The strongest P1 output had a disabled notification loop | Replacement signals live; replay pending | CS Escalations v55-v57 added unresolved-follow-up and account-repeat coverage. The original no-channel-switch loop remains disabled. | Replay Consensys, then retain, replace, or remove no-channel-switch intentionally. |
| 6 | Ticket-level QA masked account drift | Workflow mitigation live | Account-repeat detection now evaluates related conversations together. | Add durable incident progress and ownership if the workflow proves useful. |
| 7 | Solved tickets misrepresented incident resolution | Workflow mitigation live | Account-repeat detection includes new, open, pending, hold, solved, and closed conversations. | Separate ticket closure from durable incident resolution. |
| 8 | Voicemails lacked quality and risk signals | Linking improved | Company linking can place weakly attributed records in account history; empty voicemail is context, not a qualifying contact. | Measure transcription and evaluation coverage. |
Supporting evidence
Complaint classification
No target conversation emitted complaint.created or complaint.notified,
although both events worked for other Rho conversations in the same period.
Treating every fraud dispute as a complaint would create false positives. The
account-level combination of repeated contact, fragmented handling, a serious
topic, and missed commitments was the useful signal.
Company identity
Five records had null canonical company_id; only the email was directly linked
to Consensys. The production canary repaired the incident through
company_conversations.
The relationship store may contain multiple extracted entities, including Rho, merchant names, and a duplicate Consensys record. This is acceptable for v1, but alert consumers must verify which company is the supported account. See Conversation-to-company linking plan.
Slack delivery
On August 18, the reassignment Slack action returned:
{
"posted": false,
"error": "not_in_channel"
}The workflow still wrote ticket_reassignment_guideline_alerted_at. This
happened after 274 reassignment failures from July 5 through August 22. The
required state model is:
detected -> attempted -> delivered -> stamped
-> failed -> retryableDisabled P1 delivery
On August 19 at 20:30 and 21:30 UTC, no-channel-switch classified the email as
P1, citing four client-CS exchanges over four business days, the fraud report,
and the urgent request for a call. Its agent ran, but
no_channel_switch_loop was disabled, so neither Slack action executed. This
was recovery evidence, not the earliest intervention.
Jira context
The output FRAUDTM-4046 remains new exposed missing context: it omitted when
the issue was created, what Rho requested, and what progress occurred.
ENG-2555 and
PR #10563 add
get_jira_issue_context, backed by organization-scoped synced Jira fields and
comments. Synced data keeps workflow reads fast, replayable, and independent of
live Jira availability.
Live CS Escalations configuration
Production is live at v57:
| Node | State | Decision |
|---|---|---|
#9 Find multi-ticket companies | Disabled | Keep disabled. Two open tickets, especially unrelated ones, do not establish escalation. |
#10 Find account repeat-contact patterns | Enabled hourly | Primary account-level detector. |
#11 Find wire repeat contacts | Effectively disabled: schedule disabled | Audit historical precision before folding its useful coverage into #10 or removing it. |
#12 Find txn repeat contacts | Disabled | Audit historical precision before folding or removing it. |
#13 Find unresolved follow-ups | Enabled hourly | Keep separate because it evaluates message meaning, not contact volume. |
#24 Find no-channel-switch cases | Agent enabled; notification loop disabled | Decide after replaying #10 and #13. |
Account repeat-contact thresholds
#10 returns one strongest subtype per incident:
| Subtype | Threshold |
|---|---|
high_risk_surge | 2 distinct conversations in 6 hours about one high-risk issue, excluding an orderly callback requested by Rho |
acute_surge | 3 distinct conversations in 6 hours about one unresolved issue |
sustained_recurrence | 4 distinct conversations in 7 days about one unresolved issue |
chronic_recurrence | 5 distinct conversations in 45 days about one unresolved issue |
It unions conversations.company_id with company_conversations.company_id,
counts distinct conversations, includes solved and closed tickets, excludes
non-customer and empty-voicemail activity, verifies the supported account and
same incident, then posts to #cs-quality-rulebase-alerts before stamping
chronic_repeat_contact_alerted_at.
The v57 field writer still receives only anchor_conversation_id, not all
related_conversation_ids. This can allow a later related conversation to
alert again.
Unresolved follow-up subtypes
#13 no longer requires emotional frustration. It treats calm follow-up such
as “We did not receive any update yesterday” as evidence and returns one of:
missed_promiseno_responserepeat_requestno_meaningful_progressre_explanation
Remaining work
- Replay Consensys against v57.
- Verify
#10alerts by the third substantive call without treating the requested authorization callback as a failure. - Verify
#13classifies the August 18 message asmissed_promise. - Stamp all
related_conversation_ids, not only the anchor. - Verify failed Slack delivery does not stamp dedupe.
- Audit historical precision for
#9,#11, and#12before consolidation. - Decide whether no-channel-switch adds independent value.
- Verify automatic production enqueueing before broad Rho backfill.
The regression passes when account identity resolves, the third substantive contact produces one Slack alert, the missed promise is classified correctly, and failed delivery remains retryable.