Rho Escalations: Agents Plan

Date: 2026-05-07 Last updated: 2026-05-14 Status: Phase 1 foundation shipped; agent #1 live in production; the ticket_updated path was silently broken in prod until 2026-05-14 and is now fixed; agent #2 (first variant) is the next build. Scope: Map Rho’s 24-row CS escalations matrix onto a small set of api2 Agents, grouped by trigger cadence and shared tool surface, with a phase plan that mirrors Rho’s row-level priorities.

Status — as of 2026-05-14

Shipped (Phase 1 foundation)

• Dedup state — moved off the original plan of “tag column on the api2 conversations mirror”. api2.conversations is a thin synced mirror and was empty in production for most rows, so dedup writes were silently no-oping (caught from Braintrust trace 21ef7d6b-…). Replacement: agent_alert_tags text[] column on Rails conversations (GIN-indexed) plus POST/DELETE /internal/conversations/:id/alert_tags endpoints. The <agent-slug>/<flag-key> namespacing convention from the original design is unchanged. PR #6361. The tags column on api2.conversations was subsequently dropped.
• agent_alert_tags exposed in q_conversations (v06) so the LLM’s query_data SELECTs can filter out already-alerted rows with NOT (agent_alert_tags @> ARRAY['<agent-slug>/<flag-key>']). Guidance added to q_tables_schema.json and buildToolDocumentation. PR #6377.
• alert_conversation tool — atomic check-then-post-then-tag. As of PR #6383 it posts rich Block Kit messages via Rails’s Slack::ConversationNotificationMessage (header + alert-type context + body + “View in Rulebase” / “Resolve” / “Assign to…” buttons + ticket-detail footer), matching Rulebase’s complaint and sentiment notifications. Posting an alert escalates review_status from none to unresolved so the action buttons render. Input shape is (conversationId, tag, title, summary?) — no free-form message, the type label is derived from the tag namespace.
• Memory tool — promoted from “deferred”. Five operations (memory_view, memory_create, memory_update, memory_delete, memory_search), two tiers (/memories/core/<slug>.md auto-injected, /memories/notes/<slug>.md on-demand), caps (100 entries/agent, 8000-byte core total, 2000-byte per-entry) with LRU eviction and suggested_eviction.path hints. Surface mirrors Anthropic’s memory_20250818 provider-defined tool so the storage stays put when a Claude-backed agent lands later. PR #6387.
• ticket_updated ingestion path — Rails WorkflowTriggerEvent#enqueue_api2_forward_if_needed → ForwardRulebaseWebhookJob → POST /webhooks/rulebase on api2 → matches agents with triggers: [{ type: "ticket_updated" }], filtered to status = 'active' (PR #6538) → inserts an agent_runs row + enqueues a run-agent BullMQ job per match. The path was silently broken in prod from launch until 2026-05-14 — enqueue_api2_forward_if_needed called organization.prefix_id, which raises NoMethodError because Organization is friendly_id-slugged rather than has_prefix_id-prefixed; the surrounding rescue StandardError only Rails.logger.error’d (no Sentry capture), and no ForwardRulebaseWebhookJob row had ever been enqueued for any org. Fixed in PR #6540 (use organization.slug to match api2’s organizations.rulebase_id, plus Sentry.capture_exception in the rescue so the next regression is visible). See the 2026-05-14 postmortem for the full debug trail.
• Webhook-layer hardening (PR #6538) — two safety improvements layered onto the ingestion path the same day:
  • Webhook handler now filters on agents.status = 'active' (mirrors the scheduler in worker.ts). Paused agents — including the freshly-cloned-and-paused outputs of PR #6529 — no longer fire from real-time events, closing a gap where the clone safety story only held for scheduled runs.
  • Webhook now persists a structured trigger_event: envelope into agent_runs.input so a real-time agent knows which conversation triggered it. A system-prompt-level TRIGGER_INPUT_CONTRACT (in managed-agent.ts) documents the envelope shape so per-agent instructions can rely on it: when trigger_event is present, scope to that one conversation; when the task is the default string, run the periodic scan.
• Agent #1 — Stale Ticket Auditor — live, with one deliberate deviation from the plan: each of the six concerns is its own scheduled agent (Stale Ticket Notifier, etc.) rather than one digest agent. Same 09:00-ET business-morning cadence, posts via alert_conversation per-concern with namespaced tags (stale-ticket-auditor/<concern-key>), into #cs-quality-rulebase-alerts. Validated against Braintrust trace 70d65ceb-…: dedup hits return posted: false, reason: "already_tagged"; first-time hits return posted: true; tag namespacing correct; calendar-day proxy for business days used; filter-before-aggregate convention followed.

Next

Agent #2 (first variant) — Ownership Watcher. Buildable today end-to-end on the Phase 1 infra above. The plan previously bundled four ownership heuristics under a single “Ownership Anomaly Watcher” agent; we’re now shipping the highest-value one first and treating the rest as follow-up variants:

• Name: Ownership Watcher (slug: ownership-watcher).
• Trigger: ticket_updated (real-time) + weekday 09:00-ET schedule (daily backstop, same cadence as agent #1).
• Tools: query_data + alert_conversation + memory.
• Destination: #cs-quality-rulebase-alerts.
• Heuristic: row 5 only — 3+ distinct agents have touched an open ticket → tag ownership-watcher/three-plus-agents.
  • “Touched by an agent” = customer-facing repliers + current/past assignees. Internal-note-only authors do not count for v1.
  • Single band (no escalation by agent count); each ticket alerts at most once across runs.
  • Title format: Multi-agent ticket (3+): {short subject line}.
  • Real-time path: when the input contains a trigger_event envelope, scope to that single conversation. Scheduled path: full scan as a backstop.
  • Ticket-level for v1; case-level grouping (by BID / application, per Olivia’s note on the matrix) deferred until Sara + Olivia confirm the definition.

The remaining three heuristics from the original plan move to follow-up variants once the row-5 agent has lived in prod for a few days:

• Row 6 (P1): ticket reassigned 3+ times → tag ownership-watcher/three-plus-reassignments.
• Row 7 (P1): agent-to-agent transfer without required internal-note format → tag ownership-watcher/missing-internal-note-on-transfer.
• Row 23 (nice): application touched by 4+ people across teams → tag ownership-watcher/four-plus-people-on-app.

Known gap (does not block agent #2)

POST /webhooks/rulebase does not coalesce — every ticket_updated event creates a new agent_runs row + BullMQ job per matched agent. agent_alert_tags protects against duplicate Slack posts but not against duplicate agent-run cost. With the trigger envelope from PR #6538, an agent now exits cheaply after a single WHERE id = <conversation_id> query if the ticket doesn’t match its threshold, so the per-fire cost is dominated by one SQL round-trip rather than an LLM-driven full scan. Defer a webhook-time (conversation_id, agent_id) debounce until Braintrust traces show the actual fan-out cost from Rho’s volume is meaningful.

Engineering follow-ups (surfaced by 2026-05-14 debug session)

Source matrix

The 24-row matrix from Rho’s CS leadership lives in two places:

• Live source: Rulebase Alert Matrix — Notification Matrix  (Google Sheet, edited by Rho). This is the source of truth — when Rho updates priorities, thresholds, or status, the change happens here first.
• In-repo snapshot: /data/rulebase-alert-matrix.csv. Last exported 2026-05-08. Useful for offline reading, agent runs, and diffing changes between the live sheet and what this plan was written against. Re-export the CSV when material updates land in the live sheet.

Columns (both): #, Category, Flag Description / Summary, Threshold & Criteria, Data Source, Priority to flag in Message, Priority - Chidi, Rulebase ETA, Slack Routing Channel, Tags, Frequency, Action Required, Escalation Rule, Status, Notes / Manager Feedback, Alerts to Flag, Links to Include.

This plan references rows by # throughout but does not duplicate every column.

Context

Rho’s CS leadership (Sara Twiner, Zorica Tasic, Olivia Podos) gave us a 24-row matrix of quality alerts they want surfaced into #cs-quality-rulebase-alerts (and a few adjacent channels / DMs). See Source matrix above for the full data. A few things stand out when reading the sheet end-to-end:

• Most rows are alerting, not complaints. They are “read ticket / case state, apply a heuristic, ping a channel” — not human-reviewed escalations.
• Triggers cluster. Roughly half are real-time on a ticket update or new customer reply; the rest are on-threshold SLA scans (every business day) over open tickets.
• Destinations cluster too. ~16 rows go to #cs-quality-rulebase-alerts; 4 go to #cs-service-helpdesk; the remainder are DMs to Sara or interim posts to #c-compliance-ops.
• Several rows depend on a “case” abstraction we don’t have yet. Olivia’s notes on rows 5, 6, 7, 8, 10 ask us to define case = grouped by BID/application vs individual ticket. That’s a precondition for accurate triggering on those rows; we should not block the rest of the matrix on it.
• The matrix’s “Escalation Rule” column is out of scope for v1. Things like “Escalate to Sara Twiner if no owner assigned within 1 BD of flag” require us to remember the time of our flag and check a follow-up condition later. That’s a meaningfully harder state-machine problem than the rest of the alerting work; tackling it would force a generic alert ledger into the design before we’ve shipped anything. We’ll revisit once the basic alerts are in production and Rho confirms the dedup primitives below are working.
• All net-new state and tool surface lands in api2, not Rails. Per the default ownership rule in the architecture plan: new features ship on api2 by default, falling back to Rails only for primitives Rails fundamentally owns (RLS-enforced SQL, integration sync, encrypted credentials). This work touches Rails for query_data (RLS) and credential fetching (Slack OAuth), and for nothing else.

This shape is exactly what the api2 Agents primitive is for — a row in api2.agents with instructions, a triggers array, and a tools array. The existing trigger types (schedule, ticket_updated/closed/solved, jira_*, email_received, slack_message_received) and the existing tool surface (rulebase workspace data via query_data / get_schema, slack, email) cover most of the matrix; gaps are called out below.

Mapping principle

Group by trigger cadence + tool surface, not by row.

A single scheduled scan that runs at 09:00 ET on business days can answer many “ticket has been open > N BD” questions in one pass — same SQL surface, same channel, same audience. A single ticket-update agent can run several language/ownership heuristics on the latest reply with the same workspace-data context loaded once.

Splitting by row would give us 24 nearly-identical agents that all query q_conversations and post to the same Slack channel. Splitting by trigger gives us ~10 agents, each with one clear job.

What stays separate:

• Different trigger types (real-time ticket_updated vs daily schedule vs Jira event) → different agents.
• Different destination channels / audiences → different agents (so the ops team can mute/own them independently).
• Heuristics that need a different tool entirely (Salesforce read for churn, RAP read for second-ticket-same-BID) → their own agent so we can keep the tool list tight.

Constraints belong in tool shape, not prompts. Anywhere we’d otherwise write “you MUST always do X after Y” in an agent’s instructions is a smell. Make X impossible without Y by collapsing both into one tool call. The dedup story below is the first concrete instance — see Dedup state.

Dedup state

Even with the escalation-rule semantics out of scope, the event-style agents (#2–#6) still need to answer “have we already alerted about this exact thing?” — otherwise every ticket_updated event will re-fire all the heuristics that have been continuously true since the first time they fired (e.g., “3+ agents have responded” stays true forever once it’s true). The schedule-style agents need it less because we’re going to lean on a digest framing for them (see agent #1 below).

Where the state lives

A tags text[] column added to the existing api2.conversations table:

export const conversations = api2.table("conversations", {
  // existing columns: id, rulebaseId, connectionId, connectionUid, organizationId, ...
  tags: text().array().notNull().default(sql`ARRAY[]::text[]`),
});

api2.conversations is already a synced mirror of Rails conversations (added in migration 0008_loud_gauntlet, with rulebase_id text unique linking back to the Rails source-of-truth row). So the entire schema change here is one column — no new table, no new Rails endpoint, no new sync function.

Tag values use a <agent-slug>/<flag-key> namespacing convention (e.g. ownership-watcher/three-plus-agents, stale-ticket-auditor/stale-20bd). Cheap, namespaced per-agent without an explicit agent_id column, queryable with the Postgres array operators @> and &&.

We considered six other placements and rejected them:

• A new dedicated api2.conversation_alert_flags table with (agent_id, organization_id, rails_conversation_id, flag_key, fired_at). The previous draft of this plan. Strictly more schema for the same dedup behavior; tags-on-the-existing-mirror gets there with one column. We’d choose the dedicated table later only if we need typed analytics (“how many alerts per agent per week”) that array-of-strings doesn’t support cleanly.
• A jsonb column on Rails conversations. Adds Rails surface area we’d then have to migrate; biases new dedup state toward Rails by default. Inconsistent with the default ownership rule.
• A separate Rails conversation_agent_flags table. Same objection as above.
• Zendesk tags. New write-tool tooling per customer, pollutes the customer’s tag taxonomy, doesn’t work for non-Zendesk sources.
• Slack threads. Requires a Slack search/read tool we don’t have, ties durability to a channel we don’t control.
• A jsonb on the api2 agents row. Conflates dedup state with agent config; would need to be unpacked the moment more than one agent’s worth of flags accumulates.

The agent’s query_data SELECT can’t filter on tags inline (cross-schema joins between Rails q_conversations and api2.conversations are not allowed). That’s fine — the atomic alert tool below re-checks at write time, so even if the SELECT pulls already-tagged tickets the alert just no-ops on them. Cost: maybe 10–50 extra rows per scan. Cheap.

What we lose vs the dedicated flag table: per-tag fired_at timestamp and an explicit agent_id. The namespacing convention recovers the agent_id. The timestamp is genuinely gone — we can recover it later by upgrading tags to tags jsonb of {tag: {set_at, set_by_agent_id}} if/when the deferred escalation-rule semantics (“escalate if no owner within 1 BD of flag”) come back.

Lifecycle: when a Rails conversation is hard-deleted, its synced api2 row is orphaned (or removed, depending on the existing sync) and its tags go with it. When a conversation reopens, tags stay in place — under-alerting on reopen is the safer failure mode.

Where the complexity of using it lives

Layered, with the heaviest layer being the tool’s shape:

1. Per-agent instructions carry the strategy + the tag. This is intrinsically per-agent and has nowhere else to live. Example: “Each business morning, find tickets open ≥10/20/30 BD with no resolution. Alert each one once per band, using tags stale-ticket-auditor/stale-10bd, …/stale-20bd, …/stale-30bd.”

2. Tool docs (auto-injected into the system prompt) carry the mechanism. Example: “alert_conversation is the only way to post per-conversation alerts. It dedup-checks api2.conversations.tags for the given (rulebase_id, tag) and no-ops if already tagged.”

3. The atomic alert tool carries the actual enforcement. Don’t ship mark_conversation_tag as a separate tool the agent must remember to call after slack_send. Ship one tool that checks-then-posts-then-tags atomically:

   alert_conversation({
     conversationId,   // Rails conversation id (the agent gets this from q_conversations.id)
     tag,              // namespaced tag, e.g. "ownership-watcher/three-plus-agents"
     channel,          // "#cs-quality-rulebase-alerts"
     message,          // markdown body
   })
   // returns: { posted: true, slackTs } | { posted: false, reason: "already_tagged" }

   Implementation:

   UPDATE api2.conversations
   SET tags = array_append(tags, $tag)
   WHERE rulebase_id = $conversationId
     AND NOT (tags @> ARRAY[$tag])
   RETURNING id;

   0 rows returned = already-tagged, no-op. 1 row = call sendSlackNotification (the existing Rails proxy in lib/rails-client.ts); on Slack failure, run the inverse array_remove to make the next run retry. Race-safe across overlapping scheduled runs because Postgres serializes the row UPDATE.

4. Keep slack-send-<channel> for non-conversation alerts. Digests, summaries, anything not tied to a single conversation use the plain Slack tool — no dedup overhead, no awkwardness around “which conversationId do I pass for a 12-row digest?”. This already works today via sendSlackNotification in rails-client.ts.

So the per-conversation event agents get query_data + alert_conversation; the digest agent gets query_data + slack-send-<channel>. Two patterns, no overlap. Both Slack write paths share the same Rails proxy under the hood until we migrate Slack to TS-native (separate work, not a Phase 1 prerequisite).

Memory

Status (2026-05-10): Shipped in PR #6387 — see the Status block above. Surface matches Anthropic’s memory_20250818 provider-defined tool (five commands instead of the single multi-command tool sketched below, two path tiers, byte/entry caps with LRU eviction). The single-tool sketch and core.md-as-table-of-contents framing below stays as the original design reference; the actual ship is shape-compatible but more granular.

Even with the dedup story sorted, the agent still has to write the SQL to find candidates each run, and there’s no inherent guarantee it writes the same query twice. We don’t want it discovering the schema and re-deriving the right predicates from scratch on every scheduled invocation — that’s wasted tokens, drift risk on the heuristic itself (“3+ agents responded” interpreted slightly differently each run), and zero ability for a human to inspect what the agent is actually doing in steady state.

We considered three more constraining answers and rejected them:

• Specialized detector tools (find_stale_tickets({ thresholds })) — defeats the whole point of the agents primitive being one configurable row with a generic tool surface. We’d be back to writing code for every new heuristic.
• Promoting heuristics to view columns (q_conversations.distinct_agent_responder_count) — ossifies the schema around one customer’s interpretation. New heuristic = Rails migration. Doesn’t generalize across orgs whose definitions differ.
• A typed query catalog with draft / approved status — too much workflow ceremony for what’s actually just “the agent should remember what worked”.

What we actually want is the simplest thing the cookbook describes: a generic string-keyed memory tool the agent maintains itself.

The tool

One memory tool per agent, four operations:

memory({
  command: "view" | "update" | "delete" | "search",
  path: string,                    // e.g. "queries/stale_10bd", "notes/data-quirks"
  content?: string,                // for update
  mode?: "append" | "overwrite",   // for update; defaults to overwrite
  query?: string,                  // for search
})

Backed by a per-agent table — api2.agent_memory_entries (agent_id, path, content, updated_at). The “filesystem” is just a path convention; the storage is rows. Path conventions emerge from instructions (queries/, notes/, templates/) — not enforced.

Core memory injection

A single entry at path: "core.md" is auto-injected into the system prompt on every turn via prepareCall. It’s the agent’s own table of contents:

# Core memory

## Queries (use verbatim unless something feels off)
- queries/stale_10bd: tickets open ≥10 BD with no resolution
- queries/three_plus_agents: tickets where ≥3 distinct agents have replied

## Notes
- notes/zendesk-quirks: q_conversation_parts.author_type values for Rho
- notes/business-day-calc: how holidays are handled

## Templates
- templates/digest: morning stale-ticket digest format

The agent maintains core.md itself. Detail entries (the SQL, the prose notes, the templates) only get loaded into context on demand via memory.view. Keeps the prompt tight; the catalog’s description, not the catalog’s contents, is what’s always-on.

How a run actually flows

1. Agent receives trigger; reads core.md (auto-injected).
2. Sees queries/stale_10bd exists and matches the task at hand.
3. Calls memory.view({ path: "queries/stale_10bd" }) → gets SQL string.
4. Calls query_data with that exact SQL. Same query as yesterday, no drift.
5. Calls alert_conversation per result row.
6. If something looks wrong (zero rows when many expected, error from query_data), agent writes new SQL via query_data, then memory.update({ path: "queries/stale_10bd", mode: "overwrite", content: <new sql> }).

The system prompt biases hard toward “if a memory entry exists for this task, use it verbatim — only rewrite if it actually fails or returns clearly wrong results”. Whether the agent obeys is still LLM judgment, not enforced; the upside is that humans editing the entry directly is the audit and override surface.

Honest caveats

• First-run discovery is non-deterministic. Empty memory → LLM writes the first query freehand. If it’s subtly wrong it gets stored and reused. Mitigation: humans review memory in the Rulebase UI after the first few runs and edit directly. No status workflow, just edit.
• Memory hygiene. Grows unbounded unless the agent prunes. Bake a “review and tidy” instruction into the agent template; revisit if entries get unwieldy.
• Concurrency. Two scheduled runs of the same agent at once can race on writes. Last-write-wins is fine for now; revisit if it bites.
• No type safety on stored content. A query stored as a string can become invalid if the schema changes. Agent has to handle execution failures gracefully (try stored query → on error, rewrite via query_data → update memory).
• Cross-agent / cross-org reuse: no. Each agent’s memory is scoped to itself. Different orgs have different schemas; a human can copy entries between agents out of band if needed. Reusable, system-curated knowledge is the skills layer below — different lifecycle, different table, same conceptual surface to the agent.
• Memory and dedup are orthogonal layers. Memory is “what’s the SQL I figured out for this task”. Dedup (conversations.tags + alert_conversation) is “have I told you about this ticket yet”. Both are needed.

Skills (deferred)

The buildToolDocumentation block we ship today auto-injects per-tool guidance (SQL conventions, Slack mrkdwn, email formatting, reporting conventions) into the system prompt at every turn. That works for the small surface we have now, but a few moves push us toward externalizing this prose into proper “skills” stored alongside (not inside) the codebase:

1. The prompt-block list will keep growing. Memory tool docs, case-abstraction guidance, agent-specific cookbooks, customer-specific tone, etc. Stuffing more strings into a TS function gets painfully wrong fast.
2. Customers will want to customize. Rho’s Slack tone, Sara’s preferred digest cadence wording, an org’s escalation language vocabulary — none of that should require a code PR.
3. Versioning + authorship matter. Once skills are real prose curated by humans, knowing “who edited this, when, and what was the previous version” is table-stakes — exactly the metadata that doesn’t fit a plain string-keyed memory store.
4. Braintrust evals want stable targets. Scoring against “the slack-formatting skill v3” is much cleaner than scoring against the contents of buildToolDocumentation at a particular git SHA.

Why a separate DB table from agent_memory_entries

Memory and skills look superficially identical (string-keyed, path-shaped, agent-readable) and sharing a table is tempting. We’re not going to:

• Skills need versioning, creator metadata, and source tracking; memory does not. Memory is high-churn, agent-self-curated, last-write-wins. Skills are low-churn, human-curated, every change is an event worth recording. Sharing a table forces every memory write through the versioning code path and every skill row to handle an agent_id IS NULL branch — both pay tax for the other’s needs.
• Skills will eventually grow review/approval, sharing across orgs, publishing. Memory will not. Different roadmaps, different table.
• The agent doesn’t care about the lifecycle distinction; the database does. We unify at the agent interface (one knowledge tool that surfaces both as a virtual filesystem) and split at the storage layer.

Schema sketch (build later, pinning shape now)

-- Versioned, human-curated knowledge
CREATE TABLE api2.skills (
  id uuid PRIMARY KEY,
  organization_id uuid NULL REFERENCES api2.organizations(id),  -- NULL = built-in / system
  name text NOT NULL,                      -- slug, e.g. "slack-formatting"
  description text NOT NULL,               -- one-liner shown in the always-on TOC
  current_version_id uuid REFERENCES api2.skill_versions(id),
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (organization_id, name)           -- letting an org override a built-in by re-using the name
);
 
CREATE TABLE api2.skill_versions (
  id uuid PRIMARY KEY,
  skill_id uuid NOT NULL REFERENCES api2.skills(id) ON DELETE CASCADE,
  version int NOT NULL,                    -- 1, 2, 3, ...
  content text NOT NULL,                   -- the skill markdown
  created_by_user_id uuid NULL,            -- NULL when seeded by the admin script for built-ins
  created_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (skill_id, version)
);

Same shape as the existing artifacts / artifact_versions pair in api2 — same primitive, different content type.

Sources

• Built-in skills: authored as markdown in rulebase-api2/src/prompts/skills/<name>.md and seeded into the DB by an admin script (pnpm db:seed:skills). Script diffs filesystem against DB, upserts changed system rows, bumps version, records the commit SHA in created_by_user_id’s sibling field (or a separate source_commit column — TBD). Reviewable as regular markdown PRs.
• Custom skills: authored by org admins in the Rulebase UI. Stored with organization_id set; created_by_user_id is the editing user.

Read surface (unified)

The agent sees one tool, not two:

knowledge({
  command: "view" | "list" | "search",
  path: string,                            // e.g. "system/slack-formatting", "org/escalation-language", "agent/queries/stale_30bd"
  query?: string,                          // for search
})

Three path prefixes correspond to the three ownership tiers:

• system/<name> — built-in skills, read-only, visible to every agent
• org/<name> — this org’s custom skills, read-only at runtime (writes via UI), shadowing same-named system skills
• agent/<path> — this agent’s memory, read/write via memory.update / memory.delete (existing tool, kept separate to make writability obvious)

Resolution for skill lookups: org/<name> is checked first; if absent, system/<name> is returned. If both exist, the admin UI surfaces a conflict warning when the org skill is created. This preserves the “customer override” path without silent merging.

The always-on system-prompt injection becomes a tiny TOC of every visible skill name + description (the equivalent of core.md); full content is fetched on demand. This is the same hybrid pattern as memory’s core.md, and it’s the only model that scales as skills accumulate.

When we build it

Not Phase 1. The trigger conditions, in priority order:

1. The third prompt block we want to add. When buildToolDocumentation would need a fourth or fifth conditional section (memory tool docs, case-abstraction guidance, alert language vocabulary), stop adding strings to TS and ship the skills primitive instead.
2. First customer asks to customize. When Rho (or a future customer) wants to override the slack-formatting block with their own house style, that’s the moment to ship the org-skill tier.
3. Wiring Braintrust evals. Skills give scorers stable, addressable prompt targets that survive prose edits.

Until any of those triggers, the snapshot tests on buildToolDocumentation are the right level of investment. They go away when skills land.

Honest caveats

• The seed-script story is fiddly. Diffing markdown files against DB rows, deciding when to bump version vs. ignore whitespace, handling deletes. We’ve punted this until we actually need it.
• Override semantics are simple now, hard later. “Org skill shadows system skill” is fine until a customer wants composition (“use the system skill but add this paragraph at the end”). We’ll cross that bridge when it shows up; current bias is to refuse composition until a real request justifies the merge logic.
• Versioning of the current_version_id pointer creates orphans. Old skill_versions rows live forever unless we add a retention policy. Cheap to ignore in v1; revisit if storage matters.
• Cross-org sharing is out of scope. A skill authored by one customer can’t be promoted to another customer or to the system tier without an explicit copy operation. We don’t plan to build a “marketplace”; if we ever do, it’s its own design.

Proposed agents

Every agent below carries memory in addition to the listed tools — omitted from the table for readability. The Phase column reflects the highest-priority row each agent covers, using Rho’s phase assignments on the matrix.

* blocked on input from Rho — see Open questions / dependencies.

A few rows ship in an earlier phase than Rho asked because they fall into the same agent’s bucket as a higher-priority row:

• Row 4 (P2) ships in Phase 1 inside agent #1.
• Row 14 (P3) ships in Phase 1 inside agent #1.
• Row 22 (nice) ships in Phase 1 inside agent #1.
• Row 23 (nice) ships in Phase 1 inside agent #2.
• Row 11 (P2) ships in Phase 1 inside agent #4.
• Row 13 (P2) and row 21 (P2) ship in Phase 1 inside agent #3.

This is a free win for Rho — they get coverage on lower-priority rows earlier than asked, with no additional cost to us, because the higher-priority rows in the same agent already paid for the agent’s existence.

10 agents covering 23 of 24 matrix rows. Row 9 (“client opened second ticket while first is open, same BID”) is on hold pending RAP — fold into agent #4 once the RAP signal is available.

Why this grouping

• #1 collapses six SLA scans into one daily digest, not per-ticket events. All six rows it covers — three at P1, one at P2, one at P3, one nice-to-have — ask the same query shape (“open tickets where some clock has crossed N business days”) against the same q_conversations view. One LLM call iterates the buckets and posts a single structured Slack message, like a morning standup. No per-ticket dedup needed because the digest is a snapshot of current state; resolved tickets fall off naturally, and row 1’s “every 10 BD thereafter” semantics are satisfied because age keeps growing and the ticket keeps appearing in the right band. Cheap; low blast radius if a heuristic misfires; ships rows Rho put across four priority tiers in one shot.
• #2 and #3 both fire on ticket_updated but have different audiences (Sara’s DM vs the channel) and different prompt scope (ownership state vs message language), so they stay separate. Within each, multiple heuristics share the same loaded ticket context.
• #4 is the single-ticket “is this a duplicate of something already open?” check. Same trigger + tool surface for rows 11, 12, 17. Row 9 will join when RAP exposes a programmatic second-ticket-same-BID signal.
• #5 needs Rho-event correlation (was there an automated transaction-failure email in the last 24h?) — close to #3 in shape but distinct enough that we keep its prompt clean.
• #6 is the only agent that needs Salesforce. Keep it isolated so we don’t drag the Salesforce tool into the broad ticket_updated agents.
• #7 is a multi-ticket clustering scan — different from #1 (SLA per ticket) and from #4 (duplicate per org). Needs its own cadence (every 2h is faster than daily but cheaper than real-time).
• #8, #9, #10 are blocked on clarification — they still get slots in the plan so we know where they’ll land once cleared, but build only when their definitions are confirmed.

Phase plan

Phases mirror Rho’s row-level priority assignments. Within each phase, the build order is set by dependency on shared infrastructure (foundational primitives first, then progressively more specialized tool surfaces).

Phase 1 (six unblocked agents)

This is the bulk of the work. The event-style agents (#2–#6) need three pieces of net-new infrastructure that have all now shipped:

1. Per-conversation dedup state — conversations.agent_alert_tags text[] on Rails (GIN-indexed) plus POST/DELETE /internal/conversations/:id/alert_tags. Replaced the original api2.conversations.tags plan after that mirror was found unpopulated in production. PR #6361. Surfaced in q_conversations_v06 so SELECTs can filter on it. PR #6377.
2. alert_conversation tool — atomic check-then-post-then-tag, posts rich Block Kit messages via Slack::ConversationNotificationMessage. PR #6383.
3. ticket_updated webhook ingestion path — Rails → ForwardRulebaseWebhookJob → api2 /webhooks/rulebase → BullMQ. Live.

Plus the originally-deferred fourth piece:

4. Memory tool — shipped in PR #6387. Five commands, two tiers, LRU eviction. Modeled after Anthropic’s memory_20250818 so storage survives a later Claude swap.

The digest agent (#1) needed only query_data + a Slack send path and shipped first (and as it turned out, split into one agent per concern rather than one digest agent — see Status block). Slack send still flows through the sendSlackNotification Rails proxy in lib/rails-client.ts; migrating Slack to TS-native per the default ownership rule remains desirable but is not a Phase 1 dependency.

1. Agent #1 — Stale Ticket Auditor. ✅ Shipped (with deviation: split into one scheduled agent per concern rather than one combined digest agent — see Status block). Foundational. Exercised query_data + alert_conversation end-to-end on a schedule trigger. The plan originally suggested staging via email-to-a-single-Rulebase-recipient before flipping to Slack; in practice we went straight to alert_conversation into #cs-quality-rulebase-alerts because the per-conversation dedup story (PR #6361) made the blast radius bounded. Iteration loop in use: trigger from the agent chat (“Run agent” button) — chat-triggered runs enqueue via BullMQ identically to scheduled runs (PR #6330), so <AgentRunStatus> renders status + output inline and writes a real agent_runs row with source: "chat". Same Braintrust trace, same DB persistence as a 09:00-ET schedule run. There is also a direct POST /agents/{id}/runs endpoint that mirrors the schedule path (sets source: "manual") for cases where the chat surface isn’t convenient.
2. Agent #2 — Ownership Anomaly Watcher. ← Next. First event-style agent. Exercises ticket_updated trigger + alert_conversation + agent_alert_tags end-to-end. Heuristics (3+ agents responded, reassigned 3+ times, missing internal-note format on transfer) are all derivable from q_conversation_parts / q_conversation_assignments. Ships rows 5, 6, 7 (P1) plus 23 (nice). Build via the UI; no code change required for v0.
3. Agent #3 — Client Language & Sentiment Monitor. Same trigger pattern as #2, extends the prompt surface to message-text heuristics. Sentiment drift specifically — start with a simple “last 3 customer replies trending negative” prompt; no separate scoring service. Ships rows 15, 19 (P1) plus 13, 21 (P2).
4. Agent #4 — Duplicate / Repeat-Contact Detector. Adds the case-grouping logic (initial pass: org + topic-similarity over open tickets in the last N days). Keep the bar simple: “is there another open ticket on the same wire / same transaction / same BID?”. Ships rows 12, 17 (P1) plus 11 (P2). Row 9 (P1, RAP-blocked) folds in once RAP exposes the second-ticket signal.
5. Agent #5 — Transaction-Failure Rage Detector. Narrow scope, high signal. Needs a Rho-event source — confirm Rails already records automated-failure notifications before starting. Ships row 18 (P1).
6. Agent #6 — Credit-Limit Churn-Signal Watcher. First agent that needs Salesforce read access. Build the Salesforce TS-native tool here (account tier, AM/AE/Growth Manager lookup) since it’ll be reused for Sara’s weekly report and other roadmap items. Ships row 16 (P1).

Phase 1, blocked:

• Agent #8 — Repeat-Contact Pattern Scanner. Rho marked row 10 as P1 but it’s blocked on Olivia clarifying “how does Rulebase determine repeat contact across tickets” (matrix note on row 10). Once cleared, this is shape-identical to the digest agent #1 — minimal incremental work. Don’t actively build until cleared, but it slots into Phase 1 the moment it is.

After the first 2-3 Phase 1 agents ship, reassess: the most useful next agent will likely be obvious from CS team feedback in #cs-quality-rulebase-alerts (signal-to-noise ratio per agent will tell us where to invest). The plan order above is a default, not a contract.

Phase 3

7. Agent #7 — Platform Bug Cluster Detector. First clustering agent (scan, group by tag + feature, alert on 2+ in 48h). Pure SQL + LLM summarization; no new tool surface beyond what Phase 1 already built. Ships row 20.

Phase 4 (blocked)

8. Agent #9 — Compliance Freeze Acknowledgment Tracker. Blocked on Olivia defining “compliance freeze” + “no acknowledgment sent” (matrix note on row 2). Reuses the hourly schedule + ticket_updated dual trigger pattern.

Nice to have (blocked)

9. Agent #10 — Application Stall Watcher. Blocked on Olivia clarifying the 7-day back-and-forth definition + Derek’s escalation policy with Onboarding (matrix note on row 3). Interim posting to #c-compliance-ops can be a lightweight scheduled SQL cron until the agent’s prompt is well-defined.

Notes on Rho’s priority assignments

• Phase 2 has no dedicated agents. Every Phase 2 row (4, 11, 13, 21) folds into a Phase 1 agent’s bucket and ships earlier than asked. Phase 2 is effectively absorbed into Phase 1.
• Two Phase 1 agents (#5 and #6) cover only one row each. They look “expensive per row” but they’re each unlocking a primitive (transaction-failure event correlation, Salesforce read tool) that downstream agents and the broader roadmap will reuse. Worth doing in Phase 1 even at one row per agent.
• Six agents in Phase 1 is a lot. Confirm with Rho whether they want all six concurrently or staged across two sub-phases (1a / 1b). My lean: stage as 1a (#1 + #2 — foundational, two trigger paths) → 1b (#3 + #4 — text heuristics + case grouping) → 1c (#5 + #6 — narrower-scope agents that reuse everything 1a/1b built). Each sub-phase takes a few days; the whole Phase 1 fits in a couple of weeks.

Trigger / tool gaps to close

Most of the matrix is achievable with what exists today. Status of each piece:

• ✅ Per-conversation dedup state — conversations.agent_alert_tags text[] on Rails with GIN index plus POST/DELETE /internal/conversations/:id/alert_tags endpoints (PR #6361). Surfaced in q_conversations_v06 for inline filtering (PR #6377). Original api2.conversations.tags plan was reverted (column dropped in migration 0014_fine_payback) because the api2 mirror was unpopulated in production.
• ✅ alert_conversation tool in api2. Now calls Rails’s POST /internal/conversations/:id/slack_notifications, which composes a rich Block Kit message via Slack::ConversationNotificationMessage and escalates review_status: none → unresolved so Slack action buttons render (PR #6383). Tool input is (conversationId, tag, title, summary?) — type label is derived from the tag namespace; the agent doesn’t see the Block Kit shape.
• ✅ Memory tool — shipped in PR #6387 as five granular tools (memory_view, memory_create, memory_update, memory_delete, memory_search) against api2.agent_memory_entries, mirroring Anthropic’s memory_20250818. Core entries under /memories/core/<slug>.md are auto-injected into the system prompt on every run.
• ✅ ticket_updated webhook ingestion. Rails WorkflowTriggerEvent#enqueue_api2_forward_if_needed → ForwardRulebaseWebhookJob → api2 POST /webhooks/rulebase → matching agents → agent_runs insert + BullMQ enqueue. Live.
• ⏳ ticket_created trigger. A few rows (9, 11, 12, 17) want “any new ticket” as the entry signal; ticket_updated will fire on creation too but is noisier than necessary. Cheap to add to agentTriggerSchema + a sibling forward path in Rails. Not a blocker for agent #2.
• ⏳ Webhook coalescing. /webhooks/rulebase fans out a run per matched agent per event. agent_alert_tags protects the Slack write side, not the run cost. Build the (conversation_id, agent_id) debounce once Braintrust traces from agent #2 in production show the actual fan-out rate.
• Salesforce tool. Needed for agent #6 (and for the back-office-agent-opportunities roadmap). Read-only, scoped to account + AM/AE/Growth Manager lookup at first. Per the default ownership rule the right shape is TS-native fetching credentials from a Rails internal endpoint — but that credential-fetch endpoint doesn’t exist yet (the codebase currently uses per-tool Rails proxies). For agent #6, the pragmatic path is: ship a Salesforce Rails proxy mirroring the sendSlackNotification shape now, then migrate to TS-native when the credential endpoint lands.
• Rho event source for transaction-failure emails. Either: query Rails for the corresponding outbound notification record, or have Rho’s automated-failure system stamp a Zendesk tag we can match on. Pick whichever is already there.
• Case abstraction (BID / application grouped). Per Olivia’s notes on rows 5, 6, 7, 8, 10. This is a query-time concept (group q_conversations by business_id or application_id over a window) — not a new model. Decide the join key with Rho before agent #2 ships, since #2’s “3+ agents responded” / “reassigned 3+ times” thresholds change meaning if applied at the case vs ticket level.

Evaluating agent behavior

Once Phase 1 agents are running, we’ll need a way to score whether they’re producing correct, well-formed outputs over time — both for steady-state QA and for safely iterating on prompts and the system-prompt blocks injected by buildToolDocumentation.

Per-step transcripts already exist via Braintrust: the agent loop runs through wrapAISDK(ai) (see lib/braintrust.ts), so every model call, tool call, tool result, and usage delta is auto-traced. We deliberately did not ship a parallel agent_run_steps table in Postgres — Braintrust is the durable transcript store. See Agent run observability in the architecture plan for the rationale.

The reason to defer wiring autoevals isn’t infrastructure — it’s that there’s nothing to evaluate yet. Agent #1 hasn’t produced a real output. Building eval infra without a dataset is just plumbing. The fastest feedback loop right now is “trigger from chat → watch the live <AgentRunStatus> component → read the digest in the run output → edit instructions in the UI → trigger again.” Once the prompt is trusted, the 9am-ET schedule takes over and “ask Sara if it matches what she’d flag manually” becomes the steady-state validation. A formal eval harness adds latency to that loop without shortening it.

Sequencing

1. Run Phase 1 agents 1–3 in production for ≥1 week. Gather real Braintrust traces — at least ~50 runs across the digest agent (#1), the ownership watcher (#2), and one other. Real data only; no synthetic seeding.
2. Cut a frozen dataset from those traces. ~30 inputs per agent is enough to start. Manual labelling: for each input, write the expected digest content / expected alert / expected no-alert. Sara reviews #1’s labels; we review #2 and #3.
3. Wire Braintrust autoevals against the frozen dataset. One project per agent (or one project with per-agent scorers). Combination of:
   • Deterministic scorers for formatting (one message per run, all configured sections present, oldest-first ordering, mrkdwn / HTML well-formed, links resolve, no raw IDs where a URL exists).
   • LLM-as-judge scorer for content fidelity (“does this digest correctly identify tickets matching the matrix heuristic?”). Use the matrix row’s threshold language as the rubric.
   • Regression scorer that re-runs the dataset on every prompt change and surfaces deltas.
4. Wire Braintrust into the iteration loop.
   • System-prompt edits (buildToolDocumentation in lib/managed-agent.ts) affect every agent. Run the full Braintrust eval as a CI step on these PRs and require the score not to regress.
   • Per-agent instruction edits happen in the Rulebase UI against rows in api2.agents — not in code. For these, run Braintrust manually before saving, or auto-replay on save (UX TBD).

Cheaper adjacent move (already shipped)

Snapshot tests on buildToolDocumentation itself — one fixture per relevant tool combination, assert against a .snap file. Not Braintrust, just Vitest. Catches regressions in the system-prompt builder when we extend it again. Lives in rulebase-web/rulebase-api2/src/lib/managed-agent.test.ts; runs in CI via the test-api2 job in .github/workflows/web.yml.

Open questions / dependencies

Carried forward from the matrix’s “Needs Clarification” / “On Hold” rows so they don’t get lost:

1. Row 2 — what counts as “compliance freeze” and “no acknowledgment sent”? Blocks agent #9.
2. Row 3 — does “stalled at same stage” mean the client is actively reaching out and going back-and-forth for 7 days, or just no internal movement? Blocks agent #10. Needs Derek + Onboarding alignment.
3. Row 9 — depends on RAP. Roll into agent #4 once RAP exposes the second-ticket-same-BID signal.
4. Row 10 — how does Rulebase determine repeat contact across tickets technically? Blocks agent #8. We have org-level grouping; “same issue / same transaction” is the harder part. Likely an embedding similarity check over recent tickets, but confirm scope first.
5. Rows 5, 6, 7, 8 — case definition (BID/application grouped vs individual ticket). Affects threshold accuracy on agents #2 and (later) #4/#8. Resolve before agent #2’s prompt is written.
6. Pricing / volume. Many of these agents will fire many times per day per org. Confirm the consumption-pricing meter (AgentRun + UsageEvent from the architecture plan) is in place before turning more than 2-3 of these on for Rho production.

Not doing

• One agent per row. Explicit anti-pattern — see “mapping principle” above.
• The matrix’s “Escalation Rule” column. “Escalate to Sara if no owner assigned within 1 BD of flag” and friends are out of scope for v1; revisit after the basic alerts are landing well. The tags text[] shape doesn’t carry per-tag timestamps, so re-introducing escalation rules will require either upgrading tags to tags jsonb or adding the dedicated conversation_alert_flags table — see “Where the state lives” for the upgrade path.
• A dedicated api2.conversation_alert_flags table. Earlier draft of this plan. Replaced by adding a tags text[] column to the existing api2.conversations mirror — same dedup behavior with one column instead of one new table. We’d graduate to the dedicated table only if typed analytics or per-tag timestamps become a hard requirement.
• A jsonb rulebase_agent_flags column on Rails conversations (an even earlier draft). Adds Rails surface area inconsistent with the default ownership rule.
• A new TS-native Slack tool in api2. Desirable per the api2-default rule, but not a Phase 1 prerequisite. The existing sendSlackNotification Rails proxy in lib/rails-client.ts is fine for both alert_conversation and slack-send-<channel> to call into. Migrate to TS-native as a separate piece of work once the generic /internal/organizations/:id/integration_credentials/:type endpoint exists (see the architecture plan).
• Separate mark_conversation_tag tool. Folded into alert_conversation so the agent cannot post-without-tagging. Tool shape over prompt enforcement.
• Specialized detector tools (find_stale_tickets, etc.) per heuristic. Defeats the genericity of the agents primitive; we’d be writing new code per heuristic per customer. Use query_data + the memory tool instead.
• Promoting matrix heuristics to first-class columns on q_conversations. Ossifies the schema around one customer’s interpretation; doesn’t generalize. Let each agent’s memory hold its own SQL.
• A typed query catalog with draft / approved status workflow. Too much ceremony for what’s actually just “the agent should remember what worked”. Plain string memory with humans editing entries directly is the audit and override surface.
• Sentinel pre-seeding agent memory at creation time. Agents start with empty memory and discover queries over the first few runs; humans review and edit in the Rulebase UI.
• A separate sentiment-scoring microservice. Use the LLM in agent #3 directly with the last N customer replies in context. Revisit only if precision is bad.
• De-escalation playbooks / response drafting. The matrix is alerting only. Composing replies on behalf of agents is a separate roadmap item; do not entangle.
• Building the case abstraction as a new Rails model. Express it as a SQL grouping over existing data first; only formalize if multiple agents start needing the same join.
• Surfacing flags inside Zendesk tickets. Rho didn’t ask for it; their workflow is Slack-alert → click through to Zendesk. We can revisit if the request comes up.
• Wiring Braintrust autoevals in Phase 1. Deferred — see Evaluating agent behavior for the sequencing argument. Short version: nothing to evaluate yet. Per-step transcripts already exist (Braintrust auto-traces via wrapAISDK), so the gating step is just real production runs. Phase 1 ships → ≥1 week of runs accumulates → cut a frozen dataset → then wire scorers.
• Building the skills primitive in Phase 1. Deferred — see Skills (deferred) for the trigger conditions. The current buildToolDocumentation snapshot tests are the right level of investment until we want a fourth prompt block, a customer asks to customize, or Braintrust scorers need stable targets. Skills get a separate DB table from agent_memory_entries because they need versioning, creator metadata, and source tracking that memory does not — but the agent sees them through one unified knowledge tool surface.
• Sharing one DB table for memory and skills. Considered — the surfaces look identical. Rejected because their lifecycles diverge (memory is high-churn agent-self-curated; skills are versioned human-curated). Sharing forces every memory write through versioning code and every skill row to handle a null agent_id. Same agent interface, different storage.
• Composition / merge between org skills and system skills. Punted — current bias is “org skill of the same name shadows the system skill, with a conflict warning in the admin UI.” We’ll only build merge semantics if a customer explicitly asks; until then, “override” is the only customization path.
• Persisting per-step transcripts in Postgres (agent_run_steps). Braintrust is the transcript store; duplicating it in our DB would be bloated and coupled to AI SDK shape with no current consumer. See Agent run observability.