Turning Moshi’s human handoffs into its product roadmap

A production evidence review of Qonto EU

Prepared by Rulebase for the Qonto and Moshi teams · 24 August 2026

Executive summary

Moshi already operates at unusual scale: it appeared in 315,749 Qonto EU conversations over 30 days. The clearest path to greater autonomy is visible in the conversations it hands to people. Human follow-ups can reveal the data Moshi could not access, the action it could not take, the guidance it handled differently, or the source state it could not trust.

Rulebase screened 50,991 unique, successfully summarized Moshi-to-human handoffs. The automated screen flagged 30,862 (60.5%) for further review under an apparent data, action, guidance, or source-state gap. A blinded, stratified review of 400 handoffs produced the more defensible estimate: 55.6% showed a transcript-visible gap, and 50.1% looked plausibly improvable through a product change. Neither figure is a count of preventable tickets.

The first area Rulebase can help with is turning that production evidence into Moshi’s next releases. The clearest candidate for validation is onboarding and account-state visibility: the automated clustering found 635 handoffs concerning registration or enrollment status and another 385 concerning document approval or activation. Other candidates include transfer-state reliability, eligible action workflows, and targeted knowledge corrections.

Rulebase continuously detects handoffs, screens for candidate gaps, consolidates recurring issues, validates their precision and eligibility, and measures each selected release. Qonto confirms policy and ships selected changes. The primary outcome is manual tickets avoided, with verified resolution and safety as guardrails.

This work can start with the conversation history already available. It does not require a complete Moshi simulator.

The Rulebase improvement loop

Rulebase turns production handoffs into a recurring product-development loop:

Production handoff → candidate gap surfaced → cluster validated → release specified → change shipped → manual tickets avoided measured → residual handoffs become the next backlog

1. Detect Moshi-to-human handoffs and compare what Moshi could do with what the human did next.
2. Screen for an apparent data, knowledge, action, or source-state gap.
3. Consolidate recurring issues, validate them, and rank candidate clusters by eligible volume, customer risk, and implementation effort.
4. Produce a capability card and a regression set drawn from the conversations that exposed the gap.
5. Measure the selected change in a controlled rollout.
6. Feed the remaining failures into the next ranked backlog.

Knowledge and deterministic lookup changes can be replayed before release. Transactional changes add production measurement once terminal action states and cohort metadata are available.

What the automated screen found

The automated screen assigned each successfully summarized handoff to one forced primary label. These are investigation hypotheses, not confirmed root causes or release decisions.

The 60.5% is the screen-positive rate. It is not the share of addressable or preventable handoffs.

What blinded review supports

Rulebase reviewed a disproportionately stratified sample of 400 handoffs and reweighted the findings to the fixed 50,991-handoff snapshot.

A product change looked capable of plausibly improving 50.1% of handoffs, with a 45.6–54.6% sampling interval. This is not an estimate of tickets Moshi could safely resolve: policy, permissions, feasibility, and expected completion still require capability-specific validation.

The screen is useful for discovery but not exclusion. Review confirmed a visible gap in 77.1% of screen-positive handoffs, while finding a visible gap in 22.5% of the handoffs screened as none.

The capability backlog

Repeated differences between Moshi and the human handling that followed become capability cards. Each card makes the opportunity concrete:

• unique 30- and 60-day handoff volume;
• the data, action, or knowledge the human used;
• classifier precision, representative examples, and counterexamples;
• eligible users, states, permissions, and exceptions;
• the proposed release path and acceptance criteria;
• a production-derived regression set; and
• expected and observed manual tickets avoided.

This is a recurring product output rather than a one-off analysis. New handoffs update the volume, surface new variants, and show whether a released capability removed the intended work or merely changed the shape of the remaining failures.

1. Missing data: start with onboarding and account state

The automated screen assigned 20,448 handoffs, or 40.1% of the snapshot, to an apparent missing-data category. Blinded review reduced the calibrated directional estimate to 28.1%, or roughly 14,309 handoffs. The largest raw screen-generated clusters included:

Onboarding and account state is the strongest first read capability. In ticket #6499558 , Moshi said it had no way to check a micro-enterprise registration directly, then inferred from ticket history that it had been approved. The customer requested a human. In ticket #6522916 , a human could see that registration had completed successfully and continue investigating the remaining problem.

Together, these conversations show a missing customer-specific state rather than a generic knowledge gap. A customer-safe read capability could return:

• the current stage;
• missing or rejected documents;
• a customer-safe blocking reason;
• whether the next action belongs to Qonto or the customer;
• the next permissible action; and
• the status timestamp.

The capability card consolidates any overlap between the two onboarding clusters, identifies the fields humans actually used, applies Qonto’s permission rules, and estimates the manual tickets this read capability could avoid.

For the onboarding proposal, the first card would make the build decision explicit:

This is the practical output of the capability backlog: not “improve onboarding,” but a quantified candidate, a proposed interface, its constraints, and the measurement that determines whether it worked.

2. Incorrect or stale state: make action reliability a release gate

The automated screen flagged 1,006 apparent bot–human state conflicts. The calibrated directional estimate is larger—4.4%, or roughly 2,260 handoffs—because reviewers also found state conflicts in other screen labels. The volume is smaller than missing data, but the risk is higher: a missing answer causes a handoff; a confident wrong answer can cause financial harm.

In ticket #6535381 , the customer asked Moshi to create a beneficiary and send €174,600. Moshi emitted a beneficiary-creation action and then said the transfer “has been executed.” Thirteen minutes later, a human could not find the transfer and explained that the customer still needed to authorize it from the account.

The available trace does not show whether Moshi created a prefilled confirmation, submitted a request, or misread an intermediate result. It does show that the customer-facing claim was stronger than the state the human could verify.

A second production case shows the same risk in status interpretation. In ticket #6513786 , Moshi described a €480 instant transfer as successfully executed. The human later explained that the instant transfer had been rejected and a subsequent standard transfer was still processing. The cause could be stale data, the wrong transfer record, or an incorrect mapping between payment states.

Financial actions need an explicit lifecycle:

Proposed → awaiting confirmation → confirmed → submitted → succeeded or failed

The transfer reliability work follows the complete sequence from selected tool and valid inputs through returned state, customer confirmation, submission, terminal backend result, and customer-facing claim. It isolates whether a mismatch comes from stale state, the wrong record, payment-rail semantics, or wording. Greater transactional autonomy should follow—not precede—reliable terminal-state observability.

3. Missing actions: automate only the eligible path

The automated screen assigned 3,991 handoffs to apparent missing tools or actions. This was the screen’s most reliable subtype: blinded review confirmed the exact label in 88.3% of its sample and produced a calibrated directional estimate of 13.7%, or roughly 6,979 handoffs across the full snapshot. The strongest recent candidates were garnishment settlement and account closure.

The garnishment data shows why a cluster label is not enough. One cluster contained 219 handoffs apparently involving settlement or account-block release. In ticket #6801272 , however, the human explained that the standard settlement runs automatically twice per day. In ticket #6800561 , a customer requested accelerated settlement and a human later initiated a €4,235.59 transfer.

The backlog therefore separates:

1. standard automatic settlement;
2. accelerated or exceptional settlement; and
3. blocked or failed settlement.

Only the confirmed exceptional path becomes an action proposal, with eligibility checks, customer confirmation, an audit trail, and explicit success or failure. Account closure follows the same process: near-duplicate clusters of 84–150 handoffs are consolidated, then filtered for balances, pending transactions, user authority, and regulatory constraints.

These human responses are evidence of divergent handling, not authoritative policy. Qonto confirms the standard and exceptional paths; the capability backlog then converts the confirmed path into decision logic, eligibility rules, and regression cases.

4. Knowledge mismatches: keep fast fixes out of the backend queue

The automated screen flagged 5,417 apparent guidance divergences. Blinded review produced a calibrated directional estimate of 9.4%, or roughly 4,788 handoffs. These corrections can move independently of new tools once Qonto confirms the authoritative rule.

In ticket #6653241 , Moshi said no token was needed to move electronic invoicing while closing an account. A human said that an Approved Platform migration token was required and supplied it. Another recent conversation received generic transfer guidance before a human explained that the new provider had to initiate and finalize the migration.

Once Qonto confirms the authoritative rule, this becomes a knowledge or routing correction paired with a regression set. Garnishment guidance is another candidate: Moshi needs to distinguish the standard automatic process from accelerated or exceptional handling.

Other recent clusters included incorrect guidance during an active transfer or payment incident and generic invoice guidance used in place of the migration procedure. Keeping these fixes in a separate release lane prevents a source or decision-logic correction from waiting behind a new endpoint.

The calibrated mix matters for the roadmap. Customer-specific data, missing actions, and source-state reliability account for roughly 83% of the estimated visible gaps; knowledge or guidance mismatches account for roughly 17%. Knowledge updates can produce fast wins, but materially higher AI handling will depend primarily on new reads, actions, and reliable state contracts.

5. Regression cases that static knowledge tests miss

Two production examples show why improving an agent requires more than checking final answers against help articles.

State continuity. In ticket #6553345 , Moshi found two invoices with the same number, client, and amount and correctly asked the customer which one they meant. After an upload error interrupted the flow, Moshi restarted and selected the wrong invoice. The regression case goes beyond “find an invoice”; it must preserve a resolved ambiguity across interruption and resume.

Grounding before action. In ticket #6484022 , the transcript represented Moshi as finalizing a €48,480 invoice with the line item “Summary with flex,” apparently copied from a PDF extraction summary rather than the source document. The regression case must verify that low-confidence extracted fields are grounded, shown to the customer, and confirmed before finalization.

Both cases begin with behavior that appears successful at the message level. The failure only becomes visible when the test follows state across turns or connects the final claim to the underlying business action. These are the kinds of production-derived cases that a static answer benchmark will miss.

Moshi already has strong foundations

The review also found behavior worth preserving:

• Moshi correctly disambiguated duplicate invoice records before the interrupted restart in #6553345 .
• It used recent contact history to recognize a repeat issue in #6553602 .
• It declined a request it could not answer reliably and routed to support in #6499864 .

Moshi already selects tools, uses history, disambiguates, and declines. The opportunity is to make these behaviors reliable across state changes and extend them into the high-volume gaps humans are filling today.

Guardrails for greater autonomy

The goal is not to minimize handoffs at any cost. Required escalation, permission boundaries, and verified backend state remain part of the product definition.

Sensitive intents. Account takeover, seizures, and account security had the highest observed routing rates. That may be correct. These issues should be evaluated for safe escalation, correct triage, prohibited advice, authorization, and unnecessary disclosure rather than scored against a blanket containment target.

Authorization and data minimization. Customer-specific access should be tested against the authenticated user, organization, and role. A new read capability should expose the smallest customer-safe projection needed to answer the request, not the complete back-office object.

Financial claims. Timing, settlement, action completion, and statements that there is “no risk” should be grounded in a current source state or appropriately qualified. The transfer and invoice examples show why action safety cannot be inferred from fluent prose.

Action controls. Every transactional capability needs customer confirmation, idempotency, an audit trail, and a terminal backend result. The customer-facing claim must match that result.

Safe escalation is a successful outcome when policy requires a person. The KPI below separates those cases from preventable manual work.

The KPI: manual tickets avoided

The headline outcome is the reduction in conversations that require a person.

For each controlled release, Rulebase reports:

Human-handled conversations per 1,000 eligible Moshi conversations

Manual tickets avoided is the difference between expected human-handled volume at the baseline rate and observed volume after release. The comparison stays within the same issue family and accounts for country, language, channel, customer tier, and Moshi version.

Eligibility is defined by issue family with Qonto’s policy input. Tests, spam, duplicate contacts, unsupported channels, required escalations, and human requests that Qonto chooses to honor are excluded or reported separately.

Verified resolution is the guardrail, not the headline. A release should not reduce human handling by increasing abandonment, wrong answers, unsafe containment, or failed actions. The supporting measures are:

• correct and complete resolution;
• terminal backend success for actions;
• incorrect or stale data;
• failed or duplicated actions;
• safe escalation in sensitive intents;
• same-topic recontact at 72 hours and seven days; and
• CSAT and complaints.

For an informational workflow, verified resolution requires a correct and complete answer, no human intervention, and no evidence of same-topic recontact. For an action workflow, it also requires terminal backend success. Recontact is supporting evidence rather than proof: no return can mean abandonment, while a return can be a legitimate follow-up.

The current closure field cannot establish this outcome. Most coded closures are timeouts, which show that a customer stopped replying—not that the issue was solved.

First 90 days

Month 1: identify and validate

Start with the conversations already available. Consolidate the leading clusters, validate precision and eligibility, establish issue-level baselines, and produce the first ranked capability backlog.

Output: candidate capability cards for onboarding visibility, transfer reliability, and the leading eligible action or knowledge gap, each with cluster-specific validation requirements.

The initial validation separates required escalations from preventable handoffs, merges duplicate topics, and checks representative cases and counterexamples. Qonto’s input is concentrated on ambiguous policy, permissions, and automation eligibility rather than raw-ticket review.

Month 2: ship and measure

Qonto selects the first changes to implement. Each release receives acceptance criteria, a production-derived regression set, and controlled-cohort measurement.

Output: two release-ready proposals and the first before-and-after results.

The likely first pair is onboarding-state visibility and either the highest-volume eligible action or transfer reliability, depending on what the state investigation finds.

Month 3: prove impact and repeat

Measure manual tickets avoided, verify that resolution and safety did not regress, and analyze the remaining handoffs.

Output: an impact report for each release and the next ranked capability backlog.

Each capability is expanded, revised, or withdrawn based on its issue-level result. The next opportunities are ranked by observed eligible volume, implementation effort, and risk.

Version metadata, terminal action states, and explicit cohort identifiers improve attribution as the loop matures. They are not prerequisites for the initial backlog.

The result is a recurring system:

Production handoff → candidate gap surfaced → cluster and eligibility validated → capability shipped → manual tickets avoided measured → residual failures become the next backlog

Appendix A: supporting performance data

Scale and tool use

A production query for the 60 days through 24 August found 637,605 Moshi-involved conversations. In a sample of 2,034 conversations, roughly one third of Moshi’s messages referred to the customer’s own account, such as a transaction, invoice, subscription, or KYC file. In a separate sample of 598 customers, 46% had contacted Qonto in the preceding 30 days. The earlier contact may have concerned another issue, but the result reinforces the value of cross-ticket outcome measurement.

Recorded tool invocations over the 60 days ending 4 August were:

These counts show invocation, not success. They demonstrate why evaluating Moshi only against public help articles would miss much of the product.

Handoff speed

In 1,500 recent Moshi conversations that eventually received a human reply, the median time from the customer’s first message to the first non-bot reply was 16 minutes; the 90th percentile was 2 hours 36 minutes. This was the lowest observed median in Rulebase’s directional comparison set.

The comparison is not controlled for channel, opening hours, staffing, geography, or routing policy. It also starts at the customer’s first message rather than the exact handoff and excludes conversations still waiting for a person. It is therefore an operational benchmark, not a pure bot-quality ranking.

Available closure outcomes

Of 315,749 Moshi conversations during the 30 days ending 4 August, 112,146 (35.5%) had one of the two available closure reasons:

A timeout can mean resolution, abandonment, a later action, or a return through another channel. Rulebase therefore keeps silence, containment, and resolution separate.

Routing by intent

The table below covers the 60 days ending 4 August. Rates use only conversations with an available outcome code.

The spread reflects different workflows and routing policies as well as bot performance. High routing can be correct for sensitive intents. Volume and rate also answer different questions: existing-company onboarding produced the largest recorded escalation count in this outcome-coded set, while account takeover had the highest rate. The unknown category contains 8.1% of the rows and 13.2% of recorded escalations in this table, making it a useful target for further decomposition.

Qonto’s “Reason for request” field provides a second view. The values below total 99,620 records: 88.8% of outcome-coded conversations and 31.5% of all Moshi conversations during the 30 days ending 4 August.

Observed rates were close across client tiers—6.2% to 7.8%—but the topic, country, channel, risk, and sample-size mix differ. A like-for-like tier comparison requires Qonto’s intended routing policy.

Appendix B: methodology and measurement requirements

Population and windows

Rulebase treated a conversation as Moshi-involved when the Qonto EU conversation-agent relationship identified Moshi. The 315,749-conversation scale and closure figures cover the 30 days ending 4 August. The intent and tool tables cover the 60 days ending 4 August. The fixed handoff screen and calibration snapshot cover conversations started from 25 June 2026 13:52:06 UTC up to, but not including, 24 August 2026 13:52:06 UTC, with summaries created by that endpoint. The current scale query covers the 60 days through 24 August.

Sahar described an operating volume of roughly 5,000–7,000 tickets per day. Rulebase’s higher conversation count may use a broader channel or lifecycle definition. Any staffing or savings forecast will align to Qonto’s weekly operating definition.

Capability-gap screen

The automated handoff screen compared Moshi’s last substantive contribution before a visible bot-to-human transition with the human work that followed. It forced one primary label from the subject and a bounded window of up to 12 bot and 12 human turns; customer turns were not included in that first pass.

The screen does not observe every back-office action, can split similar issues across clusters, and has no calibrated confidence score for individual rows. The 50,991 rows are successfully summarized handoffs in the fixed snapshot, not every production handoff.

Blinded calibration

Rulebase selected 400 handoffs disproportionately across the screen’s five labels and reweighted the results to the 50,991-handoff population. Reviewers saw customer, bot, and human turns but not the screen label, generated title, rationale, or another reviewer’s decision. The review used a strict causality rule: later human work was not counted as the cause of a handoff when the customer had requested a person before disclosing the issue.

Forty-six items received three independent reviews. A hierarchical two-of-three majority resolved the broad gap decision before the subtype; only three items required a fresh blind tie-break. All population, sample, assignment, duplicate, schema, consistency, missing-label, and adjudication checks passed.

The reviewers were independent model-assisted analyst passes, not Qonto subject-matter experts or human ground truth. The reported intervals quantify sampling error under the stratified design; they do not include correlated reviewer error, omitted back-office actions, or policy uncertainty. Exact subtypes are therefore directional even where the broad-gap estimate is useful.

Human handling is evidence of what happened, not automatically authoritative Qonto policy. Policy, permission, and automation eligibility are confirmed before a capability is recommended.

Estimating eligible volume and impact

Raw cluster volume is an upper bound. Before implementation, a capability’s planning estimate is:

Deduplicated cluster volume × cluster-specific precision × policy-eligible share

Near-duplicate clusters are consolidated and conversations deduplicated before the estimate. The actual impact is the experimentally observed reduction in human handling within that eligible cohort, with resolution and safety held as guardrails. Broad taxonomy calibration does not replace cluster-specific precision or eligibility review.

Signals that improve impact measurement

The existing Zendesk and Rulebase records are enough to produce the first capability backlog. The following signals improve attribution and verified outcome measurement later:

• Moshi, prompt, knowledge, tool, schema, and feature-cohort versions;
• action correlation IDs, customer confirmation, submission, terminal result, error, source status, and timestamp;
• the expected routing policy by issue and customer state;
• pseudonymous cross-ticket identity and same-topic linkage; and
• Qonto’s exact weekly definition of a manually handled ticket.

Production traces can deepen root-cause analysis, but they are not required to start.

Interpretation

Missing metadata remains missing; Rulebase does not impute it as resolved or escalated. Handoff timing is conditional on a later visible human reply. Other-bot comparisons are directional because operating conditions differ.

This report does not claim that every screened gap is a Moshi defect, every human action should be automated, a timeout is a resolution, or raw handoff volume equals tickets saved. It identifies high-volume production evidence from which to validate, build, and measure Moshi’s next capabilities.