Qonto Evaluation Feedback Analysis — March 2026

Methodology

Extracting ticket data for analysis

Rails console script to pull transcripts, AI evaluations, check results, and reviewer feedback for a set of conversation IDs. Run in the production Rails console.

# Set tenant to Qonto
org = Organization.find_by!(name: "Qonto")
ActsAsTenant.current_tenant = org
 
CONVERSATION_IDS = %w[
  conversation_XXXXX
]
 
def short_transcript(conversation, max_parts: 40)
  parts = conversation.conversation_parts
    .where(type: %w[chat email note call])
    .includes(:author_employee, :author_client,
              conversation_part_call_detail: :conversation_part_call_detail_transcript_messages)
    .order(:external_created_at)
 
  lines = []
  parts.limit(max_parts).each do |part|
    author = part.author_type.upcase
    name = part.author_name || part.author_employee&.name || part.author_client&.name || "Unknown"
    timestamp = part.external_created_at&.strftime("%Y-%m-%d %H:%M") || "?"
    channel = part.type
 
    if part.type_call? && part.conversation_part_call_detail&.conversation_part_call_detail_transcript_messages&.any?
      lines << "[#{timestamp}] [#{channel}] #{author} (#{name}): [CALL START]"
      part.conversation_part_call_detail.conversation_part_call_detail_transcript_messages.each do |msg|
        speaker = msg.author_type&.upcase || "?"
        lines << "    #{speaker}: #{(msg.text_english.presence || msg.text).to_s.truncate(300)}"
      end
      lines << "  [CALL END]"
    else
      body = (part.body_english.presence || part.body.presence ||
              part.html_body.to_s.gsub(/<[^>]+>/, ' ').squish).to_s.gsub(/\s+/, " ").truncate(500)
      next if body.blank?
      lines << "[#{timestamp}] [#{channel}] #{author} (#{name}): #{body}"
    end
 
    lines << "  [IMAGE: #{part.image_caption}]" if part.image_caption.present?
  end
 
  remaining = parts.count - max_parts
  lines << "... (#{remaining} more parts)" if remaining > 0
  lines.empty? ? "[No transcript parts found]" : lines.join("\n")
end
 
CONVERSATION_IDS.each do |prefix_id|
  conv = Conversation.find_by_prefix_id!(prefix_id)
  puts "=" * 120
  puts "TICKET: #{conv.external_id} | #{prefix_id}"
  puts "=" * 120
 
  puts "\n--- TRANSCRIPT ---"
  puts short_transcript(conv)
 
  ai_eval = conv.qa_agent_evaluations.effective.where(source: :ai).order(created_at: :desc).first
  next puts("\n[No AI evaluation found]\n") unless ai_eval
 
  puts "\n--- AI EVALUATION (score: #{ai_eval.overall_score}) ---"
  ai_eval.scorecard_results.includes(
    criterion_results: [:scorecard_criterion,
                        { check_results: :scorecard_criterion_check },
                        :evaluation_scorecard_criterion_result_comments]
  ).each do |sr|
    sr.criterion_results.each do |cr|
      criterion_name = cr.scorecard_criterion&.description || "Unknown"
      puts "\n  CRITERION: #{criterion_name} | Score: #{cr.score}/#{cr.max_score}"
      puts "  REASON: #{cr.reason}" if cr.reason.present?
 
      cr.check_results.each do |check|
        check_name = check.scorecard_criterion_check&.description&.truncate(80) || "Check"
        puts "    CHECK: #{check_name} | Result: #{check.result}"
        puts "    SUMMARY: #{check.summary}" if check.summary.present?
      end
 
      cr.evaluation_scorecard_criterion_result_comments.each do |comment|
        puts "    >>> REVIEWER FEEDBACK (#{comment.comment_type}): #{comment.content}"
        puts "    >>> REVIEWER: #{comment.author&.full_name}"
        puts "    >>> EXPANDED: #{comment.expanded_comment}" if comment.expanded_comment.present?
      end
    end
  end
 
  puts "\n"
end

Extracting scorecard definitions

Rails console script to dump scorecard structure including score options, justification options, and dependency rules.

org = Organization.find_by!(name: "Qonto")
ActsAsTenant.current_tenant = org
 
scorecards = Scorecard.all
scorecards.each do |sc|
  puts "SCORECARD: #{sc.description} (#{sc.prefix_id})"
  puts "=" * 120
 
  all_criteria = sc.scorecard_criteria.where(parent_id: nil).order(:position) +
                 sc.scorecard_criteria.where.not(parent_id: nil).order(:position)
 
  all_criteria.each do |criterion|
    indent = criterion.parent_id ? "    " : "  "
    label = criterion.parent_id ? "CHILD CRITERION" : "CRITERION"
    puts "\n#{indent}#{label}: #{criterion.description} (weight: #{criterion.weight})"
 
    criterion.score_options.includes(:justification_options).order(:value).each do |opt|
      puts "#{indent}  SCORE OPTION: #{opt.value} — #{opt.label}"
      opt.justification_options.each do |j|
        dep = j.depends_on_criterion_id.present? ? " [DEPENDS ON: #{j.depends_on_criterion_id}]" : ""
        puts "#{indent}    JUSTIFICATION: #{j.content}#{dep} (type: #{j.option_type})"
      end
    end
  end
  puts "\n"
end

Key model relationships

Conversation
  → QAEvaluation (container, per evaluator)
    → QAAgentEvaluation (per agent, .effective scope = non-overridden)
      → QAAgentEvaluationScorecardResult (per scorecard)
        → QAAgentEvaluationScorecardCriterionResult (per criterion)
          → QAAgentEvaluationScorecardCriterionCheckResult (per check)
          → EvaluationScorecardCriterionResultComment (reviewer thumbs up/down)

ScorecardCriterion
  → ScorecardCriterionScoreOption (Pass/Partial/Fail/Auto-fail with values)
    → ScorecardCriterionScoreOptionJustificationOption (boundary definitions)
  → ScorecardCriterionCheck (individual checks within a criterion)

Note: after human feedback, the criterion-level scores may be updated but the check-level results remain from the original AI evaluation. To see the original AI assessment, look at the check results, not the criterion scores.

Summary

Negative Feedback by Criterion

By Reviewer

Daily Breakdown

Root Cause Analysis of Negative Feedback

Six recurring failure patterns account for nearly all 63 negative comments. Many comments exhibit multiple patterns.

1. Contextual Misread of the Conversation (~15 comments) core issue

The AI misreads what actually happened in the conversation — what the customer asked for, what the agent did, or how the interaction unfolded. This is the “contextual gap” the customer is describing.

• Misunderstanding the customer’s request:
  • AI says agent understood when they didn’t — customer gave wrong merchant name (#5002793 )
  • Agent thought customer had a ready card when they meant pending cards needing PIN (#4868415 )
  • Agent missed the specific company name the customer called about (#4801082 )
  • Agent misread “dated within the last year” as “dated today” (#4924688 )
• Misunderstanding the agent’s actions:
  • Agent focused on wrong topic entirely — customer couldn’t validate from phone, not a payment issue (#4889517 )
  • Agent treated a commercial dispute as fraud (#4924549 )
  • AI missed that agent’s first written solution didn’t address the issue, only the phone call did (#4868415 )
  • AI gave pass but agent did exactly what was penalized for (#4876696 )
• Misreading conversation flow:
  • Agent asked a legitimate clarifying question the AI called unnecessary (#4829596 )
  • AI didn’t notice agent misread information in a screenshot (#4829596 )

2. Dependency Rule Misapplication (~12 comments) core issue

Qonto’s scorecard has a “dependency rule” where certain criteria (e.g., Be Efficient) are affected by related criteria (e.g., Solve the Request). The AI frequently gets this wrong.

• Applies when it shouldn’t:
  • AI penalizes via dependency when linked criterion was actually passed (#4974065 , #4936594 , #4869453 )
• Doesn’t apply when it should:
  • AI misses that a failing linked criterion should cascade (#4884019 , #4924549 )
• Contradicts itself:
  • Dependency rule conflicts with the AI’s own assessment of the linked criterion (#4937843 )

3. Severity Calibration Errors (~15 comments) systematic

The AI miscalibrates between pass / partial / fail / auto-fail.

• Too strict (auto-fail instead of partial/fail):
  • AI gives 0 when partial was warranted (#4974065 , #4916085 , #4904942 )
• Too lenient (pass instead of fail):
  • AI gives full marks when agent clearly missed something (#5002793 , #4949412 , #4886614 , #4883340 , #4868415 , #4801082 )
• Right ballpark, wrong grade:
  • AI reasoning is partially correct but final score doesn’t match (#4974065 , #4922785 , #4884019 )

4. Correct Score, Wrong Reasoning (~15 comments) quality

The AI arrives at the right score but for the wrong reasons. Reviewers give thumbs down because the reasoning doesn’t demonstrate genuine understanding.

• Pass with wrong justification:
  • Score is correct but AI mischaracterizes what happened (#4924688 , #4958406 , #4884232 , #4904360 , #4926006 , #4873185 )
• Implications: Wrong reasoning erodes trust even when scores are accurate, and makes it harder for agents to learn from evaluations

5. Criterion Scope Confusion (~7 comments) systematic

The AI uses arguments that don’t belong to the criterion being evaluated.

• Tone of Voice: AI includes efficiency or content arguments under tone assessment (#4889517 , #4949412 )
• Be Efficient: AI penalizes for not seeking confirmation when agent’s “merci” already served that purpose (#4873185 )
• Create a Human Bond: AI counts an irrelevant FAQ link as “education” (#4886614 )

6. Edge Cases / Hard Rules (~5 comments) rules

The AI doesn’t handle well-defined edge cases that have clear-cut answers.

• Voicemail = no call: All three say the same thing — reaching voicemail means call recording consent doesn’t apply. This should be a simple rule. (#4958870 , #4936594 , #4884232 )
• Context-dependent expectations: Churning customer has different efficiency expectations (#4869453 ); template rendering issues aren’t the agent’s fault (#4868415 )

Key Takeaways

• The “contextual gap” claim is valid. ~25% of negative feedback stems from the AI misreading what the customer asked or what the agent did. The AI often gives a surface-level reading that misses the real issue.
• The dependency rule is a major pain point. ~20% of negatives are about this single rule being misapplied. This could potentially be addressed with clearer rule encoding.
• Even when scores are right, wrong reasoning damages trust. ~25% of thumbs-down are on comments where the score was correct but the explanation was wrong. This suggests the AI sometimes gets lucky rather than genuinely understanding the conversation.
• Some edge cases are trivially fixable. The voicemail/call recording issue accounts for 3 comments and has a clear binary rule that could be hardcoded.
• 76% approval rate is a reasonable baseline but the 24% failure rate is concentrated in the hardest criteria (Understand, Solve, Be Efficient) — the ones that require deep contextual understanding of both the customer’s intent and the agent’s response.

Appendix: All Negative Feedback — March 2026

Fixes & Outcomes

Fix 1: Dependency rule now applies even when score is already the same shipped

Problem: When a check already had the same score as the dependency rule would assign (e.g., both are fail/0), the dependency rule was skipped entirely. This meant the dependency reasoning was never recorded, and the score was attributed to the AI’s original (often wrong) justification instead of the dependency rule.

Root cause: In apply_cross_criteria_to_check_result, the guard return if score_option.value >= existing_value skipped when values were equal. For example, if Be Efficient was already 0 (fail) and the dependency rule would also set it to 0 (fail from Solve), it was skipped.

Fix: Changed to return if score_option.value > existing_value — the dependency rule now only skips when it would make the score better (higher), not when it’s the same. This ensures the dependency justification and reasoning are always applied when the condition matches.

Impact: Addresses feedback comments like #4884019  and #4924549  where reviewers said “it’s a fail due to the dependency rule” but the dependency reason was missing from the evaluation.

File: rulebase-api/app/models/qa_agent_evaluation_scorecard_result.rb:910

Fix 2: Feasibility check for voicemail/call recording consent shipped

Problem: The AI auto-failed agents on the “Call Recording Consent” criterion when outbound calls reached voicemail. The agent can’t inform a voicemail recording about call recording — the preconditions for the check were never met, so the result should be “not_applicable”, not “failed”.

Root cause: The evaluation prompts had no concept of whether a required action was physically possible. If a check said “agent must inform the client about call recording” and a call existed, the AI would evaluate it regardless of whether a live person was on the other end.

Fix: Added a Feasibility Check as Core Principle 3 in the evaluation prompt (evaluate_scorecard_criterion_for_agent_tool.md). Before marking any check as failed, the AI must now verify that the required action was physically possible:

• Voicemail ≠ live call: If a call transcript contains only a voicemail greeting with no two-way dialogue, interaction-dependent checks (inform, consent, verify) are “not_applicable”
• Dropped/unanswered call ≠ connected call: Same treatment
• Unanswered message ≠ dialogue: Checks requiring back-and-forth are “not_applicable”

The principle was also embedded in the strict calibration prompt (evaluations/strict.md), updating the auto-fail threshold and borderline pass/fail dimensions to exclude physically impossible actions from evaluation.

Eval harness: Added a voicemail_not_live_call test case in scripts/eval_qa_evaluations.rb that asserts the call recording consent check returns not_applicable (not failed) for a conversation where both outbound calls reach voicemail. Scored via Braintrust with correct_outcome and check_result_correct scorers.

Impact: Directly addresses feedback comments #4958870 , #4936594 , and #4884232  where reviewers flagged that voicemail calls should not trigger call recording consent failures.

Files:

• rulebase-api/config/prompts/evaluate_scorecard_criterion_for_agent_tool.md (Core Principle 3)
• rulebase-api/config/prompts/evaluations/strict.md (Dimensions 1 & 2 updated)
• rulebase-api/scripts/eval_qa_evaluations.rb (eval test case)

Fix 3: KB queries must be customer-centric, not agent-centric shipped

Problem: When the AI evaluator used the knowledge base to verify procedures, it would ask questions that baked in the agent’s framing — e.g. “agent provides fraud dispute steps + card block — pass or partial?” This is a leading question: the correct procedure should emerge from the retrieved content, but the query already assumes the agent’s categorisation is correct.

Root cause: The prompt had no guidance on how to phrase KB queries. The evaluator naturally mirrored the agent’s language, which meant the KB retrieval was biased toward confirming the agent’s approach rather than independently verifying it.

Fix: Added a rule in the “Knowledge Base Lookup via Tool” section of evaluate_scorecard_criterion_for_agent_tool.md: KB queries must describe the customer’s situation and what they asked for, not the agent’s chosen procedure. Example: query “customer made a payment online and says it was a scam — what is the correct procedure?” rather than “agent provides fraud dispute steps + card block — pass or partial?”.

Eval harness: Added a fraud_vs_commercial_dispute test case in scripts/eval_qa_evaluations.rb. The check instructions are deliberately generic (“use the knowledge base to verify the correct procedure”) — the fraud vs commercial dispute distinction is only available via a stub KB, so the evaluator must query it and reason about which procedure applies. Based on real ticket #4924549 . Scored via Braintrust: correct_outcome: 100%, check_result_correct: 100%.

Impact: Directly addresses the procedure misclassification in #4924549  where the AI passed the agent for applying the fraud procedure to an authorized transaction (commercial dispute).

Files:

• rulebase-api/config/prompts/evaluate_scorecard_criterion_for_agent_tool.md (Knowledge Base Lookup via Tool section)
• rulebase-api/scripts/eval_qa_evaluations.rb (eval test case)

Fix 4: Don’t inherit agent’s categorisation when verifying procedures shipped

Problem: The AI evaluator would inherit the agent’s framing of the issue when cross-referencing procedures. In ticket #4924549 , the customer said “I made a payment and it was a scam” (authorized transaction = commercial dispute), but the agent directed them to the fraud procedure. The AI evaluator accepted the agent’s “fraud” label without verifying it against the customer’s actual words.

Root cause: The prompt’s Core Principle 6 (cross-referencing) didn’t explicitly warn against inheriting the agent’s categorisation. The evaluator treated the agent’s framing as given rather than as a claim to verify.

Fix: Added a rule in Core Principle 6 of evaluate_scorecard_criterion_for_agent_tool.md: the agent’s framing of the issue (e.g. fraud vs. commercial dispute) is a claim to be verified, not a given. The evaluator must determine the correct procedure from the customer’s actual described situation, independent of the agent’s label. Includes a concrete payment dispute example: “I made a payment” = authorized = commercial dispute, regardless of whether “fraud” was used by the agent or customer.

Impact: Generalises beyond Qonto — any organisation could have agents miscategorising issues. The payment dispute example is the clearest illustration but the principle applies to any procedure verification.

Source feedback (#4924549 ): The reviewer (Anaëlle Tabary) corrected Solve the Request from Pass (30/30) to Fail (0/30), stating: “Le client a effectué le paiement, le cas doit être traité comme un litige et non comme une fraude. Le site internet existe et ne semble pas frauduleux. Le client doit contacter le marchand et essayer de trouver une solution à son problème. S’il n’y parvient pas, nous pourrons faire un dossier litige.” (“The customer made the payment, the case should be treated as a dispute not fraud. The website exists and does not appear fraudulent. The customer should contact the merchant first. Only if that fails should we open a dispute case.”)

The expanded feedback clarifies the litige vs fraude distinction: when a customer authorized a payment themselves and the merchant appears legitimate, the agent must instruct the customer to contact the merchant first — not file a fraud dispute. The AI originally passed because it inherited the agent’s “fraud” framing without verifying it against the customer’s actual words (“I made a payment”).

Note: The reviewer referenced the merchant website as appearing legitimate, but we don’t have access to verify this externally. The core principle — that “I made a payment” = authorized transaction = commercial dispute procedure — holds regardless of merchant legitimacy.

Files:

• rulebase-api/config/prompts/evaluate_scorecard_criterion_for_agent_tool.md (Core Principle 6)

Fix 5: Severity miscalibration — precise justification option matching shipped

Problem: The AI evaluator was selecting wrong severity levels when scoring criteria. In ticket #4974065 , the agent provided a comprehensive TPE fee overview (rates, no subscription, first 5 free) but didn’t calculate the specific commission on a €69–€200 sale. The AI gave auto-fail (0/30) — “does not provide any resolution path” — when the correct score was Partial (20/30) — “core request solved but follow-up questions not addressed.” Similarly, in #4889517 , the agent addressed the entirely wrong topic (card status instead of identity verification) but the AI gave Partial (10/30) instead of Fail (0/30).

Root cause: Two issues in the global evaluation prompt (evaluate_scorecard_criterion_for_agent_tool.md):

1. No explicit comparison step: The prompt told the AI to “choose the justification option that most accurately describes the failure” but didn’t instruct it to read ALL available justification options and compare before selecting. The AI latched onto the first plausible match rather than the most specific one.
2. General leniency heuristics overrode justification boundaries: The prompt contained rules like “err on the side of the agent”, “prefer to give full marks if the scoring difference is more than 5 points”, and “Do not score 0 unless the agent gave a factually incorrect answer” — all of which prevented the AI from selecting the correct lower-scoring justification option even when the scorecard’s own definitions clearly defined it.

Fix: Four changes to the global evaluation prompt:

• Added justification option matching guidance in the check evaluation section: the AI must compare against ALL available justification options across ALL score levels before selecting one. Includes concrete boundary examples:
  • Agent addressed the correct topic but incompletely → “partial”, not “failed”
  • Agent addressed an entirely different topic → “wrong solution” (fail), not “partial”
  • Agent provided some relevant information → partial resolution, not “no resolution path”
  • “No resolution path” = zero actionable information toward the customer’s stated problem
• Removed “err on the side of the agent” from Core Principle 4 (conflicts with strict.md and undermines precise matching)
• Removed “5 points” threshold rules from Principles 4 and 9 (arbitrary, doesn’t account for scorecard-specific scales)
• Removed blanket “Do not score 0 unless factually incorrect” from Partial Responses section (overrides justification options that define other valid conditions for score 0)

Eval harness: Added a severity_calibration_core_solved_followup_missed test case in scripts/eval_qa_evaluations.rb. Transcript mirrors real ticket #4974065: agent initially answers about wrong topic (device prices), then provides correct TPE rates with a dubious “€0.01 minimum” claim, but doesn’t calculate the specific €69–€200 amounts the customer asked for. Criterion has two Partial levels (20 and 10) with distinct justification descriptions. Scored via Braintrust: correct_outcome: 100%, check_result_correct: 100%.

Impact: Addresses 4 tickets directly (#4974065 , #4889517 , #4873185 , #4829596 ) and generalises to any scorecard with multiple severity levels defined via justification options.

Files:

• rulebase-api/config/prompts/evaluate_scorecard_criterion_for_agent_tool.md (Core Principles 4, 9; Justification Option Selection Rules; Partial Responses section)
• rulebase-api/scripts/eval_qa_evaluations.rb (eval test case)

Won’t Fix: Criterion scope confusion — process/ownership args under Tone of Voice

Reported issue: In 4 tickets (#4889517 , #4974065 , #4949412 , #4922785 ), reviewers flagged that the AI used “ownership”, “proactiveness”, and “driving resolution forward” arguments when evaluating Tone of Voice, saying these belong under Efficiency or Solve the Request instead.

Why won’t fix: The AI is following the scorecard correctly. Qonto’s Tone of Voice KB document explicitly includes section 3 “Professionalism” → “At the Client’s Service and To the Point”, which covers:

• “Ownership: active voice, show of proactiveness/intent to drive the resolution forward”
• “Positivity: emphasis on the workarounds to keep the conversation on a positive track”

These are defined as Tone of Voice dimensions in the KB, not Efficiency dimensions. The AI is correctly applying the criterion as specified. The reviewers’ feedback reflects a misunderstanding of the scorecard scope, not an AI error.

Fix 6: Wrong-topic severity — agent addressing wrong problem gets no partial credit shipped

Problem: When an agent investigated a completely different problem than what the customer reported, the AI gave Partial (“solves part of core request”) instead of Fail (“wrong solution”). In ticket #4889517 , the customer explicitly stated twice that identity verification on mobile was failing with “error, please try again later” blocking all payments. The agent ignored the identity verification issue entirely, asked which virtual card was being used, and requested a screenshot of the payment error. The AI gave Partial (10/30) — should be Fail (0/30).

Root cause: Three competing prompt instructions gave the AI escape routes to award partial credit for wrong-topic work:

1. The customer abandonment override auto-passed when the customer stopped responding, treating the agent’s wrong-topic investigation as “a reasonable attempt to help”
2. The “standard troubleshooting” principle scored diagnostic questions as “effective probing” regardless of whether they targeted the customer’s actual problem
3. The justification matching guidance from Fix 5 told the AI to distinguish wrong-topic from partial, but the decision happened too late — by the time justification options were being compared, the AI had already decided the check was “partial” based on the other overrides

Fix: Four changes across the global prompt and strict.md:

• Topic-match gate in check evaluation: added a new gate (alongside the existing feasibility gate) that blocks “partial” for resolution checks when the agent addressed a different problem than the customer reported. “Partial” requires the agent to have been working on the correct problem but incompletely.
• Customer abandonment caveat: the override now explicitly excludes cases where the agent investigated the wrong topic — investigating the wrong problem is not “a reasonable attempt to help” regardless of whether the customer stopped responding.
• Standard troubleshooting caveat: diagnostic questions only count as effective probing when they target the customer’s stated problem. Asking about card status when the customer reported identity verification errors is not effective probing.
• strict.md dimension 9 — Wrong-Topic Resolution: added a new calibration dimension reinforcing that working competently on the wrong problem gets no partial credit.

Eval harness: Added a wrong_topic_fail_not_partial test case in scripts/eval_qa_evaluations.rb. Transcript mirrors real ticket #4889517: customer says identity verification fails on mobile, agent asks about virtual cards and requests a payment screenshot, then closes the chat. Expected result: Fail (“wrong solution”), not Partial. Scored via Braintrust: correct_outcome: 100%, check_result_correct: 100%.

Impact: Directly addresses #4889517  and generalises to any case where the agent works on the wrong topic. Also strengthens Fix 5 by closing the escape routes that prevented correct justification matching.

Files:

• rulebase-api/config/prompts/evaluate_scorecard_criterion_for_agent_tool.md (Topic-match gate, customer abandonment caveat, standard troubleshooting caveat)
• rulebase-api/config/prompts/evaluations/strict.md (Dimension 9)
• rulebase-api/scripts/eval_qa_evaluations.rb (eval test case)

Won’t Fix: #4829596 display discrepancy — AI anchors on SME internal note

Reported issue: In ticket #4829596 , the customer reported a display discrepancy (upcoming transactions showed +€85,000 but detail listed ~€50). The second agent (Rania) asked basic troubleshooting (reinstall the app, check missing transactions) without analyzing the screenshot or checking back-office. Reviewer wanted auto-fail — the second agent should have moved the investigation forward. The AI gave Partial.

Why won’t fix (two reasons):

1. SME internal note anchors the AI. Thibaud (SME) wrote an internal note asking “has the client tried deleting and reinstalling the app? Are the other 2 transactions visible?” Rania then asked the customer exactly those questions. The AI reads Rania as having followed the SME’s guidance and scores her positively for it, rather than independently evaluating whether her contribution was sufficient as the second agent on the ticket. This anchoring effect is outside prompt control — the AI correctly identifies that the agent did what was suggested, but fails to recognize that merely relaying SME questions without adding independent analysis is insufficient.

2. No KB document for display discrepancy investigation. Without a knowledge base procedure defining what proper troubleshooting looks like for forecast/upcoming transaction discrepancies, the AI has no independent reference standard. It falls back on “the agent did something reasonable” rather than cross-referencing a specific procedure. This is a KB content gap that Qonto would need to fill.

Recommendation: Qonto could address this by (a) adding a KB article for display/forecast discrepancy troubleshooting, and (b) adding check instructions that explicitly state “repeating another agent’s or SME’s suggestions without adding independent analysis does not constitute adequate resolution by a subsequent agent.”

Remaining: AI evaluation accuracy (no code fix)

The majority of negative feedback (~75%) stems from the AI’s contextual understanding of conversations, not from dependency rule mechanics. Key areas:

• Misreading customer intent — the AI gives a surface-level reading and misses the real issue the customer is asking about
• Wrong reasoning on correct scores — even when scores are right, the explanation doesn’t match what actually happened

Score Delta Analysis — March 2026

Per-conversation delta between inferred original AI score and human-corrected score, across all 56 reviewed conversations. Delta is calculated only on criteria with a direct thumbs-down comment (dependency-rule cascades excluded). Positive delta = AI was too harsh; negative delta = AI was too lenient.

Summary

Interpretation: The avg absolute delta of 8.3 points per conversation shows meaningful per-ticket error in both directions. The negative tail (AI too lenient) is driven by a small number of high-impact misreads — procedure misclassification and first-touch failures. The positive tail (AI too harsh) is largely the severity calibration issue — the AI penalising agents for things reviewers consider passes.

Create a Human Bond + Solve the Request

Delta restricted to the two highest-weight criteria only.