AI Spend Optimization Log

This is a running internal note for AI spend work that started from the June 2026 usage review. Add new observations to the daily log first, then update the reference sections when a decision, estimate, or rollout state changes.

Current state

In June 2026, Rulebase’s largest variable infrastructure expense was AI model usage. The first direct OpenAI optimization target was QA eligibility checks running through OpenAI Responses, mostly on gpt-5.4.

QA eligibility was first moved from gpt-5.4 to gpt-5.1; on August 6 it moved again to OpenRouter openai/gpt-5.6-terra as part of the broader effort to replace routine Bedrock Claude defaults. On August 7, after finding the new Terra default-tier path had become a visible spend bucket, QA eligibility moved to OpenRouter openai/gpt-5.6-luna with Flex provider options. We tested OpenRouter GLM 5.2 as a cheaper QA eligibility model, but reliability was not strong enough to use in production.

For full QA evaluations, Kuda was the first important migration target. June Braintrust analysis showed Kuda still using the older Rails parent-criterion evaluator, evaluate_parent_scorecard_criterion_for_agent, which repeats the transcript/prompt/tool loop per parent criterion. That was the main reason to migrate high-volume orgs from Rails per-criterion QA to the conversation-level evaluate_conversation_with_tools path, then to API2 once flex/provider instrumentation is in place. As of July 2026, Kuda, EasyPay, and Rho have all had fresh rulebase-api2 production traces on conversation_qa_evaluation; continue checking invoice data because Braintrust trace estimates do not fully reflect provider caching and flex economics.

Separately, Kuda’s managed-agent High-effort conversation scan has now moved from a legacy Opus agent loop to a workflow: the model returns structured candidates, and deterministic workflow nodes handle tagging, field updates, and Slack notifications. Re-check this workflow’s spend after a few days of production runs.

Current July data points to Qonto conversation QA as the largest remaining OpenAI driver. On July 8, Qonto was configured to use OpenRouter Claude Opus 4.8 for evaluations at about 8:30 PM ET. A July 16 investigation found that OpenRouter’s cross-model models fallback was routing almost all sampled Opus requests to BYOK OpenAI GPT-5.5 even when Bedrock Opus could complete the same request. PR #8304  removes that silent route and permits a premium-model switch only after an actual transient provider exception. Treat the intended Bedrock route as a cash-preservation experiment while credits are available, not a gross-efficiency improvement: Opus 4.8 is much more expensive than OpenAI flex on list pricing.

API2’s routine background defaults now use OpenRouter GPT-5.6 Terra instead of Bedrock Claude. Anthropic model IDs remain supported for organization-specific QA settings and legacy-model migration, but no longer serve as the shared implicit model or the default for eligibility, translation, assistants, sheet fill, work-item import, topic naming, or compliance search-term generation.

Braintrust is useful for ranking spend drivers, but it likely overstates current OpenAI cost because it does not appear to account for OpenAI flex service-tier pricing. It also does not currently expose spend by Rulebase AI function/category on child LLM spans, because the child doGenerate/doStream spans do not inherit root trace metadata. For managed-agent spend by customer and workflow, api2.workflow_runs and api2.node_executions are a better attribution source than credit_usage_events, because credit_usage_events only covers Grant credit-billed customers.

Daily log

August 8, 2026

Compacted the API2 conversation QA prompt for every organization on runConversationQaEvaluation, without touching models, reasoning effort, retrieval, feedback/KB access, criteria counts, or one-save-per-leaf semantics.

Two sources of repeated input tokens were removed:

• evaluation_work_queue_xml inlined the complete <scorecard> hierarchy in every unit of work, so a scorecard’s definition was repeated once per leaf criterion per agent. It is now serialized once under <shared_scorecards> and each unit carries <scorecard_ref id="..."/>. A definition is only shared when every unit for that scorecard has a byte-identical one; otherwise it stays inlined.
• A successful save_agent_criterion_result echoed the whole remaining queue, and every one of those tool results stayed in later model input. It now returns the saved acknowledgement, progress counts, next_pair, and an instruction. Validation failures, unknown pairs, and duplicate saves still return full corrective context.

On a synthetic 3-agent × 12-leaf queue, serialized queue size fell from 406,721 to 54,897 characters (86.5%). Real reductions depend on hierarchy size and leaf count per scorecard.

Prior offline fresh-cache replays on Rho, NALA, and EasyPay retained 100% normalized result agreement and 97.22% exact agreement, with aggregate estimated cost falling from $1.8965 to $1.2199 (35.7%); per-fixture reductions were 35.9%, 40.5%, and 32.8%. Those fixtures stubbed feedback/KB responses, so this is not evidence of full Qonto parity. Qonto is the largest dollar beneficiary, but the change is global to the API2 conversation QA runner.

August 7, 2026

Moved QA eligibility from OpenRouter openai/gpt-5.6-terra on the default tier to OpenRouter openai/gpt-5.6-luna with Flex provider options.

The reason for the change was the August 6-7 OpenAI export: gpt-5.6-terra default-tier traffic on the OpenRouter BYOK key cost about $181.79 over two days, with 5,562 requests, 71.3M input tokens, and 70.0M cache-write tokens. That pattern matches the eligibility route’s previous default-tier configuration.

Same two-day traffic repriced:

Ran the QA eligibility Braintrust eval after the switch:

Expected savings from this specific move are roughly $2.6K/month versus the Aug 6-7 Terra default-tier pace, or about $1.3K/month versus Terra Flex at the same volume. Re-check the OpenAI export after one full production day to verify that eligibility is now reported as gpt-5.6-luna on Flex and that 502/timeout rates have not increased.

Moved Rho CS Escalations from Normal to Lite in production by replacing the saved OpenRouter openai/gpt-5.6-sol model IDs with openai/gpt-5.6-terra in both the editable workflow graph and the published live version 51. Verification showed 18/18 model fields on Terra in each graph. This exposed a product-modeling issue: workflow graphs persist concrete provider model IDs, while Normal/Lite are only labels over those IDs. That makes global model repricing require saved-graph and agent_versions migrations; a better durable shape would persist a stable tier such as normal or lite and resolve it to the current model at runtime.

Conversation QA Luna downgrade for Rho, Lesaka, and NALA

Set up offline replay fixtures from production for the next full-QA downgrade decision. The fixtures live locally under tmp/conversation-qa-offline-fixtures/ and are intentionally git-ignored because they contain production conversation and scorecard context.

Fixture sample:

Ran the offline replay with openrouter/openai/gpt-5.6-luna through Infisical so OPENROUTER_API_KEY was present. The valid experiment is conversation-qa-offline-luna-fixtures-openrouter-2026-08-07.

Results:

The model route logs confirmed actual OpenRouter/OpenAI Luna calls: openai/gpt-5.6-luna with provider OpenAI for all three fixtures. Ignore the earlier local experiment conversation-qa-offline-luna-fixtures-2026-08-07: local .env did not include OPENROUTER_API_KEY, so the development fallback routed the requested OpenRouter Luna model to native gpt-4o. That run was stopped after the EasyPay fixture and should not be used for model-quality decisions.

EasyPay emitted a few rejected unknown pair warnings. That means the model attempted to save a result for an (agent_id, scorecard_id, leaf_criterion_id) triple outside the work queue. The runner rejected those tool calls and asked the model to retry with a valid remaining pair. This did not corrupt output and the final normalized verdicts still matched, but it is the main watch item after the downgrade.

Based on the canary, switched full conversation QA evaluation models from Terra to Luna for all three orgs at about 8:16 PM ET on August 7, 2026 / 00:16 UTC on August 8, 2026:

Pricing implication: Luna is roughly 90% cheaper than Terra for the same token mix across input, cached input, cache writes, and output. The August 1-7 production volume for these three orgs was 1,012 completed AI QA evaluations: 206 Lesaka/EasyPay, 629 Rho, and 177 NALA. Using the Luna fixture cost per org and scaling that seven-day volume to a 30-day month gives an estimated Terra run-rate of about $1.0K/month, a Luna run-rate of about $0.1K/month, and expected savings of roughly $0.9K/month. Treat the practical range as $0.8K-$1.1K/month depending on evaluation volume, retry rate, and provider accounting. This estimate covers only the Rho/NALA/Lesaka full-conversation QA downgrade; it does not include QA eligibility, Kuda API2 consolidation, or OpenAI storage cleanup.

After the first NALA fixture review, we found the “one response / do not evaluate” rule was mistakenly stored in custom_evaluation_instructions. That made Luna repeat eligibility-style exclusion wording across every criterion, which looked like a model-quality issue but was actually prompt contamination. At about 8:35 PM ET on August 7, 2026 / 00:35 UTC on August 8, 2026, those NALA rules were moved into custom_eligibility_instructions, leaving only the reassignment-latency exception in evaluation instructions. At about 8:40 PM ET / 00:40 UTC, NALA eligibility instructions were rewritten into one coherent policy covering real customer requests, Intercom chat scope, investigated/resolved human-agent exchanges, and the single-reply exclusion.

Clean NALA replay with the corrected evaluation prompt: conversation-qa-offline-luna-nala-clean-eval-instructions-2026-08-08. Results stayed production-safe for this canary: run_completed 100%, normalized check-result match 100%, exact check-result match 91.67%, selected justification-option match 100%, errors 0, estimated cost about $0.02. The repeated exclusion boilerplate disappeared, and the saved summaries became specific to the account-deletion outcome, so the earlier tone issue should not be attributed to Luna alone.

OpenAI vector-store and file cleanup

Found a separate OpenAI storage leak: file-search request volume was tiny, but vector-store storage was large enough to show up as a real daily charge. Recent OpenAI exports showed the Default Project storing roughly 460 GiB/day of vector-store data, which priced to about $46/day or $1.38K/month at OpenAI’s $0.10/GB/day file-search storage rate after the first free GB.

Live object inventory was split across project-scoped keys:

The scoped-project vector stores were dominated by unnamed or legacy per-object stores:

Deleted all vector stores visible to the local project key: 3,326 deletes succeeded, 1 store was already gone, and the final re-list returned 0 stores.

For the scoped project, took a full cursor snapshot and deleted all 206,163 stores visible in that remaining snapshot after earlier partial delete passes. The final delete phase reported 206,159 successful deletes, 4 already-gone stores, and 0 failures. Immediate final verification hit OpenAI 503 Unable to list vector stores right now, likely because the list API was catching up after the mass delete. Re-run the vector-store usage export after the next billing refresh before recording the realized daily savings.

Also inventoried and deleted raw OpenAI Files API objects. These files were visible separately from vector stores, but the large billed line we found was vector-store/file-search storage, not a generic raw-file storage line. Treat raw-file deletion as cleanup and privacy/data-retention hygiene unless OpenAI later exposes a separate generic Files API storage charge.

Total raw file inventory was 177,624 files and 15.350 GiB. The linked example file, file-GfJFbPkrg9DjJHr69wvFnV, was a 5.3 KB QA manual eval criterion feedback file with purpose = user_data, created on August 7.

Deleted all 177,624 files visible to the scoped project key. The Files API deletion pass hit heavy rate limiting (42,272 429 retries) but completed with 0 hard failures. Final verification returned 0 files and 0.000 GiB remaining for that project.

Expected realized savings from the storage cleanup are still about $46/day or $1.35K-$1.4K/month, because the measured billable source was roughly 461 GiB of vector-store/file-search storage at OpenAI’s $0.10/GB/day rate after the first free GB. The raw-file deletion removed another 15.350 GiB of objects, but do not count that as incremental savings unless OpenAI bills generic Files API storage separately.

Handed off the code-side migration work to Devin so this does not recur: move QA feedback store retrieval off OpenAI vector stores, stop creating OpenAI vector-store/files for KB/procedure writes, and move framework/compliance retrieval to internal chunks. The cleanup above removed existing storage objects; the code migration still needs to prevent new persistent vector-store usage.

August 6, 2026

Reconciled the surprising AWS Bedrock credit burn against Braintrust. The conclusion: the spend is real model inference, not an AWS dashboard artifact or ordinary infra noise.

• AWS Billing showed roughly $16.5K month-to-date spend by the morning of August 6, with July’s finalized-looking AWS total at $58.6K and about $49.9K of July coming from Bedrock Claude model SKUs.
• Braintrust BTQL over the rulebase-api2 project from August 1, 2026 00:00 UTC through August 6, 2026 13:00 UTC showed about $17.9K of cache-aware gross Bedrock-equivalent API2 model usage. That is close enough to AWS month-to-date after normal AWS billing lag to treat the spend as materially real.
• The Braintrust rollup used prompt_tokens, completion_tokens, prompt_cached_tokens, and prompt_cache_creation_tokens, then priced Claude cache reads/writes separately instead of treating every input token as uncached.

Model split for the August 1 to August 6 13:00 UTC rulebase-api2 window:

Drilled into August 5, 2026 00:00 UTC through August 6, 2026 13:00 UTC, where cache-aware Claude spend was about $8.9K:

Important interpretation: the current AWS Bedrock burn does not look primarily like Instruct or managed-agent usage. In this sampled window it is dominated by conversation QA evaluations, with a meaningful secondary issue in translation/language detection. The sampled heaviest traces had ask_feedback_store, ask_knowledge_base, and save_agent_criterion_result spans, which matches the API2 conversation QA tool loop.

The translation surprise was Kuda language detection: in the 37-hour sample, conversation_language_detection for kuda-pilot wrote about 295M prompt cache creation tokens and priced to about $1.16K by the cache-aware estimate. This needs a separate check because language detection should not be creating that much gross Claude cache-write cost.

Reconciliation caveats:

• We still do not persist OpenRouter generation IDs, so we cannot ask OpenRouter’s /api/v1/generation?id=... endpoint to reconstruct historical feature-level cost exactly.
• Braintrust BTQL can aggregate logs, but it does not support CTEs or joins, so exact root-to-child span attribution requires client-side trace reconstruction or better metadata on the cost-bearing LLM child spans.
• The top-level agent_run, conversation_qa_evaluation, and translation task spans do not carry aggregate token cost. The dollars live on child doGenerate / doStream spans.
• Treat the numbers above as a strong scope estimate, not invoice-grade accounting. They are good enough to say the AWS spend is real and mostly QA/translation in this window.

Simplified Instruct and managed-agent model tiers

On August 6, PR #9374  replaced provider-specific model pickers with two product-level choices across both Instruct and managed-agent workflow nodes:

Normal is now the default for new Instruct chats and new managed-agent nodes. Existing saved selections and workflow model IDs migrate at resolution time: Auto, Claude Opus/Sonnet, GPT-5.5, and GPT-5.4 map to Normal; GPT-5.4 mini and GPT-5.2 map to Lite. The fallbacks are directional, so explicitly reaching GPT-5.5 or GPT-5.6 Luna as a fallback does not route back upward.

Managed-agent runs and workflow router nodes already send OpenAI/OpenRouter Flex service-tier options, so this change moves their default background inference away from Bedrock Claude toward OpenAI Flex while keeping a reliability fallback. Instruct is interactive and does not use the background Flex options. This should reduce gross managed-agent inference cost and slow AWS credit burn, but it may increase direct OpenAI cash spend relative to Bedrock-credit-backed Claude. Reconcile the first full post-deploy week across OpenRouter actual routes, OpenAI service tiers, AWS Bedrock usage, Braintrust, and workflow_runs before recording a dollar saving.

Moved remaining API2 defaults off Bedrock

After reviewing remaining Anthropic pins on August 6, moved routine API2 defaults to OpenRouter openai/gpt-5.6-terra, with its directional fallback now set to openai/gpt-5.6-luna:

• QA eligibility, conversation translation, and language detection.
• Agent chat when no supported picker model is supplied; agent chat now honors the route’s supplied Normal/Lite model instead of discarding it.
• Company-comment and Slack assistants, including Slack reply classification.
• Sheet filling, work-item import, topic naming, and compliance population search-term generation.
• The shared no-argument createModel() default used by scorecard generation, skill generation, agent-run reflection, and chat-title generation.
• Local sheet-fill and workflow-chat eval defaults, while retaining environment-variable overrides.

This removes the known routine production defaults that were sending these features to Bedrock Claude. Organization-configured Anthropic QA models still route to Bedrock intentionally, and legacy Anthropic aliases/fallback support remains in the model router. Check OpenRouter actual-provider logs, AWS Bedrock daily usage, and OpenAI daily usage after deployment before assigning realized savings.

August 5, 2026

Reviewed Blacksmith CI spend against the usage already included in Rulebase’s GitHub Enterprise plan, then reduced unused GitHub licenses:

• GitHub Enterprise includes 50,000 standard GitHub-hosted Actions minutes per month. The August billing API showed only 55 minutes consumed at the time of review, leaving 49,945 included minutes unused while most CI ran on Blacksmith. Actions storage was 15.3 GB of 50 GB included.
• Blacksmith’s July baseline was 118,180 runner minutes and $1,090.98 of Actions spend. Standard 4-vCPU Linux jobs accounted for 112,116 minutes and about $896.93; the 16-vCPU Rails test runner accounted for the remaining 6,064 minutes and about $194.05.
• Opened PR #9312  / ENG-2285  to move 43 blacksmith-4vcpu-ubuntu-2404 declarations across 18 workflows to ubuntu-latest. The performance-sensitive Rails test job stays on blacksmith-16vcpu-ubuntu-2404.
• At equal runtime, the July 4-vCPU workload would cost about $372.70 on GitHub after the 50,000 included minutes, versus $896.93 on Blacksmith: approximately $524.23/month saved. Allowing for potentially slower GitHub-hosted runners, use $400-$525/month as the CI savings range until production timings settle.
• Reduced the GitHub Enterprise subscription from 8 licenses to 4. The downgrade is effective August 30, 2026 and reduces the subscription from $168 to $84/month, an exact $84/month saving.
• Combined working estimate: about $550/month, with a practical range of $484-$609/month, or roughly $6.6K/year at the midpoint.

Post-rollout checks:

• Compare GitHub-hosted job duration and queue time with the prior Blacksmith runs.
• Confirm the required Rails, web, docs, deployment, and maintenance checks remain reliable on ubuntu-latest.
• Re-check GitHub included-minute consumption and Blacksmith spend after one full billing month.
• Move a job back to Blacksmith only when measured runtime or reliability outweighs the cost difference; keep the 16-vCPU Rails suite on Blacksmith unless a separate benchmark supports changing it.

July 16, 2026

Investigated why direct OpenAI spend remained high after moving Qonto conversation QA to OpenRouter Claude Opus 4.8:

• Direct OpenAI spend was $8,847.89 for July 1-16, a $17.1K 31-day run-rate. July 16 spend was about $553.66, of which GPT-5.5 contributed $432.16. Most GPT-5.5 spend was flex-billed, so the remaining problem was model routing rather than standard-tier leakage.
• Braintrust grouped Qonto QA child spans under the requested claude-opus-4.8 model, but the response payload exposed modelId = openai/gpt-5.5 and OpenRouter provider OpenAI. On July 16, 3,579 sampled API2 Opus-requested child spans actually used GPT-5.5/OpenAI; only one exposed an Opus/Amazon Bedrock response.
• The code configured explicit Opus and GPT-5.5 models as an OpenRouter fallback tier. For Opus requests this sent models: ["openai/gpt-5.5"], allowing OpenRouter to choose GPT-5.5 without exposing a failed Bedrock attempt to the application.
• Replayed a real Qonto QA generation from conversation_0rpo9ZqB41wUGqeaRyD4KaNM. Bedrock-only Opus succeeded with the original prompt and tools. The same request also stayed on Bedrock when both BYOK providers were available but no cross-model models array was supplied. Adding the GPT-5.5 model fallback routed it to OpenAI instead. This ruled out an inherent Opus/Bedrock failure for the sampled production request.
• Opened PR #8304  / ENG-1813 . Explicit Opus 4.8 and GPT-5.5 requests no longer send OpenRouter cross-model fallback candidates. API2 retries the alternate premium model only after a classified transient provider exception; 400-class request errors do not retry. The user-selected Auto chain remains unchanged.
• Added structured QA completion logging for each generation step’s requested model, actual response model, and OpenRouter provider. This gives us an application-side audit trail when Braintrust groups spend by the requested model.
• Expected cash impact after deployment: eliminate most of the Qonto-driven direct OpenAI GPT-5.5 route, potentially reducing the July 16-style OpenAI day from about $554 toward roughly $120 before unrelated variance. This is not a net gross-spend saving: the sampled forced-Bedrock Opus child cost about $0.46 upstream versus about $0.16 on GPT-5.5. The rationale is to consume AWS credits and preserve cash, then monitor Bedrock credit burn and QA quality.

July 15, 2026

Checked OpenRouter Bedrock BYOK routing for Claude Sonnet structured outputs:

• Sonnet 5 plain text requests stayed on Bedrock BYOK, but response_format: json_schema requests returned 400 on the Bedrock BYOK attempts and then fell back to non-BYOK Bedrock. This means AI SDK Output.object(...) can silently burn OpenRouter credits instead of AWS credits on Sonnet 5.
• The same minimal tests on Claude Sonnet 4.6 passed on Bedrock BYOK for plain text, JSON schema structured output, require_parameters: true, forced tool calls, and a translation-shaped nested schema.
• Switched API2 Sonnet defaults and configured Sonnet workflows back to openrouter/anthropic/claude-sonnet-4.6: default/auto Sonnet route, agent chat default, sheet-fill/workflow eval defaults, company and Slack assistant Sonnet calls, conversation translation, language detection, and critical-failure clustering.
• If we need Sonnet 5 on Bedrock later, prefer a forced-tool structured-output wrapper or wait for OpenRouter’s Bedrock BYOK route to translate response_format: json_schema correctly for Anthropic.

July 8, 2026

Reviewed July OpenAI usage and the latest Braintrust traces after the Kuda/EasyPay/Rho API2 migration work:

• July 1-8 OpenAI raw usage was about $4.17K, which projects to roughly $16.2K/month if the rate stays flat. That is lower than the June OpenAI run-rate, but not low enough to stop optimizing.
• Top July 1-8 model spend was still concentrated in gpt-5.4 flex, with notable gpt-5.5 standard/default spend. The largest named workflow in the current view is Qonto conversation QA, not Kuda.
• Qonto evaluate_conversation_with_tools was roughly $1.36K of the July 1-8 OpenAI export, about a third of total raw OpenAI spend in that window and more than half of gpt-5.4 flex spend. Kuda was the June consolidation target; after the Kuda migration, Qonto is the current visible QA driver.
• Fresh Braintrust/API2 traces verified that Kuda is now running conversation QA through rulebase-api2, not the old Rails parent-criterion path. The working estimate from matched conversation-level samples is about $0.19/eval on the new API2 path versus about $0.64/eval on the old Rails path, or roughly 70% per-eval savings before invoice reconciliation.
• At the observed Kuda volume, that points to an approximate $5K/month savings opportunity from the API2 consolidation. This remains an estimate because Braintrust generic cost, provider-reported cost, OpenAI flex pricing, OpenRouter routing, and prompt caching do not all line up cleanly in traces.
• Braintrust child LLM spans still do not carry function_id or spend_category, so Braintrust Monitor cannot show spend by Rulebase AI feature yet. The practical fix is to keep wrapAISDK and add attribution directly on child doGenerate/doStream spans, likely via AsyncLocalStorage plus AI SDK model middleware. Estimate: about 0.5-1 day. A fuller OpenTelemetry migration is more like 1-3 days depending on how much wrapAISDK we remove.
• Rho still showed material gpt-5.5 agent-run spend in the July data, even though we expected some Rho work to have moved to Sonnet/Bedrock. Re-check the deployed workflow/model config and OpenRouter provider routing before counting Rho Bedrock savings.
• Confirmed OpenRouter Claude Opus 4.6 and 4.8 are effectively the same list price class for the regular model. If using Opus for a cash-preserving Bedrock-credit route, prefer Opus 4.8 unless compatibility or provider availability says otherwise.
• Confirmed the current API2 OpenRouter Opus setup can pass OpenRouter reasoning/effort options, but the conversation QA runner does not currently map organization.evaluation_reasoning_effort into OpenRouter. It only maps the org setting to OpenAI reasoningEffort. If Qonto stays on OpenRouter Opus, add openrouter.verbosity or openrouter.reasoning.effort wiring so org-level effort settings are not silently ignored.
• Qonto was changed to OpenRouter Claude Opus 4.8 for evaluations at about 8:30 PM ET on July 8, 2026. Verify the first post-change traces for project, model, provider route, cost, cache behavior, and quality before treating the switch as a durable savings move.
• Outside AI, Better Stack has mostly been cut down: services were disabled and responders were reduced to Chidi, while keeping status pages available. This should reduce observability spend, but it is a smaller lever than AI model routing.

July 2, 2026

Updated the QA eligibility plan after testing alternatives:

• Kept QA eligibility on OpenAI instead of OpenRouter GLM 5.2.
• Pinned the eligibility model to gpt-5.1, because it is the same price class as gpt-5 and passed the local Braintrust eval run with 100% verdict_matches, 100% reason_does_not_leak_instruction_wrapper, and 0 errors.
• Treated OpenAI flex as the primary savings lever. Braintrust still appears to estimate these spans at standard list pricing, so invoice reconciliation needs OpenAI dashboard data.

Logged follow-up model-routing work:

• Confirmed language detection is not worth moving to Bedrock Haiku on gross economics. Keep it on OpenAI gpt-5-mini with flex and optimize with deterministic/local detection first if needed.
• Reviewed Rho managed-agent workflow spend. Historical WIP workflow rows were scary but mostly paused/deleted on June 23, 2026; current attention stays on CS Escalations and, secondarily, Treasury mention daily report.
• Turned on production Flipper api2_conversation_qa_evaluation for Organization;38 and Organization;19.
• Braintrust already showed EasyPay and Rho using evaluate_conversation_with_tools, but the sampled spans were in the Rulebase project and did not prove API2 execution. Re-check after fresh post-flag runs.
• Re-checked after fresh production runs. Braintrust now shows EasyPay and Rho in the rulebase-api2 project with functionId = conversation_qa_evaluation and runner = api2_conversation_qa.
• Early post-flag traces are directionally consistent with lower realized cost when using provider-reported upstream cost, but the sample is short and Braintrust’s generic estimated_cost still overstates the likely invoice cost on cached/flex-like traffic.
• Checked whether cached Claude Sonnet 5 on Bedrock would beat the current GPT/OpenAI route for EasyPay/Rho conversation QA. It can get close only with extremely high cache-read coverage, and the current provider-reported upstream cost is still lower than the realistic cached-Sonnet cases. Keep EasyPay/Rho on the current GPT/OpenAI route for now; use Sonnet on Bedrock as a canary or fallback, not the default production path.
• Re-read the July 1 Codex analysis for Kuda QA evaluations. Kuda was the larger gpt-5.4 evaluation source in Braintrust, with the costly part being Rails evaluate_parent_scorecard_criterion_for_agent, not the managed-agent high-effort workflow. Kuda is still the main full-QA migration target.
• Migrated Kuda’s High-effort conversation scan managed agent from legacy_instruction mode to workflow mode. The model now returns structured high-effort candidates, and deterministic workflow nodes handle the High effort tag, High Effort Count field, and Slack alert. Re-check spend after a few days of production runs.
• Enabled Kuda (Organization;25, kuda-pilot) in production Flipper for both qa_evaluation_conversation_runner and api2_conversation_qa_evaluation, alongside EasyPay (Organization;38) and Rho (Organization;19). Re-check Braintrust/API2 after fresh Kuda QA runs.

June 2026 baseline review

Established the baseline spend picture:

• OpenAI dashboard reported $18,261.48 of June spend across all projects.
• Braintrust estimated direct OpenAI-like provider spend at about $20,044, roughly 9% above the raw OpenAI dashboard.
• AWS Billing showed $13,534.04 of gross Bedrock marketplace usage, fully offset by credits.
• QA eligibility was the clearest direct OpenAI optimization target, with Braintrust estimating $3,042.68/month on the old path.
• Green Desk was canceled after the June charge had already landed, so the cash benefit should start after June 2026 if it stays canceled.

Reference findings

Direct OpenAI spend

The OpenAI dashboard reported $18,261.48 of June 2026 spend across all projects, with 21.997B input tokens and 2.407M requests.

Braintrust estimated direct OpenAI-like providers at about $20,044 for June 2026:

Braintrust was about 9% higher than the raw OpenAI dashboard. That makes Braintrust useful for ranking sources of spend, but not reliable enough to use as invoice-grade savings math.

OpenRouter and Bedrock spend

OpenRouter Claude usage was not the source of QA evaluation spend. The large Opus spend was managed-agent and agent-run work, routed through OpenRouter BYOK to Amazon Bedrock.

AWS Billing showed $13,534.04 of gross June 2026 Bedrock marketplace usage, fully offset by credits:

This validates that OpenRouter’s Claude BYOK usage is landing on Bedrock credits. It is still real gross usage, but it is not currently cash out the door while credits remain.

QA eligibility spend

Braintrust showed QA eligibility checks as the best direct OpenAI optimization target:

At OpenAI gpt-5.1 flex pricing of $0.625/M input tokens, $0.0625/M cached input tokens, and $5.00/M output tokens, the same token volume would cost about $815.92/month if all input tokens are treated as uncached:

That is $2,226.76/month lower than the Braintrust standard-pricing estimate, or about 73% savings. The estimate is conservative for the new gpt-5.1 path because it does not assume any cached input discount.

OpenRouter GLM 5.2 was the cheaper-model candidate we tested before correcting the flex math. At GLM 5.2 list pricing of $0.93/M input tokens and $3.00/M output tokens, the same token volume would cost about $1,038.81/month. For this input-heavy eligibility workload, gpt-5.1 flex is about $222.89/month cheaper than GLM 5.2 while keeping the reliability profile on OpenAI. GLM 5.2 remains a discarded option because it was not reliable enough for QA eligibility without more prompt/eval work.

The flex pricing caveat

Braintrust does not appear to account for OpenAI flex service-tier pricing.

The codebase should use flex in background OpenAI paths:

• Rails QA evaluation runner sends service_tier: 'flex'.
• Background QA calls start on flex and immediately retry the same model on auto for narrowly classified capacity failures.
• API2 managed agents use serviceTier: "flex" for OpenAI.
• API2 workflow router and task execution paths should also use background flex provider options.
• API2 QA eligibility pins to gpt-5.1 and sends serviceTier: "flex".

Braintrust spans did not expose service_tier or serviceTier in metadata, span attributes, or inputs. A sampled gpt-5.4 Responses span with 7,655 input tokens and 202 output tokens had a Braintrust estimated cost of $0.0221675, which matches standard list pricing. With a 50% flex discount, that span would be about $0.01108.

That means the true incremental savings depend on how much of the old gpt-5.4 QA eligibility volume was already billed at flex rates.

The practical estimate for QA eligibility alone is $0.7K-$2.2K/month in savings from moving to OpenAI gpt-5.1 flex, depending on how much of the old path was already flex-billed. A midpoint assumption of 50% old flex usage gives about $1.5K/month in savings. Because GLM was both less reliable and more expensive than gpt-5.1 flex for this token mix, GLM should stay behind evals or a canary rather than ship as the first optimization.

Braintrust attribution gap

Braintrust Monitor currently cannot break spend down by Rulebase AI feature for the API2 LLM child spans. In the last check, the Monitor view showed almost all spend under generic doGenerate / doStream, while “Spend by AI category” and “Spend by AI function” were None.

The sampled cost-bearing spans in the rulebase-api2 project had child LLM metadata like model, provider, finish_reason, and Braintrust internals, but not function_id or spend_category. The root generateText call often has experimental_telemetry.functionId and metadata, but Braintrust does not automatically propagate root metadata down to child LLM spans. Because Monitor aggregates spend on the child spans, the root metadata is not enough.

Implementation options:

Do this before the next major spend review. Without child-span attribution, Braintrust is still useful for sampling traces and ranking models, but not for reliable spend-by-feature dashboards.

Decision

Ship OpenAI gpt-5.1 with flex pricing for QA eligibility, not OpenRouter GLM 5.2.

Eligibility is a routing heuristic. It decides whether a conversation should enter the heavier QA evaluation path. It should be cheap, predictable, and decoupled from the organization’s full evaluator model. But the cheaper model still needs to be reliable enough not to distort downstream QA volume.

A local Braintrust eval run for gpt-5.1 reported 100% verdict_matches, 100% reason_does_not_leak_instruction_wrapper, and 0 errors. That makes gpt-5.1 the preferred reliability baseline while still capturing the flex-tier savings.

The current decision:

• Roll back the GLM eligibility change in this branch.
• Pin QA eligibility to OpenAI gpt-5.1 with serviceTier: "flex".
• Set OpenAI flex service tier for API2 background managed-agent, workflow router, and task execution paths.
• Keep OpenAI as the reliability baseline while using flex to reduce cost.
• Revisit cheaper eligibility models only after prompt improvements and evals show stable pass/fail behavior.

Conversation QA migration status

Conversation-level QA evaluation is also a major cost-control lever because it avoids the older per-criterion evaluate_parent_scorecard_criterion_for_agent path. The cheaper path is evaluate_conversation_with_tools, where the model evaluates the conversation with tools and saves criterion results during the loop.

The motivating customer for this was Kuda. On July 1, we checked June Braintrust gpt-5.4 attribution:

The Kuda gpt-5.4 spend was mostly the parent-criterion path:

The architecture explains the cost difference. evaluate_conversation_with_tools builds all (agent, scorecard, leaf_criterion) pairs up front, sends the transcript once, and lets the model save each leaf result through save_agent_criterion_result. The parent-criterion path enqueues one job per root criterion and repeats the transcript, prompt, custom instructions, KB context, and tool loop for each parent criterion. The comparison is not perfectly apples-to-apples because Kuda and Qonto have different scorecards and conversation shapes, but it is directionally strong enough to treat Kuda as the highest-impact full-QA migration candidate.

As of a July 2 re-check before the flag change, Kuda still had recent Rulebase project rows on evaluate_parent_scorecard_criterion_for_agent and no rulebase-api2 conversation_qa_evaluation rows. Later on July 2, Kuda was enabled for both production Flipper flags: qa_evaluation_conversation_runner and api2_conversation_qa_evaluation. A later July 8 re-check found fresh Kuda production traces in the rulebase-api2 project, so the migration is now verified at the trace level.

On July 2, 2026, production Flipper was set to conditionally enable api2_conversation_qa_evaluation and qa_evaluation_conversation_runner for three organization actors:

Braintrust data before this flag change already showed the target orgs using the conversation-level prompt:

That initial query verified the spend-relevant prompt migration, but it did not prove API2 execution. The sampled pre-flag Braintrust rows were in the Rulebase project, not rulebase-api2, and the provider spans did not include functionId: "conversation_qa_evaluation" or runner: "api2_conversation_qa". The post-flag verification below now confirms API2 execution for sampled EasyPay, Rho, and Kuda runs.

Do not count a separate API2 savings line yet. The savings comes from avoiding per-criterion evaluation and using flex-priced OpenAI calls. Moving the runner to API2 is still useful because it centralizes the background AI path and should make flex/provider instrumentation easier to audit.

Post-flag verification

A follow-up Braintrust query on July 2, 2026 verified fresh production runs in the rulebase-api2 project after the Flipper change:

The API2 root spans carried:

• input.experimental_telemetry.functionId = "conversation_qa_evaluation"
• input.experimental_telemetry.metadata.runner = "api2_conversation_qa"
• input.experimental_telemetry.metadata.organizationSlug = "easypay" or "rho-tnkw"

The same time window had no EasyPay/Rho evaluate_conversation_with_tools or evaluate_parent_scorecard_criterion_for_agent rows in the old Rulebase Braintrust project, so the flag appears to be routing these orgs through API2.

Kuda API2 verification and estimate

A later July 8 Braintrust/API2 check found fresh Kuda production traces in the rulebase-api2 project with runner = api2_conversation_qa. This verifies that Kuda’s full QA evaluations are now using API2, not the old Rails parent-criterion path.

The working per-eval estimate uses conversation-level matching, because Kuda almost never evaluates the same conversation twice:

At the observed Kuda volume, this points to roughly $5K/month of savings from the API2 consolidation. Treat that as an operational estimate, not invoice-grade accounting, because Braintrust generic estimates, provider-reported upstream cost, OpenAI flex pricing, OpenRouter routing, and prompt caching do not currently reconcile perfectly.

Post-flag cost sanity check

The first API2 sample is small, but it is useful for checking whether the savings thesis is directionally right.

For comparison, the pre-flag Rulebase project window from July 2, 2026 17:00-21:30 UTC showed Braintrust-estimated evaluate_conversation_with_tools cost of about $13.80 for EasyPay and $12.70 for Rho. Using distinct QA evaluations in that sampled window, that was roughly $0.73/eval for EasyPay and $0.28/eval for Rho.

The best read is that the API2 traces are on track against the cost-saving thesis if we use provider-reported upstream cost: first-sample cost per eval is roughly half the old Braintrust list-price estimate. Do not use Braintrust’s generic API2 estimated_cost for invoice math here; it is much higher than the provider-reported upstream cost and appears to miss the caching/provider economics. Also note that the API2 child spans currently report provider = openrouter, so reconcile this path against OpenRouter/OpenAI billing before treating the trace-level estimate as final cash savings.

Qonto Opus 4.8 experiment

Qonto is now the largest visible OpenAI QA driver in the July data. On July 8, 2026 at about 8:30 PM ET, Qonto was configured to use OpenRouter Claude Opus 4.8 for evaluations. This did not reliably reach Bedrock: the July 16 route audit found that almost all sampled Opus-requested Qonto child spans actually completed on BYOK OpenAI GPT-5.5 because explicit Opus was configured with a GPT-5.5 OpenRouter model fallback.

This is best understood as a cash-preservation experiment while Bedrock credits remain, not as a gross-spend improvement. OpenRouter Claude Opus 4.8 and Opus 4.6 are effectively the same regular price class, and both are much more expensive on gross list pricing than OpenAI flex for the same token volume. If OpenRouter routes this through Bedrock BYOK, the cash bill may be offset by credits, but the gross usage will still burn AWS credits quickly.

PR #8304  removes the cross-model fallback candidates from explicit Opus 4.8 and GPT-5.5 requests. It retains an application-level retry only for a real transient provider exception and logs the actual response model/provider for every conversation QA generation step. Do not count the Qonto cash saving until post-deploy logs and billing show sustained Amazon Bedrock routing.

Important implementation caveat: the API2 conversation QA runner currently maps organization.evaluation_reasoning_effort only into OpenAI reasoningEffort. The OpenRouter provider can pass Claude effort controls, but this path does not yet set openrouter.verbosity or openrouter.reasoning.effort. If Qonto remains on OpenRouter Opus, add that wiring before assuming org-level thinking settings are honored.

Post-change checks to run:

• Confirm Qonto traces land in rulebase-api2, not the old Rulebase Rails project.
• Confirm model id, provider route, and whether traffic lands on Bedrock credits.
• Compare per-eval cost against the pre-change gpt-5.4 flex baseline.
• Check quality and failure rate before expanding Opus to other evaluation workloads.
• Verify whether OpenRouter/Braintrust reports cache reads or whether cache accounting only appears in provider billing.

Bedrock Sonnet 5 caching check

We also checked whether moving EasyPay/Rho conversation QA to Claude Sonnet 5 on Bedrock would be better if prompt caching is enabled.

The key lesson: caching changes Sonnet economics materially, but it only wins if nearly all input tokens are cache reads. Conversation QA has a large amount of fresh per-conversation transcript and evidence, so the reusable portion is likely the rubric, system prompt, criteria schema, and stable tool instructions rather than the whole request.

Using the first post-flag API2 sample:

AWS lists Claude Sonnet 5 promo pricing at $2/M input and $10/M output through August 31, 2026, then $3/M input and $15/M output. Anthropic prompt caching charges cache reads at 10% of base input price and 5-minute cache writes at 1.25x base input price, so cold writes are not free and have to be amortized across repeated calls.

Approximate Sonnet 5 promo costs for the same sample at different input cache-read rates:

The 100% cache-read case is an unrealistic lower bound because each QA eval includes fresh conversation content. At a more plausible 50%-75% cache-read range, Sonnet 5 on Bedrock would still be materially more expensive on gross usage than the current provider-reported upstream cost, even before accounting for cache-write overhead. AWS credits can make this attractive for cash preservation, but it is not a better gross-efficiency path unless measured cacheReadInputTokens is very high.

Recommendation: keep EasyPay/Rho API2 conversation QA on the current GPT/OpenAI route for now. If we want to use Bedrock credits here, first run a canary that records cache write/read tokens and quality metrics, with the static rubric/schema/tool instructions placed before cache breakpoints and the fresh transcript outside the cached segment.

Managed-agent workflow spend

For customer/workflow attribution, api2.workflow_runs and api2.node_executions are more useful than credit_usage_events. They include organization_id, workflow_id, model, token totals, and Braintrust root span ids for all managed-agent runs, not only credit-billed customers.

The caveat is that workflow tables store aggregate input/output tokens, not cached-vs-uncached tokens or OpenAI service tier. Dollar estimates from these tables should be treated as ranking and run-rate estimates, then reconciled against OpenAI, OpenRouter, Bedrock, and Braintrust.

Kuda high-effort workflow migration

Kuda’s High-effort conversation scan was the largest managed-agent workflow opportunity in the earlier 30-day workflow analysis. The old path was a legacy Opus agent loop that scanned recently closed conversations and performed side effects from inside the model/tool loop. Historical estimate:

As of July 2, 2026, the workflow has been migrated to graph workflow mode:

The important cost-control change is architectural: the model now only classifies and returns structured high_effort_conversations. The workflow loop handles tagging, setting High Effort Count, and sending Slack alerts deterministically, so those side effects no longer consume model tool-call context in the main reasoning loop.

Re-check spend after 48-72 hours of production runs. Compare the new workflow’s workflow_runs / node_executions token totals and provider costs against the prior run-rate for the legacy Opus path. Also verify lastRunAt/run history, because the captured workflow export showed status = active but lastRunAt = null at capture time.

Rho workflow cleanup

A workflow-run attribution query initially showed very high 30-day Rho spend, mostly from Opus 4.8 WIP workflows. A follow-up status check showed that those workflows were already paused/deleted on June 23, 2026, with next_run_at = null.

Current Rho state from production:

The scary Rho Opus rows in the 30-day report should drop out of the run rate after June 23. Do not treat them as current ongoing spend. The remaining live Rho lever is CS Escalations, with a smaller follow-up on Treasury mention daily report.

Using the workflow-token estimate and standard pricing, CS Escalations was about $1.6K/month on gpt-5.5 in the 30-day window. That estimate treats all input as uncached and does not account for flex, so actual invoice cost may be lower. With same-model flex, the run-rate should be roughly half before caching effects; moving common cases to gpt-5.1 flex would be a larger savings step but needs a canary.

Language detection

Language detection is not a good Bedrock migration target.

In June, the API2 translation path used OpenAI gpt-5-mini with reasoningEffort: "minimal", structured output, and serviceTier: "flex" for detect_conversation_language. Archived Braintrust analysis showed about 472K-484K June spans, 618M-625M input tokens, 17.4M-17.8M output tokens, and about $189-$191/month of Braintrust-estimated cost. As of August 7, both translation and language detection resolve to OpenRouter gpt-5.6-terra with flex processing, so the June model description is historical rather than the current route.

At the same token volume, Claude Haiku 4.5 on Bedrock would be worse gross economics. Using Haiku 4.5 list pricing of $1/M input and $5/M output, the same June language-detection volume would be about $705/month gross. With AWS credits, moving it would save only about $190/month of cash while burning about $705/month of credits.

Do not move language detection to Haiku/Bedrock. The August 7 canary below supports a cheaper OpenAI Luna replay first, with Gemini 2.5 Flash-Lite as the credit-backed alternative. Deterministic/local detection with an LLM fallback remains the best architectural ceiling if this path is still material after the model downgrade.

Offline QA model canaries

On August 7, 2026, we added an API2 offline replay harness for conversation QA evaluations:

cd rulebase-web/rulebase-api2
INFISICAL_ENV=prod QA_OFFLINE_DRY_RUN=1 pnpm eval:conversation-qa-offline:infisical
INFISICAL_ENV=prod pnpm eval:conversation-qa-offline:infisical

Defaults are tuned for the current downgrade question: Rho (rho-tnkw), Lesaka/EasyPay (easypay), and NALA (nala) are sampled from recent completed AI evaluations and replayed with openrouter/openai/gpt-5.6-luna. The script captures model output locally instead of posting results back to Rails, and it wraps the replay in a database transaction that is intentionally rolled back when using live database mode.

The scores compare Luna output against the already-saved AI evaluation for the same conversation: run completion, normalized check-result match, exact check-result match, and selected justification-option match. This is a practical regression canary for a cost downgrade, not a human-labeled accuracy benchmark. On August 7, the fixture canary passed with 100% normalized check-result match, 97.22% exact match, 100% justification-option match, and 0 errors across Rho, NALA, and Lesaka/EasyPay, so all three orgs were switched from Terra to Luna for production conversation QA.

August 7 alternative-model research

This section records research-only experiments. No production model routing was changed as part of this work.

Qonto conversation QA

The first human-labeled Qonto replay makes DeepSeek V4 Pro the only non-OpenAI candidate worth expanding so far:

At unchanged August 7 volume, the small DeepSeek sample projects to about $56/day, versus $214/day on Sol: approximately 74% lower, or $4.7K per 30 days. Treat this as a screening estimate until at least 12-25 human evaluations complete with near-100% run completion and acceptable agreement. The OpenRouter ZDR route used a more expensive privacy-qualified endpoint than DeepSeek’s headline first-party price, so the estimate already includes the material ZDR premium.

ZDR is not the same as EU data residency. OpenRouter documents EU in-region processing as an enterprise feature on eu.openrouter.ai; a global ZDR route can still process outside the EU. Qonto cannot move until the selected endpoint, DPA, subprocessor list, and regional-processing requirement are approved.

The attempted 12-case human-labeled expansion did run far enough to produce nine successful replays; three cases failed in the database reset/write step rather than in model generation. On the nine completed cases, Sol reached approximately 82.4% normalized agreement / 82.0% exact agreement with the subsequent human evaluation. DeepSeek V4 Pro reached approximately 75.2% normalized / 73.6% exact on the same completed subset. This is a meaningful seven-to-eight-point gap, so DeepSeek is not ready to replace Sol for Qonto even though it remains the strongest low-cost model candidate.

August 8: Qonto runner compaction

We tested a lower-risk optimization that keeps Sol and the current iterative retrieval loop. The research variant only:

1. Sends each scorecard hierarchy once instead of repeating it for every leaf criterion.
2. Returns a compact success payload from save_agent_criterion_result instead of echoing the complete remaining work queue after every save. Validation failures still return full correction context.

A fresh cache-buster was added to both control and comparison runs so earlier fixture replays could not create artificially favorable cache hits. The three fixtures retained 100% normalized agreement, and exact agreement moved from 87.22% to 97.22%. Exact agreement is noisy at this sample size; the important result is that no normalized label regressed.

The fixtures stub feedback and KB tool responses, so this proves the compact queue/save contract rather than full retrieval parity. Qonto also uses strict, detailed evaluation settings. Use 25%-40% as the working Qonto saving range until the same variant passes a larger live human-labeled replay. Applied to the measured $221.31/day Qonto Sol run rate, that is about $1.66K-$2.66K per 30 days; the sample’s 35.7% point estimate is about $2.37K/month.

The compaction experiment did not test changing save_agent_criterion_result to accept an array of agent/criterion results. Both the research variant and PR #9486 preserve one save call per leaf criterion, so the measured 35.7% reduction contains no benefit from batching tool turns. A separate candidate is a partially accepting batch-save contract: accept several results at once, return success or validation errors per item, persist the valid items, and leave invalid items in the remaining queue. Test it independently because rejecting a whole batch for one malformed result would weaken the current incremental recovery behavior. The possible saving is fewer tool-call turns, repeated wrapper fields, and continuation input; the underlying check-result output still has to be generated.

We also replayed the compacted variant on gpt-5.5 with a fresh cache namespace. It retained the same 100% normalized / 97.22% exact fixture agreement and cost $1.1900 across the three fixtures, versus $1.2199 on compacted Sol: only 2.4% less. Eliminating 5.6 cache-write charges did not create a meaningful second saving after prompt compaction converted most repeated context into cache reads. Keep Sol for Qonto unless a larger human-labeled replay demonstrates a quality-neutral reason to move to 5.5.

The same production sample contained 183 zero-save roots, but only 80 incurred model cost; the other 103 failed before recording billable tokens. The 80 paid failures spent $9.82/day before aborting. Of those, 59 roots and about $6.77 were explicitly attributed to OpenRouter/provider overload after the evaluator had already completed its first model turn and fetched feedback/KB context. Six roots failed with invalid JSON responses, two failed while processing a successful response, and 13 had no recorded terminal error despite never saving a criterion. The $295/month figure is therefore a ceiling calculated as $9.82 x 30, not a forecast. The practical fix is concurrency/backpressure plus resumable or idempotent retries that reuse already-fetched research context; initial invalid-request/no-credit failures matter for reliability but were not the source of this spend.

The 12-case human-labeled Qonto sample also exposes a second architecture candidate. Across 16 agent-level human evaluations, phone-identification was not_applicable 13/13 times, call-recording disclosure was not_applicable 14/16 times and passed twice, prohibited-bank terminology passed 13/13, and privacy handling passed 13/13. Do not hardcode those outcomes. Instead, add explicit criterion applicability/routing: use structured channel/call metadata to resolve phone-only criteria deterministically when no call occurred, run lexical or cheap-model screens for bounded terminology checks, and reserve Sol for ambiguous or high-risk cases. Safely bypassing even two of Qonto’s nine leaf criteria could add another 10%-20% saving after compaction; this needs a larger historical human-label analysis before implementation.

August 8: Braintrust/OpenAI reconciliation

For August 1-7, Braintrust grouped API2 LLM spans at approximately $1,007.95 for Sol and $451.47 for Terra. Rails attributed 59,397 generate_customer_issue_summary calls to gpt-5.2, with 444.90M input and 8.58M output tokens. Braintrust priced that Rails pool at $315.04, while the raw OpenAI export puts the matching Flex pool near $169/week. The two-times difference is the known Braintrust service-tier accounting problem, not evidence that the path was non-Flex.

Use Braintrust to identify workload ownership and token shape, then use OpenAI/OpenRouter exports for invoice economics. The reconciliation still identifies the same priorities: Qonto conversation QA dominates Sol, other organizations’ QA dominates much of Terra, and Kuda owns 94.9% of the Rails customer-issue classifier volume.

August 8: Qonto single-agent QA model replay

Qonto’s QA workload is overwhelmingly single-agent: 2,120 of 2,237 evaluations over seven days (94.77%), 4,304 of 4,519 over 14 days (95.24%), and 9,339 of 9,900 over 30 days (94.33%). This makes agent count a simple, stable routing signal without introducing a percentile-based conversation-complexity heuristic.

The first offline replay incorrectly compared pre-persistence model output with saved results after Rails had applied dependent and unconditional check rules. Those checks are deliberately omitted from the model’s work and executed deterministically after the LLM step. The replay scorer now excludes them from model parity while the production runner continues to merge/apply deterministic results normally.

After that correction, GPT-5.6 Terra completed all 20 recent Qonto single-agent cases and matched 81.5% of 200 model-owned check decisions from the saved baseline. On the identical first 11 cases, Terra reached 78.18% verdict parity versus 73.64% for a fresh GPT-5.6 Sol control. Justification-option parity was close (68.18% Terra versus 70.45% Sol). A manual review of the lowest-agreement cases found Terra’s stricter decisions defensible or preferable in several cases, including one caller whose identity did not match the ticket metadata. The saved Sol baseline is therefore noisy; this replay supports Terra as a single-agent route, not an assertion of 81.5% absolute accuracy.

The production routing design is organization complexity plus the number of distinct agents whose criteria still require model evaluation after deterministic pruning:

Rho, EasyPay, and Nala are intended to use low complexity; Kuda medium; and Qonto high. Organizations without a complexity value retain their existing raw evaluation_model, which is also the rollback path. At Qonto’s observed mix, moving single-agent work from Sol to Terra should reduce Qonto QA model spend by approximately 46%-47% while retaining Sol for the roughly 5% multi-agent tail. Verify realized savings and reviewer override rates in Braintrust/provider exports after the production values are enabled.

Kuda CX-risk attribution

The largest remaining direct Rails model pool is not legacy per-criterion QA. Braintrust and the August 1-7 OpenAI exports line up on the generate_customer_issue_summary path:

Across all organizations, the same prompt used 59,397 calls, 444.90M input tokens, 370.57M cached tokens, and 8.58M output tokens. The raw OpenAI export prices the matching gpt-5.2 Flex pool at about $169/week, or $732/month. Kuda is therefore the overwhelming owner of this pool.

The Rails CX-risk path currently makes three model passes over substantially the same ticket:

1. generate_customer_issue_summary on gpt-5.2 Flex: about $169/week.
2. Complaint classification on gpt-5.4-mini Flex: about $50/week at the observed volume.
3. Ticket summary/description on gpt-5-mini Flex: about $90/week across both summary entry points.

There is a concrete duplicate in the third category. ConversationCxRiskSnapshot#check_completion generates ticket_summary and description with conversation_cx_risk_prompt. Separately, every completed QAEvaluationRequest enqueues conversation.summarize_later, which renders the same prompt from the transcript and related conversations. That second path does not include QA results: it passes snapshot: nil, and the existing build_risk_checks_for_summarize helper is not used. The duplicate is intentional call-path overlap, not a GoodJob concurrency race.

The August request counts imply about 138,868 summary calls after subtracting image-caption and speaker-role traffic. Roughly 59,397 align with CX-risk completion and about 79,471 with QA completion. Gating the QA completion summary on a transcript/related-conversation fingerprint, or reusing the fresh CX-risk summary, therefore has an estimated no-quality-loss saving of $51/week / $223/month. Confirm the overlap against production rows before shipping, because Braintrust sampling prevents invoice-perfect call matching.

Combining customer-issue classification, complaint classification, and ticket summary into one structured pass could remove another $400-$700/month, but this is not yet a no-risk change. Complaint recall is safety-relevant and needs a matched production replay. Do not re-enable the commented lexical complaint-keyword bypass without a recall benchmark.

Kuda OSS classifier canary

A research-only harness replayed real Kuda generate_customer_issue_summary prompts from Braintrust. The first fixed sample had 12 incumbent positives and 12 incumbent negatives:

GPT-OSS was expanded to 100 balanced production cases and returned valid structured output on all 100, but agreement fell to 79% for issue existence, 77% for risk, and 74% for type. This is agreement with the incumbent gpt-5.2, not human accuracy. Manual inspection showed both kinds of disagreement: GPT-OSS missed clear failed-transfer and delayed-support issues, while the incumbent incorrectly marked at least one third-party card-refund case as a company issue under its own rubric.

Two additional 50-case balanced replays did not improve the replacement case. DeepSeek V4 Pro completed all 50 but reached only 80% issue-existence, 66% risk, and 70% type agreement. Its OpenRouter ZDR requests were distributed across DigitalOcean, Ionstream, and CoreWeave, which also makes latency and cost less predictable than the headline first-party route. Kimi K2.6 completed 0/50 because the available ZDR route repeatedly returned temporary-unavailability errors from Decart. DeepSeek remains useful for the bounded compliance workloads where it passed the dedicated suite, but neither result supports replacing Kuda’s customer-issue classifier today.

GPT-OSS is therefore not ready as a drop-in Kuda model. It is, however, the best cheap-model distillation candidate found so far. Kuda already produces enough weekly labels to build a reviewed gold set, and Bedrock offers Qwen/GPT-OSS model customization. A small human-labeled fine-tune or confidence-gated cascade is more credible than adding another prompt sentence: a targeted “waiting means service issue” instruction caused GLM to over-flag routine negative cases.

The broader GPT-OSS tests set a clear boundary. It passed the four-case agent-sheet tool suite at 100%, but scored only 41.67% on workflow editing and produced invalid submissions in 6/25 sales-compliance cases. Treat it as a single-turn structured classifier/translator candidate, not as a general managed-agent or compliance-loop replacement.

Bedrock and Azure OSS routes

The earlier Bedrock conclusion was too Anthropic-specific. Bedrock now hosts DeepSeek, Qwen, GPT-OSS, Gemma, MiniMax, Kimi, GLM, Mistral, and Nemotron models, and many support Flex or Batch at a 50% discount. Examples from the August 7 price sheet include Qwen3 Next 80B at $0.15/M input and $1.20/M output standard, GPT-OSS 120B around $0.15/M and $0.60/M on Bedrock, GLM 4.7 Flash at $0.07/M and $0.40/M, and Gemma 4 31B at $0.14/M and $0.40/M in US regions.

At the observed 444.90M input / 8.58M output customer-issue volume, GPT-OSS 120B on Bedrock is roughly $72/week standard or $36/week Flex gross, versus about $169/week for current OpenAI gpt-5.2 Flex. If covered by remaining AWS credits, the near-term cash cost is effectively zero. Quality, not price, is the blocker.

AWS states that Bedrock does not share prompts or outputs with third-party model providers and does not use them for model training. This is materially cleaner for customer data than calling DeepSeek directly. OpenRouter can use Bedrock BYOK, but its public service-tier forwarding currently lists OpenAI and Google rather than Bedrock. Use direct Bedrock invocation when the 50% Flex tier is part of the economics; otherwise OpenRouter BYOK can still consume AWS credits at the served Bedrock tier.

Azure is a credible second credit-backed route. Azure Direct Models include DeepSeek V3.2, Kimi, Mistral, and other third-party weights hosted by Microsoft; Microsoft states that prompts and outputs are not available to the model providers and can be processed within the selected geography. This makes an Azure startup-credit application more useful than merely duplicating direct OpenAI capacity.

AWS also launched OpenAI GPT-5.6 Sol, Terra, and Luna on Bedrock on July 13. This is a new cash-preservation option that did not exist during the earlier Anthropic/Bedrock comparison: it keeps the same OpenAI model quality, pricing matches OpenAI’s first-party standard rates, and usage can consume AWS commitments/credits. The current Bedrock Mantle endpoint only offers on-demand Standard inference for these models, not OpenAI Flex, so gross cost is roughly twice the equivalent direct OpenAI Flex route before cache-shape differences. Use this only as a deliberate credit-to-cash conversion, not as a gross-efficiency optimization. With approximately $45K of estimated AWS credits left, moving all current AI volume there would exhaust the balance quickly and compete with core infrastructure; first ask the AWS account team about the newer invite-only $200K+ AI-startup credit tier.

Cross-workload canaries

We ran the same 25-case sales-call compliance suite and several existing API2 tool-use suites across lower-cost models:

DeepSeek V4 Pro also scored 100% on the four-case agent-sheet tool suite and both workflow-edit cases. It scored 95% on work-item import planning and 96% on compliance-population tool use, each with one regression. It is therefore promising for bounded classification/compliance agents, but it is not yet a blanket managed-agent replacement. DeepSeek V4 Flash, MiniMax M3, and GLM 5.2 all failed the current multi-step QA/tool contract badly enough to reject as drop-in replacements.

Language detection and translation

A small structured-output canary used 12 language-detection cases and 6 translation cases:

This is only a synthetic canary, not a production-language benchmark. Still, Gemini 2.5 Flash-Lite deserves a historical replay because its public price is $0.10/M input and $0.40/M output before credits. At the previously observed roughly $190/month language-detection run rate, the likely cash-saving ceiling is only in the low hundreds per month, so this is a secondary optimization behind Qonto QA.

Feedback-store conclusion

The Qonto discrepancy was not demonstrated to be caused primarily by missing feedback. In one matched human-labeled conversation, limiting feedback to the top two items produced 9/13 exact checks (69.23%), while allowing the top eight produced 10/13 (76.92%); tool calls barely changed (48 versus 49). That one-check improvement proves feedback depth can matter, but one conversation cannot establish it as the dominant cause. A separate one-shot structured run with the full queue and no feedback reached 75%, while a compact version reached 69.2%.

The evidence points to the broader architecture change as the larger risk: the one-shot design removed the production-style iterative retrieval, per-criterion validation, correction, and incremental save loop at the same time that it changed feedback availability. Keep the feedback store for now. The next architecture experiment should retrieve KB and scoped feedback once, build a compact evidence packet, score bounded criterion shards, validate deterministically, and then save. Compare that against the current runner on the same human-labeled Qonto set before removing any source of calibration context.

Closed PR #8112  contains a useful longer-term direction: after a manual evaluation, diff the human and AI check results and propose reviewed updates to check instructions, justification options, KB documents, or organization evaluation instructions. That should be treated as a way to distill recurring feedback into stable calibration context, not as evidence that feedback can be discarded. A practical hybrid would keep raw feedback as the source of truth, promote repeated approved corrections into the scorecard/KB prompt, and call raw feedback search only for novel or low-confidence criteria. This could remove many repeated retrieval/tool turns without losing the information that currently improves Qonto agreement.

Working incremental savings ranking

The lowest-risk path to another 20% total reduction is now a stack rather than one model swap: compact Qonto while retaining Sol, remove incomplete-run waste, gate duplicate Kuda summaries, and stop unchanged suggested-agent regeneration. A 50% gross reduction at similar accuracy is not supported by the current evidence; reaching that level would require a successful Qonto model cascade/replacement or a larger redesign of evaluation applicability and evidence retrieval.

Credit programs to pursue

Credits should be tracked separately from gross efficiency. Google credits can make Gemini nearly free in cash terms while active, but a model should still pass the same eval and privacy gates so the system does not become uneconomic when credits expire.

Recommended outreach order: ask every existing investor, including YC contacts, for an OpenAI startup referral code first; ask the AWS account team for incremental AI credits; apply directly to Google for Startups AI and Microsoft for Startups; then pursue Cloudflare/NVIDIA partner credits. Azure is now worth parallel pursuit because its credits cover Azure Direct third-party models with Microsoft-hosted privacy controls, not just Azure OpenAI. Do not self-host frontier MoE models at the current company size; hosted Bedrock/Azure/OpenRouter endpoints already provide most of the economic upside without GPU utilization risk.

Follow-up checks

After the flex changes are live, review the first full week of data:

• Compare OpenAI dashboard spend, OpenRouter usage, Bedrock gross usage, and Braintrust traces rather than relying on Braintrust cost estimates alone.
• Use api2.workflow_runs and api2.node_executions to rank active customer/workflow spend.
• Re-check Kuda High-effort conversation scan spend after 48-72 hours on the workflow version. Confirm runs are firing, side-effect nodes are deterministic, and token spend dropped versus the legacy Opus run-rate.
• Filter active workflow analysis on workflows.deleted_at is null, active status, and next_run_at where scheduled-run rate matters.
• Continue checking api2_conversation_qa_evaluation for Organization;38, Organization;19, and Organization;25; the trace-level migration is verified, but the next check should compare a full week of invoice/provider data.
• Re-check Qonto after the July 8 Opus 4.8 switch. Compare quality, cost/eval, provider route, cache behavior, and Bedrock credit burn against the prior OpenAI flex baseline.
• After PR #8304 deploys, verify Qonto QA completion logs show anthropic/claude-opus-4.8 on Amazon Bedrock, then compare daily OpenAI GPT-5.5 spend and AWS credit burn for at least 48-72 hours.
• If Qonto remains on OpenRouter Opus, wire organization.evaluation_reasoning_effort into OpenRouter effort controls with openrouter.verbosity or openrouter.reasoning.effort.
• Compare Rho CS Escalations quality, retry behavior, and token usage before downgrading from gpt-5.5.
• Re-check why Rho still shows gpt-5.5 agent-run spend if the intended route was Sonnet/Bedrock; verify deployed model config and OpenRouter provider routing.
• Re-test cheaper QA eligibility models only with eval coverage for structured output quality and downstream QA volume.
• Add explicit service_tier metadata to OpenAI traces where possible so future spend reviews can separate flex and standard-tier usage.
• Add child LLM span attribution for function_id and spend_category so Braintrust Monitor can show spend by Rulebase AI feature.
• Persist OpenRouter generation IDs alongside Braintrust root span IDs, function/category metadata, org, workflow, agent, conversation, and QA evaluation IDs so future AWS/OpenRouter/Braintrust reconciliation is exact rather than sampled.
• Investigate why Kuda conversation_language_detection created about 295M prompt cache-write tokens in a 37-hour Braintrust sample, and move language detection back toward OpenAI flex or deterministic/local detection if the Sonnet route is still active.
• Re-run OpenAI vector-store and file-search storage exports after the August 7 mass delete. Expected outcome: vector-store storage should fall from roughly 460 GiB/day toward zero once OpenAI’s usage export catches up. If it does not, look for another project-scoped key or hidden store owner before assuming the cleanup worked.
• After the code migration away from OpenAI vector stores lands, check for any newly created stores or raw user_data files from the interim window and delete or expire them. The August 7 cleanup removed the old raw file inventory of 177,624 files / 15.350 GiB, mostly framework and knowledge-base artifacts.

Other spend notes

Green Desk was canceled for the month after the June charge had already landed. That should reduce future monthly spend by about $1,141/month starting after June 2026, assuming the service stays canceled.

The next-largest cash spend outside AI in the June card export was Greptile, Vercel, ElevenLabs, Blacksmith, Flightcontrol, Sentry, Braintrust, and Better Stack. Those are worth reviewing separately, but they were smaller than direct OpenAI spend and did not have the same immediate model-routing lever.

Better Stack has already been reduced: services were disabled and responders were removed except for Chidi, while preserving status pages. Treat observability as mostly handled for now unless the next card export still shows material spend.