We sell the layer between your AI and your tools. Our own benchmark says you don't always need it

We measured it on our own company: seven real questions, HubSpot, Linear and Slack plugged in directly against our company brain, and against the brain limited to one call. 84 runs. Full setup at the end.

One tool: direct wins

On the four one-tool questions, direct was cheaper or level on three. Median $0.076 per question against $0.098 through the brain, about 22% less. The assistant calls the tool, gets the record and answers. A layer in the middle only adds its own overhead.

Two or three tools: cost multiplies

Same chart, middle group. "Customer status" cost $0.35 direct against $0.15 through the brain (2.3x). "Pricing commitments", $0.27 against $0.10 (2.7x). The meeting question cost the most of all, but direct had no meeting source to search, so it's not a fair comparison: the assistant kept searching for something that wasn't there.

The reason is in how an agent loop works. At every step the model rereads the whole context: system prompt, tool definitions, and every tool output so far. A Slack search returns whole threads (20k characters in one call here), HubSpot returns full records, and all of it stays in context for the rest of the question.

Each bar is one model call, and its height is what the model had to read at that step. The total cost is roughly the sum of the bars. This run is a good one for direct. In the other repetition of the same question it took 13 calls and read 862k tokens.

Long conversations: it compounds

Inside one conversation, the context of the previous questions comes along too. Median cost per question: $0.32 direct, $0.13 brain, $0.05 one call. That's 2.5x and 6.6x. Direct was the most expensive of the three on six of the seven questions, including three of the one-tool questions that were cheap in a fresh session.

The whole chain: 48 model calls direct against 17 with one call, with the context climbing to 358k tokens. In tokens read that's 12x (10.4M against 853k). In money it's 8x ($3.23 against $0.40), because most of those tokens are cache reads, billed at a tenth of normal input.

Limits

• └Long conversation, cost, direct vs brain: holds statistically (p=0.047).
• └Long conversation, cost, direct vs one call: holds statistically (p=0.016).
• └Long conversation, model calls (9 vs 4 per question): doesn't hold (p=0.09).
• └Fresh session, cost (brain ~19% cheaper): doesn't hold (p=0.38).

Seven questions is a small sample. Where p is above 0.05, only read the direction. It's also our own company and our own product, and another setup will give other numbers. The shape is what I'd expect to travel: one-tool questions don't need a layer, cross-tool questions and long sessions are where raw tool output eats the budget.

Setup

Seven questions from our work: four answered in one tool, two across CRM, tickets and Slack, one only in meeting notes. Three setups on Claude Sonnet 5: direct (HubSpot, Linear and Slack MCP servers, read-only), brain (fed by those same tools), and brain limited to one call per question. Each question in a fresh session and all seven in one conversation, two repetitions, 84 runs. Protocol written before the first run.

Early rounds were contaminated by my own Claude Code setup (settings hooks, organisation-level instructions), so we now run on an API key with user settings off and a smoke test before each round.