Where Your LLM Token Bill Actually Comes From

Short answer: an agent's LLM token bill has five line items. The system prompt, the tool schemas, the conversation history, the context it re-establishes because it forgot, and the retries. Most cost-cutting advice attacks the first item, which is usually the smallest. The bill lives in the other four, and two of them grow structurally as your setup grows.

Let's itemize it like an invoice, because once you see the slices, the fixes pick themselves.

The five line items

The system prompt. Your instructions, persona, and rules. A few hundred to a few thousand tokens, sent on every request. It is the item people optimize first because it is the one they can see. Trimming it helps, a little, once.

Tool schemas. Every connected MCP server ships definitions for its tools, and the model holds all of them in context to know what it can call. One schema runs roughly 200 to 500 tokens. This is a standing charge: it is paid on every turn whether or not any tool gets called, and it grows with every server you connect. We call this the context tax, and it earned [its own article](/blog/the-context-tax).

Conversation history. Agent loops re-send the transcript on every turn. Turn twenty carries turns one through nineteen, including every tool result along the way. This is why long agent runs cost quadratically more than their length suggests: each new turn pays for all the old ones again.

Re-established context. The quiet one. Every new session, the agent rebuilds what it knew yesterday: re-reading files, re-fetching records, re-asking you for the project background. None of that produces new work. It reproduces old knowledge, at full price, because the previous session's understanding was wiped.

Retries and dead ends. A failed tool call, a wrong path, a malformed output that needs a second attempt. Each one replays a full context load. Cheap individually, real in aggregate.

A worked example, so the shape is visible

Take a modest agent session: a 1,500-token system prompt, six connected servers exposing 90 tools at 350 tokens per schema (31,500 tokens), and 20 turns of work.

The schemas alone contribute 31,500 tokens to the input of every turn. Over 20 turns, that is 630,000 input tokens spent on tool definitions, before counting the actual conversation. Add a history that grows by, say, 1,000 tokens per turn, and turns 1 through 20 carry an average of about 10,000 tokens of transcript each: another 200,000. The system prompt, re-sent 20 times: 30,000.

Now the number nobody budgets: if the first ten minutes of the session were the agent re-reading the repo and re-deriving decisions from last week, everything in that stretch (the fetches, the summaries, the back-and-forth) was re-established context. In long-running projects that stretch is routinely the biggest slice of the session, and it repeats every session.

Prompt caching, where available, discounts repeated prefixes like schemas and system prompts, and it genuinely helps. But cached tokens still occupy the context window, the discount vanishes whenever the tool list changes, and caching does nothing for re-established context, because that content is newly generated every time.

Which slices grow, and which fixes actually move the bill

The system prompt is flat. History grows with session length. But two items grow with your setup itself, and those are the structural ones:

• Tool schemas grow with every connected server. The fix is loading less per turn: connect fewer servers per project, trim what loads, or put a gateway in front that federates servers and serves tools on demand. The tactics are collected in [how to reduce MCP token usage](/blog/reduce-mcp-token-usage).
• Re-established context grows with how much your agent is allowed to forget. The fix is memory: store decisions, IDs, and project facts once, recall them next session, and start from a distilled summary instead of a full rebuild. Why forgetting maps directly onto spend is the argument of [our essay on agents that keep forgetting](/blog/agents-keep-forgetting-token-bill-pays-for-it).

We build a gateway with a memory layer at Tulimoa, so we are not neutral here, and we will also not hand you an invented percentage. What we can hand you is the audit method: your own bill, itemized against these five lines, tells you within an hour which slice is eating you.

Frequently asked questions

What uses the most tokens in an agent setup?

It depends on the setup, but the two chronically underestimated items are tool schemas (a standing per-turn charge that scales with connected servers) and re-established context (rebuilding yesterday's knowledge every session). Conversation history dominates only in very long single sessions.

Does prompt caching solve the token cost problem?

It softens it. Caching discounts repeated prefixes like schemas and system prompts, but cached content still fills the context window, cache breaks when the tool list changes, and freshly re-derived context is never cached. Structural fixes still matter.

How do I find out where my own token bill goes?

Log one representative session and tag every input chunk as one of the five items: system prompt, schemas, history, re-established context, retries. Most providers show per-request token counts, so the arithmetic is an afternoon, and the result usually surprises the team.