Why Token Costs Are the New Bottleneck
When you run AI at scale, the bill is not per server. It is per token. Every character you send, every word you get back, and every tool result you append goes into the context window and gets billed. As agents move from single-turn chat to multi-step workflows, token consumption becomes the primary architectural constraint.
The good news is that token costs are one of the most optimizable parts of an AI system. The bad news is that most teams stop at “use a cheaper model.” That is only the last layer. The real savings come from prompt hygiene, context compression, structured outputs, model routing, and caching.
This post is about how to stack those layers.
Layer 1: Prompt Hygiene and Data Format
Before you optimize inference, optimize the prompt itself. Every token you never send is a token you never pay for.
Ditch JSON for Structured Data
JSON is the default for APIs, but it is terrible for LLM context. Repeating object keys, braces, and quotes add no semantic value. For uniform arrays, a tabular or delimited format is far cheaper.
| Format | Token Overhead | Best For |
|---|---|---|
| JSON | Baseline | Backend persistence, APIs |
| YAML | Slightly lower | Configuration, simple structures |
| CSV | Much lower | Flat, tabular data |
| TOON | 30–60% lower | RAG payloads, reference arrays, agent arrays |
TOON (Token-Oriented Object Notation) extracts repeated keys into a header and separates values into flat rows. Studies show it can cut token usage for uniform arrays by 40% while sometimes improving accuracy by reducing noise.
Example: JSON vs. TOON
[
{"name": "Alice", "role": "engineer", "team": "platform"},
{"name": "Bob", "role": "designer", "team": "product"}
]name|role|team
Alice|engineer|platform
Bob|designer|productThe second version carries the same information in a fraction of the tokens. Keep JSON for your database, but convert to TOON just before the LLM call.
Be Direct, Not Polite
Instruction-tuned models are trained to be helpful and explanatory. Politeness increases token count. For automated pipelines, use a terse persona.
You are a terse assistant. Answer in fragments. Skip preamble.
Do not explain unless asked. Prioritize raw results.Research on forced brevity shows reductions in output tokens of 20–65% while maintaining accuracy on technical tasks. Use this only for machine-readable pipelines, not user-facing chat.
Layer 2: Constrain the Output
Output tokens are usually billed at 3–5x the rate of input tokens. Wasted output is expensive output.
Use Structured Outputs
Instead of asking the model to return JSON and hoping it complies, supply a JSON Schema. Modern inference engines compile the schema into a finite-state machine that masks invalid tokens at each step. This:
- Eliminates conversational filler
- Prevents malformed JSON
- Removes the need for retry loops
- Cuts output tokens by enforcing the exact shape you need
{
"type": "object",
"properties": {
"summary": { "type": "string", "maxLength": 100 },
"action_items": {
"type": "array",
"items": { "type": "string", "maxLength": 50 }
}
},
"required": ["summary", "action_items"]
}Cap Output Length
Always set max_tokens or max_output_tokens to the smallest value that can still satisfy the task. A 200-word summary should not have a 4,000-token budget. Reasoning models are especially dangerous here because they emit hidden “thinking tokens” billed at output rates. On identical hard problems, one model may use 500 thinking tokens while another uses 11,000.
Layer 3: Compress Multi-Turn Context
Agents are stateless. Every turn, the entire conversation history is sent back to the model. By turn ten, the context window is full of stale tool outputs the agent will never reference again.
Observation Masking
Instead of summarizing old turns, mask them. Replace large, stale tool outputs with a placeholder once the agent has moved on.
[tool output hidden: 10,546 chars]This is lossless for the current step because the agent already consumed that output. Studies show observation masking cuts input tokens by 40–60% and total cost by 20–35% without reducing success rates.
Filter CLI Output
For coding agents, utilities like RTK intercept ls, cat, git diff, and pytest output, remove boilerplate whitespace, aggregate similar errors, and truncate noisy logs before passing them to the model. This can reduce token footprint by 60–90% on common dev tasks.
Algorithmic Compression
For contexts that must be retained, tools like LLMLingua use a small encoder to estimate token importance and drop low-information tokens. It can compress prompts 3–20x with minimal accuracy loss on retrieval tasks. But be careful: compression can break reasoning on math or complex logic. Use it for context, not for problems that require step-by-step deduction.
Layer 4: Route by Complexity
Not every query needs a frontier model. Simple extraction, classification, and summarization can run on smaller models at 50–100x lower cost.
Routing Approaches
| Approach | Cost | Accuracy | Best For |
|---|---|---|---|
| Rule-based | Near-zero | Low | Obvious categories, keyword detection |
| Embedding-based | Low | Medium | Domain-specific semantic intent |
| Classifier-based | Low | High | Production routing at scale |
Frameworks like RouteLLM use preference data to train a router that predicts the optimal model for each query. In benchmarks, well-calibrated routers can send 85% of traffic to smaller models while preserving 95% of frontier performance.
Start Simple
You do not need a learned router to start. A simple heuristic works:
if "explain" in prompt or "design" in prompt:
return "claude-opus"
if "extract" in prompt or "classify" in prompt:
return "claude-haiku"
return "claude-sonnet"Then measure quality per task and refine the rules. The point is to stop sending every request to the most expensive model.
Layer 5: Cache Everything Static
AI workloads are repetitive. Caching is the highest-ROI optimization once the prompt is clean.
Semantic Caching
Store responses by embedding the query. If a new request is semantically similar to a cached one, return the cached answer. This is ideal for support bots, RAG reranking, and FAQ-style queries. Enterprise deployments report hit rates up to 68%, cutting total spend by two-thirds.
Prompt Prefix Caching
For requests that share a large static prefix — system prompts, tool definitions, RAG documents — providers can cache the KV cache for that prefix. The prefix must be byte-identical, so front-load static content and append dynamic inputs at the end.
| Provider | Write Cost | Read Discount | TTL |
|---|---|---|---|
| Anthropic | 125–200% of base input | 90% off | 5 min to 1 hour |
| OpenAI | Free | 50–75% off | Up to 24 hours |
| Storage hourly rate | 75% off | Configurable |
The Anthropic model is punishing for infrequent cron jobs because you pay a write surcharge every time. It is excellent for high-frequency agent loops.
Layer 6: Batch and Async
Not every task needs a synchronous response. For bulk processing, use the provider’s Batch API. It gives a flat 50% discount on input and output tokens. Stack it with prompt caching and you can pay roughly 5% of the standard synchronous rate for large cached workloads.
What This Looks Like in Practice
At Google, I saw cost optimization treated as a first-class engineering problem. The same is true for LLM inference today. A production pipeline might look like this:
- Convert reference data to TOON before the LLM call.
- Apply observation masking to the agent history.
- Route simple queries to a small model.
- Cache the system prompt and static documents.
- Enforce a JSON Schema for output.
- Run bulk jobs through the Batch API.
Each layer compounds. One layer alone might save 10–20%. All of them together can save 70–90%.
Frequently Asked Questions
How much can I realistically save?
Most production teams can cut 50–80% by stacking the techniques in this post. The exact number depends on your workload mix, model choice, and how repetitive your prompts are.
Should I always use the cheapest model?
No. Use the cheapest model that can reliably solve the task. Routing wrong queries to a weak model creates rework, which costs more than running the right model once.
Does compression hurt accuracy?
It depends on the technique. Observation masking and format changes generally preserve accuracy. Algorithmic compression can hurt reasoning and math tasks. Test before shipping.
What is the easiest win?
Set output limits and use structured outputs. These require no infrastructure changes and immediately reduce the most expensive tokens.
Is caching worth the complexity?
Yes, for repetitive workloads. Semantic caching is high effort for low-traffic systems. Prompt prefix caching is low effort for any system with large static prompts.
References & Further Reading
- LLMLingua — prompt compression using token importance
- RouteLLM — open-source cost-aware model routing
- Anthropic Prompt Caching — official docs on Claude prompt caching
- OpenAI Structured Outputs — schema-based constrained decoding
- TOON Format — reducing token usage with TOON
What to Read Next
Token optimization is a foundation, not a one-time fix. The more automated your AI workflows become, the more these costs compound. The next step is designing those workflows so they do not waste context in the first place.
- 📖 Related: How to Save on AI Tokens: A Practical Guide for Engineers
- 📬 Newsletter: Get AI engineering posts like this weekly → Subscribe free
- 📥 Resource: The AI Mastery Roadmap for Software Engineers