01 · Introduction
What a cache is
A cache is a stored copy of something that was expensive to produce, kept so it does not have to be produced again.
Your browser caches images so a page loads faster the second time. A website caches a report so a thousand visitors do not each trigger the same database query.
With AI models there are two kinds you will meet:
- Response cache: you save the model's answer to a question. If the exact same question comes again, you return the saved answer and skip the model entirely.
- Prompt cache: the provider keeps the model's reading of the start of your prompt. If your next request starts with the same text, that part is not processed again. You pay a small fraction for it and the answer comes back sooner.
Prompt caching is the one that matters for agents and automations, because they send the same long instructions and tools on every call.
02 · When to use it
When caching pays off
Prompt caching helps when the beginning of your prompt is long and repeats:
- A long system prompt with rules and examples, reused for every user.
- A big document (a manual, a contract, a code base) that you ask many questions about.
- A list of tools an agent carries into every step of a long task.
- A chat that keeps growing: each turn resends the whole history, and the history is the repeated part.
It does not help when every request is short and different, or when the repeated part sits at the end of the prompt.
A response cache helps for FAQ-style traffic: the same few questions asked again and again, where a slightly stale answer is acceptable.
03 · Setup
Turning it on
In chat apps and agent tools you usually do nothing. Claude Code, for example, caches automatically.
With the API you mark where the stable part of the prompt ends. In the Claude API this is a cache_control marker on a block of the system prompt or a message:
{
"system": [
{ "type": "text", "text": "...long rules and documents...",
"cache_control": { "type": "ephemeral" } }
],
"messages": [{ "role": "user", "content": "Today's question" }]
}
The first call writes the cache (it costs a little more than normal). Later calls that start with the same text read it, at roughly a tenth of the normal input price. The entry lives for a few minutes after its last use, and a longer option exists for traffic that comes in bursts. Very short prompts are not cached at all; the provider's docs list the minimum.
Response cache is just a lookup table in your own app: key = the question, value = the answer, with an expiry date.
04 · How to use it
Day-to-day use
- Put what never changes first (rules, tools, documents) and what changes every time last (the user's question, today's date).
- Check the numbers. The API reply reports how many tokens were read from cache. If it is zero on the second call, something in the prefix changed.
- Common culprits: a timestamp in the system prompt, a random ID, tools listed in a different order.
- One marker at the end of the stable part is enough for most apps.
- For response caches, set an expiry and include the model name in the key, so an old answer is not served after you switch models.
05 · Pitfalls
Pitfalls
- Any change breaks the match. One extra space before the marker and the whole cache is missed, with no error shown.
- Writes cost extra. Caching a prompt that is used once costs more than not caching it.
- It expires. A cache that is not used for a few minutes is gone. Bursty, once-an-hour jobs may never hit it.
- Stale answers. A response cache will happily return last week's price. Cache only what may be stale.
- Personal data. Never serve one user's cached answer to another user. Include the user or account in the key.
06 · Resources
Where to go next
Caching is one of three levers on cost: what you send (context), what each piece costs (tokens), and whether you pay for it again (cache).