Modern LLMs will 'cache' prompts on the server side. It works like this:

  • The agent sends a prompt
  • The LLM server front end may parse the prompt into an internal representation
  • The text is tokenized
  • If the LLM or hosting platform supports cache checkpoints, it will identify and isolate the prefix of the prompt that is eligible for caching.
  • At this point the LLM has a “cacheable prefix”.
  • The cache layer creates a “cache key”, typically a hash of the contents of that cacheable prefix (plus model ID and some options).
  • The cache layer looks up this key in its cache store to see if there is a match.
    • If a match is not found, the prompt is processed normally to produce a KV prefill for that prefix; the resulting KV snapshot is then added to the cache before decoding continues.
    • If a match is found, the cache layer fetches the stored KV snapshot. It can then skip recomputing the transformer layers for the prefix and directly load the prefetched KV data.transformer layers for the prefix and directly load the prefilled KV data.

What this does is potentially speed up inference and reduce LLM load. The user benefits because, on many hosted LLMs, cache hits have a lower effective cost per token than uncached input. It’s up to the agent (and its developer) to structure prompts so there is a “static header” at the front of prompts in a particular session.

There are two broad classes of implementations you’ll see in the wild:

  1. Automatic prefix caching (no explicit hints from the agent)
  2. Systems like vLLM with enable_prefix_caching=True, or similar engines, automatically store KV blocks for shared prefixes across requests.

  3. Internally, they detect shared prefixes between the new prompt and previously seen prompts (for example via block-level hashes or token-level radix trees); once the prefix diverges, reuse stops and normal computation resumes.

  4. APIs with explicit cache checkpoints / controls

  5. Some hosted platforms (e.g., Bedrock prompt caching, certain managed LLM APIs) let you mark cache checkpoints or specify “cache this part of the prompt” via API parameters or message attributes.
  6. On the first request, everything up to a checkpoint is computed and saved as a reusable internal state snapshot (KV tensors for those tokens).
  7. On later requests, if the prefix up to that checkpoint is byte-for-byte (token-for-token) identical, the engine reloads that cached state instead of recomputing it.

Further Reading

https://redis.io/blog/what-is-prompt-caching/ https://bentoml.com/llm/inference-optimization/prefix-caching https://medium.com/tr-labs-ml-engineering-blog/prompt-caching-the-secret-to-60-cost-reduction-in-llm-applications-6c792a0ac29b https://caylent.com/blog/prompt-caching-saving-time-and-money-in-llm-applications https://www.digitalocean.com/community/tutorials/prompt-caching-explained https://ngrok.com/blog/prompt-caching https://www.linkedin.com/pulse/prompt-caching-mechanics-guarantees-failure-modes-sanjay-basu-phd-iyiqf https://www.youtube.com/watch?v=u57EnkQaUTY https://community.openai.com/t/how-prompt-caching-works/970979 https://sankalp.bearblog.dev/how-prompt-caching-works/ https://developer.furiosa.ai/latest/en/furiosa_llm/prefix-caching.html https://oneuptime.com/blog/post/2026-01-30-llm-prompt-caching/view https://www.datacamp.com/tutorial/prompt-caching https://medium.com/data-science/prompt-caching-in-llms-intuition-5cfc151c4420 https://humanloop.com/blog/prompt-caching