Production LLM applications rarely receive a question nobody has asked before. Support assistants and RAG pipelines field the same intents thousands of times a day, each phrased differently, and most stacks treat every phrasing as a fresh, fully billed request. Redis LangCache is a fully managed semantic caching service that sits between the application and the model, matches incoming prompts against previously answered ones by meaning rather than exact text, and returns the stored response when a close enough match exists. Redis reports API cost savings of up to 90% and cache-hit responses up to 15x faster than re-querying the model.
Is it deployable? Yes. LangCache is available today as a public preview on Redis Cloud, accessed through a REST API with Python and JavaScript SDKs, and Redis notes that features and behavior may change during the preview.
The Problem: Paraphrases Are Still Full LLM Calls
Consider three requests to a customer-support assistant:
- “Can I get a refund after buying the monthly plan?”
- “Is the monthly subscription refundable?”
- “Can I cancel the plan and get my money back?”
The wording differs, but the question and answer are identical. Without a semantic cache, each version triggers a complete generation: input tokens processed, output tokens decoded, user waiting.
Prefix caching only removes part of that cost. When requests share a system prompt or context, the engine reuses the KV states computed for that prefix, but the request still reaches the LLM, new tokens still get processed, and the full answer still gets decoded. A prefix-cache hit is a cheaper generation call, not an avoided one.
How LangCache Works
LangCache moves the cache outside the model and stores the generated response itself. The architecture is a two-call loop:
- Before invoking the model, the app sends the prompt to
POST /v1/caches/{cacheId}/entries/search. - LangCache generates an embedding for the prompt and runs a vector search over stored entries.
- If a semantically similar entry clears the configured similarity threshold, the cached response is returned and no LLM call occurs.
- On a miss, the app calls its chosen LLM as usual, then stores the prompt and new response through
POST /v1/caches/{cacheId}/entriesfor future matches.
Embedding generation is handled by the service, with default models or bring-your-own. Cache behavior is controlled through similarity thresholds, TTLs, and eviction policies, plus adaptive controls that tune precision and recall. Built on Redis’s vector database and exposed as a REST API, it works with any LLM provider and language. Hit rates and savings are monitored from the Redis Cloud console.
