Blog/launchweek

# Semantic caching

Faster, cheaper LLM API calls

Written by

Dom Eccleston

Published on

Jun 26, 2024

Large language models are getting faster and cheaper. The below charts show progress in OpenAI's GPT family of models over the past year:

Cost per million tokens ($)

Tokens per second

Recent releases like Meta's Llama 3 and Gemini Flash have pushed the cost / speed frontier further:

Below data is for May 2024

As cost and latency have decreased, more complex LLM workflows have become increasingly viable, which in turn bring their own set of challenges.

Since these workflows often involve multiple steps and retries, taking measures to reduce cost and latency remains helpful. One way to do so is via semantic caching.

## Example use case: RAG

Say that you want to build a RAG (retrieval-augmented generation) chatbot for customer support. It can respond to user queries and suggest articles from your support center.

This workflow would involve multiple API calls:

1. Generate embedding of user query
2. Search vector DB for relevant embeddings and append to prompt
3. Query LLM API with enhanced prompt

Through caching, you could avoid doing this work except for when users ask questions that haven't been asked before - otherwise, re-use existing work.

## Caching

A simple caching solution would be to cache results using the user query as the key, and the result of the workflow as the value.

This would enable re-use of responses between identical queries. But it would rely on user queries being phrased identically.

For instance, if two users asked "How do I cancel my subscription" then the second would be served the cached response. But if another user was to ask "I need to cancel my subscription - how?" then we'd see a cache miss. The question is the same in terms of its intent, but the phrasing is different.

This is how semantic caching can be useful: through caching based on the embedding of the query, we can ensure that all users who ask the same question receive a cache hit. The below diagrams give an overview of the architecture:

## Try it now

In response to feedback from our AI customers, we're offering semantic caching now as part of Unkey. You can enable it now through signing up and changing the baseUrl parameter of the OpenAI SDK:

    1const openai = new OpenAI({
    2  apiKey: process.env.OPENAI_API_KEY,
    3  baseURL: "https://<gateway>.llm.unkey.io", // change the baseUrl parameter to your gateway name
    4});

Unkey's semantic caching is free, supports streaming, and comes with built-in observability and analytics. In future we will offer deeper integration with Unkey's API keys for enhanced security and performance.

If you'd like to read more about how it works, check out our documentation.

Written by

Dom Eccleston

Published on

Jun 26, 2024

Contents

- Example use case: RAG
- Caching
- Try it now

Suggested

Vncod — Hành trình định nghĩa lại tiêu chuẩn vận hành Game

Mar 26, 2026

How to ratelimit public pages

Dec 11, 2025

Building a flow editor

Dec 02, 2025

## Sẵn sàng bứt phá doanh thu.
Bắt đầu ngay hôm nay.

Liên hệ với chúng tôi
Server Game

Hỗ trợ cài đặt trọn gói. Cam kết vận hành ổn định 24/7.

We value your privacy
This site uses cookies to improve your browsing experience, analyze site traffic, and show personalized content.