Semantic Caching and Live Internet Search: Tools That Let AI Cache Answers to Save Money, and Pull Data From the Web When It Needs To

Say you open the logs of your chat system and they are full of questions like these:

• "How do I change my password?"
• "Forgot my password, what do I do?"
• "Where do I reset my password?"

All three mean the same thing. But the system sent each one to the AI, had it write a fresh answer three times, and paid for the tokens three times. Multiply that by thousands or tens of thousands of users, and the API bill gets... sigh.

At the same time, there is another kind of question the AI cannot answer from memory at all: "What's the gold price today?", "Is it going to rain today?", or "What promotions are on this month?" These answers change every day, and the data a model was trained on stops at a certain date, its knowledge cutoff. Anything new after that date, the model simply does not know.

This article covers two techniques, one for each side. Semantic caching lets an AI remember answers to questions that keep coming back with the same meaning, so you stop paying for them again. Live internet search lets the AI pull fresh data from the web, or a search engine, when a question needs it. And we will look at how to decide which questions should be "remembered" and which should be "looked up again."

The short version

• Semantic caching keeps past answers. When a new question is close enough in meaning, it sends the old answer straight back, without calling the AI again and again.
• If around 90% or more of your questions can be answered from the cache, you might save about 10x on tokens. The research I found reached roughly 60 to 70%, which works out to about 3x cheaper.
• A cache has three main risks: a wrong answer because two questions looked like the same question, an outdated answer, and someone else's answer.
• Live internet search lets an AI look up and answer current questions with sources. But each call costs more, because you also pay for the search API, and answers are slower because the system has to search and summarize first. Web pages it finds can also contain harmful instructions.
• The two work together, as long as you have a step that sorts questions first: which ones can use the cache, which ones need a fresh search, and which ones must never share an answer with other users.

Think of your AI as a call-center agent with a notebook

A call-center agent who has been on the job for a while knows customers ask the same things over and over, so they write the answers down in a notebook. When someone calls and asks "I lost my card, what do I do?", they open the notebook and answer right away, without digging through the policy manual again. The idea here is pretty much the same.

For example:

The customer asks	What a good agent does	In an AI system
"I lost my card. What do I do?"	Opens the notebook and answers	Semantic cache
"What's the exchange rate today?"	Checks whether today's rate is already in the notebook. If not, looks at the live screen, then writes it down again	Live internet search + cache (per day)
"What's my outstanding balance?"	Never answers from another customer's notes; opens this customer's account	Fetch from the real system with permissions; no cache shared with other users
"I lost my credit card" vs. "I lost my debit card..."	Listens carefully first, because the steps may differ	Watch out for wrong matches (false hits)

What a cache is, and why it pays off with AI

A cache is a place where you keep results you have already worked out, so next time you can grab them instead of doing the work again. Websites and apps have used this technique for a long time.

With AI it pays off even more, because every time a model writes an answer, you pay per token and wait several seconds. Every word you send in and every word that comes back costs money, and output tokens cost more than input. If you want the background, I wrote an earlier article on tokens.

Three kinds of cache in the AI world

1. Exact-match cache: the whole question is the key, so it only hits when every character matches. "How do I change my password" and "How do I change my password?" count as two different questions.
2. Prompt caching from model providers: providers such as Anthropic and OpenAI remember the beginning of a prompt you send repeatedly, such as a long system prompt or the same set of documents, and charge less for that part. On many models, the part read from cache costs about 0.1x the normal input price. The catch: it has to match character for character from the start, and the model still writes a new answer every time, so you pay full price for output.
3. Semantic cache: you keep "question and answer" pairs on your own application side and compare questions by meaning. If the meaning is close enough, you send the old answer back and skip the model call.

Type	Matches on	Saves	Limitation
Exact-match cache	Identical text	The whole AI call	Type it slightly differently and it misses
Prompt caching	Identical prompt prefix	Input cost for the repeated part, and input processing time	You still pay for output and still wait for the model
Semantic cache	Similar meaning	The whole AI call	May answer in the wrong context; needs care with stale data and data permissions

When you see "10x cheaper," part of it usually comes from that 0.1x price on prompt caching. But that discount only covers repeated input. It is not 10x off the whole bill.

How semantic caching works

Most setups have four steps.

1. Turn the question into numbers that represent its meaning (an embedding). This is the same method as vector search. I covered it in my earlier article on hybrid search.
2. Search the cache for the old question closest in meaning.
3. Compare it against a similarity threshold. If it is closer than the threshold, that is a cache hit, and the old answer goes straight back.
4. If it does not reach the threshold, it is a miss. The question goes to the AI as usual, and the new question and answer pair is saved for next time.

The clearest win: on a hit, you pay nothing for the model and you do not wait for it, so users get their answer much faster. Turning the question into an embedding and searching the cache cost far less than having the AI answer.

So how much can you actually save?

To get a feel for it, let's do some simple math. If we ignore the cost of embeddings and the cache system itself, your AI cost comes down to the questions that miss.

Share of questions that hit	Still sent to the AI	Roughly cheaper by
30%	70%	1.4x
50%	50%	2x
69%	31%	3.2x
90%	10%	10x

To get 10x, about 90% of requests need to hit. That means almost every question has to repeat the meaning of one you have seen before, like a bot that answers FAQs.

The GPT Semantic Cache paper (Regmi and Pun, 2024) tested repetitive, customer-service-style questions. It got cache hit rates of about 61.6% to 68.8%, cut API calls by up to 68.8%, and over 97% of the answers served from cache were correct, by the authors' own measurement. Run that through the table above and it works out to roughly 3x cheaper. (More in the references at the end.)

Even 3x is a very good deal.

The risks of a semantic cache you need to know first

1. A wrong answer because two questions looked like the same question (false hit)

"How do I cancel my credit card?" and "How do I cancel my debit card?" differ by one word. Their embeddings may be very close, but the answers may be completely different. The same goes for sentences that differ only slightly, by a number, a date, or even a different context.

Redis's documentation says it plainly: semantic caching does not guarantee that a hit is always correct. The vCache paper (Schroeder et al., 2025) makes the same point from another angle: using one fixed similarity threshold for every question does not guarantee correctness either.

To lower the risk, start with a strict threshold, then loosen it step by step, and spot-check the answers that hit the cache along the way. Test with real questions in the languages your users actually write in, because every embedding model understands each language a bit differently.

2. An outdated answer (stale answer)

Some things have already changed, a promotion for example, but the cache keeps serving the old answer. The fix is to give each answer an expiry time (TTL) that matches how often the source data changes. Redis gives the example that real-time data like stock prices or weather should expire within 15 to 30 minutes, while FAQs that rarely change can be kept for days or weeks. And when a source document gets edited, clear the related cache entries right away.

3. Someone else's answer

This one is the most serious. If a customer asks "What's my outstanding balance?" and the answer gets saved into a shared cache, the next customer who asks something similar could get the first customer's answer. That is bad. Redis gives a multi-tenant example too: Company A's policy could end up answering Company B if the cache is not separated.

There is also research (Song et al., 2024) showing that by watching for responses that come back unusually fast, because they hit the cache, someone may be able to guess what other people asked. So a cache shared across users is a privacy issue as well, not only an accuracy issue.

The fix is to separate the cache by user, organization, or permission level, and to not cache answers built from personal data at all unless you really need to.

4. Questions where users want a new answer

Requests like "Help me come up with a caption" or "Give me name ideas for my shop" should get a fresh answer each time. Serving a repeat from the cache feels weird. GPTCache itself skips the cache more often when the temperature is set high.

Same question, same context

The same question asked at a different time or in a different context may need a different answer. So the cache key should be tied to its context, for example:

• Model version
• System prompt version
• Language
• The asker's permissions
• Version of the documents used to answer (if you run a RAG system)

If any one of these changes, the old answer may no longer be up to date.

Live internet search: going out to find real data

Even with a RAG system that can search your company's documents (I wrote an article explaining RAG, if you want to go back to it), some questions still need the latest data: news, prices, newly issued regulations, or software versions. That information is neither inside the model nor in your internal documents.

A typical live internet search flow looks like this:

1. Decide whether this question needs a search
2. Turn the question into a search query, then call the search
3. Read the search results or the content of the relevant pages
4. Pick the passages that answer the question
5. Answer with sources

There are mostly two ways to do it: 1. use the web search tool your model provider builds in, or 2. use a separate search API service and wire up the steps yourself.

The limits of live search

• Cost: search APIs charge per call, and results are long, so the tokens sent into the model are many times higher than for a normal question.
• Speed: you wait for the search results, then load them into the model.
• Source quality: results may come from unreliable sites, or the information may contradict itself.
• Web pages may carry harmful instructions: the text on a web page is data we do not control.

To lower the risk, you can restrict domains for topics that need accuracy, for example only searching official sites or a whitelist of trusted ones, and show sources users can check. Cap the number of searches per question. And do not give an AI that reads the web permission to do important things at the same time, because it may end up running or doing something unsafe.

Sort first: cache it or search again?

Put a router in front that sorts the questions first.

Question type	Example	Best approach	Answer lifetime (TTL)
Data that rarely changes	"What is RAG?", "How do I change my password?"	Semantic cache	Long, days or weeks, and cleared when the documents change
Data that changes over time	"Gold price today", "Latest news on..."	Live search, then cache the search result briefly	Minutes, with the data's timestamp shown
Personal or permission-bound data	"What's my outstanding balance?"	Fetch from the real system with permissions	No shared cache, or a per-user cache
Design or creative work	"Help me come up with a caption"	Always let the AI answer fresh	No cache

Takeaway: cache what rarely changes, search what keeps updating, and be careful with caches shared between users

Semantic caching really can lower costs and make AI answer faster. But caching has to be checked carefully and tightly. You need to weigh the benefit against the risk for each kind of question. If the AI gives a wrong answer without having any idea it is wrong, the savings may not be worth it compared with simply calling the LLM or the API again.

Live internet search lets an AI look things up to answer current questions, but it costs both money and time. And the key point is that we cannot control anything about content that comes from the web, so we need a way to handle that too.

Once you have a solid step that sorts questions into "cache" or "search again," your AI starts working like a great call-center agent: quick to answer, remembers what rarely changes, checks or looks up new information when it makes sense, answers accurately, does not leak data, stays safe, and saves money. If you can get all of that, that is the complete system everyone wants.

References

• GPTCache (Zilliz) on GitHub
• Regmi and Pun: GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching
• Schroeder et al.: vCache: Verified Semantic Prompt Caching
• Song et al.: The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems
• Redis: LangCache concepts
• Redis: 10 techniques for semantic cache optimization
• Anthropic: Prompt caching
• OpenAI: Prompt caching
• Anthropic: Web search tool
• Microsoft: Bing Search API retirement