Semantic Caching and Live Internet Search: When Should AI Remember an Answer, and When Should It Search Again?

---

Let me start with an example from a chat system. Imagine messages like these coming in all day:

“How do I change my password?” “I forgot my password. What do I do?” “Where do I reset my password?”

All three mean the same thing. Yet the system sends each one to the AI to work out a fresh answer, and we pay for tokens three times. With thousands or tens of thousands of users, that becomes a problem we need to solve.

Now flip it around. If the question is “What's the gold price today?” or “Will it rain today?”, reusing an old answer can definitely become a problem, because the information is no longer current.

• Semantic caching: Store answers and reuse them for new questions with a similar meaning, so the AI doesn't have to answer from scratch every time.
• Live internet search: Let the AI look things up on the web when a question needs the latest information.

---

Think of our AI as a call-center agent with a notebook

A call-center agent who has been on the job for a while knows customers keep asking the same questions, so they write the answers down in a notebook. When someone calls and asks, “I lost my card. What should I do?”, they simply open the notebook and answer. No need to dig through the policy manual again. Our approach works in much the same way.

• “I lost my card. What should I do?”: If it's the same card type and the procedure hasn't changed, the saved answer works.
• “What's the exchange rate right now?”: Check whether the saved figure is recent enough. If not, check the latest source.
• “What's my outstanding balance?”: Open the caller's own account, with the right permissions. Never read out the previous customer's balance.

---

What is a cache, and why does it pay off especially well with AI?

A cache stores results of work that has already been done, so next time we can grab them and use them right away instead of doing the work again. Websites and apps have used this technique for a long time.

Having a model generate an answer costs money per token and takes several seconds. Every word you send in and every word that comes back costs something, and output tokens are the more expensive ones. You can read more in my earlier article on tokens.

There are roughly three kinds of cache in AI

1. Exact-match cache: Uses the whole question as the key, and only finds a match when every character is identical. “How do I change my password?” and “How do I change my password, please?” count as different questions.
2. Prompt caching from model providers: Providers such as Anthropic and OpenAI remember the beginning of a prompt that is sent repeatedly, such as a long system prompt or the same set of documents, and charge less for that part. For many models, input read from the cache costs about 0.1 times the normal input price. The catch: it must match character for character from the very start, and the model still generates a new answer every time, so you pay full price for output.
3. Semantic cache: Stores “question–answer” pairs on our own application side and compares questions by meaning. If the meaning is close enough, it returns the saved answer straight away without calling the model again.

Back to our call-center agent: prompt caching is like having the manual open and ready to read, while semantic caching is like having the answer already written down. Even then, you still need to check that it fits the question.

---

How does semantic caching work?

Most implementations follow four steps:

1. Turn the question into numbers that represent its meaning (an embedding): The same approach used in vector search. You can read more in my earlier article on Hybrid Search.
2. Search the cache: Find the previous question with the closest meaning.
3. Compare against a similarity threshold: If it's closer than the threshold, that's a cache hit, and the saved answer is returned immediately.
4. If it falls short, that's a miss: Send the question to the AI to answer as usual, then save the new question–answer pair in the cache for next time.

The clearest benefit: on a hit, you don't pay for the model and you don't wait for it to answer, so users get their answer much faster. Turning the question into an embedding and looking it up in the cache costs far less than having the AI answer.

---

Similar questions that can't use the cache

1. Similar, but actually a different topic

“How do I cancel my credit card?” “How do I cancel my debit card?”

To reduce the risk, start with a strict threshold, then loosen it gradually, and spot-check the answers that hit the cache along the way. If your users ask in a language other than English, such as Thai, test with real questions in that language, because embedding models don't all understand every language equally well.

2. Serving outdated information (a stale answer)

Some information has already changed, such as a promotion, but the cache keeps serving the old answer. The fix is to give each answer an expiry period (a time to live, or TTL) that matches how often the source information changes. Redis, for example, suggests that real-time data like stock prices or weather should expire within 15–30 minutes, while FAQs that rarely change can be kept for days or weeks. You should also clear the related cache entries as soon as the source document is updated.

3. The same question, but someone else's data

A customer asks, “What's my outstanding balance?”, and the answer gets saved to a shared cache. The next customer who asks something similar could receive the first customer's answer, which is a serious problem. Redis gives a similar example for systems serving multiple organizations: company A's policy could end up in an answer for company B if the cache isn't separated.

The fix is to separate the cache by user, organization, or permission level, and not to cache answers based on personal data at all unless you really need to.

4. The user wants a new answer, not the old one faster

Requests like “Help me write a caption” or “Give me ideas for a shop name” usually call for variety. Answers like these shouldn't be cached. Let the model think of something new instead.

---

The same question also has to be in the same context

The same question asked at a different time or in a different context may need a different answer. That's why the cache key should be tied to the context too, for example:

• Model version
• System prompt version
• Language
• The asker's permissions
• Version of the documents used to answer (in a RAG system)

---

Live internet search usually means searching the actual internet

In general, it involves these steps:

1. Decide whether this question needs a search
2. Turn the question into a search query, then call the search
3. Read the search results or the content of relevant web pages
4. Pick out the passages that answer the question
5. Answer with sources cited

The limits of live search

• Cost: Search APIs charge per search, and the results are long, so the tokens sent into the model are many times higher than for a normal question.
• Speed: You have to wait for the search results, then load them into the model.
• Source quality: Results may come from unreliable websites, or the information may conflict.
• Web pages may contain malicious scripts: The text on a web page is data we can't control.

---

Wrapping up

Semantic caching can genuinely cut costs and make AI respond faster, but caching has to be checked carefully and kept tight. You need to weigh the trade-offs and risks of what to cache and what not to. And if the AI gives a wrong answer without any idea that it's wrong, the savings may not be worth it compared with simply calling the LLM or the API again.

Live internet search lets AI find answers about the latest events, but it costs both money and time. Most importantly, we can't control content that comes from the web, so we need mechanisms to handle that too.

Once we have a complete mechanism for deciding up front whether to use the cache or search again, our AI can work like a great call-center agent: answering quickly by remembering things that rarely change, checking or searching for new information when it makes sense, and answering correctly and accurately, without leaking data, safely, and at lower cost. Achieve all of that, and you have the kind of system everyone wants.

---

Further reading

• GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Caching: Experimental results on cache hits and fewer model calls. The figures are research results, not guarantees.
• Redis LangCache concepts: How meaning-based matching differs from exact matching, and why the threshold is a trade-off between hit rate and the risk of wrong answers.
• OpenAI: Prompt caching: How the beginning of a prompt is reused, and how that differs from storing ready-made answers.
• Anthropic: Web search tool: An example of a web search tool with source citations, domain restrictions, and limits on the number of searches.