Blog14 min read
Build a Low-Cost Agentic RAG Support Bot with Mem0 and OpenAI
Build a lower-cost customer support agent that uses Agentic RAG to retrieve company knowledge and customer memory only when the current question needs them.
By Ntense
A customer support chatbot usually needs three different kinds of context:
- Conversation history — what happened in the current chat.
- Customer memory — useful facts remembered across conversations.
- Knowledge base — company policies, documentation, troubleshooting guides, pricing, refunds, and other authoritative information.
A straightforward implementation is easy to build with Mem0 and an LLM. However, the obvious design has an important weakness: it can significantly reduce LLM prompt-cache efficiency.
This article starts with the simple implementation, explains why it becomes expensive, and then improves the architecture by letting the LLM decide when to search memory, when to search the knowledge base, and when something is worth saving to long-term memory.
The lower-cost design is to keep the system prompt and tools static, then let the LLM call memory and knowledge searches only when needed and save only durable facts.
1. The Simple Solution
A first implementation might look like this:
User
│
▼
Next.js API
│
├── Search Mem0 customer memory
│
├── Search knowledge base
│
▼
Build prompt
│
▼
LLM
│
▼
Response
│
▼
Save conversation to Mem0For every message, the backend performs:
1. Search customer memory
2. Search knowledge base
3. Add both results to the prompt
4. Send everything to the LLM
5. Save the interaction to memoryIt is simple, predictable, and easy to implement.
Mem0 setup
Install the SDK:
npm install mem0ai openaiCreate a Mem0 client:
import { MemoryClient } from "mem0ai";
export const mem0 = new MemoryClient({
apiKey: process.env.MEM0_API_KEY!,
});We can use separate Mem0 scopes for customers and the shared support knowledge base.
Customer memory:
userId = customer_123
Company knowledge:
agentId = support-kb-v1Mem0 supports entity-scoped memories such as user_id, agent_id, app_id, and run_id. The application supplies the customer or knowledge-base scope when it writes and searches memory. (Mem0)[3]
2. Add the Knowledge Base
Suppose the company has these support documents:
shipping.md
refunds.md
subscriptions.md
troubleshooting.mdAfter chunking them, we can store the content in Mem0.
const SUPPORT_KB = "support-kb-v1";
await mem0.add(
[
{
role: "user",
content: `
Refund Policy
Customers can request a refund within 30 days of purchase.
Approved refunds normally appear on the original payment method
within 5-10 business days.
`,
},
],
{
agentId: SUPPORT_KB,
infer: false,
metadata: {
document: "refunds.md",
type: "knowledge-base",
},
}
);infer: false is useful here because Mem0 stores the supplied text directly instead of running its normal memory-extraction process. (Mem0)[1]
For a small support knowledge base, this is convenient.
For a very large documentation system, I would normally replace this part with a dedicated RAG system such as:
Qdrant
pgvector
Pinecone
Elasticsearch
OpenAI File Searchwhile continuing to use Mem0 for customer memory.
3. Search Customer Memory
When a customer sends:
What happened with my refund?we can search their memories:
const customerMemory = await mem0.search(
"What happened with my refund?",
{
filters: {
userId: "customer_123",
},
topK: 5,
}
);Possible result:
Customer requested a refund for order #A123 last week.
Refund #R891 was approved.
Customer prefers communication by email.Mem0 supports semantic memory search and scoped retrieval for individual users. (Mem0)[2]
4. Search the Knowledge Base
We can separately retrieve relevant company information:
const knowledge = await mem0.search(
"refund processing time",
{
filters: {
agentId: SUPPORT_KB,
},
topK: 5,
}
);Possible result:
Approved refunds normally take 5-10 business days
to appear on the customer's original payment method.5. Build the LLM Prompt
The obvious implementation combines everything:
const systemPrompt = `
You are the customer support assistant for Example Company.
CUSTOMER MEMORY:
${customerMemory.results
.map((x) => x.memory)
.join("\n")}
KNOWLEDGE BASE:
${knowledge.results
.map((x) => x.memory)
.join("\n")}
Use the knowledge base as the authoritative source.
Do not invent company policies.
`;Then send it together with conversation history:
const response = await openai.responses.create({
model: "gpt-5-mini",
instructions: systemPrompt,
input: messages,
});Finally, save the conversation:
await mem0.add(
[
{
role: "user",
content: userMessage,
},
{
role: "assistant",
content: answer,
},
],
{
userId,
}
);Mem0's default infer: true pipeline can extract useful facts, resolve conflicts, and store the resulting memories. (Mem0)[1]
This works.
But it is not the architecture I would use for a high-volume customer support service.
6. The Problem: We Are Hurting the LLM Cache
Modern LLM APIs can cache repeated prompt prefixes.
Imagine the first request is:
SYSTEM PROMPT
CUSTOMER MEMORY
- Customer owns Product A
- Customer lives in Sydney
KNOWLEDGE BASE
- Shipping takes 3-5 days
CHAT
User: Where is my order?The next request might become:
SYSTEM PROMPT
CUSTOMER MEMORY
- Customer owns Product A
- Customer lives in Sydney
- Customer's order #123 is delayed
KNOWLEDGE BASE
- Delayed orders should be investigated after 7 days
- Shipping takes 3-5 days
CHAT
User: Where is my order?
Assistant: ...
User: Can you check again?The dynamic content near the beginning changed.
That matters because prompt caching fundamentally benefits from repeated prefixes.
Conceptually:
Request 1:
AAAAAAAAAAAAAAAA BBBBBBBBBB
Request 2:
AAAAAAAAAAAAAAAA CCCCCCCCCC
^^^^^^^^^^^^^^^^
cacheable prefixBut if we construct the prompt like this:
STATIC SYSTEM
DYNAMIC MEMORY
DYNAMIC KNOWLEDGE
CHAT HISTORY
NEW MESSAGEa memory change can make the prompt diverge relatively early.
Instead, we want as much stable information as possible at the beginning:
STATIC SYSTEM
STATIC TOOL DEFINITIONS
UNCHANGED CONVERSATION
NEW MESSAGEOpenAI exposes the number of cached input tokens in usage.input_tokens_details.cached_tokens, and provides prompt_cache_key to improve cache routing for related requests. (OpenAI Platform)[4]
Prompt caching can reduce both cost and latency, but only when we design prompts with reusable prefixes in mind.
7. Think of the Prompt as an Append-Only Log
A good chatbot prompt naturally looks like:
System
Tools
User message 1
Assistant message 1
User message 2
Assistant message 2
User message 3At turn three, most of the previous request is unchanged:
████████████████████████████████████████
System
Tools
User message 1
Assistant message 1
User message 2
Assistant message 2
████████████████████████████████████████
User message 3The large prefix can potentially be reused from cache.
This is one reason append-only conversation structures are attractive.
Deleting old messages from the beginning of a conversation can also hurt cache reuse because the prefix changes. OpenAI explicitly notes this effect for conversation truncation. (OpenAI Platform)[4]
8. There Is Another Problem With the Simple Architecture
Consider these messages:
Thanks!OK.Yes.Where is the login button?Do we really need to do all of this?
Mem0 search
+
Knowledge-base search
+
LLM
+
Mem0 memory extractionProbably not.
The application is spending money retrieving information the model doesn't need.
And this is even worse:
await mem0.add(conversation);after every single message.
Does Mem0 need to remember:
Customer said thanks.Probably not.
What we actually want is something closer to:
Customer owns Product Pro.
Customer's order #123 is currently missing.
Customer already tried reinstalling the application.
Customer prefers email support.
Refund #891 was approved.
The customer's billing issue remains unresolved.Long-term memory should contain important durable state, not a second copy of the entire conversation.
9. Improved Architecture: Make Memory and RAG Tools
Instead of automatically injecting memory into every prompt, give the LLM tools.
User
│
▼
┌───────────────┐
│ LLM │
└───────┬───────┘
│
Does it need data?
│
┌───────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Search Memory Search KB Save Memory
│ │ │
▼ ▼ ▼
Mem0 RAG/Mem0 Mem0The LLM now decides:
Do I need customer history?
Do I need company documentation?
Did I learn something worth remembering?This changes the entire cost model.
This pattern is often called Agentic RAG: the LLM decides whether retrieval is needed instead of the application running RAG before every request.
10. Keep the System Prompt Static
Instead of dynamically inserting memory:
const SYSTEM_PROMPT = `
You are the customer support assistant for Example Company.
Rules:
- Help the customer solve their problem.
- Never invent company policies.
- Search the knowledge base when company-specific information is required.
- Search customer memory when previous customer context is required.
- Save durable customer information when it will likely be useful in a future conversation.
- Do not save greetings, acknowledgements, or temporary conversation details.
- If information cannot be verified, escalate to human support.
`;This stays exactly the same between customers and messages.
That is ideal for caching.
11. Create Three Memory Tools
The LLM gets three important tools:
search_customer_memory
search_knowledge_base
save_customer_memoryFor example:
const tools = [
{
type: "function",
name: "search_customer_memory",
description:
"Search long-term information previously saved about this customer. " +
"Use when previous conversations, preferences, orders, unresolved issues " +
"or other historical context is needed.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
},
},
required: ["query"],
additionalProperties: false,
},
strict: true,
},
{
type: "function",
name: "search_knowledge_base",
description:
"Search authoritative company documentation including policies, " +
"products, billing, refunds, shipping and troubleshooting instructions.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
},
},
required: ["query"],
additionalProperties: false,
},
strict: true,
},
{
type: "function",
name: "save_customer_memory",
description:
"Save an important durable fact about the customer that will likely " +
"be useful in a future conversation. Do not save greetings, small talk, " +
"temporary statements or information already present in recent chat history.",
parameters: {
type: "object",
properties: {
memory: {
type: "string",
description:
"A concise standalone fact worth remembering.",
},
},
required: ["memory"],
additionalProperties: false,
},
strict: true,
},
];These tool definitions are static too.
So the beginning of the prompt remains:
SYSTEM PROMPT
TOOL DEFINITIONSacross potentially millions of requests.
12. Implement search_customer_memory
The backend controls the real customer ID.
Never allow the model to choose it.
async function searchCustomerMemory(
userId: string,
query: string
) {
const result = await mem0.search(query, {
filters: {
AND: [{ user_id: userId }],
},
topK: 5,
});
return result.results.map((item) => ({
memory: item.memory,
score: item.score,
}));
}The tool might be called when the user says:
What happened with the problem I had yesterday?The model can search:
{
"query": "customer's unresolved problem from yesterday"
}and retrieve:
Customer reported that order #A123 had not arrived.
A shipping investigation was opened yesterday.13. Implement search_knowledge_base
async function searchKnowledgeBase(query: string) {
const result = await mem0.search(query, {
filters: {
AND: [{ agent_id: SUPPORT_KB }],
},
topK: 5,
});
return result.results.map((item) => ({
text: item.memory,
metadata: item.metadata,
score: item.score,
}));
}A question such as:
How long does a refund take?does not require customer memory.
The model can simply call:
search_knowledge_base(
"refund processing time"
)and answer from the retrieved policy.
14. Let the LLM Decide When to Save Memory
This is where the design becomes more interesting.
Suppose the customer says:
Please contact me by email in the future. I don't normally answer phone calls.That is probably worth remembering.
The model can call:
{
"memory": "Customer prefers email communication instead of phone calls."
}The backend then stores it.
async function saveCustomerMemory(
userId: string,
memory: string
) {
await mem0.add(
[
{
role: "user",
content: memory,
},
],
{
userId,
infer: false,
metadata: {
source: "customer-support-agent",
},
}
);
}Why infer: false here?
Because the first LLM has already done the reasoning:
Should this information be remembered?
Yes.
What is the concise durable fact?
"Customer prefers email communication instead of phone calls."Running another LLM inside the memory system to extract the same fact may be unnecessary.
Mem0 supports direct storage with infer: false, which bypasses its extraction LLM. (Mem0)[1]
There is a tradeoff, however.
With:
infer: trueMem0 can perform extraction and conflict resolution.
With:
infer: falseyou get a cheaper direct write, but your application becomes more responsible for duplicate and conflicting memories. Mem0's documentation specifically notes that direct imports bypass automatic deduplication/conflict resolution. (Mem0)[1]
For a cost-sensitive system, I would often use:
LLM decides whether memory matters
↓
LLM generates normalized memory
↓
optional duplicate check
↓
Mem0 infer=falserather than sending every conversation through another memory-extraction LLM.
15. The LLM Becomes the Memory Controller
Now different messages produce very different workflows.
Example 1: Greeting
User:
HiLLM:
Hello! How can I help?Calls:
Memory search: 0
KB search: 0
Memory save: 0Example 2: Simple knowledge question
User:
How long do refunds take?LLM decides:
I need company policy.Calls:
Memory search: 0
KB search: 1
Memory save: 0Example 3: Previous customer issue
User:
What happened with my refund from last week?LLM decides:
I need customer history.Calls:
Memory search: 1
KB search: possibly 0If it finds:
Refund #891 was approved last Tuesday.that may already be enough.
Example 4: Previous issue plus company policy
User:
My refund from last week still hasn't arrived. Is that normal?The agent may need both:
search_customer_memory
↓
Refund #891 was approved six days ago.
search_knowledge_base
↓
Refunds normally arrive within 5-10 business days.Then it can answer:
Your refund #891 was approved six days ago.
Our normal processing time is 5-10 business days,
so it is still within the expected window.Calls:
Memory search: 1
KB search: 1
Memory save: probably 016. Not Every New Fact Should Become Memory
Suppose the customer says:
I'm at the airport right now.That information is probably not useful next month.
Don't save it.
But:
I always use our Enterprise plan and manage billing for the company.may be useful later.
Save it.
A useful memory rule is:
Save information because future conversations will benefit from it, not simply because the user said it.
Examples worth considering:
preferences
owned products
account relationships
important identifiers
long-running support issues
previous troubleshooting attempts
customer communication preferences
important decisions
ongoing projectsUsually avoid:
greetings
thanks
temporary moods
one-time situational context
information already visible in recent history
generic questions
assistant-generated explanations17. Complete Agent Flow
The resulting system looks like:
┌────────────────────┐
│ Static System │
│ Static Tools │
└─────────┬──────────┘
│
cache-friendly
│
▼
Conversation
│
▼
New message
│
▼
LLM
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
Need memory? Need KB? Save memory?
│ │ │
▼ ▼ ▼
Mem0 RAG/Mem0 Mem0
│ │ │
└───────────┼───────────┘
│
▼
LLM
│
▼
AnswerThat is very different from:
Search everything
↓
Inject everything
↓
LLM
↓
Save everything18. Example Tool Execution Loop
A simplified implementation might look like this:
async function supportAgent(
userId: string,
messages: ChatMessage[]
) {
let response = await openai.responses.create({
model: "gpt-5-mini",
instructions: SYSTEM_PROMPT,
input: messages,
tools,
prompt_cache_key: `support:${userId}`,
});
while (true) {
const calls = response.output.filter(
(item: any) =>
item.type === "function_call"
);
if (calls.length === 0) {
return response.output_text;
}
const outputs = await Promise.all(
calls.map(async (call: any) => {
const args = JSON.parse(call.arguments);
switch (call.name) {
case "search_customer_memory": {
const result =
await searchCustomerMemory(
userId,
args.query
);
return {
type: "function_call_output",
call_id: call.call_id,
output: JSON.stringify(result),
};
}
case "search_knowledge_base": {
const result =
await searchKnowledgeBase(
args.query
);
return {
type: "function_call_output",
call_id: call.call_id,
output: JSON.stringify(result),
};
}
case "save_customer_memory": {
await saveCustomerMemory(
userId,
args.memory
);
return {
type: "function_call_output",
call_id: call.call_id,
output: JSON.stringify({
saved: true,
}),
};
}
default:
throw new Error(
`Unknown tool: ${call.name}`
);
}
})
);
response = await openai.responses.create({
model: "gpt-5-mini",
instructions: SYSTEM_PROMPT,
previous_response_id: response.id,
input: outputs,
tools,
prompt_cache_key: `support:${userId}`,
});
}
}The important point is not the specific SDK syntax.
This is the standard OpenAI function-calling loop: the model requests a tool, the application runs it, and the tool result is sent back to the model. (OpenAI Platform)[5]
If this workflow later needs explicit retrieval grading, query rewriting, retries, or approval steps, LangGraph can model those branches. It is optional; the OpenAI SDK is enough for the beginner version in this article. (LangGraph)[6]
It is the architecture:
LLM reasoning
↓
selective tool usage
↓
small relevant contextinstead of:
retrieve everything
↓
large dynamic context
↓
LLM19. Keep Full Chat History Outside Mem0
I would also avoid treating Mem0 as the primary conversation database.
Use something such as PostgreSQL:
PostgreSQL
↓
complete conversations
tickets
messages
audit history
timestamps
attachmentsand use Mem0 for:
durable semantic memorySo the storage architecture becomes:
Customer
│
▼
Next.js
│
┌───────────┼────────────┐
│ │ │
▼ ▼ ▼
PostgreSQL Mem0 RAG
│ │ │
full chat long-term company
history memory knowledge
│ │ │
└───────────┼────────────┘
│
▼
LLMEach system has one clear responsibility.
20. We Can Optimize Further With Multiple Cache Layers
For a high-volume support service, I would eventually use several layers:
User request
│
▼
L1: Exact response cache
│
▼
L2: Semantic FAQ cache
│
▼
L3: LLM prompt cache
│
▼
L4: On-demand KB retrieval
│
▼
L5: On-demand customer memory
│
▼
L6: Selective memory writesFor example:
User:
How do I reset my password?If this exact FAQ has been safely answered thousands of times, the application may not even need an expensive model call.
But:
My password reset still doesn't work after the steps
you gave me yesterday.now requires:
LLM
+
customer memory
+
possibly KBThe system spends more only when the problem actually requires more reasoning.
21. The Main Principle
The naive approach is:
More context = better AIBut production AI systems need a slightly different principle:
Provide the minimum context necessary
for the model to make the correct decision.Memory should therefore not mean:
put everything the system knows
into every promptIt should mean:
give the model the ability
to retrieve what it needs
when it needs itThe same applies to knowledge bases.
Instead of:
RAG → LLMfor every request, think:
LLM → decide whether RAG is required → RAG → LLMAnd instead of:
conversation → automatically save memorythink:
conversation
↓
LLM decides whether something matters
↓
save only useful long-term stateFinal Architecture
For a production customer support chatbot, my preferred architecture is therefore:
Customer
│
▼
Next.js
│
▼
Support Agent
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
PostgreSQL Mem0 RAG
Chat History Memory Knowledge
│ ▲ ▲
│ │ │
│ search/save tool search tool
│ │ │
└──────────────┼──────────────┘
│
▼
LLMWith the prompt designed as:
STATIC SYSTEM INSTRUCTIONS
STATIC TOOL DEFINITIONS
──────────────────────────
CONVERSATION HISTORY
NEW MESSAGErather than:
SYSTEM
DYNAMIC MEMORY
DYNAMIC KNOWLEDGE
CONVERSATIONThe first architecture gives the model access to memory.
The improved architecture gives the model control over memory.
That distinction matters.
A good AI support system should not constantly carry its entire memory and knowledge base around with it. It should behave more like a human support engineer:
Remember what matters, look things up when necessary, and otherwise just answer the question.
That produces a system that is simultaneously more scalable, more cache-friendly, and potentially much cheaper to operate.
Sources
- Add Memory — Mem0 Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Defines the add operation, JavaScript MemoryClient usage, entity write identifiers, the default inference path, direct storage with infer set to false, and the duplicate risk when direct and inferred writes are mixed.
- Search Memory — Mem0 Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Describes semantic memory search, filtering, ranking and reranking, provides current JavaScript Platform examples, and recommends scoping searches to the correct user.
- Entity-Scoped Memory — Mem0 Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Documents user, agent, application, and run scopes, how writes and reads apply those identifiers, and why user and agent records should not be assumed to occupy one combined entity scope.
- Prompt caching — OpenAI Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). States that cache hits require exact prefix matches, recommends stable content before variable content, explains that tools must match, and documents cached_tokens and cache_write_tokens for measuring reuse.
- Function calling — OpenAI Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Documents the model, application, and tool-output loop; automatic tool choice; strict JSON Schema function definitions; and the application's responsibility for executing tool calls.
- Build a custom RAG agent with LangGraph — LangChain Accessed Thu Aug 20 2026 00:00:00 GMT+0000 (Coordinated Universal Time). Shows a JavaScript Agentic RAG graph that decides whether to retrieve, uses ToolNode and conditional routing, grades retrieved evidence, rewrites insufficient queries, and generates a final answer.