Persist long‑term context by storing embeddings in a vector DB keyed to a thread‑ID and augmenting it with a lightweight summary cache. Combine this with LangChain’s ConversationBufferWindowMemory (or AutoGen’s ChatMemory) that reads from the DB on each turn.
Step‑by‑step implementation
1. Select a vector store – Chroma (local), Pinecone, Weaviate, or RedisVector. Choose based on latency, cost, and scaling needs.
2. Create a thread identifier – UUID v4 per conversation. Store it as metadata thread_id with every embedding.
3. Insert each turn – Encode user_message + agent_response with OpenAIEmbeddings(model="text-embedding-3-large") and upsert into the DB using upsert(ids=[msg_id], embeddings=[vec], metadatas=[{"thread_id": thread_id, "timestamp": ts}]).
4. Summarize periodically – Every 10 turns run summarize_chain = LLMChain(prompt=SUMMARIZE_PROMPT) and store the result in a ConversationSummaryMemory attached to the same thread_id.
5. Load on resume – On new request, query the DB for the latest k relevant chunks (k=5) with similarity_search(query, k=5, filter={"thread_id": thread_id}), prepend the cached summary, and feed the combined list to the LLM.
Tool comparison (text table)
| Tool | Persistence | Query latency (ms) | Cost tier |
|------|-------------|--------------------|----------|
| Chroma | Local files | 5‑10 | Free |
| Pinecone | Managed | 15‑30 | Pay‑as‑you‑go |
| Weaviate | Hybrid (cloud/on‑prem) | 10‑20 | Tiered |
| RedisVector | In‑memory + RDBMS backup | 2‑5 | Low |
LangChain config example
from langchain.memory import ConversationBufferWindowMemory
from langchain.vectorstores import Pinecone
from langchain.embeddings import OpenAIEmbeddings
emb = OpenAIEmbeddings(model="text-embedding-3-large")
vectorstore = Pinecone.from_existing_index(index_name="agent‑mem", embedding=emb)
memory = ConversationBufferWindowMemory(k=5, return_messages=True, memory_key="chat_history")
memory.load_memory_variables({"thread_id": thread_id}) # pulls latest chunks & summary