Building AI with Memory & Context
How do we build agents that can remember past conversations and learn over time? We'll explore memory and context engineering techniques to create AI systems that maintain state across interactions.
Project details
Open in GitHub🦄 ai that works: Context Engineering and memory deep dive
A deep dive into building effective memory systems for AI agents, focusing on context engineering, scalable memory architectures, and practical implementation patterns.
Links:
- 12 factor agents: Context Engineering
- Bryan's Blog Post on triggers and memory - Building Proactive AI Agents
- Previous Episode with deeper dive on structured outputs as context eng: Cracking the Prompting Interview
- OWL Ontology Time relationships
Episode Highlights
"Treat RAG, memory, and prompts as a single, unified context engineering problem. Think about how to best assemble all necessary information into the final set of tokens for the model."
"Don't try to make your agent remember everything. Implement a summarization strategy like Decaying Resolution Memory (DRM) to create a focused, scalable memory that surfaces what's important over time."
"Give your agent semantically meaningful, human-like tools (e.g., 'check_calendar', 'search_inbox') instead of a generic 'retrieve_memory' function. Sandbox these tools to the current user to improve security and simplify the agent's task."
"Before writing code, clearly define your success criteria and the specific user experience you want to create. Your memory architecture should be a direct solution to that well-defined problem."
"When creating summarization tasks, provide the model with the existing memory context. This allows it to understand what is 'notable' in the new information relative to the entire history."
"For tasks where you know the agent will always need certain information (e.g., today's date, user profile), fetch it deterministically and inject it into the context yourself. Don't rely on the agent to ask for it every time."
"Avoid solving complex, deterministic problems like timezone conversions inside a prompt. Handle that logic in your application code and provide the model with a normalized, simple format to work with."
Key Takeaways
- "Context Engineering" is the unifying paradigm for building with LLMs. All inputs—prompts, RAG, memory, agent history—are simply different ways of assembling the tokens that go into the model. The quality of your output is a direct function of the quality of this input context.
- Effective memory is not about remembering everything. It's an engineered, lossy process designed to retain the most relevant information for a specific use case. Techniques like Decaying Resolution Memory (DRM) manage this by summarizing information over time, making memory scalable and focused.
- Offload memory and state to sandboxed, stateful tools. Instead of stuffing all data into the prompt, give the agent tools that mirror human workflows (e.g., a calendar, an inbox, a notepad). This makes the agent's task more intuitive, improves security, and reduces prompt size.
- Before engineering a complex memory system, you must deeply understand your user and define the problem. Identify the specific 'wow factor' or core value proposition (e.g., proactivity, personalization) and design the memory system to enable that behavior. It's a system design problem, not just a prompt tuning exercise.
Resources
- Session Recording
- Discord Community
- Sign up for the next session on Luma













































































