🦄 Evaluating Prompts Across Models
AI That Works #16 will be a super-practical deep dive into real-world examples and techniques for evaluating a single prompt against multiple models. While this is a commonly heralded use case for Evals, e.g. 'how do we know if the new model is better' / 'how do we know if the new model breaks anything', there's not a ton of practical examples out there for real-world use cases.
Project Details
Open in GitHubA practical deep dive into evaluating single prompts against multiple models for real-world use cases
Episode Summary
This week's session focused on systematically deciding which LLM is right for your specific use case. We demonstrated how to build a simple evaluation tool to benchmark models side-by-side on your specific prompts, weighing output quality against latency and cost.
Key Points
- Be open to trying new models but ensure they fit your specific needs before adopting them.
- Automate evaluations when possible to minimize the manual effort involved in testing.
- Consider the business use case when defining what constitutes 'accuracy' in AI generated outputs.
- Build tools to simplify the evaluation of multiple models and prompt variations to streamline the analysis process.
Main Topics
- Model evaluation strategies
- User experience in AI applications
- Benchmarking LLM performance
- Prompt engineering and its impact on model accuracy
Key Takeaways
-
Evaluate new LLM models based on performance, cost, and speed. When a new model drops, don't just look at its upper bound - benchmark it against your specific use cases.
-
User experience often drives the decision to switch models. A slightly "less accurate" but significantly faster model can often provide a better user experience. Automating these evaluations allows you to confidently decide when a model switch is justified by both the metrics and the end-user's delight.
-
Build bespoke evaluation tools. Vibe code UIs that help you understand the comparisons you need to make. Model vs Model for the same prompt/task. Task vs Task for the same model/prompt. Prompt vs Prompt for the same task/model. There's a lot of parameterization, don't get overwhelmed and make a 1 tool for everything. Bespoke tools all the way (at least when you start).
-
Establish personal benchmarks to assess new models effectively. The importance of context engineering and iterative evaluation in improving model outputs.
The One Thing to Remember
Systematically evaluate new models against your own benchmarks for performance, cost, and speed. The 'best' model is the one that best serves your specific use case and user experience.
Running the Code
All Episodes
No Vibes Allowed: Using CodeLayer to Build CodeLayer
Live coding with CodeLayer, we'll use Research / Plan / Implement live to ship 3 new features to CodeLayer.
Building an Animation Pipeline
We do a lot of work with Excalidraw, and this session shows the AI-first workflow for turning any sketch into a finished animation. We'll blend Claude Code with custom TypeScript scripts, wire up interactive slash commands, and add browser automation to existing OSS tools to export polished WebM assets.
Dates, Times, and LLMs
How do you make an LLM amazing at dates? Relative dates, absolute dates, timezones, all that madness. Let's talk dates, times, and all that goodness.
Event-driven agentic loops
Key takeaway: treat agent interactions as an event log, not mutable state. Modeling user inputs, LLM chunks, tool calls, interrupts, and UI actions as a single event stream lets you project state for the UI, agent loop, and persistence without drift. We walk through effect-ts patterns for subscribing to the bus, deriving “current” state via pure projections, and deciding when to persist or replay events—plus trade-offs for queuing, cancelation, and tool orchestration in complex agent UX.
Ralph Wiggum under the hood: Coding Agent Power Tools
We've talked a lot about how to use context engineering to get more out of coding agents. In this episode, we dive deep on the Ralph Wiggum technique and why this different approach can reshape your coding workflow. We explore how Ralph handles greenfield work, refactors, and spec generation—surprise: it's all about higher-quality context engineering.
Agentic RAG + Context Engineering
In this conversation, Vaibhav Gupta and Dex explore the intricacies of building an Agentic Retrieval-Augmented Generation (RAG) system. They discuss the differences between traditional RAG and Agentic RAG, emphasizing the flexibility and decision-making capabilities of the latter. The conversation includes a live demo of a coding agent, insights into the coding architecture, challenges faced during tool implementation, and the iterative process of refining the system. They also touch on the integration of web search functionalities and the evaluation of tool effectiveness, providing a comprehensive overview of the development process and the underlying principles of Agentic RAG systems. In this conversation, Vaibhav Gupta and Dex discuss the intricacies of building dynamic AI systems, focusing on tool implementation, user interface optimization, and model performance. They explore the importance of reinforcement learning in training models, the challenges of debugging AI systems, and the significance of writing code to enhance understanding and efficiency in AI development. The dialogue emphasizes the balance between different AI approaches and the necessity of real use cases in building effective solutions.
No Vibes Allowed - Live Coding with AI Agents
Vaibhav Gupta and Dex demonstrate the power of AI-assisted coding by implementing a complex timeout feature for BAML (a programming language for AI applications) in a live coding session. Starting from a GitHub issue that had been open since March, they showcase a systematic workflow: specification refinement, codebase research, implementation planning, and phased execution. Using Claude and specialized coding agents, they navigate a 400,000+ line codebase, implementing timeout configurations for HTTP clients including connection timeouts, request timeouts, idle timeouts, and time-to-first-token for streaming responses. The session highlights key practices like context engineering, frequent plan validation, breaking complex features into testable phases, and the importance of reading AI-generated code. In under 3 hours of live coding, they achieve what would typically take 1-2 days of engineering time, successfully implementing parsing, validation, error handling, and Python integration tests.
Anthropic Post Mortem
In this conversation, Vaibhav Gupta and Aaron discuss various aspects of AI model performance, focusing on the recent downtime experienced by Anthropic and the implications for AI systems. They explore the sensitivity of models to context windows, the challenges of output corruption, and the complexities of token selection mechanisms. The discussion also highlights the importance of debugging and observability in AI systems, as well as the role of user-friendly workflows and integrations in making AI accessible to non-technical users. The conversation concludes with thoughts on the future of AI development and the need for effective metrics to monitor product performance.
Dynamic Schemas
In this episode, Dex and Vaibhav explore the concept of dynamic UIs and how to build systems that can adapt to unknown data structures. They discuss the importance of dynamic schema generation, meta programming with LLMs, and the potential for creating dynamic React components. The conversation also delves into the execution and rendering of these dynamic schemas, highlighting the challenges and opportunities in this evolving field. They conclude with thoughts on future directions and the importance of building robust workflows around schema management.
Evals for Classification
In this episode of AI That Works, hosts Vaibhav Gupta and Dex, along with guest Kevin Gregory, explore the intricacies of building AI systems that are ready for production. They discuss the concept of dynamic UIs, the challenges of large-scale classification, and the importance of user experience in AI applications. The conversation delves into the use of LLMs for enhancing classification systems, the evaluation and tuning of these systems, and the subjective nature of what constitutes a 'correct' classification. The episode emphasizes the need for engineers to focus on accuracy and user experience while navigating the complexities of AI engineering. The speakers also discuss model upgrades, user feedback, and the importance of building effective user interfaces, emphasizing iterative development and rapid prototyping for chatbot performance evaluation.
Bash vs. MCP - token efficient coding agent tooling
In this conversation, Dex and Vaibhav delve into the intricacies of coding agents, focusing on the debate between using MCP (Model Control Protocol) and Bash for tool integration. They explore the importance of understanding context windows, token management, and the efficiency of using different tools. The discussion emphasizes the significance of naming conventions, dynamic context engineering, and the engineering efforts required to optimize performance. They also share real-world applications, best practices for using MCPs, and engage with the community through a Q&A session.
Generative UIs and Structured Streaming
We'll explore hard problems in building rich UIs that rely on streaming data from LLMs. ​Specifically, we'll talk through techniques for rendering **STRUCTURED** outputs from LLMs, with real-world examples of how to handle partially-streamed outputs over incomplete JSON data. We'll explore advanced needs like * Fields that should be required for stream to start * ​Rendering React Components with partial data ​* Handling nullable fields vs. yet-to-be-streamed fields * ​Building high-quality User feedback * ​Handling errors mid-stream
Voice Agents and Supervisor Threading
Exploring voice-based AI agents and supervisor threading patterns for managing complex conversational workflows.
Claude for Non-Code Tasks
On #17 we talked about advanced context engineering workflows for using Claude code to work in complex codebases. This week, we're gonna get a little weird with it, and show off a bunch of ways you can use Claude Code as a generic agent to handle non-coding tasks. We'll learn things like: Skipping the MCP and having claude write its own scripts to interact with external systems, Creating internal knowledge graphs with markdown files, How to blend agentic retrieval and search with deterministic context packing
Interruptible Agents
Anyone can build a chatbot, but the user experience is what truly sets it apart. Can you cancel a message? Can you queue commands while it's busy? How finely can you steer the agent? We'll explore these questions and code a solution together.
Decoding Context Engineering Lessons from Manus
A few weeks ago, the Manus team published an excellent paper on context engineering. It covered KV Cache, Hot-swapping tools with custom samplers, and a ton of other cool techniques. On this week's episode, we'll dive deep on the manus Article and put some of the advice into practice, exploring how a deep understanding of models and inference can help you to get the most out of today's LLMs.
Context Engineering for Coding Agents
By popular demand, AI That Works #17 will dive deep on a new kind of context engineering: managing research, specs, and planning to get the most of coding agents and coding CLIs. You've heard people bragging about spending thousands/mo on Claude Code, maxing out Amp limits, and much more. Now Dex and Vaibhav are gonna share some tips and tricks for pushing AI coding tools to their absolute limits, while still shipping well-tested, bug-free code. This isn't vibe-coding, this is something completely different.
PDFs, Multimodality, Vision Models
Dive deep into practical PDF processing techniques for AI applications. We'll explore how to extract, parse, and leverage PDF content effectively in your AI workflows, tackling common challenges like layout preservation, table extraction, and multi-modal content handling.
Implementing Decaying-Resolution Memory
Last week on #13, we did a conceptual deep dive on context engineering and memory - this week, we're going to jump right into the weeds and implement a version of Decaying-Resolution Memory that you can pick up and apply to your AI Agents today. For this episode, you'll probably want to check out episode #13 in the session listing to get caught up on DRM and why its worth building from scratch.
Building AI with Memory & Context
How do we build agents that can remember past conversations and learn over time? We'll explore memory and context engineering techniques to create AI systems that maintain state across interactions.
Boosting AI Output Quality
This week's session was a bit meta! We explored 'Boosting AI Output Quality' by building the very AI pipeline that generated this email from our Zoom recording. The real breakthrough: separating extraction from polishing for high-quality AI generation.
Building an AI Content Pipeline
Content creation involves a lot of manual work - uploading videos, sending emails, and other follow-up tasks that are easy to drop. We'll build an agent that integrates YouTube, email, GitHub and human-in-the-loop to fully automate the AI that Works content pipeline, handling all the repetitive work while maintaining quality.
Entity Resolution: Extraction, Deduping, and Enriching
Disambiguating many ways of naming the same thing (companies, skills, etc.) - from entity extraction to resolution to deduping. We'll explore breaking problems into extraction → resolution → enrichment stages, scaling with two-stage designs, and building async workflows with human-in-loop patterns for production entity resolution systems.
Cracking the Prompting Interview
Ready to level up your prompting skills? Join us for a deep dive into advanced prompting techniques that separate good prompt engineers from great ones. We'll cover systematic prompt design, testing tools / inner loops, and tackle real-world prompting challenges. Perfect prep for becoming a more effective AI engineer.
Humans as Tools: Async Agents and Durable Execution
Agents are great, but for the most accuracy-sensitive scenarios, we some times want a human in the loop. Today we'll discuss techniques for how to make this possible. We'll dive deep into concepts from our 4/22 session on 12-factor agents and extend them to handle asynchronous operations where agents need to contact humans for help, feedback, or approvals across a variety of channels.
12-factor agents: selecting from thousands of MCP tools
MCP is only as great as your ability to pick the right tools. We'll dive into showing how to leverage MCP servers and accurately use the right ones when only a few have actually relevant tools.
Policy to Prompt: Evaluating w/ the Enron Emails Dataset
One of the most common problems in AI engineering is looking at a set of policies/rules and evaluating evidence to determine if the rules were followed. In this session we'll explore turning policies into prompts and pipelines to evaluate which emails in the massive Enron email dataset violated SEC and Sarbanes-Oxley regulations.
Designing Evals
Minimalist and high-performance testing/evals for LLM applications. Stay tuned for our season 2 kickoff topic on testing and evaluation strategies.
Twelve Factor Agents
Learn how to build production-ready AI agents using the twelve-factor methodology. We'll cover the core concepts and build a real agent from scratch.
Code Generation with Small Models
Large models can do a lot, but so can small models. We'll discuss techniques for how to leverage extremely small models for generating diffs and making changes in complete codebases.
Reasoning Models vs Reasoning Prompts
Models can reason but you can also reason within a prompt. Which technique wins out when and why? We'll find out by adding reasoning to an existing movie chat agent.
Large Scale Classification
LLMs are great at classification from 5, 10, maybe even 50 categories. But how do we deal with situations when we have over 1000? Perhaps it's an ever changing list of categories?