🦄 Designing Evals
Minimalist and high-performance testing/evals for LLM applications. Stay tuned for our season 2 kickoff topic on testing and evaluation strategies.
Project Details
Open in GitHub🦄 designing evals
minimalist and high-performance testing/evals for LLM applications
Overview
This session explores best practices for evaluating LLM applications, focusing on practical, efficient approaches that provide meaningful insights without unnecessary complexity.
Running this code
installing dependencies
# Install dependencies
uv sync
run the code
# Run the code
python hello.py
Key Topics
- Why evals are great - what you can do with an answer key
- How to get the answer key
- we all start out with no answer key
- how do you build it up over time
- Structured Data vs. Unstructured data
- people view as one or the other, but its often semi-structured / a blend
- json with sentences
- markdown with json
- using rubrics to design evals
- llm as judge
- Enron email dataset
- Visualizing Eval Results
Session Notes
Checklist
- Vibe evals - run your prompt (e.g. in playground) and look at the output
- write in a few test cases that work
- write a few end to end tests that run your prompt chain (e.g. with pytest)
- great for tone
- capture intermediate steps of your pipeline as probes and individual testable components
- alternative to probes
- structured outputs from an llm
- helps you break your problems down into smaller components
- e.g. lesson plan output --> "list of biases", "estimated cost"
- don't use numbers for confidence, use a rubric
- categorical, "slow" vs "medium" vs "fast" - enum-based evals
- use prod data to build up your golden dataset over time
- review diffs in either/both of RAW OUTPUT and the STRUCTURED EVALUATION of your pipeline outputs
Links
- (using only) integrated tests are a scam https://www.youtube.com/watch?v=VDfX44fZoMc
All Episodes
Voice Agents and Supervisor Threading
Exploring voice-based AI agents and supervisor threading patterns for managing complex conversational workflows.
Claude for Non-Code Tasks
On #17 we talked about advanced context engineering workflows for using Claude code to work in complex codebases. This week, we're gonna get a little weird with it, and show off a bunch of ways you can use Claude Code as a generic agent to handle non-coding tasks. We'll learn things like: Skipping the MCP and having claude write its own scripts to interact with external systems, Creating internal knowledge graphs with markdown files, How to blend agentic retrieval and search with deterministic context packing
Interruptible Agents
Anyone can build a chatbot, but the user experience is what truly sets it apart. Can you cancel a message? Can you queue commands while it's busy? How finely can you steer the agent? We'll explore these questions and code a solution together.
Decoding Context Engineering Lessons from Manus
A few weeks ago, the Manus team published an excellent paper on context engineering. It covered KV Cache, Hot-swapping tools with custom samplers, and a ton of other cool techniques. On this week's episode, we'll dive deep on the manus Article and put some of the advice into practice, exploring how a deep understanding of models and inference can help you to get the most out of today's LLMs.
Context Engineering for Coding Agents
By popular demand, AI That Works #17 will dive deep on a new kind of context engineering: managing research, specs, and planning to get the most of coding agents and coding CLIs. You've heard people bragging about spending thousands/mo on Claude Code, maxing out Amp limits, and much more. Now Dex and Vaibhav are gonna share some tips and tricks for pushing AI coding tools to their absolute limits, while still shipping well-tested, bug-free code. This isn't vibe-coding, this is something completely different.
Evaluating Prompts Across Models
AI That Works #16 will be a super-practical deep dive into real-world examples and techniques for evaluating a single prompt against multiple models. While this is a commonly heralded use case for Evals, e.g. 'how do we know if the new model is better' / 'how do we know if the new model breaks anything', there's not a ton of practical examples out there for real-world use cases.
PDFs, Multimodality, Vision Models
Dive deep into practical PDF processing techniques for AI applications. We'll explore how to extract, parse, and leverage PDF content effectively in your AI workflows, tackling common challenges like layout preservation, table extraction, and multi-modal content handling.
Implementing Decaying-Resolution Memory
Last week on #13, we did a conceptual deep dive on context engineering and memory - this week, we're going to jump right into the weeds and implement a version of Decaying-Resolution Memory that you can pick up and apply to your AI Agents today. For this episode, you'll probably want to check out episode #13 in the session listing to get caught up on DRM and why its worth building from scratch.
Building AI with Memory & Context
How do we build agents that can remember past conversations and learn over time? We'll explore memory and context engineering techniques to create AI systems that maintain state across interactions.
Boosting AI Output Quality
This week's session was a bit meta! We explored 'Boosting AI Output Quality' by building the very AI pipeline that generated this email from our Zoom recording. The real breakthrough: separating extraction from polishing for high-quality AI generation.
Building an AI Content Pipeline
Content creation involves a lot of manual work - uploading videos, sending emails, and other follow-up tasks that are easy to drop. We'll build an agent that integrates YouTube, email, GitHub and human-in-the-loop to fully automate the AI that Works content pipeline, handling all the repetitive work while maintaining quality.
Entity Resolution: Extraction, Deduping, and Enriching
Disambiguating many ways of naming the same thing (companies, skills, etc.) - from entity extraction to resolution to deduping. We'll explore breaking problems into extraction → resolution → enrichment stages, scaling with two-stage designs, and building async workflows with human-in-loop patterns for production entity resolution systems.
Cracking the Prompting Interview
Ready to level up your prompting skills? Join us for a deep dive into advanced prompting techniques that separate good prompt engineers from great ones. We'll cover systematic prompt design, testing tools / inner loops, and tackle real-world prompting challenges. Perfect prep for becoming a more effective AI engineer.
Humans as Tools: Async Agents and Durable Execution
Agents are great, but for the most accuracy-sensitive scenarios, we some times want a human in the loop. Today we'll discuss techniques for how to make this possible. We'll dive deep into concepts from our 4/22 session on 12-factor agents and extend them to handle asynchronous operations where agents need to contact humans for help, feedback, or approvals across a variety of channels.
12-factor agents: selecting from thousands of MCP tools
MCP is only as great as your ability to pick the right tools. We'll dive into showing how to leverage MCP servers and accurately use the right ones when only a few have actually relevant tools.
Policy to Prompt: Evaluating w/ the Enron Emails Dataset
One of the most common problems in AI engineering is looking at a set of policies/rules and evaluating evidence to determine if the rules were followed. In this session we'll explore turning policies into prompts and pipelines to evaluate which emails in the massive Enron email dataset violated SEC and Sarbanes-Oxley regulations.
Twelve Factor Agents
Learn how to build production-ready AI agents using the twelve-factor methodology. We'll cover the core concepts and build a real agent from scratch.
Code Generation with Small Models
Large models can do a lot, but so can small models. We'll discuss techniques for how to leverage extremely small models for generating diffs and making changes in complete codebases.
Reasoning Models vs Reasoning Prompts
Models can reason but you can also reason within a prompt. Which technique wins out when and why? We'll find out by adding reasoning to an existing movie chat agent.
Large Scale Classification
LLMs are great at classification from 5, 10, maybe even 50 categories. But how do we deal with situations when we have over 1000? Perhaps it's an ever changing list of categories?