🦄 code generation with small models

large models can do a lot, but so can small models. we'll discuss techniques for how to leverage extremely small models for generating diffs and making changes in complete codebases.

Diagrams

Overall Ownership - User vs. Agent

Architecture

Context Window Management

Pipelining Updates

Optimize - Serve most users with small, fast models

Start with big expensive models, improve coverage with smaller models over time

Project Structure

This session contains two main components:

1. Calculator Project (/project)

A simple calculator application that demonstrates a complete, well-structured Python codebase. Features include:

• Basic arithmetic operations (+, -, *, /)
• Memory functionality (store, recall, clear)
• Interactive command-line interface
• Clean separation of concerns (operations, calculator logic, user interface)

2. Agent Project (/agent)

A BAML-based project that shows how to use small models to generate and modify code. The agent demonstrates:

• Code analysis and understanding
• Targeted code modifications
• Working with existing codebases

Running the Code

Calculator Project

cd project

# Install dependencies
uv sync

# Run the calculator
python main.py

Agent Project

cd agent

# Install dependencies
uv sync

# Generate BAML code
uv run baml-cli generate

# Run the agent
python hello.py