Boundary | The hyperobservable software stack

Hyperobservability is observability that is always on, by default, for everything.

Observability

More work, less data.

support.ts11 lines

const tracer = trace.getTracer("support");

const replies = meter.createCounter("replies");

const latency = meter.createHistogram("reply_ms");

async function draftReply(ticket) {

return tracer.startActiveSpan("draftReply", async (span) => {

const start = performance.now();

try {

span.setAttribute("ticket.id", ticket.id);

const label = await classify(ticket);

if (label === "Spam") return "";

const reply = await writeReply(ticket, label);

replies.add(1, { status: "ok" });

logger.info("drafted", { traceId: span.spanContext().traceId });

return reply;

} catch (err) {

span.recordException(err);

span.setStatus({ code: SpanStatusCode.ERROR });

replies.add(1, { status: "error" });

throw err;

} finally {

latency.record(performance.now() - start);

span.end();

}

});

}

what you get

draftReply1842.0ms

http.post auto611.4ms

http.post auto1227.2ms

exceptionerror

repliescounter

reply_mshistogram

draftedlog

Hyperobservability

No work, more data.

support.baml7 lines

function draft_reply(ticket: Ticket) -> string {

let label = classify(ticket);

match (label) {

Label.Spam => "",

_ => write_reply(ticket, label),

}

what you get

draft_reply1842.0ms

classify612.3ms

classify.render_prompt0.3ms

classify.build_request0.1ms

ai.Client.invoke611.2ms

POST api.openai.com/v1/chat/completions610.8ms

connection200

dataevent

endevent

classify.parse0.4ms

write_reply1228.6ms

write_reply.render_prompt0.4ms

write_reply.build_request0.1ms

ai.Client.invoke1227.4ms

POST api.openai.com/v1/chat/completions1227.0ms

connection200

dataevent

endevent

write_reply.parse0.5ms

garbage collector6.1ms

collection_levelmajor

reclaimed_slots1,204

live_slots3,817

live_payload_bytes96 KB

Observability makes decisions, on-call, fixes slow. Hyperobservability makes them fast.

Observability only answers the questions you thought of before you shipped. Hyperobservability answers all of them.

Observability

Time to a decisionstill waiting, day 1

# supportday 1

Monday, November 2

Vaibhav6:12 PM

@claude the support agent refunded a customer twice. How?

ClaudeAPP6:13 PM

The LLM trace shows one issue_refund tool call. Inside the tool, nothing is traced.

ClaudeAPP6:14 PM

I'll add tracing inside the tool.

Tuesday, November 3

ClaudeAPP9:21 AM

Opened a PR to trace refunds.

acme/support-agent#1287 Trace issue_refund and its retriesOpen+52−23 files

ClaudeAPP10:40 AM

Merged. Waiting for another double refund ⏳

Friday, November 6

Vaibhav2:05 PM

Finance found two more 😬

ClaudeAPP2:09 PM

Checked the traces: no new double refunds since Tuesday. Finance's two must be older, from before we traced. Still waiting ⏳

Tuesday, November 10

ClaudeAPP9:40 AM

Caught one. Inside issue_refund, POST /refunds took 31s. The client retried at 30s.

Vaibhav9:45 AM

How many customers did it hit before this?

ClaudeAPP9:46 AM

Can't tell. We weren't recording then. Finance would have to dig through the payment records.

Vaibhav9:52 AM

Make refunds idempotent, and ask finance who else got two 👍

Decision

ClaudeAPP12:20 PM

Fix is through CI and deployed. Finance is still working on the list.

Message #support

Hyperobservability

Time to a decision3 minutes

# supportday 1

Monday, November 2

Vaibhav6:12 PM

@claude the support agent refunded a customer twice. How?

ClaudeAPP6:13 PM

The model called the issue_refund tool once. The AI part is fine.

ClaudeAPP6:13 PM

Inside the tool, POST /refunds took 31s. The client retried at 30s, and both went through.

ClaudeAPP6:14 PM

Same thing hit 23 customers since October 3. $4,180 in all.

Vaibhav6:15 PM

Make refunds idempotent, then email those 23 👍

Decision

ClaudeAPP8:40 PM

Fix is through CI and deployed. Emailed all 23 customers.

Message #support

Mon 2Tue 3Wed 4Thu 5Fri 6Sat 7Sun 8Mon 9Tue 10Wed 11Thu 12

Observability

Asked

Inside the tool isn’t traced

Tracing merged

Now wait for a repeat

Finance finds 2 more

From before the tracing

Caught one

Decided: make refunds idempotent

Fix shipped after CI, 12:20 PM

Customers not emailed

Hyperobservability

Answered in 3 minutes

Decided: make refunds idempotent

Fix shipped after CI, 8:40 PM

All 23 customers emailed

Mon 2

Drag across the timeline to move through the days.

No person could read this much data. Agents can, and they need to.

Automated systems are only as good as their data. Give an agent gaps and it fills them with assumptions. With hyperobservability, there are no gaps.

One run. You can read this.

1 run, 200 spans

Timeouts0

Bad tool calls0

Retry loops0

GC spikes0

Is hyperobservability practical?

Recording every function means overhead on every call and far more data to store.

We made observability fast.

The overhead of observing a function.

32 nsbaseline

One Date.now()

91 ns2.8× slower

OTel+Rust

186 ns5.8× slower

OTel+TypeScript

702 ns22× slower

OTel+Python

9,288 ns290× slower

And we're making it even faster.

Measured on Apple M2 Max, 64 GB.

We made observability tiny.

OpenTelemetry storage

16 TB

Boundary storage

3.5 TB

4.6×

smaller

5× more functions recorded, in 4.6× less storage.

OpenTelemetry+Python

Functions recorded

200B

Storage

16 TB

Sampling

drops 80%

Drag to change the sampling

Errors lost

80%

Functions recorded

1T

Storage

3.5 TB

Sampling

why would you?

Drag to change the sampling

Errors lost

0%

We made observability multimodal.

OpenTelemetry + your blob store

Traces

Unified storage engine

Blob store

Every kind of data
Images, audio, video, PDFs, large strings and bytes.
Recursive deduping
Runtime and cloud collaborate to minimize duplicated data.
RedactionComing soon
Withhold a value, and the trace records why.
Keeps your types
Classes and enums stored with their definitions.

Runs on your laptop. Runs in the cloud.

describe_photo

s3://a8f3.png

caption_photo

s3://c19b.png

2nd copy

watch_clip

s3://91ad.mp4

link expired

read_contract

s3://e20c.pdf

PDF

draft_reply

s3://04bd.txt

file moved

"dear customer, your order has shipped..."

We made observability multimodal.

Traditional stackOpenTelemetry + your blob store

Traces

Unified storage engine

Blob store

Every kind of data
Images, audio, video, PDFs, large strings and bytes.
Recursive deduping
Runtime and cloud collaborate to minimize duplicated data.
RedactionComing soon
Withhold a value, and the trace records why.
Keeps your types
Classes and enums stored with their definitions.

Runs on your laptop. Runs in the cloud.

describe_photo

s3://a8f3.png

caption_photo

s3://c19b.png

2nd copy

watch_clip

s3://91ad.mp4

link expired

read_contract

s3://e20c.pdf

PDF

draft_reply

s3://04bd.txt

file moved

"dear customer, your order has shipped..."

BoundaryOpenTelemetry + your blob store

Traces

Unified storage engine

Blob store

Every kind of data
Images, audio, video, PDFs, large strings and bytes.
Recursive deduping
Runtime and cloud collaborate to minimize duplicated data.
RedactionComing soon
Withhold a value, and the trace records why.
Keeps your types
Classes and enums stored with their definitions.

Runs on your laptop. Runs in the cloud.

describe_photo

s3://a8f3.png

caption_photo

s3://c19b.png

2nd copy

watch_clip

s3://91ad.mp4

link expired

read_contract

s3://e20c.pdf

PDF

draft_reply

s3://04bd.txt

file moved

"dear customer, your order has shipped..."

So yes, hyperobservability is practical. By design.

But to make hyperobservability possible, we had to build a programming language.

BAML, the hyperobservable language.Boundary Cloud, the hyperobservable stack.

BAML is designed like the languages you already trust.

9.5k stars
Open source, Apache 2.0
Free to use
No internet required

TypeScript-like syntax
Rust-like correctness
Go-like concurrency
C#-like reflection
Python, well, you know, Python.

1// An agent is an AI function with tools.

2function support_agent(ticket: string) -> Answer | FollowUp {

3 client: anthropic.Client.new(model = "claude-sonnet-5-5")

4 tools: [lookup_customer, search_tickets]

5 prompt: `

6 Answer this ticket, or ask one question.

7 Look up the customer first.

8 ${ticket}

9 `

10}

11

12class Answer {

13 reply: string,

14}

15

16class FollowUp {

17 question: string,

18}

19

20// Tools are plain functions. Their docs tell the model what they do.

21/// Look up a customer's plan and billing status.

22function lookup_customer(email: string) -> string {

23 `${email}: pro plan, paid through March`

24}

25

26/// Search past tickets for ones like this one.

27function search_tickets(query: string) -> string[] {

28 ["Charged twice", "Wrong plan on invoice"]

29}

1// Classify every ticket at once. Each spawn is a green thread.

2function triage_all(tickets: string[]) -> Label[] {

3 let jobs = tickets.map((t) -> {

4 spawn { classify(t) }

5 });

6 await baml.future.all(jobs)

7}

8

9type Label = "billing" | "bug" | "other";

10

11function classify(ticket: string) -> Label {

12 client: openai.OllamaClient.new(model = "llama3.3")

13 prompt: `

14 Classify this support ticket.

15 ${ticket}

16 `

17}

1// handle can't throw. baml check proves every error is handled.

2function handle(ticket: string) -> string throws never {

3 route(ticket) catch (e) {

4 Unroutable => "triage-queue",

5 ai.errors.Failure | baml.errors.Timeout => "retry-queue",

6 _ => "on-call",

7 }

8}

9

10// The match has to cover every Label, or baml check fails.

11function route(ticket: string) -> string {

12 match (classify(ticket)) {

13 "billing" => "support",

14 "bug" => "engineering",

15 "other" => throw Unroutable { label: "other" },

16 }

17}

18

19class Unroutable {

20 label: Label,

21}

22

23type Label = "billing" | "bug" | "other";

24

25function classify(ticket: string) -> Label {

26 client: google.GeminiClient.new(model = "gemini-3-pro")

27 prompt: `

28 Classify this support ticket.

29 ${ticket}

30 `

31}

1// Make a type at runtime from a spreadsheet's columns.

2// The model fills it in, and its reply is checked against it.

3function extract_row(doc: string, columns: string[]) -> unknown {

4 let row = reflect.Class.builder("Row");

5 for (let c in columns) {

6 row.field(c, reflect.Type.of<string>());

7 }

8 let type Row = row.build();

9 extract<Row>(doc)

10}

11

12function extract<T>(doc: string) -> T {

13 client: openai.ResponsesClient.new(model = "gpt-6-luna")

14 prompt: `

15 Fill this in from the document.

16 ${doc}

17 `

18}

1// Run an agent's code. eval<int> type-checks it before it runs.

2function run(code: string) -> string {

3 let n = reflect.Session.new().eval<int>(code) catch (e) {

4 reflect.errors.CompilationError => {

5 return "rejected";

6 }

7 reflect.errors.EvaluationError => {

8 return "threw";

9 }

10 };

11 `got ${n}`

12}

13

14test "an int runs" {

15 assert.equal(run("6 * 7"), "got 42")

16}

17

18test "a string never runs" {

19 assert.equal(run(`"forty-two"`), "rejected")

20}

21

22test "a runtime error is caught" {

23 assert.equal(run(`baml.json.decode<int>("nope")`), "threw")

24}

25

26test "a session remembers" {

27 let session = reflect.Session.new();

28 session.eval<void>("let total = 40;");

29 assert.equal(session.eval<int>("total + 2"), 42)

30}

1// A test sits in the same file as the code it tests.

2test "sarcasm is billing" {

3 assert.equal(classify("Charged twice again. Love that."), "billing")

4}

5

6// One test per ticket in the file, named for the ticket:

7// production tickets::Charged twice for one order

8// production tickets::App crashes on login

9testset "production tickets" {

10 let cases = baml.json.decode<Case[]>(baml.fs.read("evals/cases.json"));

11 for (let c in cases) {

12 test c.ticket {

13 assert.equal(classify(c.ticket), c.label)

14 }

15 }

16}

17

18class Case {

19 ticket: string,

20 label: Label,

21}

22

23type Label = "billing" | "bug" | "other";

24

25function classify(ticket: string) -> Label {

26 client: anthropic.Client.new(model = "claude-haiku-5-5")

27 prompt: `

28 Classify this support ticket.

29 ${ticket}

30 `

31}

1// Every value can be asked how it feels.

2function inbox(email: string) -> string {

3 if (email.feels("urgent")) {

4 "page on-call"

5 } else {

6 "later"

7 }

8}

9

10// .feels() comes from one blanket implementation, for every type.

11interface Vibes {

12 function feels(self, quality: string) -> bool throws unknown;

13}

14

15implements<S> Vibes for S {

16 function feels(self, quality: string) -> bool {

17 Feels(baml.json.stringify(self), quality)

18 }

19}

20

21function Feels(state: string, quality: string) -> bool {

22 client: typesafeai.Client.new(model = "jev-latest")

23 prompt: `

24 Does this feel ${quality}?

25 ${state}

26 `

27}

Can agents write high-quality BAML?

BAML isn't even close to a mainstream language, so we overcompensate by building amazing backpressure into the toolchain.

Compiler errors built for every reader

61% less context260 characters

Claude Code
⏺ Bash(baml check)
⎿Error: Exit code 4
main.baml:15:5-19:6 error[E0062]: non-exhaustive match on type Label; missing: Label.Refund
main.baml:23:17-23:31 error[E0007]: type `Ticket` has no member `priorty`
main.baml:28:24-28:39 error[E0001]: mismatched types
  primary: expected `string`, found `int`

Every function is a CLI

The fastest way to understand a function is to call it.
For AI functions, it's the only way.

$ baml run apply_discount -- --total 8000 --coupon SAVE20

6400

$ baml run classify_sentiment -- --text "Charged twice and nobody replied."

Negative

1 tool call

Instead of script, run, delete

LLM functions too

Try non-deterministic code

0 scratch files

No left-over slop

Our GitHub Action measures code quality on every pull request.

No model in the loop, so the token bill is $0.

- uses: BoundaryML/baml-action@v1check, fmt, test and codequality

Your agent removes a feature and opens a pull request.

Remove the legacy keyword router #482

•BAML / check•BAML / fmt•BAML / test•BAML / code health

a3f91c2Simplify routing now the agent picks the teamyour agent

github-actionsbotcommented on your agent's push

A feature was removed, and the code got longer: 3 functions got worse. Lines 1,240 → 1,556.

mainyour branch

worseroute_ticketcognitive 12 → 41, nesting 3 → 6

worsebuild_promptcognitive 8 → 27, NPath 16 → 240

worseescalatecognitive 5 → 19

newlegacy_router_shimcognitive 23

Honest compile times

30,000 lines, cold

259 ms

baseline

TypeScript

100 ms

2.6× faster

Go

151 ms

1.7× faster

Rust

799 ms

3.1× slower

Not our proudest chart. We'll get there, we just need a bit more time.

Measured on Apple M2 Max, 64 GB.

So yes, agents can write high-quality BAML. By design.

There's no bad BAML in the training data to learn from. And we make sure the only BAML that gets checked in is great BAML.

Blog: Teaching Agents to Write Maintainable Code →

Avery Townsend

Everyone writes code now.
Boundary gives them great defaults.

With BAML it takes zero config. Every team's data lands in one place, so anyone can ask any question, or hand it to their agent.

Your code

Support

routing.bamlhyperrouter.pyrouter.bamlhyper

Ops

extract.bamlhyper

Engineering

agent.tsagent.bamlhypercheckout.goIsFraud()hyper

Data

etl.pyDedupe()hyper

Sales

leads.pyleads.bamlhyper

Marketing

draft.ipynbdraft.bamlhyper

LangSmith

Datadog

Sentry

S3 bucket

nowhere

boundary cloud

What you want to know

Evals on real traffic

✗ export, then join by hand✓ one query

Accuracy over time

✗ only runs someone labeled✓ one query

Exactly what happened

✗ sampled, across 3 tools✓ one query

Record and replay

✗ only if you built it in ahead of time✓ one query

Cost per feature

✗ prompts and users live apart✓ one query

Traditional stack

LangSmithsupport/routing.bamlhypereng/agent.tseng/agent.bamlhyper

Datadogops/extract.bamlhypereng/checkout.goIsFraud()hyper

Sentryops/extract.bamlhypereng/checkout.goIsFraud()hyper

S3 bucketdata/etl.pyDedupe()hyper

nowheresupport/router.pysupport/router.bamlhypereng/agent.tseng/agent.bamlhyperdata/etl.pyDedupe()hypersales/leads.pysales/leads.bamlhypermarketing/draft.ipynbmarketing/draft.bamlhyper

Evals on real traffic

✗ export, then join by hand

Accuracy over time

✗ only runs someone labeled

Exactly what happened

✗ sampled, across 3 tools

Record and replay

✗ only if you built it in ahead of time

Cost per feature

✗ prompts and users live apart

Boundary

boundary cloud

Supportrouting.bamlhyperrouter.pyrouter.bamlhyper

Opsextract.bamlhyper

Engineeringagent.tsagent.bamlhypercheckout.goIsFraud()hyper

Dataetl.pyDedupe()hyper

Salesleads.pyleads.bamlhyper

Marketingdraft.ipynbdraft.bamlhyper

Evals on real traffic

✓ one query

Accuracy over time

✓ one query

Exactly what happened

✓ one query

Record and replay

✓ one query

Cost per feature

✓ one query

Observe

Create dashboards and alerts from your traces.

Make custom dashboards that are 100% tied to your product by using the SQL-based query API.

Calls per minute

SELECT count(*) FROM spans GROUP BY minute

last 4 hours

Labels

SELECT output_value, count(*) FROM spans WHERE span_name = 'classify' GROUP BY hour, output_value

billingbugother

Latency by function

SELECT span_name, p50(duration), p95(duration) FROM spans GROUP BY span_name

route

p50 1.4s

p95 2.9s

classify

p50 1.3s

p95 2.4s

Share labelled other

SELECT avg(output_value = 'other') FROM spans WHERE span_name = 'classify' GROUP BY hour

alert firing on classify

ImproveComing soon

More things to look forward to.

In progressPR #5218 ↗

Interpret for instant builds. Compile for instant execution.

quicksort, 65,536 random ints · run on Apple M2 Maxshorter is better

BAML interpreted

152 ms

Python

112 ms

Node

6.56 ms

BAML compiled

6.07 ms

↳ tracing on

6.08 ms

Go

4.93 ms

Rust

4.00 ms

Weekend prototypePR #4956 ↗

Pause a run on one machine and pick it up on another.

local

cloud-1

cloud-2

✗

✓ one run, finished: cloud-1 died, cloud-2 picked it up

Watch the proof of concept ✓It works. BEP incoming to design it for reliability.↗

ProposedBEP-063 ↗

Milliseconds are too slow, megabytes too big.<20 µsto start<50 KBof memoryZeroinfrastructure

let sb = sandbox.new(  max_memory = 2 GB,  deadline = 30s,  allow = [FsRead])sb.run(() -> agent(task))

fs.read("docs/faq.md")✓ allowed

http.get("pastebin.com")✗ IoDenied

spawn 50k tasks✗ MemoryExceeded

✓ caught, your program keeps running

For stateless code. Long-lived, stateful environments still need a traditional VM sandbox.

Designing

Roll out experiments 100x faster than CI/CD.

gpt-6-luna3.1% labelled other, $0.03 per 1k calls

gpt-6-sol0.3% labelled other, $0.65 per 1k calls

Not started

Prove a rule holds for every input, not just the ones you tested.

refund(order) ≤ order.total

tests5 inputs

proofevery input

✓ holds for every input

Use Boundary where feedback loops matter most.

Rewrite your stack in BAML

Start today→

Everything you write gets hyperobservability.

Heads up
Not recommended yet. BAML’s ecosystem is still young, and a rewrite trades working code for risk.

Recommended

Adopt BAML incrementally

Start today→

BAML generates a typed SDK for most languages.

OTel → Boundary

Send OTel to Boundary Cloud

Learn more→

Keep your code. Send the traces you already have.

Get ready
Prepare to take advantage of BAML and hyperobservability, at your own pace.

The hyperobservable software stack

Hyperobservability is observability that is always on, by default, for everything.

Observability makes decisions, on-call, fixes slow. Hyperobservability makes them fast.Observability makes decisions slow.Hyperobservability makes decisions fast.

No person could read this much data. Agents can, and they need to.

Is hyperobservability practical?

But to make hyperobservability possible, we had to build a programming language.

BAML, the hyperobservable language.Boundary Cloud, the hyperobservable stack.

BAML is designed like the languages you already trust.

Can agents write high-quality BAML?

Everyone writes code now.Boundary gives them great defaults.

Create dashboards and alerts from your traces.

More things to look forward to.

Use Boundary where feedback loops matter most.

Rewrite your stack in BAML

Adopt BAML incrementally

Send OTel to Boundary Cloud

When humans wrote code, it was okay to move in days or weeks.

Observability makes decisions, on-call, fixes slow. Hyperobservability makes them fast.

Everyone writes code now.
Boundary gives them great defaults.