Podcast · Past conversation
Unit Test Benchmarks
This talk has ended. The recording hasn’t been added here yet.
Unit tests can make strong benchmarks. Joe Barrow and I will discuss why they are useful, when they are a good fit, what their results can and cannot tell you, and how we would design them.
We will look at concrete examples such as charts, tables, content formatting, and grounding.

What we discussed
- Why checking specific behaviors can make evaluation useful and failures easier to understand.
- When unit tests are an appropriate benchmark, and what passing a test leaves unmeasured.
- How we would choose documents, write checks, and decide what to count as correct.
- Examples from olmOCR, OmniParseBench, and ParseBench.