Podcast · Live conversation
Unit Test Benchmarks
Loading registration…
Register to get the calendar invitation and viewing link, plus the recording and notes by email after the talk.
Unit tests can make strong benchmarks. Joe Barrow and I will discuss why they are useful, when they are a good fit, what their results can and cannot tell you, and how we would design them.
We will look at concrete examples such as charts, tables, content formatting, and grounding.
What we’ll discuss
- Why checking specific behaviors can make evaluation useful and failures easier to understand.
- When unit tests are an appropriate benchmark, and what passing a test leaves unmeasured.
- How we would choose documents, write checks, and decide what to count as correct.
- Examples from olmOCR, OmniParseBench, and ParseBench.