LLM Evals for Structured Outputs
Lesson Notes are notes I use to present the videos inside the course.
You can check out the free preview course to see what this would look like.
Introduction
Why learn about Structured Outputs
Pros and Cons of using OpenRouter
OpenRouter Response Schema vs Structured Outputs
VAERS dataset
Understanding the VAERS data format
Why VAERS is well suited for this course
The benchmark task
Calculating an upper limit for NUMDAYS
Why the empty NUMDAYS value is a good test case
Calculating the accuracy
Send 100 requests to an LLM using OpenRouter
Measuring schema compliance using Structured Output Percentage Stats
Run the same experiment using four LLMs
Consolidate multiple results into a single CSV file