Measuring and improving LLM accuracy for structured outputs
This course explores techniques for measuring and improving the accuracy of Large Language Models (LLMs) when generating structured outputs.
Students will learn to evaluate and compare the performance of GPT and Gemini models across different data types including numerical values, dates, and boolean values. The curriculum covers practical implementation with hands-on exercises in data extraction, accuracy measurement, and result comparison.
Special emphasis is placed on using explanation-based approaches to enhance model responses and streamline annotation processes. By the end of the course, participants will understand how to effectively assess LLM accuracy and implement strategies for obtaining more reliable structured outputs.
Curriculum
- 6 Sections
- 31 Lessons
- 0m Duration
Introduction
4 Lessons
Getting started
3 Lessons
- Install libraries
- Set environment variables
- Download Jupyter notebook
Numerical values
6 Lessons
- Exploring numerical values in the dataset
- Extracting numerical values using Gemini
- Measuring Gemini accuracy for numerical values
- Extracting numerical values using GPT
- Measuring GPT accuracy for numerical values
- Comparing GPT and Gemini accuracy for numerical values
Date values
6 Lessons
- Exploring date values in the dataset
- Extracting date values using Gemini
- Measuring Gemini accuracy for date values
- Extracting date values using GPT
- Measuring GPT accuracy for date values
- Comparing GPT and Gemini accuracy for date values
Boolean Values
6 Lessons
- Exploring boolean values in the dataset
- Extracting boolean values using Gemini
- Measuring Gemini accuracy for boolean values
- Extracting boolean values using GPT
- Measuring GPT accuracy for boolean values
- Comparing GPT and Gemini accuracy for boolean values
Why use an Explanation
6 Lessons
- Downsides of using the Explanation class
- Explanation provides a future reference
- Explanation can speed up annotation for spaCy Prodigy
- Explanation can provide more accurate responses
- Better responses: an example
- What we can infer from the quality of GPT and Gemini explanations