In my Udemy course, I build a system where I use four different Gemini models to provide an example of LLM eval – where an LLM is a judge for whether or not an answer is correct.
Basic idea:
- Gemini now has more than four fairly decent LLMs
- Ask four LLMs to do the same data extraction task
- If there is a majority (3 or 4 agree) vote, use that as the correct answer
- For the rest, manually select the correct answer using the LLM Eval tool
- Save everything into a gold dataset
- Use the gold dataset to calculate accuracy of each LLM
This system can be extended to any dataset and to any LLM that you want to benchmark.