In this task, we want to know if the patient ever had a documented COVID19 infection Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received…
Once you create the gold dataset, you can do the following: Once you create a copy of the majority vote CSV file which includes all corrections, it becomes your gold dataset. Now you can use this gold dataset to calculate the accuracy of all the LLMs for the given task. Here are the results for…
I have created a custom LLM eval tool for my example, but you can build a similar one which best matches the dataset you are interested in. This is what my custom eval tool looks like for VAERS reports The Eval tool will help you quickly provide the correct value for cases where there is…
Once you run the same data extraction (for example ONSET_DATE) using multiple Gemini models, you can consolidate all the results into a CSV file. This is what the Python code look like The basic idea is that for each VAERS_ID you will get the corresponding value extracted by each Gemini model and check to see…
In a recent article, Simon Willison wrote this In my opinion, Google Gemini models make great choices for LLM evals
We are interested in extracting the AGE information from a VAERS writeup. This must be an integer. Since there isn’t anything ambiguous about the age, we expect that this will be an easy task for the current generation of LLMs. Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable…
The field called ONSET_DATE describes the date of onset for the earliest symptom (after vaccination). Here is an example Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A…
Here are some tips you can use to improve the accuracy of structured outputs from LLM. Note: you can use the Python instructor library to get structured outputs from nearly any Large Language Model. Use a more descriptive field name You can improve the accuracy of structured information extraction by providing a more descriptive field…