Click here to learn more about the system I have used to do this calculation Model Accuracy Comparison Model Accuracy Comparison Comparison gemini-1.5-flash gemini-2.5-flash-preview-05-20 Age 99 100 Onset Date 99 100 Vaccination Date 100 100 Number of Vaccine Doses 100 100 Vaccine Manufacturer 100 100 Vaccine Lot Number 100 100 Had COVID-19 100 100 Was…
In my Udemy course, I build a system where I use four different Gemini models to provide an example of LLM eval – where an LLM is a judge for whether or not an answer is correct. Basic idea: This system can be extended to any dataset and to any LLM that you want to…
This task is to extract whether the patient was healthy before taking the vaccine In the example below patient_was_healthy will be False, because they did have a medical history. Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of…
This task is to extract the number of COVID19 vaccine doses a patient has taken Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received first…
This task involves extracting the vaccine lot from VAERS Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received first dose of bnt162b2 (COMIRNATY, lot number:…
This task is to extract the vaccine manufacturer – for example Pfizer, Moderna etc – from a VAERS report. For example, in the report below, we should extract Pfizer (which made Comirnaty) Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority…
This task is to find out of the person who filed the VAERS report (the reporter) was a healthcare professional. For example in the report below, the answer should be “Yes” since it was filed by a physician. Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received…
This task involves extracting the date of the most recent vaccination from a VAERS report Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received first…
In this task, we want to know if the patient ever had a documented COVID19 infection Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received…
Once you create the gold dataset, you can do the following: Once you create a copy of the majority vote CSV file which includes all corrections, it becomes your gold dataset. Now you can use this gold dataset to calculate the accuracy of all the LLMs for the given task. Here are the results for…
I have created a custom LLM eval tool for my example, but you can build a similar one which best matches the dataset you are interested in. This is what my custom eval tool looks like for VAERS reports The Eval tool will help you quickly provide the correct value for cases where there is…
Once you run the same data extraction (for example ONSET_DATE) using multiple Gemini models, you can consolidate all the results into a CSV file. This is what the Python code look like The basic idea is that for each VAERS_ID you will get the corresponding value extracted by each Gemini model and check to see…