Author: aravindmc

This task involves extracting the vaccine lot from VAERS Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received first dose of bnt162b2 (COMIRNATY, lot number:…

This task is to extract the vaccine manufacturer – for example Pfizer, Moderna etc – from a VAERS report. For example, in the report below, we should extract Pfizer (which made Comirnaty) Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority…

This task is to find out of the person who filed the VAERS report (the reporter) was a healthcare professional. For example in the report below, the answer should be “Yes” since it was filed by a physician. Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received…

This task involves extracting the date of the most recent vaccination from a VAERS report Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received first…

In this task, we want to know if the patient ever had a documented COVID19 infection Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A 47-year-old female received…

Once you create the gold dataset, you can do the following: Once you create a copy of the majority vote CSV file which includes all corrections, it becomes your gold dataset. Now you can use this gold dataset to calculate the accuracy of all the LLMs for the given task. Here are the results for…

I have created a custom LLM eval tool for my example, but you can build a similar one which best matches the dataset you are interested in. This is what my custom eval tool looks like for VAERS reports The Eval tool will help you quickly provide the correct value for cases where there is…

Once you run the same data extraction (for example ONSET_DATE) using multiple Gemini models, you can consolidate all the results into a CSV file. This is what the Python code look like The basic idea is that for each VAERS_ID you will get the corresponding value extracted by each Gemini model and check to see…

In a recent article, Simon Willison wrote this In my opinion, Google Gemini models make great choices for LLM evals

We are interested in extracting the AGE information from a VAERS writeup. This must be an integer. Since there isn’t anything ambiguous about the age, we expect that this will be an easy task for the current generation of LLMs. Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable…

The field called ONSET_DATE describes the date of onset for the earliest symptom (after vaccination). Here is an example Feeling cold; Chills and shivering; high sweating; This is a spontaneous report from a contactable physician, received from the Regulatory Authority, regulatory authority report number of v20100815 and v20100816, and also received information via System. A…