JSON Metrics
We use four metrics related to the JSON responses coming back from LLMs
url = "https://openrouter.ai/api/v1/chat/completions"
headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"HTTP-Referer": "https://botflo.com/", # Optional. Site URL for rankings on openrouter.ai.
"X-Title": "LLM Accuracy Comparisons for Structured Outputs",
}
payload = {
"model": model_name,
"messages": [
{"role": "system", "content": f'{system_prompt}'},
{"role": "user", "content": f'{prompt}'}
],
"reasoning": {
"effort": "high"
}
}
response_full = requests.post(url, headers=headers, data=json.dumps(payload))
response = response_full.json()
after = time.time()
elapsed = after - before
inner_response_text = response['choices'][0]['message']['content']
full_response_json = response_full.json()api_error
If response_full.json() throws an error, that is considered an api_error
is_pure_json
If json.loads(inner_response_text) does not throw any error, then is_pure_json is true
contains_valid_json
Sometimes you have inner_response_text which looks like this:
"```json\n{\n \"timeline\": {\n \"date_of_vaccination\": {\n \"year\": 1996,\n \"month\": 5,\n \"citations\": [\n 47\n ],\n \"explanation\": \"Extracted from Vax_date field.\"\n },\n \"regression_onset_date\": {\n \"year\": 1996,\n \"month\": 5,\n \"citations\": [\n 47\n ],\n \"explanation\": \"Extracted from Onset_date field. Occurred on the same day as vaccination.\"\n },\n \"diagnosis_date\": null,\n \"age_at_vaccination_months\": {\n \"value\": 15,\n \"citations\": [\n 3\n ],\n \"explanation\": \"Calculated from 'Age_yrs: 1.25' (1.25 * 12 = 15 months).\"\n },\n \"age_at_diagnosis_months\": {\n \"value\": 45,\n \"citations\": [\n 7\n ],\n \"explanation\": \"The follow-up report states the patient is a '3 year, 9 month old child with a diagnosis of Autism Spectrum Disorder', which is 45 months. This is the age at which the diagnosis is confirmed in the report, not necessarily the exact date of diagnosis.\"\n }\n },\n \"developmental_milestones\": [],\n \"regression_record\": {\n \"pre_regression_language_level\": {\n \"value\": \"babbling_single_words\",\n \"citations\": [\n 5\n ],\n \"explanation\": \"Patient experienced 'loss of... language, comprehension' at 15 months old, which implies that some language skills, at least consistent with that age, were present prior to the regression.\"\n },\n \"pre_regression_social_communication_level\": {\n \"value\": \"basic_engagement\",\n \"citations\": [\n 5\n ],\n \"explanation\": \"Patient experienced a 'loss of skills' at 15 months, which implies that some social skills consistent with that age were present before the regression.\"\n },\n \"pre_regression_adaptive_behavior_level\": {\n \"value\": \"basic_self_help\",\n \"citations\": [\n 5\n ],\n \"explanation\": \"Patient experienced a 'loss of skills' at 15 months, which implies that some adaptive/play skills consistent with that age were present prior to the regression.\"\n },\n \"post_regression_language_level\": {\n \"value\": \"babbling_single_words\",\n \"citations\": [\n 12,\n 14\n ],\n \"explanation\": \"After regression, the patient has 'gotton some words' and uses 'mam, go, bye, and dada' with some consistency.\"\n },\n \"post_regression_social_communication_level\": {\n \"value\": \"basic_engagement\",\n \"citations\": [\n 18,\n 42,\n 43\n ],\n \"explanation\": \"While 'eye contact' has improved, the patient is 'still not using his language or words to initiate communication' and 'non-verbal communication appears to be somewhat sparse'.\"\n },\n \"post_regression_adaptive_behavior_level\": {\n \"value\": \"basic_self_help\",\n \"citations\": [\n 22\n ],\n \"explanation\": \"Patient 'is now able to take off his shoes and socks and is otherwise able to help with dressing'.\"\n },\n \"has_language_loss\": {\n \"value\": true,\n \"citations\": [\n 5\n ],\n \"explanation\": \"The report states the patient experienced 'loss of skills, language, comprehension etc.'\"\n },\n \"has_social_loss\": {\n \"value\": true,\n \"citations\": [\n 5,\n 18\n ],\n \"explanation\": \"The report mentions a general 'loss of skills' and that 'eye contact' had 'improved' in follow-up, implying a prior loss.\"\n },\n \"has_adaptive_loss\": {\n \"value\": true,\n \"citations\": [\n 5,\n 18\n ],\n \"explanation\": \"The report mentions a 'loss of skills' and that 'Play skills have like wise improved', implying a prior loss.\"\n },\n \"has_repetitive_behaviors\": {\n \"value\": true,\n \"citations\": [\n 19,\n 21,\n 39\n ],\n \"explanation\": \"The report notes a 'tendency to become fixated', 'perserverate behavior', 'self stimulatory behavior', and 'hand shaking behavior'.\"\n }\n },\n \"early_vaccination_symptoms\": {\n \"had_autism_features\": {\n \"value\": true,\n \"ongoing\": true,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49,\n \"Order: 1\"\n ],\n \"explanation\": \"Initial symptoms included 'loss of skills, language, comprehension'. The official symptom list also includes 'Autism' with an onset of day 0.\"\n },\n \"had_fever\": {\n \"value\": true,\n \"ongoing\": false,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49\n ],\n \"explanation\": \"The report explicitly states 'The pt experienced a fever' as an initial symptom with an onset of day 0. It is not mentioned as ongoing.\"\n },\n \"had_speech_loss\": {\n \"value\": true,\n \"ongoing\": true,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49,\n \"Order: 1\"\n ],\n \"explanation\": \"Initial symptoms included 'loss of ... language, comprehension'. The symptom list includes 'Aphasia' with an onset of day 0.\"\n },\n \"had_irritability\": {\n \"value\": true,\n \"ongoing\": false,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49\n ],\n \"explanation\": \"The report states the patient experienced 'fussiness, irritable' as an initial symptom. It is not mentioned as ongoing.\"\n },\n \"had_eye_contact_loss\": {\n \"value\": false,\n \"ongoing\": false,\n \"explanation\": \"Loss of eye contact is implied by later improvement but not explicitly listed as an initial symptom within 30 days.\"\n },\n \"had_diarrhea\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_seizures\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_lethargy\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_screaming\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_behavioral_changes\": {\n \"value\": true,\n \"ongoing\": true,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49\n ],\n \"explanation\": \"Patient exhibited 'fussiness, irritable, loss of skills' starting on the day of vaccination.\"\n },\n \"had_rash\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_head_banging\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_social_withdrawal\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_injection_site_reaction\": {\n \"value\": true,\n \"ongoing\": false,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 5,\n 49,\n \"Order: 1\"\n ],\n \"explanation\": \"The patient had a 'knot at the injection site for an extended period'. 'Injection site mass' is on the symptom list with an onset of day 0. It is not described as currently ongoing.\"\n },\n \"had_balance_loss\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_appetite_loss\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_staring_spells\": {\n \"value\": false,\n \"ongoing\": false,\n \"explanation\": \"Staring spells are mentioned [38] but not specified as an early symptom occurring within 30 days of vaccination.\"\n },\n \"had_hand_flapping\": {\n \"value\": false,\n \"ongoing\": false,\n \"explanation\": \"'Hand shaking behavior' is mentioned [39] but not specified as an early symptom occurring within 30 days of vaccination.\"\n },\n \"had_toe_walking\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_gastrointestinal_problems\": {\n \"value\": true,\n \"ongoing\": true,\n \"days_to_symptom_onset\": 0,\n \"citations\": [\n 6,\n 35,\n 49,\n \"Order: 1\"\n ],\n \"explanation\": \"The symptom list includes 'Gastrointestinal disorder' with an onset of day 0. Follow-up text mentions ongoing 'GI issues' and gastrointestinal problems.\"\n },\n \"had_hyperactivity\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_ear_infections\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_vomiting\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_sleep_problems\": {\n \"value\": false,\n \"ongoing\": false\n },\n \"had_encephalopathy\": {\n \"value\": false,\n \"ongoing\": false\n }\n },\n \"diagnosis_record\": {\n \"asd_severity\": {\n \"value\": 2,\n \"citations\": [\n 7,\n 14,\n 28,\n 29,\n 42\n ],\n \"explanation\": \"Severity level is not explicitly stated. Level 2 ('Requiring substantial support') is assigned based on the description of significant but not complete deficits: limited language to single words, sparse non-verbal communication, and need for multiple ongoing therapies (speech, OT, home program).\"\n },\n \"diagnosis_name\": [\n {\n \"value\": \"Autism Spectrum Disorder\",\n \"citations\": [\n 7\n ]\n },\n {\n \"value\": \"Pervasive Developmental Disorder\",\n \"citations\": [\n 6,\n 7\n ]\n },\n {\n \"value\": \"Autism\",\n \"citations\": [\n \"Order: 1\"\n ]\n }\n ],\n \"diagnosing_professional\": null\n },\n \"intervention_record\": {\n \"interventions\": [\n {\n \"value\": \"speech_therapy\",\n \"citations\": [\n 29,\n 32\n ],\n \"explanation\": \"Patient receives 'speech and language services' and is seen by a 'speech and language pathologist'.\"\n },\n {\n \"value\": \"occupational_therapy\",\n \"citations\": [\n 29,\n 31\n ],\n \"explanation\": \"Patient receives 'occupational therapy services' and his therapist is trained in 'sensory integration'.\"\n },\n {\n \"value\": \"other\",\n \"citations\": [\n 28\n ],\n \"explanation\": \"Patient 'continues to participate in a home program'.\"\n }\n ],\n \"is_intervention_ongoing\": {\n \"value\": true,\n \"citations\": [\n 28,\n 29\n ],\n \"explanation\": \"The report states the patient 'continues to participate in a home program' and 'also receives speech and language services and occupational therapy services'.\"\n },\n \"has_recovery\": {\n \"value\": false,\n \"citations\": [\n 47\n ],\n \"explanation\": \"The 'Recovd' field is marked 'N' (No).\"\n }\n },\n \"birth_record\": {\n \"was_normal_pre_vaccination\": {\n \"value\": true,\n \"citations\": [\n 5\n ],\n \"explanation\": \"The report of a 'loss of skills, language, comprehension etc.' implies that development was progressing prior to the event.\"\n },\n \"apgar_score\": null\n },\n \"behavior_record\": {\n \"repetitive_behaviors\": [\n {\n \"value\": \"Fixated interests\",\n \"citations\": [\n 19\n ],\n \"explanation\": \"He has a tendency to become fixated on the same four viseo tapes.\"\n },\n {\n \"value\": \"Perseverative behavior\",\n \"citations\": [\n 21\n ],\n \"explanation\": \"SOme perserverate behavior and play behavior however, can be exhibited.\"\n },\n {\n \"value\": \"Self-stimulatory behavior\",\n \"citations\": [\n 39\n ],\n \"explanation\": \"he comes back with self stimulatory behavior.\"\n },\n {\n \"value\": \"Hand shaking\",\n \"citations\": [\n 39\n ],\n \"explanation\": \"has been known during around these times, to have hand shaking behavior.\"\n },\n {\n \"value\": \"Auditory sensitivity\",\n \"citations\": [\n 36\n ],\n \"explanation\": \"pt tends to be sensitive to noises and frequently cover his ears.\"\n }\n ]\n },\n \"heavy_metal_tests\": null,\n \"comorbidities\": [\n {\n \"value\": \"eczema\",\n \"citations\": [\n 25,\n 51\n ],\n \"explanation\": \"Patient has a history of eczema and 'continues to have eczema, which is 'itchy all of the time''.\"\n },\n {\n \"value\": \"allergies\",\n \"citations\": [\n 51\n ],\n \"explanation\": \"Patient has a history of allergies.\"\n },\n {\n \"value\": \"other\",\n \"citations\": [\n 6\n ],\n \"explanation\": \"Annual follow-up states the pt has metabolic disturbances, mitochondrial problems.\"\n },\n {\n \"value\": \"other\",\n \"citations\": [\n 6,\n 35\n ],\n \"explanation\": \"Patient has 'GI issues' / 'gastrointestinal problems'.\"\n },\n {\n \"value\": \"other\",\n \"citations\": [\n 26\n ],\n \"explanation\": \"Patient has 'elevated IGE immune titers'.\"\n }\n ],\n \"environmental_exposures\": [],\n \"legal_matter\": {\n \"value\": false,\n \"explanation\": \"No mention of a lawsuit or legal matter in the report.\"\n },\n \"diagnosis_reporting_delay_reason\": null\n}\n```"This would be perfectly valid JSON once you remove the markdown prefix:
```json
{
"timeline": {
"date_of_vaccination": {
"year": 1996,
"month": 5,
"citations": [
47
],
"explanation": "Extracted from Vax_date field."
},
"regression_onset_date": {
"year": 1996,
"month": 5,
"citations": [
47
],
"explanation": "Extracted from Onset_date field. Occurred on the same day as vaccination."
},
"diagnosis_date": null,
"age_at_vaccination_months": {
"value": 15,
"citations": [
3
],
"explanation": "Calculated from 'Age_yrs: 1.25' (1.25 * 12 = 15 months)."
},
"age_at_diagnosis_months": {
"value": 45,
"citations": [
7
],
"explanation": "The follow-up report states the patient is a '3 year, 9 month old child with a diagnosis of Autism Spectrum Disorder', which is 45 months. This is the age at which the diagnosis is confirmed in the report, not necessarily the exact date of diagnosis."
}
},
"developmental_milestones": [],
"regression_record": {
"pre_regression_language_level": {
"value": "babbling_single_words",
"citations": [
5
],
"explanation": "Patient experienced 'loss of... language, comprehension' at 15 months old, which implies that some language skills, at least consistent with that age, were present prior to the regression."
},
"pre_regression_social_communication_level": {
"value": "basic_engagement",
"citations": [
5
],
"explanation": "Patient experienced a 'loss of skills' at 15 months, which implies that some social skills consistent with that age were present before the regression."
},
"pre_regression_adaptive_behavior_level": {
"value": "basic_self_help",
"citations": [
5
],
"explanation": "Patient experienced a 'loss of skills' at 15 months, which implies that some adaptive/play skills consistent with that age were present prior to the regression."
},
"post_regression_language_level": {
"value": "babbling_single_words",
"citations": [
12,
14
],
"explanation": "After regression, the patient has 'gotton some words' and uses 'mam, go, bye, and dada' with some consistency."
},
"post_regression_social_communication_level": {
"value": "basic_engagement",
"citations": [
18,
42,
43
],
"explanation": "While 'eye contact' has improved, the patient is 'still not using his language or words to initiate communication' and 'non-verbal communication appears to be somewhat sparse'."
},
"post_regression_adaptive_behavior_level": {
"value": "basic_self_help",
"citations": [
22
],
"explanation": "Patient 'is now able to take off his shoes and socks and is otherwise able to help with dressing'."
},
"has_language_loss": {
"value": true,
"citations": [
5
],
"explanation": "The report states the patient experienced 'loss of skills, language, comprehension etc.'"
},
"has_social_loss": {
"value": true,
"citations": [
5,
18
],
"explanation": "The report mentions a general 'loss of skills' and that 'eye contact' had 'improved' in follow-up, implying a prior loss."
},
"has_adaptive_loss": {
"value": true,
"citations": [
5,
18
],
"explanation": "The report mentions a 'loss of skills' and that 'Play skills have like wise improved', implying a prior loss."
},
"has_repetitive_behaviors": {
"value": true,
"citations": [
19,
21,
39
],
"explanation": "The report notes a 'tendency to become fixated', 'perserverate behavior', 'self stimulatory behavior', and 'hand shaking behavior'."
}
},
"early_vaccination_symptoms": {
"had_autism_features": {
"value": true,
"ongoing": true,
"days_to_symptom_onset": 0,
"citations": [
5,
49,
"Order: 1"
],
"explanation": "Initial symptoms included 'loss of skills, language, comprehension'. The official symptom list also includes 'Autism' with an onset of day 0."
},
"had_fever": {
"value": true,
"ongoing": false,
"days_to_symptom_onset": 0,
"citations": [
5,
49
],
"explanation": "The report explicitly states 'The pt experienced a fever' as an initial symptom with an onset of day 0. It is not mentioned as ongoing."
},
"had_speech_loss": {
"value": true,
"ongoing": true,
"days_to_symptom_onset": 0,
"citations": [
5,
49,
"Order: 1"
],
"explanation": "Initial symptoms included 'loss of ... language, comprehension'. The symptom list includes 'Aphasia' with an onset of day 0."
},
"had_irritability": {
"value": true,
"ongoing": false,
"days_to_symptom_onset": 0,
"citations": [
5,
49
],
"explanation": "The report states the patient experienced 'fussiness, irritable' as an initial symptom. It is not mentioned as ongoing."
},
"had_eye_contact_loss": {
"value": false,
"ongoing": false,
"explanation": "Loss of eye contact is implied by later improvement but not explicitly listed as an initial symptom within 30 days."
},
"had_diarrhea": {
"value": false,
"ongoing": false
},
"had_seizures": {
"value": false,
"ongoing": false
},
"had_lethargy": {
"value": false,
"ongoing": false
},
"had_screaming": {
"value": false,
"ongoing": false
},
"had_behavioral_changes": {
"value": true,
"ongoing": true,
"days_to_symptom_onset": 0,
"citations": [
5,
49
],
"explanation": "Patient exhibited 'fussiness, irritable, loss of skills' starting on the day of vaccination."
},
"had_rash": {
"value": false,
"ongoing": false
},
"had_head_banging": {
"value": false,
"ongoing": false
},
"had_social_withdrawal": {
"value": false,
"ongoing": false
},
"had_injection_site_reaction": {
"value": true,
"ongoing": false,
"days_to_symptom_onset": 0,
"citations": [
5,
49,
"Order: 1"
],
"explanation": "The patient had a 'knot at the injection site for an extended period'. 'Injection site mass' is on the symptom list with an onset of day 0. It is not described as currently ongoing."
},
"had_balance_loss": {
"value": false,
"ongoing": false
},
"had_appetite_loss": {
"value": false,
"ongoing": false
},
"had_staring_spells": {
"value": false,
"ongoing": false,
"explanation": "Staring spells are mentioned [38] but not specified as an early symptom occurring within 30 days of vaccination."
},
"had_hand_flapping": {
"value": false,
"ongoing": false,
"explanation": "'Hand shaking behavior' is mentioned [39] but not specified as an early symptom occurring within 30 days of vaccination."
},
"had_toe_walking": {
"value": false,
"ongoing": false
},
"had_gastrointestinal_problems": {
"value": true,
"ongoing": true,
"days_to_symptom_onset": 0,
"citations": [
6,
35,
49,
"Order: 1"
],
"explanation": "The symptom list includes 'Gastrointestinal disorder' with an onset of day 0. Follow-up text mentions ongoing 'GI issues' and gastrointestinal problems."
},
"had_hyperactivity": {
"value": false,
"ongoing": false
},
"had_ear_infections": {
"value": false,
"ongoing": false
},
"had_vomiting": {
"value": false,
"ongoing": false
},
"had_sleep_problems": {
"value": false,
"ongoing": false
},
"had_encephalopathy": {
"value": false,
"ongoing": false
}
},
"diagnosis_record": {
"asd_severity": {
"value": 2,
"citations": [
7,
14,
28,
29,
42
],
"explanation": "Severity level is not explicitly stated. Level 2 ('Requiring substantial support') is assigned based on the description of significant but not complete deficits: limited language to single words, sparse non-verbal communication, and need for multiple ongoing therapies (speech, OT, home program)."
},
"diagnosis_name": [
{
"value": "Autism Spectrum Disorder",
"citations": [
7
]
},
{
"value": "Pervasive Developmental Disorder",
"citations": [
6,
7
]
},
{
"value": "Autism",
"citations": [
"Order: 1"
]
}
],
"diagnosing_professional": null
},
"intervention_record": {
"interventions": [
{
"value": "speech_therapy",
"citations": [
29,
32
],
"explanation": "Patient receives 'speech and language services' and is seen by a 'speech and language pathologist'."
},
{
"value": "occupational_therapy",
"citations": [
29,
31
],
"explanation": "Patient receives 'occupational therapy services' and his therapist is trained in 'sensory integration'."
},
{
"value": "other",
"citations": [
28
],
"explanation": "Patient 'continues to participate in a home program'."
}
],
"is_intervention_ongoing": {
"value": true,
"citations": [
28,
29
],
"explanation": "The report states the patient 'continues to participate in a home program' and 'also receives speech and language services and occupational therapy services'."
},
"has_recovery": {
"value": false,
"citations": [
47
],
"explanation": "The 'Recovd' field is marked 'N' (No)."
}
},
"birth_record": {
"was_normal_pre_vaccination": {
"value": true,
"citations": [
5
],
"explanation": "The report of a 'loss of skills, language, comprehension etc.' implies that development was progressing prior to the event."
},
"apgar_score": null
},
"behavior_record": {
"repetitive_behaviors": [
{
"value": "Fixated interests",
"citations": [
19
],
"explanation": "He has a tendency to become fixated on the same four viseo tapes."
},
{
"value": "Perseverative behavior",
"citations": [
21
],
"explanation": "SOme perserverate behavior and play behavior however, can be exhibited."
},
{
"value": "Self-stimulatory behavior",
"citations": [
39
],
"explanation": "he comes back with self stimulatory behavior."
},
{
"value": "Hand shaking",
"citations": [
39
],
"explanation": "has been known during around these times, to have hand shaking behavior."
},
{
"value": "Auditory sensitivity",
"citations": [
36
],
"explanation": "pt tends to be sensitive to noises and frequently cover his ears."
}
]
},
"heavy_metal_tests": null,
"comorbidities": [
{
"value": "eczema",
"citations": [
25,
51
],
"explanation": "Patient has a history of eczema and 'continues to have eczema, which is 'itchy all of the time''."
},
{
"value": "allergies",
"citations": [
51
],
"explanation": "Patient has a history of allergies."
},
{
"value": "other",
"citations": [
6
],
"explanation": "Annual follow-up states the pt has metabolic disturbances, mitochondrial problems."
},
{
"value": "other",
"citations": [
6,
35
],
"explanation": "Patient has 'GI issues' / 'gastrointestinal problems'."
},
{
"value": "other",
"citations": [
26
],
"explanation": "Patient has 'elevated IGE immune titers'."
}
],
"environmental_exposures": [],
"legal_matter": {
"value": false,
"explanation": "No mention of a lawsuit or legal matter in the report."
},
"diagnosis_reporting_delay_reason": null
}
```is_valid_schema
This will be set to true if you can validate the JSON that you extracted against the given Pydantic schema