I tested sending 100 requests each to both GPT-5 and Gemini Pro 2.5 to extract information from clinical narratives into a very complex Pydantic schema I used the following metrics: is_pure_json contains_valid_json is_valid_schema to measure how well they adhered to the Pydantic schema Sometimes the LLMs will not even produce valid JSON as a substring…
No matter how much money you are spending on OpenRouter, it is possible to increase the RoI by writing some additional Python code. Why? Because no matter which LLM you use, some percentage of responses will return invalid JSON. Usually, you will do a retry, which costs (say) $X Most of the time, the invalid…
Cogito V2 Preview Llama 109B is less expensive than Deep Cogito Created Sep 2, 202532,767 context$0.18/M input tokens$0.59/M output tokens And in fact, it was able to provide schema compatible JSON for 84 out of the 99 test VAERS reports. One file did not return a response, and I had to stop the program after…
I have been calling the OpenRouter API directly from my code recently. Earlier my code used to look a bit like this And now they look more like this The second option makes a lot more sense for my use case – extracting structured outputs. Here are some reasons why The direct API call is…
We use Gemma 3N 4B model for the task of extracting the NUMDAYS field when both VAX_DATE and ONSET_DATE are specified, but the NUMDAYS field is empty in the VAERS CSV file. The code has some extra steps for caching previous results and validating schema and such, but you should be able to paste this…