Skip to content
- Send 100 requests to the same LLM using OpenRouter
- Sample code
- OpenRouter allows you to reuse the code and only change the model name
- How many have pure JSON
- Sometimes the response is Markdown JSON which is still valid JSON once you strip out the Markdown annotation
- Some LLMs like Claude preface the answer with “Here is the information you requested” 🙂
- How many contain valid JSON
- you can use regular expressions to extract JSON out of the string response
- and often it will be valid JSON even if the response itself isn’t pure JSON
- How many have valid schema?
- then check to see if the JSON complies with the schema provided in the input
- How many extract the accurate value?
- only after doing all these steps, you will be able to calculate if the extracted value is accurate
- First, calculate the Structured Output Percentage Stats
- not all LLMs behave the same way
