No matter how much money you are spending on OpenRouter, it is possible to increase the RoI by writing some additional Python code.
Why?
Because no matter which LLM you use, some percentage of responses will return invalid JSON.
Usually, you will do a retry, which costs (say) $X
Most of the time, the invalid JSON is due to an identifiable pattern which depends on
a) the Pydantic schema you are using
b) the LLM
Fix Markdown JSON
Surprisingly often, the LLM will return Markdown friendly JSON instead of pure JSON.
This is the most common “error” although this can be fixed with a single line of code.
Strip away the Markdown JSON annotation from the string and process the rest of it.
Common Error Types
Here are some other examples for the Pydantic schema I use
You can find the error by pasting the JSON into an IDE like PyCharm (which will highlight the error)
Some common JSON error modes:
Text outside the JSON brackets
Unquoted strings
Citations are supposed to be lists of integers in my schema. The LLM returns a string instead, but the JSON would have been valid if the string was inside quotes [1]
Improper escaping for nested quotes
This is a surprisingly common issue – when the LLM adds quotes (which would be inside a quoted string already), it needs to apply proper escaping but does not.
There are also a few other common types of errors which would be specific to the LLM and the schema you are using
Fixing this does require writing more complex code, but you will be doing it offline on your local machine without incurring any additional cost. If you need help with implementing this for your LLM project, please get in touch.
Measuring the cost savings
First send a small batch of representative requests to the API (say 10 requests) and make a note of the common types of invalid JSON.
Now fix the invalid JSON issues and try with a slightly larger batch (say 50 requests) and see if existing fixes cover all invalid JSON errors.
Repeat this process for the most common errors but I also recommend you don’t overdo it.
Remember that it is quite common for LLMs to fix the problem on their end in just the first retry.
Every retry you avoid will allow you to save approximately $X - the cost of the input and output tokens for that request.
In other words, fixing invalid JSON is a measurable way to save cost and thus increase the RoI of your OpenRouter investment
[1] Note: If this string did have quotes, the JSON would be syntactically valid but semantically invalid