Once you have it all set up, it is clear that you can use the transcript summaries to speed up your learning We are pasting the transcript manually for each video in the playlist Can we make it even more agentic by asking Notion AI itself to fetch all the transcripts for a YouTube playlist
Database on its own page vs Inline database Database has content corresponding to each row Notion Database has a lot of no-code database features such as sort and filter The page content can include Notion’s builtin database is an important reason why Notion AI is truly agentic
For each LLM, you need 100 valid API responses All the responses should have valid JSON Do a retry until each of the 100 reports contains valid JSON
Code
Code
I used Claude to generate this code walkthrough from the Python script. This script validates the API response and marks contains_valid_json as True or False. You can download the Python script (.py file) from my Measuring LLM Accuracy course. The script is part of a bigger project and cannot be run without pulling in code…
I used Claude to generate this code walkthrough from the Python script. This code saves the response from the LLM API to the local folder. It is also used for doing retries in case previous calls to the API did not produce valid JSON. You can download the Python script (.py file) from my Measuring…
A comparison Demo
Use the system defined in the previous chapter for a single field Evaluate and compare the accuracy for each field (which is comparable) Use Citations and Explanation for the adjudication
We use four metrics related to the JSON responses coming back from LLMs api_error If response_full.json() throws an error, that is considered an api_error is_pure_json If json.loads(inner_response_text) does not throw any error, then is_pure_json is true contains_valid_json Sometimes you have inner_response_text which looks like this: This would be perfectly valid JSON once you remove the…