How do you measure your Dialogflow bot’s accuracy

You may be familiar with the term error matrix or confusion matrix. If not, don’t worry! It is a way to measure if classification techniques work well, and it is quite appropriate in this case because under the hood, Dialogflow takes the user’s input and classifies it to the nearest matching intent.

Consider the last 100 user messages to your bot. If you don’t have that many, get a few beta testers to try out your bot for a few minutes.

A while back, I wrote a post which linked to Chatbase’s UMM method which provides a way to reason about your chatbot’s accuracy. While it is a good idea and I do derive some ideas from it, it is not particularly useful because there isn’t any way to measure the accuracy using the UMM method.