How to measure the deflection rate in Dialogflow ES

In a previous article, I discussed how to measure the accuracy of your Dialogflow ES chatbot.

However, the accuracy usually needs to be combined with the deflection rate of your chatbot to get the full picture of its usefulness.

Note: The deflection rate is defined as the number of times the bot deflects the query from the human. 

Let us take a look at the accuracy mapping diagram again.

A true positive and a false positive are both considered deflections.

Let us suppose users sent 100 messages to the bot, and the breakdown is as follows:

True Positives50
True Negatives40
False Positives5
False Negatives5
Total Messages100
Accuracy = 90%, Deflection rate = 55%

Let us consider a different breakdown

True Positives70
True Negatives20
False Positives5
False Negatives5
Total Messages100
Accuracy = 90%, Deflection rate = 75%

As you can see, the accuracy is 90% in BOTH cases, but we obviously prefer the second case because it has a higher true positive rate. In other words, the bot is able to correctly answer many more questions on its own.

You can intuitively note that while we do want as high a deflection rate as possible, it should not be because we have a lot of false positives.

Conversely, we do not want a lot of true negatives just to maintain a high level of accuracy, because it means you are not making the most of your chatbot deployment.

Note: This is my old website and is in maintenance mode. I am publishing new articles only on my new website. 

If you are not sure where to start on my new website, I recommend the following article:

Is Dialogflow still relevant in the era of Large Language Models?

Leave a Reply