How to measure the deflection rate in Dialogflow ES

In a previous article, I discussed how to measure the accuracy of your Dialogflow ES chatbot.

However, the accuracy usually needs to be combined with the deflection rate of your chatbot to get the full picture of its usefulness.

Note: The deflection rate is defined as the number of times the bot deflects the query from the human. 

Let us take a look at the accuracy mapping diagram again.

A true positive and a false positive are both considered deflections.

Let us suppose users sent 100 messages to the bot, and the breakdown is as follows:

True Positives50
True Negatives40
False Positives5
False Negatives5
Total Messages100
Accuracy = 90%, Deflection rate = 55%

Let us consider a different breakdown

True Positives70
True Negatives20
False Positives5
False Negatives5
Total Messages100
Accuracy = 90%, Deflection rate = 75%

As you can see, the accuracy is 90% in BOTH cases, but we obviously prefer the second case because it has a higher true positive rate. In other words, the bot is able to correctly answer many more questions on its own.

You can intuitively note that while we do want as high a deflection rate as possible, it should not be because we have a lot of false positives.

Conversely, we do not want a lot of true negatives just to maintain a high level of accuracy, because it means you are not making the most of your chatbot deployment.

About this website

BotFlo1 was created by Aravind Mohanoor as a website which provided training and tools for non-programmers who were2 building Dialogflow chatbots.

This website has now expanded into other topics in Natural Language Processing, including the recent Large Language Models (GPT etc.) with a special focus on helping non-programmers identify and use the right tool for their specific NLP task.

1 BotFlo was previously called MiningBusinessData. That is why you see that name in many videos

2 And still are building Dialogflow chatbots. Dialogflow ES first evolved into Dialogflow CX, and Dialogflow CX itself evolved to add Generative AI features in mid-2023