Home / DialogFlow ES / Why you should regression test your Dialogflow (API.AI) chatbot
DialogFlow ES

Why you should regression test your Dialogflow (API.AI) chatbot

Website Name Change

I have changed the name of this website from Mining Business Data to BotFlo. I am offering a 60% off discount on both my Dialogflow ES and Dialogflow CX courses till April 20th 2021 for people who can help me spread the word about my new website.

If you are building a commercial grade Dialogflow chatbot, you should definitely consider adding some regression testing to your chatbot.

What is regression testing?

The idea behind regression testing is that you will run a series of tests and verify if the performance of your chatbot is consistent over time.

How to do regression testing in Dialogflow?

You will make use of the /query endpoint in the REST API exposed by Dialogflow.

By writing scripts which will send specific, pre-determined user phrases to the /query endpoint, you can verify if the phrases are getting mapped to the same intent (by using the intentID).

Why do regression testing

This brings us to the next question: why should you do regression testing for your Dialogflow chatbot? I will provide you a few reasons why it would really help.

1 Keep the chatbot’s behavior consistent as you make changes

You will keep modifying your chatbot over time once users interact with it. How do you know if a recent change you made didn’t break some existing conversation flow that worked well previously? By setting up regression tests, you can confidently make changes to your chatbot

2 Dialogflow updates

Sometimes, the Dialogflow team makes updates and changes to their service, and occasionally it has ended up changing the behavior of their own entity detection.

To guard against these kind of issues, you can set up regression tests that verify if the most important entity detection you wish to do is running consistently. In other words, if the city “Chennai” was being recognized as @sys.geo-city before Dialogflow rolled out an update, you want to make sure it is still being recognized as an entity of the same type after the update is rolled out.

3 Adjust your ML Threshold score

You might be familiar with the ML Threshold score setting in your Dialogflow agent.

When you do your regression tests, you can inspect the JSON response to see the score associated with an intent match. Watch the video below to understand how this works.

For a group of phrases, suppose the lowest matching score is 0.75. This means you can afford to set the ML threshold fairly high, say 0.5, because the matching score is well above such a threshold for most of the phrases you are interested in. (Of course, this also depends on how flexible you want your chatbot to be)

4 Improve your chatbot’s functionality

When your user’s phrase gets mapped to a Fallback intent, that phrase can be used in either an existing intent or added to a new intent to improve the functionality of your chatbot.

If you already had regression tests in place, you will do this analysis a lot more systematically, which means your chatbot’s functionality is continuously improving over time.

5 Reverse Engineer your chatbot’s behavior

One nice thing about setting up a large set of regression test phrases is that as you gain more experience building out your regression test cases, you will also be able to reverse engineer your agent’s behavior quite a bit. Considering that Dialogflow is not open source and its Machine Learning is a bit of a black box, you can learn a lot by doing some text analysis on top of your regression tests.

Want to see some examples of reverse engineering which will give you more insights into Dialogflow’s intent mapping? Check out my Dialogflow Tips course (specifically, the “Dissecting Intent Mapping” section)

Similar Posts

2 Comments

  1. Hey Arvind, how do we set up a regression test? Can you share a video on the same? I am subscribed to your courses but I can’t find anything.