Home / DialogFlow ES / 10+ Dialogflow failure modes for automated testing
Conversation Testing | DialogFlow ES

10+ Dialogflow failure modes for automated testing

Recently, a couple of students/clients asked me about creating scripts for testing their Dialogflow agent’s conversations.

In this article, I list the things you can test for when you are writing these scripts.

Prerequisite: You need to use the explicative approach

I have written about the explicative approach to Dialogflow development in an earlier article. One of the reasons I recommend that approach is because it actually makes automation of various Dialogflow tasks a whole lot easier.

As an example, many of the items in this list will become untestable (programmatically) if you were to use slot filling.

REST API

You will be testing your agent by calling the detectIntent method using Dialogflow’s REST API v2.

The test script will be quite straightforward:

1 Write a set of utterances (which form a single conversation)

2 Send each utterance to your agent’s detectIntent method one by one

3 Check the response to see if all the fields are as expected

4 If not, flag the utterance for review

Obviously it is a little more involved than that, but that is the gist of how you would approach the task.

List of things you can test for

Let us look at the list of things you can test using this approach.

Please note that not all of these issues are caused by Dialogflow failures. In fact many of them are human errors.

1 Fallback intent is mapped

You expect a specific intent to be mapped, but the fallback intent is mapped.

2 Wrong intent is mapped

Again, this issue is when a wrong intent is mapped, but the wrong intent isn’t the Fallback intent either.

Two things can help you when you run into this issue:

a) understand how candidate intents work

b) use the candidate explorer tool to see why the wrong intent is a candidate in the first place

3 Incorrect action is set

The action that is set by an intent will determine the logic which is used by the webhook. You can also use the intent name for deciding the webhook logic, but the action is more likely to remain unchanged.

4 Incorrect output contexts are set

When an intent fires, you can set specific output contexts to help move the conversation in a direction you want. If the wrong output context is set, it might lead to wrong intents being mapped in the future steps in your conversation.

As you are checking the output contexts, it wouldn’t be a bad idea to also verify if their lifespans are set to 1. Since Dialogflow autocompletes context lifespans to use a value of 5, you might have accidentally changed the lifespan when trying to modify a context value.

5 Incorrect or incomplete parameters

You will notice something quite surprising in Dialogflow. Sometimes, even when an intent fires, it doesn’t extract the parameter value correctly. You can, in theory, use required parameters to force Dialogflow to get those values, but the benefit isn’t worth the cost, in my opinion.

There are actually two failure modes here: one is Dialogflow not extracting the correct value of the parameter, and the other is where Dialogflow extracts no value at all (that is, if you inspect the parameter field, it will be empty).

6 Incorrect values of parameters from previous steps

You might use something like session variables to store parameter values across long conversations.

In that case, at each step, your session-vars output context will actually hold parameter values extracted during previous steps.

You need to check if those values are still available for use as they might be used in future conversation steps.

7 Webhook call fails with timeout

You have probably experienced this, especially when you make changes to your agent and test inside your simulator.

Remember that when the webhook call returns, it also contains a response code. You can inspect the value inside this code to check if there was a timeout. The image below shows some possible error codes.

For a timeout, I believe you get the 503 error code.

8 Webhook call fails with internal server error

Usually this indicates a more serious issue with your Dialogflow webhook code.

Sometimes you might receive this error code because the format of the JSON file you have constructed might be slightly off (e.g. missing a curly brace).

This error code is usually 500.

9 Webhook call fails with 404

If you get a 404 error code, it is usually a simple fix. Your fulfillment URL is usually pointing to the wrong resource (e.g. moving the folder where you deployed the code). Also make sure that your URL doesn’t redirect, because Dialogflow doesn’t allow redirect URLs to be used for fulfillment.

10 Webhook call returns but sets the wrong output context

In this case, you should get a normal 200 status code for the webhook call.

However, the webhook code might be trying to set an output context and might not be setting the value correctly. If your webhook does modify or update output contexts, you need to have this test in place because it can easily change the course of your conversation and send it down the wrong path.

11 Webhook call returns but invokes the wrong followupEvent

You can jump from one intent to another from your webhook by using the followupEvent feature.

When you do this, make sure to have a test in place which checks if the correct event is being invoked. Since the event is a string value that you set inside your Dialogflow intent, it is quite possible to make a silly mistake when writing out these string values.

12 Intent settings changed

There are also some miscellaneous settings inside a Dialogflow intent such as “does it call a webhook?”, “does it represent the end of conversation (actions on google)” etc.

It is very easy to accidentally change these settings (for example you might be temporary changing these values to test something inside the DF simulator).

So having some tests in place for these types of settings isn’t a bad idea, since the values for these fields can be verified as they are part of the JSON response coming back.

Free: Dialogflow ES Crash course for non-programmers

I recently changed the name of my website from Mining Business Data to BotFlo. To help spread the word, and to hopefully get people to bookmark my website, I am making a set of my training courses free for a few weeks. 

Learn more
"The magic key I needed as a non-programmer"

The custom payload generator was the magic key I needed (as a non-programmer) to build a good demo with rich responses in DialogFlow Messenger. I've only used it for 30 minutes and am thrilled. I've spent hours trying to figure out some of the intricacies of DialogFlow on my own. Over and over, I kept coming back to Aravind's tutorials available on-line. I trust the other functionalities I learn to use in the app will save me additional time and heartburn.

- Kathleen R
Cofounder, gathrHealth

Similar Posts