Home / DialogFlow ES / How to bulk upload intents from a CSV file in Dialogflow
DialogFlow ES

How to bulk upload intents from a CSV file in Dialogflow

If you are creating a large FAQ chatbot in Dialogflow, typing out all the intents one by one can be a slooooooooow and painful process.

Thankfully, Dialogflow has an agent ZIP file import feature, and it is possible to programmatically create this ZIP file.

Note: this article talks about the use case where you have already identified the Dialogflow intents. If you haven’t yet defined the intents and the training phrases inside them, check out my tool which can help you automatically extract Dialogflow intents from your chat logs.

Alternatively, you can use the knowledge base feature. But there are some important differences.

I have created a table summarizing the differences between knowledge based FAQ bots and intent based FAQ bots.

Intent Based FAQ BotKnowledge Based FAQ Bot
How we build itUsing intents and training phrasesUsing the knowledge base feature
Support for multiple responses for the same queryNoYes
Multiple training phrases per intentYesNo
Supports entities in training phrasesYesNo
Support for contexts (and follow up)YesNo
Easy to construct rich responses (e.g. for Dialogflow Messenger)YesNo
Use term reinforcement for adding weights to specific words and phrasesYesBlackbox
Allows us to minimize intra-intent variance and maximize inter-intent variance (as recommended by Google)YesBlackbox
Read full article

As you can see, unless you are sure you need multiple responses from your chatbot, you are much better off going with the Intent Based FAQ Bot.

In this article, I describe how you can type out your intent information into a spreadsheet and then use a CSV export of your spreadsheet to generate a Dialogflow agent ZIP file. I also discuss the different aspects you need to consider as you try to use this approach for more complex FAQ bots.

By following this template, you should be able to build a tool which can create a Dialogflow FAQ chatbot with the push of a button.

I also have an app which can do this for you, if you don’t want to build such a tool by yourself.

Want to build it yourself?

First understand the agent ZIP file format. Then write some code in your favorite programming language to construct the agent ZIP file. 

Steps:
1 Construct the package.json file
2 Construct the agent.json file
3 Create a folder called intents
4 For each intent (based on the CSV data), construct the response JSON and the training phrase JSON files and add them into the intents folder
5 ZIP them into a single file

This article talks about text-only FAQ chatbots. If you would like to see how to do something similar for rich response channels such as Dialogflow Messenger, Facebook Messenger and Google Assistant check out the follow up article.

Simple question and answer FAQ chatbot

If you just want to create a simple FAQ bot with one question corresponding to one answer, it requires a simple CSV format. There is a column with a user says message (Query), and then another column with the text response (Response).

What you want to do: You upload this CSV format, and you have an agent at the end of it.

You can have a very simple format with a single question and a single answer

How to convert your CSV file into Dialogflow agent ZIP File

These are the steps to take your 2 Column CSV file specified in the format above and convert it into a Dialogflow agent ZIP file in BotFlo.

1 Download the 2 Column CSV file

2 Upload your CSV file

Now upload the CSV file you downloaded in the previous step.

Upload your CSV file (make sure it is in the correct format)

3 Download agent ZIP file

Once you click on Convert, you will be able to download the agent ZIP file to your local computer

4 Restore agent ZIP file in Dialogflow

Restore the agent ZIP file inside your Dialogflow console

5 Test your agent

And test your agent using the builtin Dialogflow Messenger integration

How to add multiple training phrases per intent

Now, the simple 2 Column CSV format is not sufficient. For example,

  • how can you handle multiple user says (training) phrases per intent?
  • what if you want to add multiple text responses?
  • what should be used as the intent’s name?

Suppose we create the following CSV column structure to accommodate the points from the previous section. You have an intent ID, followed by intent name, followed by userSays phrase, and the final column will be your text response.

For example, this is what the CSV format would look like (screenshot taken from my BotFlo app). All rows must have an intentID to identify the intent. Notice that you can have multiple queries (training phrases) and multiple responses per intent.

Four column format

While this gives you the ability to use multiple userSays phrases as well as multiple text responses, it is also more complex to process this format.

How to add input and output contexts

If you would like to add input and output contexts into the CSV format, you need to consider the following:

  • you can have multiple input and output contexts per intent
  • every output context should have an associated context lifespan

Here is the format I came up with for this purpose (again taken from my BotFlo app):

Adding more columns to accommodate contexts, actions, and webhook setting

What about entities?

Specifying entities inside this CSV format is more complex.

Why is this?

It’s because when you add a training phrase containing an entity, Dialogflow automatically parses the entity value (annotates it, that is). For example, you can see below that the word tomorrow is automatically annotated as a date entity.

Dialogflow automatically parses and annotates entities when it can

If you download and inspect the JSON of this intent, it looks like this:

JSON file output of the entity annotation

So adding entities via a CSV format will have two issues:

  • it makes the CSV format more complex (not such a big problem)
  • the annotation will not automatically work (bigger problem)

For example, you might annotate a name as @sys.given-name in your CSV file, for example. But that will not automatically be accepted by Dialogflow which might reject the entity value since its system entity cannot handle it.

(I am working on a solution to this also, and will update this post soon).

What about followup intents?

Sometime around May 2017, Dialogflow (which was then API.AI) introduced the followup intents feature. These allowed you to have a hierarchy of intents, like below.

Example follow up intent hierarchy

Since the number of hierarchical levels is (theoretically) unlimited, you can imagine that it is not possible to capture this very complex structure in the CSV format at all.

So an app which can convert a CSV file into a Dialogflow agent ZIP file will NOT be able to capture followup intents.

If you want to use followup intents in your Dialogflow agent (make sure you read this article to learn about some situations where you should not use follow up intents), you cannot use this approach.

Summary

Here is a summary of how the BotFlo app works.

Feature2 Column CSV (Simple FAQ Bot)4 Column CSV (Basic FAQ Bot)10 Column CSV (Advanced FAQ Bot)
Multiple training phrases per intentNoYesYes
Multiple responses per intent (from which one will be randomly selected by Dialogflow)NoYesYes
Input and output contextsNoNoYes
Support for ActionNoNoYes
Support for entitiesNoNoNo
Support for follow up intentsNoNoNo

<— End of article —>


This website contains affiliate links. See the disclosure page for more details. 
"The magic key I needed as a non-programmer"

The custom payload generator was the magic key I needed (as a non-programmer) to build a good demo with rich responses in DialogFlow Messenger. I've only used it for 30 minutes and am thrilled. I've spent hours trying to figure out some of the intricacies of DialogFlow on my own. Over and over, I kept coming back to Aravind's tutorials available on-line. I trust the other functionalities I learn to use in the app will save me additional time and heartburn.

- Kathleen R
Cofounder, gathrHealth
"Much clearer than the official documentation to be honest"

Thanks a lot for the advice (of buying and following your videos)! They helped a lot indeed. Everything is very clear when you explain, much clearer than the official documentation to be honest 🙂

Neuraz T
Review for Learn Dialogflow CX
"I will strongly recommend this course because even I can learn how to design chatbot (no programming background)"

I think Aravind really did a great job to introduce dialogflow to people like me, without programming background. He organizes his course in very clear manner since I have been a college professor for 20 years. It is very easy for me to recognize how great Aravind’s course is! Very use-friend and very easy to follow. He doesn’t have any strong accent when he gives the lectures. It is so easy for me to understand. Really appreciate it.

Yes, I will strongly recommend this course because even I can learn how to design chatbot (no programming background) after studying Avarind’s course, you definitely can!

Ann Cai
Review for Learn Dialogflow ES

Similar Posts

    • This article is about using a spreadsheet software like a CMS for creating your Dialogflow FAQ chatbot. That is, you will be manually assigning the Intent ID which groups all information for a single intent. Its goal is to speed up input for large-ish Dialogflow FAQ bots.

      I don’t think that is what you are talking about. If you tell me your end goal, maybe I can give some suggestions.

  • Great tutorials!

    Do the intent IDs need to be in sequential order, or can they be any random number?

    Can I use your tool to upload new intents? Where I select the “Import from zip” option?

    I am looking to train the agent, by uploading new intents, or even new training phrases.

    In order to train the agents, I would need to reupload a new zipfile and restore agent?

    Or would I need to:

    1- export the agent,

    2-convert the Json file back to csv (in order to get the list of intents)

    3. then add in my new intents,

    4. then convert back to the zip file using your tool?

    • >> Do the intent IDs need to be in sequential order, or can they be any random number?

      They can be any random number, but you do need to use a unique number per intent obviously.

      I think it is fair to say the rest of your questions boil down to whether or not my app can help with two way updates. Before I answer that question, can you tell me a little bit more about your use case
      a) what kind of bot are you creating?
      b) what kind of training will you be doing? (i.e. using system entities, or developer entities)?

      • Cool thanks for your reply!!

        a) I will be creating an FAQ bot in a sense. That explains which products are needed based on the user input.

        So I would need to update the bot with new intents (new products).

        b) for the training, I will be using Developer entities. Each new product is an entity.

        I would need to train the bot in various ways, the user will ask for this product.

        • I wrote a reply to your question.

          Also, is there any reason you cannot use a single entity to represent all your products, and use a webhook to fetch the answer? That might make your life much simpler.

  • Hello Aravind,
    I have a question about your application. I would like to be able to update the intents that I have on a regular basis. Is it possible to update the intents and integrate new intents into existing hierarchies and contexts with your app?

    Manuel

    • 1 If you mean update the intents from the CSV file, then yes.

      2 You cannot, however, update the intents in Dialogflow and convert the ZIP file in Dialogflow into a CSV format, because of the inherently flat structure of the CSV file.

      3 As for hierarchies, the CSV format cannot naturally support hierarchical intents (i.e. followup intents) again because of its flat structure.

      4 While in theory it is possible to modify the app to support followup intents, it will also make the UX a lot more confusing.

      5 My app does support contexts though, and you can update the contexts (and their lifespans) in the CSV and recreate the ZIP file.

    • In theory, this could happen if the language of your agent isn’t actually English, and you use the _en suffix for the user says JSON files. If you use a non-English language you should look up the corresponding language code and use it in the usersays file names.

  • Excellent tool! I had to wait 2 days before the app was unlocked on your free course, but was worth the wait. You should note: DialogFlow will not accept ZIPs with any intent names greater than 100 characters.

    • A note for those reading this comment: the app is now a paid app (with a very tiny, token payment). This stops the spammers who are using disposable email addresses to sign up.

      • Another update. I don’t offer the limited app anymore as it confused people a lot. If you want to see a demo of the app, you can check out the link mentioned in this post.

  • Hi Aravind.
    Your tool is awesome, but it doesn’t work with more than one response. I think that it should be possible if you find a way to make a comma in between responses (this should still just be in one column). For example:
    IntentID=1; IntentName=Hello; Query=Hello; Response=Hi, Hello;

    • Hi Ame,
      I have tested that it works with multiple responses, but only if you split the multiple responses into multiple lines in the original CSV file. Also the purpose of putting it into two different lines is that otherwise you will need to deal with the exact same issue you are talking about here – you need a separator to indicate the multiple lines (for example a comma like you have used here), and a) it is hard for people to write such text compared to simply putting it into multiple lines and b) you can run into parsing issues.

  • Thanks for a great documentation Aravind!
    I have a question though. Do you know if it’s possible to return with a custom payload instead of the text?

    • Hi Filip,
      This means you need to specify the custom payload JSON inside the CSV file, which I think is very hard to do. Do you mind posting your sample CSV file online where I could take a look at the custom payload’s format?