05 GPT4 vs Gemini Pro: Comparing Retrieval Augmented Generation (RAG)
How does Gemini Pro compare with GPT-4 when it comes to Retrieval Augmented Generation (RAG)?
What is Retrieval Augmented Generation (RAG)?
The search app I have built provides a simple example if you would like to understand how RAG works.
This is the context – my website contains over 100 articles on the topic of Dialogflow.
Of these, I have written a series of articles about Dialogflow flowcharts.
One of them explains why I think Mindomo is the best tool for creating Dialogflow ES flowcharts.
The series also provides a step-by-step process to convert a Mindomo flowchart into a Dialogflow ES chatbot.
In addition, you can also use this same flowchart to later migrate your Dialogflow ES bot to Dialogflow CX if you like.
GPT-4 Response
Suppose the user asks this question:
How to convert Mindomo flowchart to Dialogflow ES chatbot?
This is the response from GPT4 (from the SiteGPT bot)
To convert a Mindomo flowchart to a Dialogflow ES bot, you can follow these steps:
- Write the Conversation Script:
- Start by writing out your conversation script directly into your mindmap in Mindomo.
- Create nodes for each part of the conversation and format them as needed.
- Add Context Information:
- In Dialogflow ES, contexts are used to manage state within conversations.
- Add context information from your flowchart into corresponding parts of the dialog in Mindomo.
- Define Intent Names:
- Each node or step in your flowchart should correspond to an intent name that will be used later when creating intents in Dialogflow ES.
- Translate Flowchart Nodes Into Intents: 1- Identify each unique interaction point (node) on the chart and define it as an “intent” with training phrases representing what users might say at this stage.5- Define Input/Output Contexts For The Intents Check input/output contexts associated with every defined intention ensuring they match those specified inside respective sections/nodes present throughout entire mapped-out dialogue structure
6-Test Your Bot’s Happy Paths Write test scripts simulating user interactions verifying if everything works correctly after migration process completion
7-Make Adjustments As Needed Based On Testing Results Ensuring All Functionalities Remain Unaffected Post Migration Process Completion
And these are the “links” provided in the chatbot as references:
Here are the corresponding links:
https://botflo.com/04-the-easiest-way-to-migrate-your-dialogflow-es-bot-to-dialogflow-cx
https://botflo.com/03-write-the-conversation-script
https://botflo.com/02-why-mindomo-is-the-best-tool-for-creating-a-dialogflow-flowchart
https://botflo.com/06-translate-the-flowchart-into-dialogflow-es-agent
This is a good example of retrieval augmented generation.
Under the hood, this is what happened.
1 GPT-4 selected the 4 articles you see in the Sources as relevant to the question (retrieval)
2 Then it combined them into a single document (augmentation)
3 Then it provided an answer based on the contents of all the retrieved documents appended together (generation).
But there is a small problem.
Notice that the first link talks about migrating ES bot to CX and not about converting the flowchart to ES.
Should that document have been selected (retrieved)?
And after it was selected, should it have been included in the response?
Gemini response
I used Gemini 1 Pro 2 as the LLM for my Vertex AI Search app.
This is the response from Gemini 1 Pro 2.
Mindomo offers a convenient way to design and manage your Dialogflow ES bot’s flowchart. [ 3 ] Here’s how to convert your Mindomo flowchart into a functional Dialogflow ES bot:
1. Design the Flowchart:
2. Convert to Dialogflow ES:
3. Additional Tips:
And these are the links provided:
-
You can migrate the bot step by step by adding icons into “intents” Inside each Mindomo intent, you can add a Completed Icon (like the Green one I use, but you can use any icon you want) and then check off the intent as soon as you migrate it over to ES.
-
Use a rectangle for representing training phrase and bot response You should put both the training phrase and the bot response inside a single rectangle, since it is conceptually a single block in Dialogflow. For example, here is what it looks like in the Dichotomous Key bot.
-
I used the Search tool to convert my agent ZIP files into a CSV, which I then imported into Airtable. Design a Flowchart I used Mindomo to design the bot flowchart and maintain versions. It made it much easier for me to both design the bot and to translate it into a Dialogflow ES agent.
-
Dialogflow ES Flowcharts The first step is to write out the conversation script into your mindmap directly. This is based on the decision tree bot which can identify a vertebrate class. The flowchart of the decision tree bot looks like this: Inside Mindomo, click on Create Hover your mouse over the Organigram.
-
I have compiled a list based on this article Features draw.io XMind desktop Mindomo Multiline nodes Yes Yes Yes Format different lines inside a single node with different styles Yes No Yes Support different shapes for different nodes Yes Yes Yes Automatically reformat layout to org-tree structure No Yes Yes Support drill down into individual branches No Yes Yes Node labels No Yes No Support adding notes to individual nodes No Yes Yes Easy to share Yes No Yes Easy to collaborate Yes No Yes Aesthetics Possible Good Good Price Free One time payment Recurring payment You can draw 3 free mindmaps using Mindomo and decide for yourself if it works for you.
As you can see, both GPT and Gemini provide similar summary responses.
In fact they even make the same mistake, which I discuss in the Precision vs Recall section
The results from Vertex AI Search are better than the one provided by the GPT-based chatbot, but this is an apples-to-oranges comparison because they are two completely different interfaces.
However, for a search application, the Vertex AI features are clearly more useful for the end user.
This is what I wrote in a previous article:
In my opinion, Vertex AI Search has the potential to be something of a paradigm shift when it comes to implementing custom search applications.
Precision and recall
There are a few things you would expect in the ideal response.
You want the LLM summary response to include information from only the relevant Dialogflow ES articles, but not include anything from the article which talks about migrating ES bot to CX.
Why?
Because even though it refers to the use of Mindomo, a human can perceive that the question does not ask about future migration from ES to CX.
In the context of retrieval, recall refers to fetching ALL the documents which are relevant to the user’s query. If a document is clearly relevant (for example it contains the keyword Mindomo) but is not selected in the retrieval step, then your search application has poor recall.
In the context of retrieval, precision refers to fetching NONE of the documents which are irrelevant to the user’s query.
As it turns out, fetching a document about migrating a Dialogflow ES bot to Dialogflow CX is not precise for the given query “How to migrate Mindomo flowchart to Dialogflow ES bot?” since the user has not mentioned anything about Dialogflow CX.
There are two ways to look at the results provided by GPT and Gemini.
You can either say the overall summary is still pretty useful, and the user can always click on the sources to find out more about that topic.
You can also say that the recall is very good, but the precision isn’t, for the given user query.
Summary
My example shows that both GPT 4 and Gemini Pro perform equally well when it comes to Retrieval Augmented Generation (RAG).
Since I could not implement a direct apples-to-apples comparison by using a Vertex AI Search alternative for GPT-4, this is not a complete comparison.
If someone has done a similar comparison using Azure AI Search (which I believe is the equivalent to Vertex AI search), please provide the relevant link in the comments.
About this website
I created this website to provide training and tools for non-programmers who are building Dialogflow chatbots.
I have now changed my focus to Vertex AI Search, which I think is a natural evolution from chatbots.
Note
BotFlo was previously called MiningBusinessData. That is why you see that watermark in many of my previous videos.