Home / Learn spaCy / How to use custom stop words in spaCy
Learn spaCy

How to use custom stop words in spaCy

This article is part of the Learn spaCy series

Stop words are very common words like “a”, “the” etc which do not provide much information (low information density words) about the text you are analyzing.

spaCy allows you to check if a given word is a stop word. Let us see how we can do that.

First, create a file called stop_words.py and add the following code to it.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp(
    "Dialogflow, previously known as api.ai, is a chatbot framework provided by Google. Google acquired API.AI in 2016.")

full_text = ''
for tok in doc:
    tok_text = tok.text
    if tok.is_stop:
        tok_text = f'[{tok.text}]'
    full_text += tok_text + ' '

print(full_text)

As you can see, I am simply iterating over all the tokens, and if the token is a stop word, I enclose it in square brackets.

This is the output when you run this script

spaCy allows you to modify the list of stop words.

Create a file called custom_stopwords.py and add the following code:

import spacy

cls = spacy.util.get_lang_class('en')
cls.Defaults.stop_words.remove('by')
cls.Defaults.stop_words.add('google')
nlp = spacy.load("en_core_web_sm")
doc = nlp(
    "Dialogflow, previously known as api.ai, is a chatbot framework provided by Google. Google acquired API.AI in 2016.")

full_text = ''
for tok in doc:
    tok_text = tok.text
    if tok.is_stop:
        tok_text = f'[{tok.text}]'
    full_text += tok_text + ' '

print(full_text)

Notice that I have added the word ‘google’ as a stop word, and removed the word ‘by’ from the existing list.

Very important: you must modify the list of stop words before you load the model

This is what the output looks like now

<— End of article —>


This website contains affiliate links. See the disclosure page for more details. 
"The magic key I needed as a non-programmer"

The custom payload generator was the magic key I needed (as a non-programmer) to build a good demo with rich responses in DialogFlow Messenger. I've only used it for 30 minutes and am thrilled. I've spent hours trying to figure out some of the intricacies of DialogFlow on my own. Over and over, I kept coming back to Aravind's tutorials available on-line. I trust the other functionalities I learn to use in the app will save me additional time and heartburn.

- Kathleen R
Cofounder, gathrHealth
"Much clearer than the official documentation to be honest"

Thanks a lot for the advice (of buying and following your videos)! They helped a lot indeed. Everything is very clear when you explain, much clearer than the official documentation to be honest 🙂

Neuraz T
Review for Learn Dialogflow CX
"I will strongly recommend this course because even I can learn how to design chatbot (no programming background)"

I think Aravind really did a great job to introduce dialogflow to people like me, without programming background. He organizes his course in very clear manner since I have been a college professor for 20 years. It is very easy for me to recognize how great Aravind’s course is! Very use-friend and very easy to follow. He doesn’t have any strong accent when he gives the lectures. It is so easy for me to understand. Really appreciate it.

Yes, I will strongly recommend this course because even I can learn how to design chatbot (no programming background) after studying Avarind’s course, you definitely can!

Ann Cai
Review for Learn Dialogflow ES

Similar Posts