Wrangling Text Data
Text data are one of the most common and information-rich forms of unstructured data, and Natural Language Processing (NLP) provides techniques for helping computers understand, analyze, and extract useful information from human language. Unlike structured numerical data, text can contain punctuation, different word forms, irrelevant words, entities, sentiment, and contextual meaning, which makes it necessary to transform and prepare text before it can be effectively analyzed by machine-learning algorithms.
Text processing is widely used in applications such as search engines, chatbots, recommendation systems, document classification, sentiment analysis, information extraction, and customer-feedback analysis. In this episode, we take a step-by-step approach to handling text data, including cleaning and preparing text, removing punctuation and stop words, tokenizing and stemming words, tagging parts of speech, identifying named entities, converting text into numerical representations, weighting word importance, measuring text similarity for search, and finally using a sentiment-analysis classifier to extract meaning from text.
Objectives
Clean and preprocess text data using Python string operations, regular expressions, and Unicode-aware techniques.
Tokenize text and apply common linguistic preprocessing techniques, including stop-word removal and stemming.
Analyze text using part-of-speech tagging and named-entity recognition to extract useful linguistic and semantic information.
Convert text into numerical representations using Bag-of-Words, n-grams, and TF-IDF, and use text vectors for similarity-based search.
Apply pretrained NLP models to perform tasks such as sentiment analysis and incorporate text-derived information into machine-learning workflows.
Instructor note
40 minutes teaching
30 minutes exercising/discussion
1. Cleaning Text with Python String Operations
Text data often needs to be cleaned before we can use it to create features or feed it into a machine-learning algorithm. This preprocessing step can involve removing unwanted characters, correcting formatting, standardizing text, or transforming strings into a more consistent form. For many basic cleaning tasks, Python’s built-in string methods are sufficient.
In the following example, we work with the titles and authors of three books. The text contains leading and trailing whitespace, punctuation, and inconsistent capitalization. We will progressively use three of Python’s core string operations strip(), replace(), and split(), to manipulate text without requiring additional libraries.
# create text data
text_data = [
" Absolutely small. By Mike Fayer. ",
" Today Is The night. By Jarek Prakash ",
" Three body. By Cixin Liu ",
" Build a large language model. By Sebastian Raschka"
]
The first step is to remove unnecessary whitespace from the beginning and end of each string. We can do this with the strip() method.
# remove leading and trailing whitespace
strip_whitespace = [string.strip() for string in text_data]
# display cleaned text
strip_whitespace
['Absolutely small. By Mike Fayer.',
'Today Is The night. By Jarek Prakash',
'Three body. By Cixin Liu',
'Build a large language model. By Sebastian Raschka']
Note
Notice that strip() removes whitespace from the beginning and end of each string, but does not change the whitespace between words.
Then we remove the periods from each string using replace() method.
# remove periods
remove_periods = [string.replace(".", "") for string in strip_whitespace]
# display cleaned text
remove_periods
['Absolutely small By Mike Fayer',
'Today Is The night By Jarek Prakash',
'Three body By Cixin Liu',
'Build a large language model By Sebastian Raschka']
1.1 Creating a custom text-cleaning function
As text-cleaning requirements become more complex, repeatedly applying individual string operations can become difficult to manage. A better approach is to place our cleaning logic inside a custom function. We can then apply that function to every item in our dataset.
For example, suppose we want to standardize the capitalization of our text. We can create a function called capitalizer() that converts a string to uppercase.
# create a custom transformation function
def capitalizer(string: str) -> str:
return string.upper()
# apply function to each string
capitalized_text = [
capitalizer(string)
for string in remove_periods
]
capitalized_text
['ABSOLUTELY SMALL BY MIKE FAYER',
'TODAY IS THE NIGHT BY JAREK PRAKASH',
'THREE BODY BY CIXIN LIU',
'BUILD A LARGE LANGUAGE MODEL BY SEBASTIAN RASCHKA']
This example illustrates an important principle in text preprocessing: rather than manually cleaning individual records, we can define a transformation once and apply it consistently to an entire collection of text.
1.2 Using regular expressions for complex cleaning
Python’s standard string methods are useful for straightforward transformations, but some cleaning tasks require us to identify patterns rather than specific characters or words. For these situations, we can use regular expressions. Regular expressions allow us to search for and manipulate text based on patterns, such as finding email addresses, URLs, numbers, repeated whitespace, or groups of characters.
Python provides regular-expression functionality through the built-in re module. Here as a simple example, we can create a function that replaces every letter with the character X.
# import regular expression library
import re
# create a function that replaces letters with X
def replace_letters_with_X(string: str) -> str:
return re.sub(r"[a-zA-Z]", "X", string)
Here the regular expression [a-zA-Z] matches any uppercase or lowercase letter. The re.sub() function then replaces each matching character with X.
# apply function
masked_text = [
replace_letters_with_X(string)
for string in remove_periods
]
masked_text
['XXXXXXXXXX XXXXX XX XXXX XXXXX',
'XXXXX XX XXX XXXXX XX XXXXX XXXXXXX',
'XXXXX XXXX XX XXXXX XXX',
'XXXXX X XXXXX XXXXXXXX XXXXX XX XXXXXXXXX XXXXXXX']
Note
The appropriate cleaning steps depend on what we intend to do with the text. For example, a search system may benefit from lowercasing and removing certain punctuation, while a sentiment-analysis system may need to preserve punctuation, emojis, or words such as “not” because they can carry important information.
The goal is therefore not to remove as much text as possible, but to remove unnecessary noise while preserving information that is useful for tasks at hand.
2. Removing Punctuation
After cleaning the basic formatting of text, the next step we may want to consider is removing punctuation. Punctuation marks such as periods, commas, exclamation marks, question marks, quotation marks, and parentheses can introduce additional variations in our text. For example, a model might otherwise treat hello, hello!, and hello? as different tokens, even though they contain the same underlying word.
Removing punctuation can simplify the text and reduce the number of unique features we need to work with. However, punctuation is not always noise, and it can carry important information about meaning, tone, or type of text we are analyzing. For example, Right? is different from Right!, and a question mark can be a useful feature when identifying questions. Therefore, punctuation should only be removed when it makes sense for the downstream task.
In the following examples, we will use Python to identify punctuation characters and remove them from our text. We will begin with a unicode-aware approach that can handle a much broader range of punctuation than simply specifying a few characters such as ., ,, and !.
First, we create a small collection of text strings containing different types of punctuation.
# create text data
text_data = [
"Hi!!!! I. Love. This. Song....!!",
"10000%%% Agree!!!! ... !!! Love??IT??!!",
"Right?!?!%%%"
]
text_data
['Hi!!!! I. Love. This. Song....!!',
'10000%%% Agree!!!! ... !!! Love??IT??!!',
'Right?!?!%%%']
Notice that our text contains several different punctuation characters, including exclamation marks !, periods ., percent signs %, and question marks ?.
2.1 Using unicode to identify punctuation
Python’s unicodedata module provides information about individual Unicode characters, including their character categories. This allows us to identify punctuation more systematically.
# load libraries
import sys
import unicodedata
# check unicode category of several characters
characters = [".", "!", "?", "A", "1"]
for character in characters:
print(
character,
unicodedata.category(character)
)
. Po
! Po
? Po
A Lu
1 Nd
The exact category codes are useful because Unicode classifies characters into groups. In particular, categories beginning with Po represent punctuation, Lu represents letter, and Nd represents number and decimal digit.
2.2 Removing punctuation
To remove punctuation, we can search through the Unicode character set and identify characters whose category starts with “P”.
# create a dictionary of punctuation characters
punctuation = dict.fromkeys(
i
for i in range(sys.maxunicode)
if unicodedata.category(chr(i)).startswith("P")
)
# display a few punctuation characters
list(punctuation.keys())[:10]
[33, 34, 35, 37, 38, 39, 40, 41, 42, 44]
The resulting dictionary contains Unicode punctuation characters as keys and None as their values.
Why use
None?Python’s
str.translate()method interprets a mapping toNoneas an instruction to remove that character.
# remove punctuation from each string
cleaned_text = [
string.translate(punctuation)
for string in text_data
]
# display result
cleaned_text
['Hi I Love This Song', '10000 Agree LoveIT', 'Right']
Note
There are other, sometimes more readable, ways to remove punctuation. For example, we could use a regular expression:
import re
cleaned_text = [re.sub(r"[^\w\s]", "", string) for string in text_data]
cleaned_text
Or we could process individual characters:
cleaned_text = [
"".join(character for character in string if character not in string.punctuation)
for string in text_data
]
These approaches may be easier to understand when first learning text processing, while translate() is often a good choice when we need to process a large amount of text efficiently.
2.3 Preserving selected punctuation
It should be noted that punctuation may contains information. For example, “Right!” and “Right?” have the same word, but the ppunctuation changes the meaning, tone, or purpose of the sentence. Therefore before automatically removing punctuation, we should consider whether it contains information that our model needs.
Depending on the specific task, rather than removing all punctuation marks, we can choose to remove only the characters that are not needed. For example, suppose we want to remove periods and commas while keeping question marks and exclamation marks.
text_data = ["I love this!", "Is this working?", "This is great, really great."]
translation_table = str.maketrans("", "", ".,")
cleaned_text = [
text.translate(translation_table)
for text in text_data
]
cleaned_text
['I love this!', 'Is this working?', 'This is great really great']
Removing punctuation can make text easier to process and can reduce unnecessary variation in the features we create.
Python’s
translate()method provides an efficient way to perform this operation, whilereand character-level processing provide alternative approaches.However, punctuation is not automatically noise.
Before removing it, we should consider downstream task.
If punctuation contains information that can help the model, such as distinguishing questions, expressing emotion, or identifying document structure, we should preserve it or selectively remove only punctuation that is not useful.
3. Tokenizing Text
After cleaning our text, we often want to break it into smaller pieces that a computer can work with. Such a process is called tokenization, which splits text into smaller units called tokens. Depending on the task, tokens can be individual words, sentences, characters, or even smaller pieces of words.
Python provides several libraries for tokenizing text. One of the most widely used is the Natural Language Toolkit (NLTK), which provides a broad collection of tools for text processing and natural language analysis, including word tokenization and sentence tokenization.
# download punkt_tab resource if needed
import nltk
nltk.download("punkt")
nltk.download("punkt_tab")
[nltk_data] Downloading package punkt to /Users/yonglei/nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package punkt_tab to
[nltk_data] /Users/yonglei/nltk_data...
[nltk_data] Package punkt_tab is already up-to-date!
True
3.1 Word tokenization
Word tokenization is a common step after cleaning text data, and it transforms a continuous string of text into a sequence of individual words. Once text has been tokenized, we can perform additional operations such as counting word frequencies, removing stop words, stemming or lemmatizing words, tagging parts of speech, or converting tokens into numerical features.
We begin with a simple text example and tokenize it into several words.
from nltk.tokenize import word_tokenize
# create text
text = """
Talk is cheap, show me the code; 💻
Code is cheap, show me the prompt! 🤖
"""
# tokenize text into words
tokens = word_tokenize(text)
print(len(tokens))
tokens
20
['Talk',
'is',
'cheap',
',',
'show',
'me',
'the',
'code',
';',
'💻',
'Code',
'is',
'cheap',
',',
'show',
'me',
'the',
'prompt',
'!',
'🤖']
Our original sentence was a single string. After tokenization, it becomes a list of individual tokens. We can work with each token separately, or we can work with multiple tokens together.
print(tokens[0], tokens[1], tokens[-1])
sentence_1 = " ".join(tokens[:8] + tokens[-1:])
sentence_2 = " ".join(tokens[10:-1] + tokens[9:10])
print(sentence_1)
print(sentence_2)
Talk is 🤖
Talk is cheap , show me the code 🤖
Code is cheap , show me the prompt ! 💻
Note
Tokenization and punctuation removal are separate preprocessing steps.
If we removed punctuation before tokenization,
!,;and,would no longer appear.If we tokenize first, punctuation can be preserved as its own token and handled later.
Which approach is preferable depends on our downstream task.
Acutally tokenization is more than simply splitting a string on spaces. From code snippet below, we can identify that split() method separates text at whitespace, whereas an NLP tokenizer applies rules designed to identify meaningful units of language, and can recognize punctuation and language-specific patterns.
text = "I don't think this is working."
tokens = word_tokenize(text)
print(tokens)
print(text.split())
['I', 'do', "n't", 'think', 'this', 'is', 'working', '.']
['I', "don't", 'think', 'this', 'is', 'working.']
Warning
It is important to recognize that there is no single “correct” way to tokenize text. Traditional NLP workflows often use word-level tokenization, while modern pretrained language models may use more sophisticated subword tokenization techniques. For example, models such as Google’s BERT use model-specific tokenization to break text into tokens that may represent complete words or parts of words. Subword tokenization helps modern language models handle unfamiliar words, different word forms, and large vocabularies. Nevertheless, word-level tokenization remains a useful and common technique when we want to work directly with individual words and construct traditional text features.
3.2 Sentence tokenization
We can also tokenize text into sentences rather than individual words. This is known as sentence tokenization or sentence segmentation. For example, we split the original text into two sentences as shown below.
# import sentence tokenizer
from nltk.tokenize import sent_tokenize
# create text containing two sentences
text = (
"Talk is cheap, show me the code 💻. "
"Code is cheap, show me the prompt 🤖!"
)
# tokenize text into sentences
sentences = sent_tokenize(text)
print(sentences)
print(len(sentences))
['Talk is cheap, show me the code 💻.', 'Code is cheap, show me the prompt 🤖!']
2
So, instead of producing individual words, sent_tokenize() identifies the boundaries between sentences and returns a list containing two sentence, which can be verified via len(sentences).
3.3 Combining sentence and word tokenizations
In some applications, we may want to first identify individual sentences and then tokenize each sentence into words. This structure can be useful when we need to preserve the relationship between words and the sentences in which they occur.
text = (
"Talk is cheap, show me the code 💻. "
"Code is cheap, show me the prompt 🤖!"
)
# split text into sentences
sentences = sent_tokenize(text)
# tokenize each sentence into words
tokenized_sentences = [
word_tokenize(sentence)
for sentence in sentences
]
tokenized_sentences
[['Talk', 'is', 'cheap', ',', 'show', 'me', 'the', 'code', '💻', '.'],
['Code', 'is', 'cheap', ',', 'show', 'me', 'the', 'prompt', '🤖', '!']]
A small tokenization exercise
Let’s try tokenizing a slightly more realistic piece of text:
text = """
Natural language processing is fascinating!
It allows computers to work with human language.
But text data can be messy.
"""
sentences = sent_tokenize(text)
for sentence in sentences:
print(word_tokenize(sentence))
Run the code and try to explore:
How many sentences were identified?
How many words are in each sentence?
Which punctuation marks became tokens?
What would happen if we removed punctuation before tokenizing?
How might contractions such as “don’t” be represented?
Why tokenization matters?
Tokenization is often the bridge between raw text and subsequent NLP processing. Tokenization converts text into manageable units that can be analyzed by an NLP system. Word tokenization is particularly useful when our features are based on individual words, while sentence tokenization allows us to analyze text one sentence at a time.
Although modern language models often use subword tokenization rather than traditional word-level tokenization, understanding word and sentence tokenization remains fundamental. It provides an intuitive introduction to how unstructured language can be transformed into structured data and prepares us for the next stages of our NLP workflow, including removing stop words, stemming words, tagging parts of speech, and creating numerical text features.
4. Handling Stop Words
After tokenizing our text, we may want to consider is removing stop words. Stop words are common words but provide little useful information for a particular task. This term mostly refers to very frequent words in natural language, such as “a,” “an,” “the,” “is,” “of,” “to,” “on,” and “and”. Removing them can reduce the number of tokens we need to process and allow a model to focus on words that are more useful for distinguishing documents or identifying topics. For example, in sentence “I am going to the store and the park.”, words such as “I”, “am”, “to”, “the”, and “and” may contain less information for some applications than words such as “store” and “park”.
However, stop-word removal is not always appropriate. Whether we should remove stop words depends on specific downstream task. For example, in sentiment analysis, words such as “not” can be extremely important. Consider these two sentences: “I like this product” and “I do not like this product.” If “not” is removed as a stop word, the two sentences could become much more similar, even though their meanings are completely opposite.
Therefore, stop-word removal should be treated as a preprocessing decision rather than a mandatory step in every NLP pipeline.
4.1 Using NLTK’s stop-word list
NLTK’s stopwords corpus contains lists of commonly used stop words for several languages. We can load the English stop-word list as follows. This gives us a predefined collection of common English words that we can use when filtering tokenized text.
# download stop-word corpus if needed
import nltk
nltk.download("stopwords")
from nltk.corpus import stopwords
# load English stop-word list
stop_words = stopwords.words("english")
stop_words[:10]
[nltk_data] Downloading package stopwords to
[nltk_data] /Users/yonglei/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
['a', 'about', 'above', 'after', 'again', 'against', 'ain', 'all', 'am', 'an']
4.2 Removing stop words from tokenized text
We start with a list of already tokenized words tokenized_words and remove any token that appears in NLTK’s English stop-word list.
tokenized_words = [
"i", "am", "going", "to", "go",
"to", "the", "store", "and", "park"
]
# remove stop words
filtered_words = [
word
for word in tokenized_words
if word not in stop_words
]
filtered_words
['going', 'go', 'store', 'park']
It turns out that common words such as “i”, “am”, “to”, “the”, and “and” have been removed, while the more content-rich words “going”, “go”, “store”, and “park” remain.
Our stop-word list contains lowercase words. If our tokens contain uppercase letters, we may need to normalize them before checking for stop words. For example we have tokenized_words, using str.lower() ensures that “The” and “the” are treated as the same word. This is one reason that lowercasing is often performed during text preprocessing.
tokenized_words = ["The", "cat", "is", "on", "the", "mat"]
filtered_words = [
word
for word in tokenized_words
if word.lower() not in stop_words
]
filtered_words
['cat', 'mat']
Callout
Stop-word removal is a common preprocessing technique used to reduce number of tokens and focus our analysis on words that provide more information.
NLTK provides a convenient starting point with its predefined English stop-word list, but that list should not automatically be treated as the correct list for every problem.
Noted that stop-word removal is task-dependent, and we should understand what information we are removing before applying transformation.
In particular, words that appear unimportant in isolation can sometimes be essential when combined with other words or when interpreting meaning of an entire sentence.
5. Stemming and Lemmatization
After tokenizing text, and optionally removing stop words, we can normalize words using either stemming or lemmatization. These are alternative approaches: stemming reduces words to a common stem, while lemmatization aims to find their proper dictionary form.
NLTK provides several algorithms for both stemming and lemmatization. One of the most widely used stemming algorithms is the Porter stemming algorithm, implemented by NLTK’s PorterStemmer. It is a relatively simple and widely used approach that applies a series of rules to remove or replace common suffixes and produce a stem.
For lemmatization, NLTK provides the WordNetLemmatizer, which uses the WordNet lexical database to reduce words to their base or dictionary forms. Unlike stemming, lemmatization aims to produce a valid word and can take the word’s part of speech into account.
5.1 Stemming with Porter Stemmer
Stemming is such a process reduces a word to a shorter form, called its stem, by removing or modifying common prefixes and suffixes. For example, the words “tradition” and “traditional” are different words, but they share the same general concept. A stemming algorithm may reduce both to the stem “tradit”. Similarly, words such as “connect”, “connected”, “connecting”, and “connection” may be reduced to forms that are much closer to one another. This can be useful when we want to compare documents, count related words, or reduce the size of our vocabulary.
Let’s use NLTK’s PorterStemmer to stem a list of tokenized words.
from nltk.stem.porter import PorterStemmer
# create word tokens
tokenized_words = [
"i", "am", "humbled", "by",
"this", "traditional", "meeting"
]
# create stemmer
porter = PorterStemmer()
# apply stemmer to each word
stemmed_words = [
porter.stem(word)
for word in tokenized_words
]
stemmed_words
['i', 'am', 'humbl', 'by', 'thi', 'tradit', 'meet']
Notice that several words have been shortened: “humbled → humbl”, “this → thi”, “traditional → tradit”, and “meeting → meet”. The results may initially look strange because “humbl” and “tradit” are not standard English words. This is expected as the purpose of stemming is not to produce grammatically correct words, but to reduce related word forms to a common representation.
5.2 Stemming a complete sentence
In practice, we usually do not stem just one word. We apply the stemmer to all tokens from a piece of text.
text = """
Natural language processing is fascinating
and it allows computers to work with human language.
"""
# tokenize text
tokens = word_tokenize(text.lower())
# stem each token
stemmed_tokens = [
porter.stem(word)
for word in tokens
]
stemmed_tokens
['natur',
'languag',
'process',
'is',
'fascin',
'and',
'it',
'allow',
'comput',
'to',
'work',
'with',
'human',
'languag',
'.']
The result contains the stemmed representation of each token, along with punctuation tokens unless punctuation has already been removed.
After stemming a complete sentence, we compare the original tokens with their stems.
for original, stemmed in zip(tokens, stemmed_tokens):
print(f"{original:11} → {stemmed}")
natural → natur
language → languag
processing → process
is → is
fascinating → fascin
and → and
it → it
allows → allow
computers → comput
to → to
work → work
with → with
human → human
language → languag
. → .
This gives us a direct view of what the stemming algorithm is doing. We see transformations such as “natural → natur”, “language → languag”, “processing → process”, “fascinating → fascin”, and “computers → comput”.
Why is stemming useful?
Suppose we have two documents: “We are studying machine learning.”, and “The researchers studied machine learning.”.
Without normalization, “studying” and “studied” are different tokens.
After stemming, they may be reduced to a common or similar stem: “studying → studi” and “studied → studi”.
This makes it easier for traditional text-processing techniques to recognize that the two documents contain related terms.
Stemming can therefore be useful when:
Reducing vocabulary size
Comparing documents
Counting related word forms
Building traditional information-retrieval systems
5.3 Stemming vs. lemmatization
Stemming is not perfect. As stemming relies on rules rather than a complete understanding of language, it can sometimes produce undesirable results. For example, the resulting stems from the following code snippet may not be complete or meaningful English words.
Another important point is that stemming focuses on reducing related words to a common form, such as “university → univers” and “universe → univers”, rather than producing a grammatically correct word. As a result, the output can sometimes look incomplete or even unnatural, such as “univers”, “studi”, or “organ”.
# stemming
words = [
"universities",
"universe",
"studies",
"study",
"organization",
"organize"
]
for word in words:
print(f"{word:12} → {porter.stem(word)}")
universities → univers
universe → univers
studies → studi
study → studi
organization → organ
organize → organ
Lemmatization, on the other hand, aims to reduce words to their proper dictionary or base forms. For example, “studies → study” and “universities → university.” So, while stemming mainly cuts words down based on predefined rules, lemmatization uses linguistic information to identify the correct base form.
# download wordnet resource if needed
import nltk
nltk.download('wordnet')
# lemmatization
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
words = [
"universities",
"universe",
"studies",
"study",
"organization",
"organize"
]
for word in words:
print(f"{word:12} → {lemmatizer.lemmatize(word)}")
universities → university
universe → universe
studies → study
study → study
organization → organization
organize → organize
Stemming vs. lemmatization
Stemming and lemmatization are generally alternative approaches.
Stemming uses a set of predefined rules to shorten words.
traditional → tradit, studies → studi
It is faster and simpler, but may produce unnatural forms.
Lemmatization attempts to determine appropriate dictionary or base form.
studies → study, better → good (depending on context and lemmatizer)
Lemmatization is more linguistically meaningful, and produces dictionary/base forms, but usually requires more computational resources and linguistic information.
6. Tagging Parts of Speech (PoS)
After cleaning, tokenizing, and normalizing our text, we can take the next step and identify the grammatical role of each word. This process is called part-of-speech (PoS) tagging. A PoS tagger assigns a grammatical category to each token based on how the word is used in its surrounding context. Common categories include nouns, verbs, adjectives, adverbs, pronouns, and prepositions.
For example, in the sentence “Chris loved outdoor running.”, we might identify “Chris” as a proper noun, “loved” as a verb, “outdoor” as an adjective, and “running” as a verb or gerund. POS information can be useful when we want to understand grammatical structure of text or create features based on particular types of words.
6.1 Using NLTK’s pretrained PoS tagger
If our text is English and comes from a general domain, the simplest approach is often to use NLTK’s pretrained POS tagger. The pretrained model has already learned patterns from a large collection of tagged text, so we do not need to create and label our own training data.
Warning
A general-purpose tagger may not perform as well on specialized domains such as medicine, law, finance, or highly technical text.
NLTK allows us to train our own tagger, but doing so requires a sufficiently large tagged corpus.
Creating such a corpus is time-consuming and expensive, so training a custom tagger is generally considered only when a pretrained model is not accurate enough for the application.
We begin with a simple example. We first tokenize our text and then pass tokens to NLTK’s pretrained POS tagger.
# download averaged_perceptron_tagger resource if needed
nltk.download("averaged_perceptron_tagger")
nltk.download("averaged_perceptron_tagger_eng")
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] /Users/yonglei/nltk_data...
[nltk_data] Package averaged_perceptron_tagger is already up-to-
[nltk_data] date!
[nltk_data] Downloading package averaged_perceptron_tagger_eng to
[nltk_data] /Users/yonglei/nltk_data...
[nltk_data] Package averaged_perceptron_tagger_eng is already up-to-
[nltk_data] date!
True
from nltk import pos_tag, word_tokenize
# create text
text_data = "Chris loved outdoor running"
# tokenize and tag text
text_tagged = pos_tag(word_tokenize(text_data))
text_tagged
[('Chris', 'NNP'), ('loved', 'VBD'), ('outdoor', 'RP'), ('running', 'VBG')]
We get a list of tuples, “Chris → NNP”, “loved → VBD”, “outdoor → RP”, and “running → VBG”. Each tuple contains (word, PoS tag). The PoS tagger is making a prediction about the grammatical role of each word based on the word itself and its context.
NLTK’s PoS tagger uses the Penn Treebank tag set, a widely used collection of PoS labels for English. Here are some commonly encountered tags.
Tag |
Meaning |
Example |
|---|---|---|
NN |
Noun, singular |
book |
NNS |
Noun, plural |
books |
NNP |
Proper noun, singular |
Chris |
VB |
Verb, base form |
run |
VBD |
Verb, past tense |
loved |
VBG |
Verb, gerund/present participle |
running |
JJ |
Adjective |
amazing |
RB |
Adverb |
quickly |
PRP |
Personal pronoun |
I |
DT |
Determiner |
the |
IN |
Preposition/subordinating conjunction |
in |
Once our text has been tagged, we can use the tags to extract particular types of words. For example, suppose we want to find all nouns. We can filter the tagged tokens using the noun tags.
# define noun tags
noun_tags = {"NN", "NNS", "NNP", "NNPS"}
# extract nouns
nouns = [
word
for word, tag in text_tagged
if tag in noun_tags
]
nouns
['Chris']
We can use the same approach to extract adjectives with adjective_tags = {"JJ", "JJR", "JJS"} or verbs with verb_tags = {"VB", "VBD", "VBG", "VBN", "VBP", "VBZ"}.
# define adjective tags
adjective_tags = {"JJ", "JJR", "JJS"}
# extract adjectives
adjectives = [
word
for word, tag in text_tagged
if tag in adjective_tags
]
adjectives
[]
7. Performing Named-Entity Recognition (NER)
After PoS tagging in text, we can take another step toward understanding the information contained in text by identifying specific entities. This process is called Named-Entity Recognition (NER). NER is the process of identifying and classifying real-world entities mentioned in freeform text, such as people, organizations, locations, dates, products, and monetary values. For example, consider the sentence “Elon Musk offered to buy Meta using $21B of his own money”. A human reader can quickly recognize that “Elon Musk” is a person, “Meta” is an organization, and “$21B” represents a monetary value.
NER allows a computer to make these same distinctions automatically. Tools such as spaCy provide pretrained NLP pipelines and machine learning models that can identify many common types of entities without requiring us to build a model from scratch.
In real-world applications, however, organizations may train or fine-tune NER models when the pretrained model does not recognize the specialized entities required by their application. This can be particularly important for domains such as medicine, law, finance, or scientific research, where domain-specific terminology may not be adequately represented by a general-purpose model. Training a custom NER model is outside the scope of this example.
7.1 Using spaCy for NER
We can use spaCy’s pretrained English pipeline to identify entities in our text. We first import spaCy, loading the pretrained English language model, and then pass our text to spaCy pipeline.
import spacy
# load pretrained English NLP pipeline
nlp = spacy.load("en_core_web_sm")
# create and parse text
text = (
"Elon Musk offered to buy Meta "
"using $21B of his own money."
)
doc = nlp(text)
The nlp() call processes the text and creates a spaCy Doc object, which contains information about individual tokens as well as any entities that the model identifies. We can access all recognized entities through the doc.ents attribute:
# display recognized entities
print(doc.ents)
(Elon Musk, Meta, 21B)
We can then loop through the entities and display both the entity text and its predicted label.
# display each entity and its label
for entity in doc.ents:
print(entity.text, entity.label_, sep=", ")
Elon Musk, PERSON
Meta, ORG
21B, MONEY
Here, spaCy has identified three entities:
Entity |
Label |
Meaning |
|---|---|---|
Elon Musk |
PERSON |
Person |
Meta |
ORG |
Organization |
21B |
MONEY |
Monetary value |
The label tells us what type of entity spaCy believes it has found. Some commonly used spaCy entity labels include:
Label |
Description |
Example |
|---|---|---|
GPE |
Geopolitical entity |
France |
LOC |
Location |
Pacific Ocean |
DATE |
Date |
August 2026 |
TIME |
Time |
5 PM |
PRODUCT |
Product |
iPhone |
EVENT |
Event |
Olympics |
CARDINAL |
Number |
21 |
7.2 Extracting entities into a structured format
Instead of simply printing these entities, we can store them in a Python data structure. This gives us structured records as shown below.
entities = [
{
"text": entity.text,
"label": entity.label_
}
for entity in doc.ents
]
entities
[{'text': 'Elon Musk', 'label': 'PERSON'},
{'text': 'Meta', 'label': 'ORG'},
{'text': '21B', 'label': 'MONEY'}]
Once entities have been identified, we can filter them by their labels, for example to extract people, organizations, and monetary values. This allows us to turn a general NER process into a targeted information-extraction task.
people = [
entity.text
for entity in doc.ents
if entity.label_ == "PERSON"
]
people
['Elon Musk']
organizations = [
entity.text
for entity in doc.ents
if entity.label_ == "ORG"
]
organizations
['Meta']
7.3 Applying NER to multiple documents
In a realistic NLP application, we will usually have many documents rather than a single sentence. The code snippet below demonstrates how the same pretrained model can be applied repeatedly to a collection of documents.
texts = [
"Elon Musk founded SpaceX.",
"Microsoft is headquartered in Redmond.",
"Apple reported strong sales in 2025."
]
for text in texts:
doc = nlp(text)
print(f"\nText: {text}")
for entity in doc.ents:
print(
f" {entity.text} → {entity.label_}"
)
Text: Elon Musk founded SpaceX.
Elon Musk → PERSON
Text: Microsoft is headquartered in Redmond.
Microsoft → ORG
Redmond → GPE
Text: Apple reported strong sales in 2025.
Apple → ORG
2025 → DATE
Warning
Important limitation: NER is a prediction
Although pretrained NER models are convenient, they are not perfect. A model may fail to recognize an entity, assign the wrong label, or behave differently when it encounters unfamiliar terminology.
For example, a general-purpose model trained primarily on news and general English may struggle with highly specialized medical or scientific entities.
We can inspect the model’s predictions, but we should not automatically assume that every prediction is correct.
This is especially important when NER is being used to create data for a high-stakes application.
NER as a data-extraction pipeline
POS tagging tells us what grammatical role a word plays, while NER tells us whether a sequence of words refers to a particular type of real-world entity.
NER transforms important pieces of unstructured text into structured information.
With a pretrained spaCy model, we can identify entities such as people, organizations, locations, dates, and monetary values with relatively little code.
Next stage will move from linguistic information to numerical representations of text, allowing machine-learning algorithms to work directly with the information contained in documents.
8. Encoding Text as a Bag of Words
After cleaning, tokenizing, and extracting linguistic information from text, we need to take another important step: turning text into numerical features that an algorithm can use. One of the simplest and most widely used approaches is the Bag-of-Words (BoW) model. A BoW model creates a feature for every unique word in the text collection. Each feature records how many times that word occurs in a particular observation. The approach is called a “bag” of words because it focuses on which words occur and how frequently they occur, while generally ignoring their order.
For example in the sentence “I love France. France!”, the word “France” appears twice, so the “france” feature has a value of 2. The word “love” appears once, so the “love” feature has a value of 1.
8.1 Creating and viewing a BoW feature matrix
Scikit-learn provides CountVectorizer, a convenient tool for converting a collection of text documents into a BoW feature matrix. We begin with a small dataset text_data, creating a CountVectorizer object and then use it to transform the text.
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
# create text data
text_data = np.array([
"I love France. France!",
"Argentina is best",
"Spain beats both"
])
# create a BoW vectorizer
count = CountVectorizer()
# create BoW feature matrix
bag_of_words = count.fit_transform(text_data)
bag_of_words
<Compressed Sparse Row sparse matrix of dtype 'int64'
with 8 stored elements and shape (3, 8)>
The output tells us that this dataset contains:
3 observations — the three sentences
8 features — the eight unique words identified by
CountVectorizer8 nonzero values — only eight word occurrences need to be stored
CountVectorizer returns a sparse matrix by default. As our dataset is very small, we can convert the matrix into a regular NumPy array so that we can see the values.
# convert sparse matrix to a regular array
bag_of_words.toarray()
array([[0, 0, 0, 0, 2, 0, 1, 0],
[1, 0, 1, 0, 0, 1, 0, 0],
[0, 1, 0, 1, 0, 0, 0, 1]])
Each row represents one observation, while each column represents one word feature.
However, the numerical matrix is difficult to interpret without knowing which word corresponds to each column.
We can retrieve the feature names using get_feature_names_out().
# show feature names
feature_names = count.get_feature_names_out()
feature_names
array(['argentina', 'beats', 'best', 'both', 'france', 'is', 'love',
'spain'], dtype=object)
With the list of feature names, we can therefore map the columns of our matrix to the corresponding words.
Text |
argentina |
beats |
best |
both |
france |
is |
love |
spain |
|---|---|---|---|---|---|---|---|---|
I love France. France! |
0 |
0 |
0 |
0 |
2 |
0 |
1 |
0 |
Argentina is best |
1 |
0 |
1 |
0 |
0 |
1 |
0 |
0 |
Spain beats both |
0 |
1 |
0 |
1 |
0 |
0 |
0 |
1 |
Note
A note about single-character words
You may have noticed that “I” does not appear in the vocabulary.
By default,
CountVectorizeruses a token pattern that requires a token to contain at least two alphanumeric characters. As a result, the single-character word “I” is ignored.We can inspect the default token pattern:
# display CountVectorizer's default token pattern count.token_pattern
If we want single-character words to be included, we can change token pattern. Check a complete version of code example.
# include single-character words count_single = CountVectorizer( token_pattern=r"(?u)\b\w+\b" ) bag_of_words_single = count_single.fit_transform(text_data) # show feature names count_single.get_feature_names_out()
This indicates that the preprocessing rules used by a vectorizer affect the features that are ultimately created.
8.2 Creating n-gram features
By default, CountVectorizer creates one feature for each individual word. These are called unigrams. However, sometimes the combination of words is more informative than individual words. For example, the phrase “machine learning” consists of two words, but together they represent a specific concept. If we treat them as separate features, “machine” and “learning”, we may lose the meaning conveyed by the phrase as a whole.
As such, we can use n-grams to create features representing combinations of words.
A 1-gram (unigram) contains one word: “machine” and “learning”.
A 2-gram (bigram) contains two consecutive words: “machine learning”.
A 3-gram (trigram) contains three consecutive words: “natural language processing”.
We can control the size of the n-grams using the ngram_range parameter.
# create a vectorizer using unigrams and bigrams
count_ngrams = CountVectorizer(
ngram_range=(1, 2)
)
bag_of_words_ngrams = count_ngrams.fit_transform(text_data)
count_ngrams.get_feature_names_out()
array(['argentina', 'argentina is', 'beats', 'beats both', 'best', 'both',
'france', 'france france', 'is', 'is best', 'love', 'love france',
'spain', 'spain beats'], dtype=object)
With ngram_range=(1, 2), we obtain both “1-word features” and “2-word features”. For example “Spain beats both” can generate features such as “spain”, “beats”, “both”, “spain beats”, “beats both”. If we just want to get bigrams, we can set ngram_range=(2, 2).
count_bigrams = CountVectorizer(
ngram_range=(2, 2)
)
count_bigrams.fit_transform(text_data)
count_bigrams.get_feature_names_out()
array(['argentina is', 'beats both', 'france france', 'is best',
'love france', 'spain beats'], dtype=object)
For our collection of documents, we can include up to trigrams. However, the trade-off is that increasing the n-gram range can dramatically increase number of features.
count_ngrams = CountVectorizer(
ngram_range=(1, 3)
)
count_ngrams.fit_transform(text_data)
count_ngrams.get_feature_names_out()
array(['argentina', 'argentina is', 'argentina is best', 'beats',
'beats both', 'best', 'both', 'france', 'france france', 'is',
'is best', 'love', 'love france', 'love france france', 'spain',
'spain beats', 'spain beats both'], dtype=object)
8.3 Restricting vocabulary
Sometimes we do not want to create features for every word in the dataset. Instead, we may have a specific vocabulary of terms that we care about. Suppose we are interested only in country names countries = ["france", "argentina", "spain"]. We can provide this vocabulary directly to CountVectorizer.
countries = ["france", "argentina", "spain"]
country_vectorizer = CountVectorizer(
vocabulary=countries
)
country_features = country_vectorizer.fit_transform(
text_data
)
country_features.toarray()
array([[2, 0, 0],
[0, 1, 0],
[0, 0, 1]])
The resulting features represent only the country names we specified. We can inspect their names via code snippet below. This can be useful when we have prior knowledge about which terms are relevant to our problem.
country_vectorizer.get_feature_names_out()
array(['france', 'argentina', 'spain'], dtype=object)
In general, CountVectorizer provides several parameters that allow us to control how text is converted into features. For code snippet shown below
lowercase=Trueconverts text to lowercase,stop_words="english"removes common English stop words,ngram_range=(1, 2)creates unigrams and bigrams,min_df=1keeps terms appearing in at least one document.
These parameters allow us to customize the feature-generation process rather than relying entirely on default settings.
vectorizer = CountVectorizer(
lowercase=True,
stop_words="english",
ngram_range=(1, 2),
min_df=1
)
X = vectorizer.fit_transform(text_data)
9. Weighting Word Importance with TF-IDF
The BoW is powerful because it gives us a simple numerical representation of text, but it has an important limitation: it treats words primarily as independent features and does not capture how important a word is across entire collection of documents. However, not every word is equally useful. A word that appears many times in one document may be particularly relevant to that document, while a word that appears in almost every document may provide very little information for distinguishing one document from another.
To address this, we can assign a weight to each word based on its importance within a document and its rarity across the entire collection of documents. A common approach is called term frequency–inverse document frequency (TF-IDF). TF-IDF combines two ideas.
First, term frequency (TF) measures how often a word occurs in a particular document. The more frequently a word appears in a document, the more likely it is to be relevant to that document.
For example, if the word “economy” appears many times in a document, that provides some evidence that the document may be related to economics.
Second, document frequency (DF) measures how many documents contain a particular word.
If a word appears in almost every document, it is less useful for distinguishing one document from another.
By combining these two ideas, TF-IDF gives us a score that reflects how important a word is to a particular document $\text{tf-idf}(t,d) = \text{tf}(t,d) \times \text{idf}(t)$, where ($t$) represents a term (word) and ($d$) represents a document. The intuition is straightforward:
High TF + Low DF → High importance
High TF + High DF → Lower importance
Low TF + Low DF → Usually lower importance
In scikit-learn, TF is based on the number of times a term appears in a document, while the default inverse document frequency is calculated as $\text{idf}(t) = \log\left(\frac{1+n_d}{1+\text{df}(t)}\right)+1$, where ($n_d$) is the total number of documents and ($\text{df}(t)$) is the number of documents containing term ($t$). The addition of 1 prevents division by zero and provides a form of smoothing. The final values therefore reflect the relative importance of words within each document rather than simply their raw counts.
In practice, we can use scikit-learn’s TfidfVectorizer to transform a collection of documents into a TF-IDF feature matrix. This gives us a representation similar to BoW, but instead of simply counting words, we weight each word according to how informative it is.
9.1 Creating and viewing a TF-IDF feature matrix
We adopt the same small collection of documents from our BoW example, and then create a TfidfVectorizer to transform the documents.
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
# text data
text_data = np.array([
"I love France. France!",
"Argentina is best",
"Spain beats both"
])
# create TF-IDF vectorizer
tfidf = TfidfVectorizer()
# create TF-IDF feature matrix
feature_matrix = tfidf.fit_transform(text_data)
feature_matrix
<Compressed Sparse Row sparse matrix of dtype 'float64'
with 8 stored elements and shape (3, 8)>
The output indicates that there are three rows for three documents, and eight columns for eight unique terms in text data.
Note
Noted that the values are now floating-point numbers rather than integer word counts. This is because TF-IDF assigns a weight to each term rather than simply recording its frequency.
As with BoW, the matrix is sparse. For this small example, we can convert it to a dense NumPy array, which combines with feature names to form a pandas DataFrame, which is similar in structure to the BoW feature matrix.
# convert sparse matrix to a dense array
dense_matrix = feature_matrix.toarray()
# get feature names
feature_names = tfidf.get_feature_names_out()
# combine them into a DataFrame
df_tfidf = pd.DataFrame(
dense_matrix,
columns=feature_names
)
df_tfidf
| argentina | beats | best | both | france | is | love | spain | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.894427 | 0.00000 | 0.447214 | 0.00000 |
| 1 | 0.57735 | 0.00000 | 0.57735 | 0.00000 | 0.000000 | 0.57735 | 0.000000 | 0.00000 |
| 2 | 0.00000 | 0.57735 | 0.00000 | 0.57735 | 0.000000 | 0.00000 | 0.000000 | 0.57735 |
The values in preceeding DataFrame should not be interpreted as simple word counts anymore. Instead, they represent the relative TF-IDF importance of each term within each document. For example, in the first document “I love France. France!”, “france” occurs twice, while “love” occurs once. “France” therefore receives a larger TF-IDF value than “love” in this small example: “france → 0.8944”, “love → 0.4472”. In addition, these values have been normalized, which is why they do not simply correspond to the raw counts 2 and 1.
Why TF-IDF can be better than raw counts?
Here we consider two words “the” and “quantum”. Suppose “the” occurs 500 times across a collection of documents, while “quantum” occurs only 20 times.
Raw word counts might suggest that “the” is more important simply because it appears more often.
But if “the” occurs in almost every document, it can’t tell us what any particular document is about.
If “quantum” appears frequently in only a small number of documents, it can be much more useful for distinguishing those documents.
TF-IDF captures this distinction via rules.
“Frequent in one document + Rare across documents → High TF-IDF”.
“Frequent across many documents → Lower IDF → Lower TF-IDF”.
This makes TF-IDF particularly useful for applications such as document search, information retrieval, document classification, and text similarity.
9.2 Looking at TF and IDF separately
To understand TF-IDF more clearly, it is useful to separate two components conceptually.
Term frequency (TF) asks “How often does this word occur in this document?”
For example “I love France. France!” has: “france → 2”, and “love → 1”.
Document frequency (DF) asks “In how many documents does this word occur?”
In our example, “france → 1 document”, “love → 1 document”, “spain → 1 document”, and “is → 1 document”.
A word appearing in only one document is more useful for distinguishing that document than a word appearing in every document.
We can inspect the IDF values learned by vectorizer. The code example below returns a mapping from each word to its column index.
# display vocabulary
tfidf.vocabulary_
{'love': 6,
'france': 4,
'argentina': 0,
'is': 5,
'best': 2,
'spain': 7,
'beats': 1,
'both': 3}
We can also inspect actual IDF weights via code snippet below.
# display IDF value for each feature
dict(zip(
tfidf.get_feature_names_out(),
tfidf.idf_
))
{'argentina': np.float64(1.6931471805599454),
'beats': np.float64(1.6931471805599454),
'best': np.float64(1.6931471805599454),
'both': np.float64(1.6931471805599454),
'france': np.float64(1.6931471805599454),
'is': np.float64(1.6931471805599454),
'love': np.float64(1.6931471805599454),
'spain': np.float64(1.6931471805599454)}
Why are all IDF values the same?
TfidfVectorizer uses smoothed IDF: $IDF(t) = \log(\frac{1+n_d}{1+df(t)}) + 1$. In this example, we have 3 documents ($n_d = 3$), and after tokenization each word appears in exactly one document ($df(t) = 1$).
Our toy dataset is very small and most words occur only in one document, many of the IDF values will be the same. In a realistic dataset containing hundreds or thousands of documents, the differences become much more meaningful.
Once we have a TF-IDF matrix, we can identify which words have the highest weights for a particular document.
# select first document
document_index = 0 # pick other documents
# get TF-IDF scores
scores = feature_matrix[document_index].toarray().flatten()
# pair words with their scores
word_scores = list(zip(feature_names, scores))
# sort from highest to lowest score
word_scores = sorted(
word_scores,
key=lambda x: x[1],
reverse=True
)
word_scores
[('france', np.float64(0.8944271909999159)),
('love', np.float64(0.4472135954999579)),
('argentina', np.float64(0.0)),
('beats', np.float64(0.0)),
('best', np.float64(0.0)),
('both', np.float64(0.0)),
('is', np.float64(0.0)),
('spain', np.float64(0.0))]
We can then display the most important terms. In this example, “france” will receive a higher weight than “love” because it occurs twice in the document.
# display top three terms
word_scores[:3]
[('france', np.float64(0.8944271909999159)),
('love', np.float64(0.4472135954999579)),
('argentina', np.float64(0.0))]
9.3 TF-IDF with stop words and n-grams
Just as with CountVectorizer, we can remove common stop words while creating TF-IDF features.
tfidf_no_stop_words = TfidfVectorizer(
stop_words="english"
)
feature_matrix = tfidf_no_stop_words.fit_transform(
text_data
)
tfidf_no_stop_words.get_feature_names_out()
array(['argentina', 'beats', 'best', 'france', 'love', 'spain'],
dtype=object)
This can reduce the number of features and prevent very common words from contributing to the representation. As always, whether stop-word removal is appropriate depends on downsteam task.
We can also combine TF-IDF with n-grams. Here, ngram_range=(1, 2) creates both unigrams as individual words, and bigrams as pairs of consecutive words. This can help capture short phrases that may carry more meaning than individual words, like “machine learning” instead of “machine” and “learning”.
tfidf_ngrams = TfidfVectorizer(
ngram_range=(1, 2)
)
feature_matrix = tfidf_ngrams.fit_transform(
text_data
)
tfidf_ngrams.get_feature_names_out()
array(['argentina', 'argentina is', 'beats', 'beats both', 'best', 'both',
'france', 'france france', 'is', 'is best', 'love', 'love france',
'spain', 'spain beats'], dtype=object)
Callout
The BoW tells us how often a word occurs, and TF-IDF goes one step further and asks how important that word is for distinguishing one document from the others.
TF rewards words that occur frequently within a document.
IDF reduces importance of words that occur across many documents.
This distinction is particularly useful when we want to compare documents or search for documents that are relevant to a particular query. We can use these numerical text vectors to calculate similarity, allowing us to determine which documents are most similar to a search query.
10. Calculating Text Similarity with Text Vectors
After converting our documents into TF-IDF vectors, we can use those numerical representations to build a simple text-search system. This is one of the most practical applications of text vectors: instead of searching only for exact keyword matches, we can compare a search query with a collection of documents and rank the documents according to how similar they are to the query.
Text vectors are useful in many NLP applications, including search engines, document retrieval, recommendation systems, and information filtering. The basic idea is straightforward. First, we calculate the TF-IDF vectors for a collection of documents. We then use the same fitted TfidfVectorizer to transform a new search query into a vector using the vocabulary and weighting scheme learned from the original documents. Finally, we compare the query vector with the document vectors using cosine similarity and rank the documents from most similar to least similar.
Cosine similarity measures the angle between two vectors rather than simply comparing their raw magnitudes. For normalized TF-IDF vectors, the similarity ranges from 0 to 1, where a value closer to 1 indicates greater similarity and a value closer to 0 indicates little or no similarity. In this way, cosine similarity gives us a convenient score that we can use to rank search results.
10.1 Calculating cosine similarity
Scikit-learn provides convenient tools for calculating similarity between vectors. We start from the TF-IDF feature matrix we have built in preceeding sections.
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
# create searchable text data
text_data = np.array([
"I love France. France!",
"Argentina is best",
"Spain beats both"
])
# create TF-IDF vectorizer
tfidf = TfidfVectorizer()
# learn vocabulary and create document vectors
feature_matrix = tfidf.fit_transform(text_data)
feature_matrix
<Compressed Sparse Row sparse matrix of dtype 'float64'
with 8 stored elements and shape (3, 8)>
Now suppose a user enters the following search query “France is the best” and want to compare it with the vectors of all three documents. We must transform this query using the same fitted vectorizer that was used for our documents.
# create a search query
query = "France is the best"
# transform search query into a TF-IDF vector
query_vector = tfidf.transform([query])
query_vector
<Compressed Sparse Row sparse matrix of dtype 'float64'
with 3 stored elements and shape (1, 8)>
Note
Note the difference: we use transform() instead of fit_transform().
When we first process our documents, we used
tfidf.fit_transform(text_data). The vectorizer learns the vocabulary and calculates the IDF weights from the document collection.When we process a new search query, we do not need to fit the vectorizer again. We simply use
tfidf.transform([query]). This represents the query using the same vocabulary and weighting scheme as the documents, so we can compare their vectors correctly.
Now we can compare the query vector with every document vector. Here we use linear_kernel from scikit-learn.
from sklearn.metrics.pairwise import linear_kernel
# calculate cosine similarity between query and every document
cosine_similarities = linear_kernel(
query_vector,
feature_matrix
).flatten()
# display similarity scores
cosine_similarities
array([0.51639778, 0.66666667, 0. ])
We get scores [0.5164, 0.6667, 0.0000]. These values correspond to the documents in their original order.
“I love France. France!” → 0.5164
“Argentina is best” → 0.6667
“Spain beats both” → 0.0000
A score of 0 indicates that the query and document have no overlapping terms in TF-IDF representation.
10.2 Ranking the search results
For a small number of documents, this ordering may be sufficient because we can simply select the document with the highest similarity score. However, when the collection contains hundreds of documents, reviewing the entire output to find the best match can be time-consuming. Therefore, we should sort the documents by their similarity scores and display the most relevant documents first.
In a real search application, we normally do not want to return every document. Instead, we might return only the top k results. NumPy’s argsort() returns the indices that would sort an array. In code snippet below, the [::-1] reverses the order so that the highest similarity score comes first. We can then retrieve the documents in ranked order.
# get indices of top three results
top_k = 3
# sort document indices by similarity
top_indices = cosine_similarities.argsort()[-top_k:][::-1]
# display top results
for index in top_indices:
print(
f"{cosine_similarities[index]:.3f} "
f"→ {text_data[index]}"
)
0.667 → Argentina is best
0.516 → I love France. France!
0.000 → Spain beats both
We have now created a very simple text-search and ranking system.
Why does “France is best” rank first?
At first glance, this result may seem surprising. The query contains the word “France”, so we might expect “I love France. France!” to be the most relevant result.
However, cosine similarity is based on the vector representation, not human interpretation of the sentence.
The query is “France is the best”, and the document “Argentina is best”. They shares two important terms with the query: “is” and “best”.
The France document shares just one term with the query.
Because of the TF-IDF weights in this small corpus, the combination of “is” and “best” produces a higher cosine similarity than the single shared term “france”.
A text-similarity algorithm does not necessarily understand the meaning of a sentence. It compares the numerical representations we provide to it.
TF-IDF gives us a numerical representation of documents, and cosine similarity allows to compare those representations.
By transforming a user’s search query with the same fitted TF-IDF vectorizer, we can compare query against a collection of documents and rank them according to similarity.
However, because TF-IDF primarily relies on shared words, it does not truly understand semantic meaning.
A query about “automobiles” may not match a document containing only “cars”, even though a human knows they are closely related.
That limitation motivates the next generation of NLP techniques: dense text vectors and semantic representations, which can capture relationships between words and documents beyond exact word matching.
Simple exercises for the text-search and ranking system
First try creating a larger collection of documents and fit a new TF-IDF vectorizer, and then creating several search queries.
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
documents = np.array([
"I would love to visit Spain someday",
"Argentina is famous for its passionate football culture",
"France is known for its rich history and cuisine",
"Sweden has many beautiful lakes and forests",
"Germany has a strong engineering and manufacturing tradition",
"Norway is well known for its fjords and natural landscapes"
])
# create TF-IDF matrix
vectorizer = TfidfVectorizer()
document_matrix = vectorizer.fit_transform(documents)
# Search function
def search_documents(query, vectorizer, document_matrix, documents, top_k=3):
query_vector = vectorizer.transform([query])
# calculate cosine similarity
similarities = cosine_similarity(
query_vector,
document_matrix
).flatten()
# get indices of top matching documents
top_indices = similarities.argsort()[::-1][:top_k]
# return documents and scores
results = [
(documents[i], similarities[i])
for i in top_indices
]
return results
# create search queries
queries = [
"Spain",
"Argentina",
"France",
"Sweden",
"Germany",
"Norway"
]
# search for each query
for query in queries:
print(f"\nQuery: {query}")
results = search_documents(
query,
vectorizer,
document_matrix,
documents,
top_k=3
)
for document, score in results:
print(f"{score:.3f} → {document}")
Try to investigate:
Which document ranks first for each query?
How does the ranking change when the query contains more words?
What happens when a query contains a word that does not appear in the documents?
Which common words receive lower influence?
Why might a search engine need more sophisticated methods than TF-IDF?
11. Performing Sentiment Analysis with a Pretrained Classifier
After preprocessing and transforming text data, we can extract higher-level information from text. Sentiment analysis provides a practical example of how we can move beyond basic text preprocessing and use a pretrained NLP model to extract meaningful information from unstructured text.
Suppose we have a collection of customer reviews, tweets, survey responses, or other text and want to determine whether each observation expresses a positive or negative sentiment. The resulting sentiment label or score can then be used directly for analysis or incorporated as a feature in a downstream machine learning or deep learning model.
Rather than building and training a sentiment classifier from scratch, we can take advantage of pretrained NLP models. The Hugging Face transformers library is one of the most widely used libraries for modern NLP and provides convenient APIs for both using pretrained models and training or fine-tuning our own models.
In this example, we use its high-level pipeline() API to create a sentiment-analysis classifier. The pretrained model takes raw text as input and returns a predicted sentiment label together with a confidence score. This provides a simple demonstration of how pretrained NLP models can be integrated into a machine learning workflow to classify text, generate features, and extract useful information from otherwise unstructured data. We will explore the transformers library and modern NLP techniques in greater depth later, but for now, the important idea is that we can use a pretrained model to perform a sophisticated NLP task with only a few lines of Python.
11.1 Build a sentiment analysis classifier
The Hugging Face transformers library provides a convenient pipeline() function that allows us to use pretrained models without having to implement the underlying neural network architecture ourselves.
import torch
from transformers import pipeline
# create an NLP pipeline for sentiment analysis
classifier = pipeline("sentiment-analysis",
model="distilbert/distilbert-base-uncased-finetuned-sst-2-english"
)
Then we provide a sentence to the sentiment-analysis classifier, which will return a result containing a sentiment label and a confidence score.
# classify a negative statement
sentiment_1 = classifier(
"I hate machine learning! It's the absolute worst."
)
print(sentiment_1)
[{'label': 'NEGATIVE', 'score': 0.9998020529747009}]
# classify a positive statement
sentiment_2 = classifier(
"Machine learning is the absolute "
"bees knees. I love it so much!"
)
print(sentiment_2)
[{'label': 'POSITIVE', 'score': 0.999546229839325}]
11.2 Classifying multiple texts
The pipeline can also process multiple observations at once. This allows us to process an entire collection of text observations rather than calling the classifier separately for every sentence.
# create a collection of text observations
texts = [
"I love this product!",
"This is the worst experience ever.",
"The product is okay.",
"Absolutely fantastic service!"
]
# classify all observations
results = classifier(texts)
# display results
for text, result in zip(texts, results):
print(text)
print(result)
print()
I love this product!
{'label': 'POSITIVE', 'score': 0.9998855590820312}
This is the worst experience ever.
{'label': 'NEGATIVE', 'score': 0.9997727274894714}
The product is okay.
{'label': 'POSITIVE', 'score': 0.9998345375061035}
Absolutely fantastic service!
{'label': 'POSITIVE', 'score': 0.9998830556869507}
As shown in preceeding results, in addition to the labels “POSITIVE” and “NEGATIVE”, the classifier also provides a confidence score. We can combine the labels and scores into a more useful structure.
# display sentiment and confidence for each observation
for text, result in zip(texts, results):
print(
f"Text: {text}\n"
f"Sentiment: {result['label']}\n"
f"Confidence: {result['score']:.3f}\n"
)
Text: I love this product!
Sentiment: POSITIVE
Confidence: 1.000
Text: This is the worst experience ever.
Sentiment: NEGATIVE
Confidence: 1.000
Text: The product is okay.
Sentiment: POSITIVE
Confidence: 1.000
Text: Absolutely fantastic service!
Sentiment: POSITIVE
Confidence: 1.000
11.3 Using sentiment as a feature
One of the most useful ideas here is that the model’s output does not have to be the final result of our analysis. Instead, sentiment can become a feature in a larger data-processing or machine-learning pipeline. For example, suppose we have customer reviews, from which we can create a simple sentiment feature.
reviews = [
"The service was excellent.",
"I waited for hours and hated the experience.",
"The product works well.",
"The quality was terrible."
]
# predict sentiment
sentiment_results = classifier(reviews)
# convert sentiment labels into a binary feature
sentiment_feature = [
1 if result["label"] == "POSITIVE" else 0
for result in sentiment_results
]
print(sentiment_feature)
[1, 0, 1, 0]
We could then combine this text-based feature with other structured data, such as customer age, purchase amount, number of previous purchases, product category, and sentiment score.
By combining these different types of information, we can create a richer feature set for downstream machine learning or deep learning tasks. For example, the resulting features could be used to predict customer behavior, classify customers, recommend products, or estimate the likelihood of a future purchase.
Exercise
Try the sentiment classifier with your own examples
examples = [
"The movie was absolutely fantastic!",
"I would never recommend this product.",
"The food was okay, nothing special.",
"This is one of the best experiences I've ever had."
]
# Classify the examples
results = classifier(examples)
# Display predictions
for text, result in zip(examples, results):
print(
f"{result['label']:>8} "
f"({result['score']:.3f}) → {text}"
)
Now experiment with sentences containing strongly positive language, strongly negative language, neutral language, mixed or contradictory opinions, and even sarcasm or humor.
test_text = [
"I absolutely love this!",
"I absolutely hate this!",
"It was fine.",
"Great job... if your goal was to make everything worse."
]
results = classifier(test_text)
for text, result in zip(test_text, results):
print(
f"{result['label']} "
f"({result['score']:.3f}) → {text}"
)
Try to run the hands-on exercises and answer:
Which examples does the model classify confidently?
Which examples produce lower confidence?
How does the model handle neutral language?
How does it handle sarcasm?
Does the model always agree with your interpretation of the sentence?
Important considerations
A pretrained sentiment classifier is convenient, but it is not universally accurate. Its performance depends on the model, training data, language, domain, and type of text it encounters.
For example, a model trained primarily on movie reviews may behave differently when analyzing medical notes, financial reports, product reviews, or social-media posts. Specialized domains may require a model that has been fine-tuned on relevant data.
We should also remember that the confidence score is the model’s confidence in its prediction, not necessarily a guarantee that the prediction is correct.
Therefore, pretrained classifiers are best viewed as useful tools for extracting signals from text rather than infallible sources of truth.
12. Summary of Text Wrangling Procedures
With raw text data in hand, we should follow three main steps to wrangle and prepare the data for analysis.
Remove punctuation → Tokenization → Remove Stop Words → Stemming/Lemmatization
These are text preprocessing/normalization steps, to get cleaned text data.
POS Tagging → NER
These are linguistic analysis tasks, to get structured linguistic information.
BoW → TF-IDF
These are are feature/vectorization methods, to get numerical representations (vectors).
Text similarity is not a text data wrangling method; rather, it is an NLP application for quantifying the semantic or lexical similarity between textual data.
Step |
Category |
Purpose |
|---|---|---|
Remove punctuation |
Text preprocessing / cleaning |
Remove unnecessary characters |
Tokenization |
Text preprocessing |
Split text into words/tokens |
Remove stop words |
Text preprocessing |
Remove common, low-information words |
Stemming / Lemmatization |
Text normalization |
Reduce words to a common/base form |
POS tagging |
Linguistic analysis |
Identify grammatical roles |
NER |
Linguistic analysis |
Identify entities such as people, organizations, locations |
BoW |
Feature extraction / vectorization |
Convert text into numerical vectors |
TF-IDF |
Feature extraction / vectorization |
Create weighted numerical vectors |
Text similarity |
NLP task / analysis |
Measure how similar two texts are |
The big picture of wrangling text data can be described by:
NLP TEXT PROCESSING
│
┌──────────────────┼──────────────────┐
↓ ↓ ↓
Text Preprocessing Linguistic Analysis Feature Extraction
│ │ │
├─ Punctuation ├─ POS ├─ BoW
├─ Tokenization └─ NER └─ TF-IDF
├─ Stop words
└─ Stemming/
Lemmatization
↓
↓
NLP Applications
│
Text Similarity
Keypoints
Text preprocessing transforms unstructured text into cleaner and more consistent data for analysis and machine learning.
Tokenization, stop-word removal, stemming, PoS tagging, and named-entity recognition provide different ways to analyze and structure textual information.
Bag-of-Words and n-gram representations convert text into numerical features, while TF-IDF assigns greater importance to informative terms.
TF-IDF vectors and cosine similarity can be used to build simple text-search and document-retrieval systems that rank documents by relevance.
Pretrained NLP models can extract higher-level information, such as sentiment, and their outputs can be used as features in downstream data-analysis and machine-learning tasks.