Week 11
Text Analysis II: Advanced Methods and Applications
This chapter moves from cleaning text to extracting structured insights: named entities, word patterns, numerical document features, latent topics, and responsible interpretation.
Entities
Counts
Topics
Learning Goals
By the End of This Chapter
By the end, you should be able to extract named entities, compare word and phrase frequencies, convert text into vectors, interpret topic-model output, and explain why ethics matter in NLP workflows.
How to Use This Page
Explore the visual panels first, then run the related Python scripts. Compare what the numerical output reveals and what it hides.
Chapter Coverage
The page preserves the core examples from the notebook while consolidating setup and making each output available directly under its script.
Part 1
Named Entity Recognition (NER)
Named Entity Recognition identifies spans of text that refer to real-world entities such as people, places, organizations, dates, and quantities.
Example
In “Marie Curie won the Nobel Prize in Physics in 1903,” a recognizer might label Marie Curie as PERSON and 1903 as DATE. Some labels are model-dependent, so results should be checked.
Common entity labels
PERSON: people, fictional or real.
ORG: organizations.
GPE: geopolitical entities such as countries and cities.
LOC: non-GPE locations such as mountains and rivers.
NORP: nationalities, religious groups, or political groups.
DATE, TIME, MONEY, PERCENT: temporal and numeric expressions.
Entity Highlighter
NER with spaCy
import spacy
nlp = spacy.load("en_core_web_sm")
text = "Barack Obama was born in Hawaii and served as President of the United States."
doc = nlp(text)
for ent in doc.ents:
print(ent.text, ent.label_)NER in a paragraph
import spacy
nlp = spacy.load("en_core_web_sm")
text = """Yesterday at 9:30 AM, Dr. Alice announced that Orion Labs
will open a new research center in Paris on 12 March 2026. During
the event, several analysts from the European Union noted that
nearly 35% of the project budget, about $12 million, will focus on
mapping rivers near the Andes."""
doc = nlp(text)
for ent in doc.ents:
print(ent.text, ent.label_)Part 2
Word Frequency
Word frequency counts how often each word appears. It is useful for quick exploration, but it ignores word order, context, and meaning.
Key point
Frequency can reveal repeated terms quickly, but words such as “run” and “running” are treated differently unless you normalize them first.
Modern alternatives
Contextual embeddings such as BERT, GPT, and spaCy Transformer models represent meaning in context and often outperform frequency-based features in NLP tasks. Frequency is still important because it is simple, interpretable, and forms the basis of Bag-of-Words and TF-IDF.
Frequency Explorer
Word frequency with Counter
from collections import Counter
import spacy
nlp = spacy.load("en_core_web_sm")
text = "Data science is a science that uses data."
doc = nlp(text)
tokens = [t.lemma_.lower() for t in doc
if not t.is_stop and not t.is_punct]
freq = Counter(tokens)
print(freq)Top words in Alice in Wonderland
from collections import Counter
import spacy
import requests
url = "https://www.gutenberg.org/files/11/11-0.txt"
alice_text = requests.get(url).text
nlp = spacy.load("en_core_web_sm")
doc = nlp(alice_text)
tokens = [
t.lemma_.lower()
for t in doc
if t.is_alpha and not t.is_stop and len(t.lemma_) > 1
]
freq = Counter(tokens)
print(freq.most_common(10))Part 3
N-grams
An n-gram is a sequence of n units, usually words. Unigrams use one word, bigrams use two, and trigrams use three.
Why n-grams matter
They capture local context and common phrases that single words miss, such as “climate change,” “machine learning,” and “artificial intelligence.”
N-gram Explorer
Create bigrams with NLTK
from nltk import bigrams
from nltk.tokenize import word_tokenize
text = "Artificial intelligence transforms industries rapidly."
tokens = word_tokenize(text.lower())
bi = list(bigrams(tokens))
print(bi)The Alice examples below use the cleaned tokens list from the word-frequency section: alphabetic words only, stop words removed, and lemmas lowercased.
Top bigrams in Alice in Wonderland
from nltk.util import bigrams
from collections import Counter
bi_tokens = list(bigrams(tokens))
bi_freq = Counter(bi_tokens)
top10_bi = bi_freq.most_common(10)
print(top10_bi)Top trigrams in Alice in Wonderland
from nltk.util import trigrams
from collections import Counter
tri_tokens = list(trigrams(tokens))
tri_freq = Counter(tri_tokens)
top10_tri = tri_freq.most_common(10)
print(top10_tri)Part 4
Vectorizing Text
Machine learning models need numbers. Bag-of-Words counts vocabulary terms, while TF-IDF weights words by how important they are in a document relative to the full collection.
\[
\begin{aligned}
\text{raw documents} &\rightarrow \text{vocabulary} \\
&\rightarrow \text{document-term matrix}
\end{aligned}
\]
Bag-of-Words example
For documents “data science is fun” and “data science uses data,” a sorted vocabulary might be ['data', 'fun', 'is', 'science', 'uses']. The vectors become [1, 1, 1, 1, 0] and [2, 0, 0, 1, 1].
TF-IDF intuition
Raw word counts make frequent words look important even when they appear in almost every document. TF-IDF keeps the term-frequency idea, but discounts words that are common across the whole corpus. A word receives a high TF-IDF score when it appears often in one document and appears in relatively few other documents.
\[
\operatorname{tfidf}(t,d) = \operatorname{tf}(t,d) \times \operatorname{idf}(t)
\]
\[
\operatorname{idf}(t) = \log\left(\frac{N + 1}{\operatorname{df}(t) + 1}\right) + 1
\]
How to read the TF-IDF formula
\(t\) means a term or word, \(d\) means a document, and \(N\) is the number of documents in the corpus. \(\operatorname{tf}(t,d)\) measures how often term \(t\) appears in document \(d\). \(\operatorname{df}(t)\) is the number of documents that contain term \(t\). If a word appears in nearly every document, \(\operatorname{df}(t)\) is large and the IDF part becomes smaller. If a word appears in only a few documents, \(\operatorname{df}(t)\) is small and the IDF part becomes larger.
Different libraries use slightly different smoothing and normalization. The version above is close to the common smoothed form used by scikit-learn. After computing TF-IDF weights, vectors are often normalized so that documents of different lengths can be compared more fairly.
What TF-IDF can and cannot do
TF-IDF is useful for search, document comparison, and classical machine learning because it is simple and interpretable. However, it does not understand word order or deep meaning. For example, not good and good may still be difficult to distinguish if the model only sees separate word weights.
Vector Matrix Explorer
Documents in this matrix
- doc1 data science includes statistics and data visualization
- doc2 data science includes machine learning and data modeling
- doc3 artificial intelligence overlaps with machine learning and learning systems
Bag-of-Words with CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer
docs = [
"data science includes statistics and data visualization",
"data science includes machine learning and data modeling",
"artificial intelligence overlaps with machine learning and learning systems"
]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(X.toarray())TF-IDF with TfidfVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
docs = [
"data science includes statistics and data visualization",
"data science includes machine learning and data modeling",
"artificial intelligence overlaps with machine learning and learning systems"
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(docs)
print(vectorizer.get_feature_names_out())
print(X.toarray().round(2))Part 5
Topic Modeling
Topic modeling groups words that frequently occur together across documents. A topic is not automatically a human concept; interpretation is still required.
Latent Dirichlet Allocation (LDA) is one common topic model. It represents each document as a mixture of topics, and each topic as a probability distribution over words. Instead of saying “this document has exactly one label,” LDA says “this document is partly about Topic 0, partly about Topic 1, and so on.”
Interpret carefully
LDA topics are clusters of co-occurring terms. They suggest themes, but they do not prove what a document is “really about.”
\[
\begin{aligned}
\theta_d &= P(\text{topic} \mid \text{document } d) \\
\phi_k &= P(\text{word} \mid \text{topic } k)
\end{aligned}
\]
Here, \(\theta_d\) is the topic mixture for document \(d\). For example, \(\theta_d = [0.70, 0.20, 0.10]\) means the document is mostly associated with Topic 0, with smaller shares of Topic 1 and Topic 2. \(\phi_k\) is the word distribution for topic \(k\). If Topic 1 gives high probability to words such as space, nasa, orbit, and launch, a reader may interpret it as a space-related topic.
LDA as a simple generative story
LDA imagines that documents are generated in two hidden steps. First, each document gets a topic mixture. Second, each word token is generated by choosing a topic from that document mixture and then choosing a word from the selected topic.
\[
\theta_d \sim \operatorname{Dirichlet}(\alpha), \qquad
\phi_k \sim \operatorname{Dirichlet}(\beta)
\]
\[
z_{dn} \sim \operatorname{Categorical}(\theta_d), \qquad
w_{dn} \sim \operatorname{Categorical}(\phi_{z_{dn}})
\]
\(z_{dn}\) is the hidden topic assignment for the \(n\)-th word token in document \(d\), and \(w_{dn}\) is the observed word. In real analysis we see the words but not the topic assignments, so the algorithm estimates the hidden topic mixtures and topic-word probabilities from repeated word co-occurrence patterns.
Practical interpretation checklist
When reading LDA output, look at the top words for each topic, inspect example documents with high topic probability, and check whether the interpretation is stable under different preprocessing choices. The number of topics is chosen by the analyst, so the model can produce topics that are too broad, too narrow, or difficult to name.
LDA uses a Bag-of-Words view of text. It notices which words tend to occur in the same documents, but it does not directly understand grammar, word order, sarcasm, or historical context. Human interpretation remains part of the method.
Topic Word Explorer
These topics come from an LDA model fit to a 20 Newsgroups subset: rec.sport.baseball, sci.space, and comp.graphics. The labels below are human interpretations of top words, not names supplied by the model.
Preview 20 Newsgroups documents
from sklearn.datasets import fetch_20newsgroups
import pprint
data = fetch_20newsgroups(remove=("headers", "footers", "quotes"))
documents = data.data
pprint.pprint(documents[0])
print("number of documents:", len(documents))
print("topics:", data.target_names)Choose a topic-modeling subset
from sklearn.datasets import fetch_20newsgroups
categories = ["sci.space", "rec.sport.baseball", "comp.graphics"]
subset = fetch_20newsgroups(categories=categories,
remove=("headers", "footers", "quotes"))
print("number of documents in subset:", len(subset.data))LDA topic modeling
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS
from gensim import corpora, models
from nltk.tokenize import word_tokenize
import re
docs = subset.data
stop_words = set(ENGLISH_STOP_WORDS)
def clean_and_tokenize(text):
text = text.lower()
text = re.sub(r"[^a-z\s]", " ", text)
tokens = word_tokenize(text)
return [t for t in tokens if len(t) > 2 and t not in stop_words]
tokenized_docs = [clean_and_tokenize(doc) for doc in docs]
dictionary = corpora.Dictionary(tokenized_docs)
dictionary.filter_extremes(no_below=10, no_above=0.5)
corpus = [dictionary.doc2bow(tokens) for tokens in tokenized_docs]
lda = models.LdaModel(
corpus=corpus,
id2word=dictionary,
num_topics=3,
random_state=314,
passes=30,
)
for idx, topic in lda.print_topics(num_topics=3, num_words=8):
print(f"\nTopic {idx}:\n{topic}")| Topic | Top words | Possible interpretation |
|---|---|---|
| 0 | year, think, don, just, good, like, game, team | Sports-like conversation |
| 1 | space, nasa, launch, earth, data, orbit, satellite, shuttle | Space / NASA / astronomy |
| 2 | image, graphics, edu, file, software, jpeg, use, files | Computer graphics and image files |
Part 6
Ethical Considerations
Text data reflects human behavior, culture, and social structures. NLP models can reproduce or amplify bias, so responsible analysis requires more than technical accuracy.
- Privacy and consent
- Bias amplification
- Over-interpreting patterns
- Cultural and linguistic transfer issues
- Deployment context
Cultural example
A sentiment model trained only on English may misread the Japanese word やばい. It can literally mean dangerous or bad, but in modern casual Japanese it can also mean amazing, cool, or impressive.
Responsible Analysis Checklist
| Bias type | Risk | Example |
|---|---|---|
| Historical and social bias | Past inequalities are reproduced. | Doctors described as “he,” nurses as “she.” |
| Representation and sampling bias | Some groups are overrepresented. | Reddit posts standing in for public opinion. |
| Evaluation bias | Tests do not match real users. | A model tested only on U.S. English. |
| Learning bias | Small topics disappear. | A topic model misses a 5% mathematics theme. |
| Deployment bias | The model is used outside its intended setting. | A research demo becomes a decision tool. |
Practice
Practice Questions
Use these questions to check your understanding and connect the examples to real text data.
Entity labels
Why might one model label “Nobel Prize” as WORK_OF_ART while another labels it differently?
Frequency limits
What important meaning is lost when we count words without considering word order or context?
TF-IDF intuition
Why can a word be frequent in one document but still receive a low TF-IDF score?
Topic interpretation
When topic words seem meaningful, what additional checks would make the interpretation more trustworthy?
Reference
Key Terms
| Term | Meaning | Section |
|---|---|---|
| NER | Detecting named entities such as people, places, organizations, and dates. | Named entities |
| GPE | Geo-political entity such as a country, city, or state. | Named entities |
| Word frequency | Counting how often words appear after selected preprocessing. | Word frequency |
| N-gram | A sequence of n words or tokens. | N-grams |
| Bag-of-Words | A document vector made from word counts. | Vectorizing text |
| TF-IDF | A weighted representation based on term frequency and inverse document frequency. | Vectorizing text |
| LDA | A topic model that represents documents as mixtures of topics. | Topic modeling |
| Bias | A systematic distortion from data, modeling, evaluation, or deployment choices. | Ethics and bias |
Slides
Week 11 Source Slides
The slides provide the original class sequence for named entities, word frequency, vectorization, topic modeling, and ethical interpretation. Use the viewer below for quick reference, open the PDF in your browser, or download it for offline study.
