Week 10: Text Analysis I

Week 10

Text Analysis I: Understanding Text as Data

School of International Liberal Studies, Waseda University

Text analysis turns messy, unstructured language into structured information that computers can count, filter, compare, and model.

1. Text is messy Sentences, slang, punctuation, typos, URLs, emails, and multilingual text do not arrive as tidy tables.

2. Encoding comes first UTF-8 lets multilingual text survive reading and writing; the wrong encoding creates mojibake.

3. Tokenization creates units NLTK and spaCy use different rules for contractions, punctuation, URLs, and abbreviations.

4. Pipelines clean text Lowercasing, punctuation removal, stopwords, lemmatization, stemming, and POS filtering prepare a corpus.

Raw text Tokens Clean corpus

Learning Goals

By the End of This Chapter

By the end, you should be able to explain why text is unstructured data, read it safely with UTF-8, tokenize with NLTK and spaCy, normalize tokens, use POS tags, and build a preprocessing pipeline.

Text analysisTurn unstructured text into structured information.

EncodingRead multilingual files with the right character representation.

TokenizationSplit text into words, sentences, or language-specific segments.

NormalizationReduce superficial differences and remove common noise.

PipelineCombine steps into a clean corpus for later analysis.

How to Use This Page

Try each explorer, then run the related script. Compare how the output changes across tokenization, encoding, normalization, and filtering choices.

Chapter Coverage

The web version consolidates repeated setup cells, keeps the essential code examples, and makes the outputs available through the Run buttons.

Part 1

What Is Text Analysis?

Text analysis, or text mining, refers to methods for turning unstructured text into structured information that computers can analyze.

Definition

Text is fundamentally messy: sentences, slang, punctuation, typos, emojis, names, URLs, and multilingual writing. Unlike numeric tables, text has no fixed structure, so we must create one.

Text analysis breaks language into smaller units, identifies patterns, and provides ways to quantify meaning.

Tokenization
Part-of-speech tagging
Named entity recognition
Text classification
Topic modeling

Why text analysis matters

Extract insights at scale. Manual reading is slow, subjective, and not scalable; automated tools can process thousands of documents in seconds.

Discover themes and structures. Text analysis can identify recurring topics, sentiment, entities, or writing style in journalism, business, policy, and academic research.

Make text usable for machine learning. Models require numerical and structured inputs, so preprocessing turns text into features for classification, clustering, and prediction.

From Messy Text to Data

Tools in this chapter

NLTK is a classic library for tokenization and linguistics. spaCy is a fast modern NLP pipeline with tagging, parsing, and entity recognition.

text becomes data

text = "Text is unstructured. We'll turn it into data."
tokens = text.split()
print(tokens)

Part 2

Prepare NLTK and spaCy

The first code cells load tokenizer resources and then test simple sentences. In this web version, each script can be run directly from the page.

NLTK setup

import nltk

nltk.download("punkt")
nltk.download("punkt_tab")

NLTK word tokenization

from nltk.tokenize import word_tokenize

print(word_tokenize("Text mining with NLP."))
print(word_tokenize("Dr. Smith's email is wang@waseda.com."))

sentence = "The url is https://www.wdaseda.com/index.html, and Mr. O'Connor's phone number is (123) 456-7890! "
print(word_tokenize(sentence))

spaCy setup

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Text mining with NLP.")

print([t.text for t in doc])

Part 3

Encoding and Representation

Computers store characters as numbers. Old ASCII encodes a small range of characters from 0 to 127; modern Unicode and UTF-8 support multilingual text.

Encoding rule

Prefer UTF-8 when reading files, and specify encoding="utf-8". Wrong encoding can produce mojibake: garbled text that no longer looks like the original language.

File-writing example

The script creates Japanese, Korean, and Chinese text files encoded as UTF-8, then reads a Japanese file correctly with UTF-8 and incorrectly with Latin-1.

Encoding Explorer

Language sample Read as

Create and read UTF-8 files

japanese = "これは日本語のテキストです。UTF-8で正しく保存されています。"
korean   = "이것은 한국어 텍스트입니다. UTF-8로 올바르게 저장되었습니다."
chinese  = "这是一段中文文本。以UTF-8格式正确保存。"

with open('jp_text.txt', 'w', encoding='utf-8') as f:
    f.write(japanese)

with open('jp_text.txt', encoding='utf-8') as f:
    text = f.read()
print(text)

with open('jp_text.txt', encoding='latin-1') as f:
    text = f.read()
print(text)

Part 4

Tokenization

Tokenization splits text into manageable units. Common levels include word tokenization, sentence tokenization, and sometimes character tokens. Languages without whitespace, such as Japanese and Chinese, require segmentation.

Tokenizer Comparison

Example sentence Or edit the sentence

NLTK-like

spaCy-like

Different tokenizers use different rules. NLTK and spaCy disagree on details such as smart apostrophes, email addresses, URLs, abbreviations, and phone-number hyphens.

Key point

There is no single universal tokenizer. A tokenizer is a set of rules and model assumptions about what counts as a meaningful unit.

Examples of NLTK word and sentence tokenization

from nltk.tokenize import word_tokenize, sent_tokenize

txt = "Don't tokenize me incorrectly! Otherwise, I'll be mad."
print(word_tokenize(txt))
print(sent_tokenize(txt))

txt = "Amazon’s CEO announced that the company would invest $5 billion in A.I. research by 2027."
print(word_tokenize(txt))
print(sent_tokenize(txt))

text = "I didn’t realize John’s ‘project update’ was actually scheduled for 9:30 a.m., not 7 p.m., on Wednesday."
print(word_tokenize(text))
print(sent_tokenize(text))

Example text	NLTK output	spaCy output
`Don't tokenize me incorrectly!`	`['Do', "n't", 'tokenize', 'me', 'incorrectly', '!']`	`['Do', "n't", 'tokenize', 'me', 'incorrectly', '!']`
`Amazon’s CEO ... A.I. research ...`	`['Amazon', '’', 's', ..., 'A.I', '.', 'research', ...]`	`['Amazon', '’s', ..., 'A.I.', 'research', ...]`
`I didn’t realize John’s ...`	`['I', 'didn', '’', 't', ..., 'John', '’', 's', ...]`	`['I', 'did', 'n’t', ..., 'John', '’s', ...]`

Part 5

Different Rules for Japanese

Tokenization in Japanese is segmentation because Japanese text is usually written without spaces between words. The same string can be segmented in different ways, so tools combine dictionaries and statistical models.

MeCab
Sudachi
spaCy with the GiNZA model

Tool note

GiNZA is a Japanese NLP model for spaCy. It segments Japanese text into tokens even when the sentence has no spaces.

Japanese Segmentation Explorer

Japanese example

GiNZA segmentation

import spacy
import ginza

nlp = spacy.load("ja_ginza")

text = "私は昨日新宿で美味しいラーメンを食べました。"
doc = nlp(text)
print([token.text for token in doc])

GiNZA segmentation: classical text

import spacy
import ginza

nlp = spacy.load("ja_ginza")

text = "春はあけぼの。やうやう白くなりゆく山ぎは、少しあかりて、紫だちたる雲の細くたなびきたる。"
doc = nlp(text)
print([token.text for token in doc])

Why this example matters

Classical Japanese uses older spellings and expressions, so segmentation can look surprising. This makes the example useful for discussing both the power and the limits of tokenization.

Part 6

Normalization

Normalization reduces superficial differences among tokens. We begin with lowercasing and punctuation removal, then contrast stopwords, lemmatization, and stemming.

raw tokens -> lowercase -> remove punctuation -> optional stopwords -> lemmas or stems

Context matters

Stopwords such as “the,” “and,” and “is” are often removed to reduce noise, but they may be important in poetry, style analysis, or careful linguistic work.

Lemmatization versus stemming

Lemmatization reduces words to valid base forms using grammar, such as running -> run and better -> good. It is usually more accurate.

Stemming chops words by simple rules without checking grammar, such as studies -> studi. It is fast but can produce non-words.

Normalization Explorer

Tokens Lowercase Remove punctuation tokens Remove common stopwords

Lowercasing and punctuation

import string

tokens = ["Data", ",", "data", "DATA", "~DATA_", "!"]
lower = [t.lower() for t in tokens]
clean  = [t for t in lower if t not in string.punctuation]

print(clean)

Lemmatization

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("The children were running faster because they had been given better shoes.")
print([w.lemma_ for w in doc])

Stemming

from nltk.stem import PorterStemmer

ps = PorterStemmer()
words = [
    "running", "runner", "runs",
    "happiness", "unhappily",
    "studies", "studying", "studied"
]
for w in words:
    print(f"{w:12s} -> {ps.stem(w)}")

Part 7

Part-of-Speech Tagging

Part-of-speech tagging assigns grammatical roles such as NOUN, VERB, and ADJ to tokens.

Filter by word type, such as nouns only.
Improve lemmatization or later named entity recognition.
Study writing style, such as adjective density.

Ambiguity

In “They will play the new play,” the first play is a verb, while the second is a noun. POS tags are contextual.

POS Tag Explorer

Sentence

POS tagging

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The quick brown fox jumps over the lazy dog.")

for t in doc:
    print(f"{t.text} -> {t.pos_}")

Part 8

From Raw Text to a Clean Corpus

The final section builds a minimal spaCy pipeline: load and tokenize, normalize by lowercasing and removing punctuation or stopwords, lemmatize tokens, and optionally keep only selected POS categories.

Load and tokenize with spaCy.
Normalize: lowercase, remove punctuation, remove stopwords.
Lemmatize tokens.
Optionally keep nouns, proper nouns, and verbs.

Pipeline principle

Every cleaning decision changes the corpus. Record your choices so later analysis is interpretable and reproducible.

Pipeline Explorer

Raw text Remove stopwords and punctuation Lemmatize selected known words Keep only NOUN, PROPN, VERB

spaCy pipeline

import spacy

nlp = spacy.load("en_core_web_sm")
text = "Apple was announcing new products, but many people prefer eating real apples."

doc = nlp(text)
print("Tokens:", [t.text for t in doc])

tokens_norm = [
    t.text.lower()
    for t in doc
    if not t.is_punct and not t.is_stop
]
print("Normalized:", tokens_norm)

lemmas = [
    t.lemma_.lower()
    for t in doc
    if not t.is_punct and not t.is_stop
]
print("Lemmas:", lemmas)

keep_pos = {"NOUN", "PROPN", "VERB"}
filtered = [
    t.lemma_.lower()
    for t in doc
    if t.pos_ in keep_pos and not t.is_stop and not t.is_punct
]
print("Filtered (NOUN/PROPN/VERB):", filtered)

Practice

Practice Questions

Use these questions to check your understanding and connect the examples to real text data.

Encoding check

Why does the Japanese sentence become garbled under Latin-1? What would happen if a dataset mixed encodings across files?

Tokenizer choice

For an analysis of emails and URLs, which tokenizer behavior is more useful: keeping wang@waseda.com as one token or splitting it into parts?

Normalization decision

When would removing stopwords damage the analysis? Think about sentiment, poetry, style, and short search queries.

Pipeline audit

Explain how the final filtered corpus changes if we keep adjectives in addition to nouns, proper nouns, and verbs.

Reference

Key Terms

Term	Meaning	Section
Text analysis	Methods for turning unstructured text into structured information.	What is Text Analysis?
Encoding	A mapping between characters and stored numbers; UTF-8 supports multilingual text.	Encoding and Representation
Tokenization	Splitting text into words, sentences, characters, or language-specific segments.	Tokenization
Stopwords	Common words often removed to reduce noise, depending on context.	Normalization
Lemmatization	Reducing words to valid base forms using linguistic information.	Stopwords, Lemmatization, Stemming
Stemming	Reducing words by rule-based chopping, sometimes producing non-words.	Stopwords, Lemmatization, Stemming
POS tagging	Assigning grammatical roles such as noun, verb, adjective, and punctuation.	POS Tagging
Pipeline	A sequence of preprocessing steps from raw text to clean corpus.	Building a Pipeline

Download Notebook