NLP by Vinod - Foundations

NLP Core Skills

Python Strings & Regex for NLP - The Real Foundation.

Before tokenization, embeddings, transformers, or BERT, every NLP pipeline starts with raw text. This post is my practical breakdown of Python strings, Unicode, regex patterns, and text cleaning for NLP.

NLP Python Regex Text Preprocessing

Python strings and regex for NLP may look like basic topics at first, but this is where real text processing begins. Before a model can classify sentiment, summarize text, detect entities, or use embeddings, Python first receives raw text as characters. If I cannot clean, inspect, slice, normalize, and extract patterns from that text, the entire NLP pipeline becomes weak.

In the NLP learning roadmap, I placed strings and regular expressions at the beginning of the Foundations Track for a reason. NLP is not only about transformers and BERT. It starts with understanding how text behaves inside Python.

Going through the notebooks properly, I realized how many NLP-specific details I had either skipped or never connected deeply: string immutability, Unicode, slicing, case normalization, whitespace cleaning, regex metacharacters, quantifiers, groups, and pattern extraction. These are not glamorous topics, but they appear everywhere in real preprocessing pipelines.

Regex was the bigger wake-up call. Earlier, I used it like many people do: search a pattern online, paste it, hope it works. This time, I wanted to actually understand what each symbol is doing and why regex is useful before tokenization, feature extraction, and model training.

"Every NLP pipeline starts with raw text."
Before you can tokenize, vectorize, or train a model, you need to understand exactly what Python is doing with that text at the character and pattern level.

Every NLP pipeline begins here: reading, cleaning, and manipulating raw text. Strings are the data structure; regex is the scalpel.

01 Python Strings for NLP - What I Actually Needed to Know

I started from the beginning, even the parts I thought I already knew. A string in Python is a sequence of characters. Each character is represented numerically under the hood, which is where ideas like ASCII, Unicode, and encoding start to matter.

ASCII covers 128 characters: English letters, digits, and common symbols. Unicode goes much wider and is designed to represent characters from many writing systems, including Hindi, Arabic, Chinese, Greek, mathematical symbols, emojis, and more. In Python 3, strings are Unicode internally, while UTF-8 is a common encoding used when storing or reading text from files.

Python

# Characters are just numbers under the hood
ord('A')        # gives 65
chr(65)         # gives 'A'

# Python strings support Unicode natively
'\u03A9'        # Greek letter Omega
'\u0041'        # same as A

This matters in NLP the moment I work with non-English datasets, scraped web text, social media posts, PDFs, or files saved with different encodings. If the text is not read correctly, the model never receives clean input in the first place.

Slicing, Indexing, and Immutability

Python string slicing is something I use constantly in NLP: pulling substrings, reversing text, extracting character-level patterns, or checking small examples before writing a full preprocessing function.

Python

name = "vinod"

name[0]       # first character
name[-1]      # last character
name[1:4]     # end index is exclusive
name[::-1]    # reverse the string

# Strings are immutable
try:
    name[0] = "V"
except TypeError as e:
    print(e)

The immutability trap: name.replace('o', 'O') does not change the original string. It returns a new one. If I forget to assign the result back with name = name.replace(...), the data stays unchanged and I may not get any error.

String Methods That Actually Matter for NLP

Python has many string methods, but for NLP work I kept noticing a few repeatedly: .replace(), .lower(), .split(), .join(), and .strip(). These are small tools, but they appear again and again in text cleaning workflows.

Python

text = "I'm a student. I'm learning NLP. Gnt, see you tomorrow."

# Expand abbreviations before processing
cleaned = text.replace("I'm", "I am").replace("Gnt", "Good night")

# Case normalization
cleaned.lower()
cleaned.upper()
cleaned.title()

# Splitting and joining
"P001, P002, P003".split(", ")

# Whitespace removal
"  vinod kumar   prajapat  ".strip()

The chained replace example above is a simple preprocessing pattern. It can be used to expand common contractions or abbreviations before running analysis. It is not always enough for production systems, but it is a good way to understand what text normalization actually means.

Text is just sequences of characters. Before any statistical or deep learning model can work with it, Python needs to clean, split, normalize, and structure it.

02 Regex for NLP - Actually Understanding Pattern Matching

Regex is one of those tools that looks like random noise until it starts making sense. I had used it before by copying patterns, but this time I wanted to understand the building blocks: metacharacters, anchors, quantifiers, character classes, groups, and substitutions.

The Python re module gives a clean interface for this. The functions I focused on were re.search(), re.findall(), re.sub(), and re.compile(). These are enough to start extracting and cleaning many common text patterns.

4

Core re functions

6+

Pattern types

5

Real NLP use cases

2

Practice notebooks

Metacharacters and Anchors

Metacharacters are symbols that give regex its power. A dot . matches any single character except a newline. Anchors do not match characters directly; they match positions. ^ means start of string, and $ means end of string.

Python

import re

re.search(r'hello.w', "hello world").group()

re.search(r'^hello', "hello world").group()

re.search(r'world$', "hello world").group()

Quantifiers - How Many Times?

Quantifiers let me match patterns of variable length. This is where regex becomes useful for real text because real text is rarely perfectly formatted.

Quantifier	Meaning	Example
*	0 or more	lo* matches l, lo, loo, looo
+	1 or more	lo+ matches lo and loo, not just l
?	0 or 1	colou?r matches colour and color
{n}	Exactly n	\d{3} matches exactly 3 digits
{n,}	n or more	\d{3,} matches 3 or more digits
{n,m}	Between n and m	\d{1,4} matches 1 to 4 digits

Python

re.findall(r'colou?r', "colour is written as color")

re.findall(r'\d{1,4}', "1234 and 56")

re.findall(r'\d+', "1234 and 56")

re.findall(r'\d*', "1234 and 56")

The * vs + distinction is subtle but important: \d* matches zero or more digits, so it can also match empty positions. \d+ requires at least one digit, so it usually behaves better when extracting real numbers from text.

Character Classes and Special Sequences

Character classes let me define my own set of characters to match. Special sequences are shortcuts for common patterns. These appear constantly in NLP text cleaning, token filtering, and pattern extraction.

Character Classes

• [aeiou] means any vowel
• [^aeiou] means any non-vowel
• [a-zA-Z] means any English letter
• [a-zA-Z0-9] means alphanumeric character
• [-.\s] means hyphen, dot, or space

Special Sequences

• \d means any digit from 0 to 9
• \D means any non-digit
• \w means word character
• \W means non-word character
• \s means whitespace
• \S means non-whitespace

Python

re.findall(r'[aeiou]', "hello world")

re.findall(r'\b[a-zA-Z]{3}\b', "abc def ghijklm")

re.findall(r'\D', "1234343abc@")

re.findall(r'\s', "hello  world  ")

Groups and Capturing

Parentheses in regex create capturing groups. They let me extract specific parts of a match instead of only getting the full matched string. There is also a non-capturing group (?:...) for grouping without storing the captured part.

Python

text = "call me at 123-456-78788"
pattern = re.compile(r'(\d{3})-(\d{3})-(\d{4,5})')
re.search(pattern, text).groups()

re.search(r'(?:\d{4})', "1234 and 5678").group()

Regex is fundamentally about pattern matching: expressing structure in text so precisely that a machine can find it reliably.

03 Where Regex Actually Shows Up in NLP

This is the part that made the topic feel practical. Regex is not just syntax practice. It appears in real NLP workflows whenever raw text contains phone numbers, emails, URLs, dates, repeated whitespace, tags, punctuation, or noisy formatting.

01

Extracting Phone Numbers

Real-world phone numbers are inconsistent. Some have parentheses, some use hyphens, and some are written without separators. Regex helps define a flexible pattern.

Python

text = "my numbers are 123-456-7890 and (987) 654-3210 or 1234567899"
pattern = r'\(?\d{3}\)?[-.\\s]?\d{3}[-.\\s]?\d{4}'
re.findall(pattern, text)

02

Extracting Email Addresses

Email extraction comes up in contact parsing, spam classification, and scraped web data. The pattern needs to handle username characters, the domain, and the top-level domain.

Python

text = "My email is test@gmail.com"
pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
re.search(pattern, text).group()

03

Extracting URLs

Web text often contains URLs that need to be extracted, replaced, or removed before modeling. This is common in scraped datasets and social media text.

Python

text = "Visit https://www.example.com or http://example.org for more info."
pattern = r'https?://(?:www\.)?\S+'
re.findall(pattern, text)

04

Extracting Dates

Date extraction is useful in documents, logs, news data, and records. A simple fixed-format date pattern is a good starting point before moving to more advanced date parsing.

Python

text = "The event is on 12/10/2022"
pattern = r'\d{2}/\d{2}/\d{4}'
re.search(pattern, text).group()

05

Cleaning Whitespace with re.sub()

The re.sub() function is regex-based find-and-replace. It becomes useful when whitespace, punctuation, or formatting problems are more complex than a simple .strip().

Python

text = "   hello world  "
cleaned = re.sub(r'^\s+|\s+$', '', text)

match = re.search(r'python', "I love Python programming!", re.IGNORECASE)
match.group()

The key insight from regex for NLP: Regex is not a replacement for a real tokenizer, parser, or NER model. It is the tool I use before those systems, and sometimes around those systems, to handle predictable patterns in messy text.

04 Strings vs Regex - When to Use Which

One thing I had to think through carefully was when a simple string method is enough and when regex is actually needed. This matters because readable preprocessing code is easier to debug, and regex should not be used just to look advanced.

Use String Methods When...

• Exact fixed replacement is enough: .replace("I'm", "I am")
• Case normalization is needed: .lower() or .upper()
• Text splits on a known delimiter: .split(", ")
• You only need a simple membership check: "word" in text
• You only need basic whitespace removal: .strip()

Use Regex When...

• You are extracting variable-format patterns like phones, emails, dates, or URLs
• The pattern has optional or repeated components
• You need to match based on character classes
• You are handling multiple format variants at once
• Whitespace or punctuation cleaning needs pattern logic

String methods are usually faster and easier to read. Regex is more powerful when the text pattern is variable or structured. In practice, a real NLP preprocessing pipeline uses both. String methods handle simple deterministic operations, while regex handles pattern extraction and flexible cleaning.

05 The Notebooks Behind This Post

The notebooks connected to this post are part of my public NLP learning repository. I am keeping the work visible because the blog explains what I understood, while GitHub shows the actual implementation, experiments, and practice code behind that understanding.

GH

github.com/vinod-kaumar/NLP-by-vinod

Notebooks: 01_strings.ipynb and 02Regex.ipynb - all code tested and annotated as part of the NLP foundations track.

Open Repository

If any pattern or concept does not work as expected, the best fix is usually to run the notebook cells in sequence and check the Python version, input string, and regex pattern carefully. This topic connects directly back to the complete NLP learning roadmap, where strings and regex are the first foundation before preprocessing, tokenization, and text representation.

06 What Comes Next in the NLP Journey

Now that I have covered Python strings and regex for NLP, the next topics move deeper into the foundations. The goal is not to rush into transformers, but to build the base properly so later models actually make sense.

01

Machine Learning Refresher for NLP

A focused review of supervised learning, unsupervised learning, classification, evaluation, and why classic ML still matters before deep learning-based NLP.

02

Linguistics for NLP

Part-of-speech tagging, syntax, semantics, morphology, and why language structure matters before building NLP models.

03

Data Acquisition for NLP

How real NLP data is collected from web pages, CSV files, JSON data, APIs, and documents before preprocessing begins.

NLP Python Regex Text Processing Strings Preprocessing NLP Foundations AI Engineering

The Foundations Come Before the Fun Stuff.

Python strings and regex may not look as exciting as transformers, but they decide whether an NLP pipeline starts clean or broken. This is the kind of foundation I want to build before moving deeper.

Read the NLP Roadmap Continue the NLP Journey

Following along? The notebooks and experiments are connected through GitHub so the learning stays visible and reproducible.