Chapter 16: Working with Data and Text

Modern programming often revolves around data: collecting it, transforming it, and sharing it in readable or structured forms. Whether that data comes from files, network responses, or user input, Python’s design makes it unusually well suited for this work. Its flexible syntax and built-in tools let you move easily between plain text and structured data such as JSON, CSV, or tabular formats.

This chapter explores how Python reads, interprets, and reshapes data in practical ways. It builds on the basics of file I/O introduced earlier and focuses on text handling, parsing, and transformation techniques. You will learn how to use built-in string methods, the re module for regular expressions, and simple iterator patterns to extract meaning and structure from raw input.

Data and text are deeply connected. Text often carries structured information (logs, reports, configuration files) that must be cleaned and parsed before it becomes usable. Python provides consistent patterns for doing so without heavy dependencies: small, readable functions that work well together. Once you grasp these, you can handle many real-world data formats with little more than the standard library.

We will also look at patterns for transforming data—filtering, mapping, grouping, and aggregating, using core features like comprehensions and itertools. The aim is to move from reading text to producing clean, structured results that other code or systems can consume.

Beyond Simple File I/O

By now you have seen how to open files, read or write their contents, and use context managers to ensure that resources are properly closed. Those patterns cover most everyday tasks, but real data rarely arrives as perfectly formatted text. It may contain inconsistent separators, nested structures, or a mixture of encodings. Moving beyond basic file handling means learning how to adapt to those differences and prepare data for further processing.

Python’s standard library makes this easier than it sounds. Modules like csv and json handle structured data, while the built-in string tools let you clean and normalise raw text. You can open files line by line to process large inputs efficiently, or use generators to stream data without loading everything into memory. The key is to combine readability with robustness, writing code that works even when the input isn’t perfect.

# Reading a large text file safely and efficiently
with open("data.txt", encoding="utf-8") as f:
  for line in f:
    line = line.strip()
    if not line:
      continue
    print(line)

This approach reads one line at a time and removes stray whitespace. It’s memory-efficient and resilient to irregularities. Most text files can be processed this way, whether they contain logs, lists, or tab-separated data.

Structured formats like JSON or CSV add a small layer of complexity but are still handled with concise, declarative code. Here is a short refresher that also demonstrates how to work with nested or irregular data.

import json
from pathlib import Path

data = json.loads(Path("users.json").read_text(encoding="utf-8"))
for user in data:
  print(user["name"], "-", user.get("email", "no email"))

When working with CSV files, the csv module supports variable delimiters and flexible field quoting. It is common to encounter non-standard files, so adapting the reader is part of professional practice.

import csv

with open("data.csv", newline="", encoding="utf-8") as f:
  reader = csv.reader(f, delimiter=";")
  for row in reader:
    print(row)

Regular Expressions with re

When simple string methods are not enough, regular expressions provide a powerful way to search, extract, and transform text based on patterns rather than fixed substrings. Python’s re module implements this capability with concise syntax and efficient matching routines. It allows you to describe text structures (words, numbers, dates, tags) and process them in a single pass.

A regular expression, or regex, is a mini-language for describing text patterns. Even a short expression can match a wide range of possibilities, making it ideal for cleaning data, validating input, or parsing semi-structured text such as logs and configuration files.

import re

text = "Email: alice@example.com, Phone: +44 1234 567890"
emails = re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}", text)
print(emails)

Here, findall() returns every substring that fits the pattern, an email address in this case. Patterns are written as raw strings (prefixed with r) so that backslashes are interpreted literally rather than as escape sequences.

Basic operations

• re.search(pattern, string) – find the first match anywhere in the text
• re.match(pattern, string) – match only at the start of the text
• re.findall(pattern, string) – return all matches as a list
• re.sub(pattern, repl, string) – replace matches with new text

log = "2025-10-22 ERROR Disk full"
m = re.search(r"(\d{4}-\d{2}-\d{2})\s+(\w+)", log)
if m:
  date, level = m.groups()
  print("Date:", date, "Level:", level)

This pattern captures two groups: a date and a word. Parentheses define capture groups, which you can retrieve later with groups() or by index. Regular expressions are terse but extremely expressive once learned.

Cleaning and substitution

text = "Price: $1,299.00"
cleaned = re.sub(r"[^0-9.]", "", text)
print(cleaned)  # 1299.00

Substitution with re.sub() is common in data cleaning: removing punctuation, trimming noise, or normalising formats. Each substitution can apply globally or selectively with flags like re.IGNORECASE or re.MULTILINE.

Precompiled patterns

For repeated operations, compile the pattern once for speed and clarity:

pattern = re.compile(r"\b\w{4}\b")
words = pattern.findall("This text has many four word units")
print(words)

Compiled expressions store settings and optimise matching internally. They also support methods like fullmatch() (entire string match) and split() (tokenise text based on pattern boundaries).

pattern = re.compile(r"""
  ([A-Za-z]+) # first word
  \s+         # one or more spaces
  (\d{4})     # a four-digit number
""", re.VERBOSE)

m = pattern.search("Invoice 2025")
print(m.groups())

Simple Parsing Patterns

Not all text needs the full power of regular expressions. Many files and logs follow simple, predictable formats that can be parsed with built-in string and iteration methods. Python’s clean text functions (split(), partition(), strip(), and join()) make it easy to extract and restructure data with minimal effort.

Parsing in this sense means turning raw strings into structured information: lists, dictionaries, or named tuples that your program can process further. The goal is clarity and reliability rather than complex pattern matching.

Splitting and partitioning

line = "id=42;name=Alice;score=91"
parts = line.split(";")
fields = dict(p.split("=") for p in parts)
print(fields)

This splits the line into key–value pairs and builds a dictionary in one step. It works because the separators are consistent and simple. For irregular formats, partition() offers a precise alternative that splits once and returns a tuple of three elements: before, separator, and after.

entry = "user: bob@example.com"
label, sep, value = entry.partition(":")
print(label.strip(), "→", value.strip())

Line-based parsing

When dealing with structured logs or configuration files, processing one line at a time keeps the logic simple and memory use low.

lines = [
  "2025-10-22 OK",
  "2025-10-22 ERROR Disk full",
  "2025-10-22 OK"
]

records = []
for line in lines:
  date, status, *message = line.split(maxsplit=2)
  records.append({
    "date": date,
    "status": status,
    "message": message[0] if message else ""
  })

print(records)

Using the star operator (*) collects any extra fields without causing unpacking errors. This approach is both readable and flexible for small parsing tasks.

Combining parsing with filtering

log = """
INFO: Starting process
WARNING: Disk space low
ERROR: Operation failed
INFO: Shutting down
""".strip().splitlines()

errors = [line for line in log if line.startswith("ERROR")]
print(errors)

Simple filtering like this often replaces complex regular expressions. When a format is regular, clarity should take precedence over compactness.

Hybrid approach: simple + regex

For semi-structured text, a mix of string methods and regular expressions often works best. Use string logic for overall structure and regex for fine-grained extraction within a field.

import re

log_entry = "User Alice (id=42) logged in at 10:45"
name_part, _, rest = log_entry.partition("(")
user = name_part.replace("User", "").strip()
user_id = re.search(r"id=(\d+)", rest).group(1)
print(user, user_id)

This pattern keeps the overall logic readable while using a small regex for precision. Together, these tools let you handle most text-based formats you will encounter in everyday scripts or data pipelines.

Data Transformation and Filtering

Once data is loaded or parsed, the next step is often to reshape, clean, or summarise it. In Python, these tasks are naturally expressed through comprehensions, higher-order functions like map() and filter(), and built-in tools from the itertools module. Each of these helps you express transformation logic clearly, without unnecessary loops or temporary variables.

Data transformation usually means taking one collection of values and producing another: filtering out unwanted entries, converting types, or computing new structures from existing ones. Python’s functional and declarative style makes these transformations concise and readable.

Filtering data

scores = [72, 85, 90, 47, 66, 100]
passed = [s for s in scores if s >= 70]
print(passed)

List comprehensions are the most direct way to filter and project data in a single expression. They are equivalent to loops but shorter and easier to reason about.

# The same using filter()
passed = list(filter(lambda s: s >= 70, scores))

The filter() function pairs naturally with lambda expressions or named functions when you want to reuse logic across transformations.

Mapping and conversion

names = ["alice", "bob", "charlie"]
uppercased = [n.title() for n in names]
print(uppercased)

The map() function performs the same pattern procedurally:

uppercased = list(map(str.title, names))

Mapping functions apply a transformation to every element of an iterable. Because they return iterators, they can chain neatly into generators and pipelines without holding everything in memory.

Combining map and filter

values = ["10", "x", "25", "7", "NaN", "40"]

def valid_int(x):
  return x.isdigit()

cleaned = [int(x) for x in values if valid_int(x)]
print(cleaned)

This pattern (filter first, then map) appears in many data-cleaning tasks. Filtering ensures you only transform valid inputs, avoiding runtime errors from invalid data.

Grouping and aggregating

Some transformations summarise or organise data rather than change individual items. The itertools.groupby() function groups adjacent items that share a key, making it ideal for simple aggregation.

import itertools

records = [
  ("A", 10), ("A", 20), ("B", 5), ("B", 15), ("B", 10)
]

for key, group in itertools.groupby(sorted(records), key=lambda r: r[0]):
  total = sum(value for _, value in group)
  print(key, total)

Grouping is especially useful after sorting data by a key. It allows you to compute totals, averages, or frequency tables with minimal code.

Chaining transformations

When multiple transformations need to occur in sequence, generator expressions and itertools utilities keep memory use low and pipelines expressive.

import itertools

nums = range(10)
pipeline = (n**2 for n in nums if n % 2 == 0)
print(list(itertools.islice(pipeline, 5)))

This pattern is both lazy and efficient: it computes only what is needed, when it is needed. This is how large datasets can be processed in streaming fashion without intermediate storage.

Sorting and restructuring

The sorted() function can be combined with key functions for flexible ordering and restructuring of data.

data = [
  {"name": "Alice", "score": 82},
  {"name": "Bob", "score": 91},
  {"name": "Charlie", "score": 77}
]

sorted_data = sorted(data, key=lambda d: d["score"], reverse=True)
for d in sorted_data:
  print(d["name"], d["score"])

Sorting by computed keys, reversing order, or chaining with transformations gives you complete control over how results are structured. These same patterns extend to tuples, dictionaries, and custom classes.

Applied “Recipes”

Data handling in Python often comes down to combining small, clear techniques into useful workflows. The following short “recipes” show how to apply the patterns from this chapter to solve practical problems, such as converting between formats, extracting information, and cleaning or summarising text data.

1. Converting JSON to CSV

import json, csv

# Read structured JSON data
with open("users.json", encoding="utf-8") as f:
  users = json.load(f)

# Write a CSV file with selected fields
with open("users.csv", "w", newline="", encoding="utf-8") as f:
  writer = csv.writer(f)
  writer.writerow(["name", "email"])
  for u in users:
    writer.writerow([u["name"], u.get("email", "")])

This combines json and csv to convert structured records into a portable tabular form. Each user dictionary becomes a CSV row, with safe defaults for missing keys.

2. Extracting email addresses from text

import re
from pathlib import Path

text = Path("correspondence.txt").read_text(encoding="utf-8")
emails = sorted(set(re.findall(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}", text)))
for e in emails:
  print(e)

This example reads a text file, finds all unique email addresses using a regular expression, and prints them sorted. Regular expressions remain one of the most concise ways to identify structured tokens inside free-form text.

3. Summarising numeric data

import csv
from statistics import mean

with open("scores.csv", newline="", encoding="utf-8") as f:
  reader = csv.DictReader(f)
  scores = [float(row["score"]) for row in reader]

print("Average score:", round(mean(scores), 2))
print("Highest score:", max(scores))

Here the csv module reads data into dictionaries, which are then transformed with comprehensions and summarised using statistics.mean(). This pattern scales to any numeric column with minimal code changes.

4. Cleaning inconsistent input

import re

lines = [
  "Alice : 82",
  "Bob, 91",
  "Charlie - 77",
]

records = []
for line in lines:
  name, score = re.split(r"[:,-]\s*", line)
  records.append((name.strip(), int(score)))

print(records)

When data is semi-structured but inconsistent, small regular expressions combined with splitting logic can restore uniformity. Once cleaned, it can easily be written to JSON or CSV for further use.

5. Counting word frequency

from collections import Counter
from pathlib import Path

text = Path("notes.txt").read_text(encoding="utf-8").lower()
words = re.findall(r"[a-z']+", text)
counts = Counter(words)

for word, freq in counts.most_common(10):
  print(word, freq)

This reads a text file, tokenises it into lowercase words, and counts occurrences. The Counter class provides a direct and efficient way to summarise unstructured data.

6. Generating a simple report

import csv

with open("sales.csv", newline="", encoding="utf-8") as f:
  reader = csv.DictReader(f)
  totals = {}
  for row in reader:
    region = row["region"]
    amount = float(row["amount"])
    totals[region] = totals.get(region, 0) + amount

for region, total in sorted(totals.items()):
  print(f"{region:10} {total:,.2f}")

This aggregates data by region and formats the results neatly. It combines file reading, grouping, and formatted output — a compact example of everyday data transformation in Python.