<< Home

Chapter 13: Iterators, Generators, and Comprehensions

In Python, iteration is more than a simple loop mechanism. It is a core language concept that powers how data is accessed, produced, and transformed. Understanding iterators, generators, and comprehensions reveals how Python achieves elegance and efficiency when working with collections or streams of data.

An iterator is any object that can be stepped through one element at a time. You encounter them constantly in Python whenever you loop over a list, tuple, or file. A generator takes this concept further, allowing you to define your own sequences that produce values lazily, only when requested. This makes them ideal for memory-efficient or infinite data flows.

Comprehensions provide a concise syntax for constructing lists, sets, and dictionaries directly from existing iterables. They combine iteration, filtering, and transformation into a single readable expression that often replaces multiple lines of code.

💡 Many Python features (such as loops, file objects, and even certain library utilities) are built on iterator principles. Once you grasp how the iteration protocol works, you gain insight into a large part of Python’s internal logic.

This chapter explores how the iteration protocol operates, how to build your own iterators and generators, and how comprehensions offer a compact yet powerful way to create and manipulate data structures.

The iteration protocol

Python separates the idea of something you can loop over from the thing that actually produces successive values. An iterable is any object that can return an iterator. An iterator is the stateful object that yields one value at a time.

The protocol is simple. An iterable implements __iter__() and returns an iterator. An iterator implements __next__() to produce the next value, and raises StopIteration when there are no more values. Iterators also implement __iter__() that returns self, which allows them to be used directly in loops.

# Built-in support
nums = [10, 20, 30]  # a list is an iterable
it = iter(nums)      # calls nums.__iter__()
next(it)             # calls it.__next__() → 10
next(it)             # → 20
next(it)             # → 30
# next(it) would now raise StopIteration

The for loop drives this protocol automatically. It calls iter(obj) once to get an iterator, then repeatedly calls next() until StopIteration is raised. The exception is handled internally, so you do not see it in normal loops.

💡 Many everyday objects are their own iterators. Open file handles, for example, return successive lines on each next() call. Lists and tuples are iterables that produce a fresh iterator each time you call iter().

You can implement a custom iterator by defining a separate iterable that creates iterator instances, or by making the object act as its own iterator. A two-class design is often clearer because it allows multiple independent passes over the same data.

# A minimal custom iterator pair

class Countdown: # iterable
  def __init__(self, start):
    self.start = start

  def __iter__(self):
    return CountdownIter(self.start)

class CountdownIter: # iterator
  def __init__(self, current):
    self.current = current

  def __iter__(self):
    return self

  def __next__(self):
    if self.current <= 0:
      raise StopIteration
    value = self.current
    self.current -= 1
    return value

for n in Countdown(3):
  print(n) # 3 2 1
⚠️ Iterators are consumable. Converting an iterator to a list, checking membership, or looping once will advance it. If you need to traverse data multiple times, start from an iterable that can produce a fresh iterator each time, or cache the data.

Making the object its own iterator is shorter, but it becomes one-shot. After a full pass, the internal state is exhausted and another loop will find nothing.

# Self-iterating, one-shot design

class Once: # iterator and iterable in one
  def __init__(self, items):
    self.items = list(items)
    self.index = 0

  def __iter__(self):
    return self

  def __next__(self):
    if self.index >= len(self.items):
      raise StopIteration
    item = self.items[self.index]
    self.index += 1
    return item

it = Once(["a", "b"])
list(it) # ['a', 'b']
list(it) # [] (already exhausted)
💡 The built-in iter(callable, sentinel) creates an iterator that repeatedly calls the function until it returns the sentinel value. This is useful for chunked reads and stream processing without manual loops. 
# Read fixed-size chunks until empty string
f = open("data.txt", "rt", encoding="utf-8")
for chunk in iter(lambda: f.read(1024), ""):
  process(chunk)
f.close()

You will usually prefer generators for custom iteration because they express the same protocol with simpler code. The next section introduces generator functions and shows how yield implements the same __next__ logic without the bookkeeping.

Generator functions and yield

Generators are a high-level, elegant way to create iterators without having to write the full class boilerplate. A generator function looks like an ordinary function but uses the yield keyword instead of return to produce a series of values over time. Each call to yield suspends the function’s state, allowing it to resume exactly where it left off the next time next() is called.

def countdown(n):
  while n > 0:
    yield n
    n -= 1

for i in countdown(3):
  print(i) # 3 2 1

When countdown(3) is called, it does not execute immediately. Instead, it returns a generator object that follows the iterator protocol, implementing both __iter__() and __next__() internally. Each iteration resumes the function’s frame until a return statement or the end of the function raises StopIteration.

💡 Generator functions are memory efficient because they produce items on demand instead of building large lists in memory. This is called lazy evaluation.

You can use yield within loops, conditionals, or even nested functions. The function can maintain local variables between yields, which makes it powerful for streaming or pipeline-style programming.

def even_numbers(limit):
  for i in range(limit + 1):
    if i % 2 == 0:
      yield i

print(list(even_numbers(10))) # [0, 2, 4, 6, 8, 10]

Generators can also receive input using send(), raise exceptions inside themselves with throw(), and be closed explicitly with close(). These advanced controls allow them to behave like lightweight coroutines, but in most use cases they are employed simply to yield sequences of data.

⚠️ A generator is exhausted once it has completed. Calling next() again will immediately raise StopIteration. To start over, you must create a new generator by calling the function again.

Just as list comprehensions build lists, generator expressions produce generator objects using a compact syntax in parentheses. They are often used for streaming transformations or to feed data directly into functions like sum() or any() without creating intermediate collections.

# Generator expression example
squares = (n * n for n in range(5))
print(sum(squares))   # 0 + 1 + 4 + 9 + 16 = 30
💡 Use generator expressions when you only need to iterate once, and list comprehensions when you need to reuse or index the result. Both rely on the same underlying iteration model.

Generator expressions

Generator expressions provide a compact, inline way to create generators without defining a separate function. They look similar to list comprehensions, but they use parentheses instead of square brackets. The result is a lazy iterator that yields items one at a time instead of constructing the whole collection in memory.

# Generator expression example
numbers = (n * n for n in range(5))
for value in numbers:
  print(value)        # 0 1 4 9 16

This syntax is particularly useful when passing data directly into another function. Because it does not build an intermediate list, it can handle large or infinite data sources efficiently.

# Streamed computation with no intermediate list
total = sum(n * n for n in range(1_000_000))
print(total)
💡 The key advantage of generator expressions is their lazy evaluation. Each item is produced only when needed, which saves memory and often improves performance on large datasets.

Generator expressions follow the same syntax and semantics as comprehensions, including support for filtering conditions and nested loops.

# With a filter condition
evens = (n for n in range(10) if n % 2 == 0)
print(list(evens))     # [0, 2, 4, 6, 8]

They are commonly used inside function calls or combined with built-ins like any(), all(), max(), and min(). When a generator expression is the only argument to a function, the parentheses can be omitted for readability.

# Parentheses not required here
result = sum(n for n in range(100) if n % 3 == 0)
print(result)
⚠️ Once consumed, a generator expression cannot be reused. Like all generators, it is exhausted after one full iteration. To iterate again, recreate the expression.

Generator expressions offer a middle ground between clarity and efficiency. They are simple enough to write inline, yet powerful enough to replace many explicit loops. In the next section, we turn to comprehensions, Python’s most concise syntax for building lists, sets, and dictionaries from iterables.

Comprehensions

Comprehensions are one of Python’s most expressive features. They let you build new collections from existing iterables using a single, readable line of code. The idea is simple: take an input sequence, apply an expression to each element, optionally filter it, and produce a new list, set, or dictionary as the result.

There are three main types of comprehensions:

# List comprehension
squares = [n * n for n in range(5)]
print(squares) # [0, 1, 4, 9, 16]

# Set comprehension
unique_lengths = {len(word) for word in ["one", "three", "seven"]}
print(unique_lengths) # {3, 5}

# Dictionary comprehension
mapping = {x: x * x for x in range(3)}
print(mapping) # {0: 0, 1: 1, 2: 4}

Comprehensions are equivalent to writing for loops that build collections manually, but they are more concise and often faster because they run in optimized bytecode.

💡 List comprehensions are ideal for transforming data in place, such as converting strings to lowercase, flattening nested lists, or applying mathematical operations to sequences. 
# Filter and transform in one expression
names = ["Alice", "Bob", "Clara", "David"]
short_lower = [n.lower() for n in names if len(n) <= 4]
print(short_lower)  # ['bob']

Comprehensions can also include multiple for clauses, which create nested loops. This makes them powerful for flattening or combining data from multiple sources.

# Cartesian product using nested comprehension
pairs = [(x, y) for x in [1, 2] for y in [3, 4]]
print(pairs)  # [(1, 3), (1, 4), (2, 3), (2, 4)]
⚠️ While comprehensions are elegant, overusing them can harm readability. If a comprehension starts to span multiple lines or requires complex conditions, a regular loop may be clearer.

Set and dictionary comprehensions follow the same pattern but use braces. In dictionary comprehensions, the expression before the for must produce key–value pairs separated by a colon.

# Set comprehension
letters = {ch for ch in "abracadabra" if ch.isalpha()}
print(letters)  # {'a', 'b', 'c', 'd', 'r'}

# Dictionary comprehension
squares = {n: n * n for n in range(5)}
print(squares)  # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Comprehensions combine clarity, speed, and power. They allow you to describe what you want to build rather than how to build it. Together with generators, they form a cornerstone of Python’s approach to expressive and efficient data processing.

💡 A good rule of thumb: use comprehensions when the intent fits comfortably on one line. If it starts needing explanation, write a loop instead.

Lazy evaluation and memory efficiency

One of Python’s design strengths lies in its ability to defer work until it is needed. This concept (known as lazy evaluation) is central to how iterators and generators operate. Instead of building large data structures all at once, Python can produce each value only when requested, saving both memory and processing time.

When you loop over a list, all its elements already exist in memory. When you loop over a generator, the values are computed on demand. This means you can work with sequences that are extremely large, or even infinite, without exhausting system resources.

# List: eager evaluation
nums_list = [n * n for n in range(1_000_000)]

# Generator: lazy evaluation
nums_gen = (n * n for n in range(1_000_000))

The list comprehension above allocates memory for one million squared integers. The generator expression, on the other hand, keeps only one value in memory at a time. The difference becomes significant in data pipelines or streaming tasks where the entire dataset cannot fit into memory.

💡 Lazy evaluation trades memory for computation. It computes each value only when you iterate, which allows you to process massive datasets efficiently or even handle continuous data streams.

Many built-in functions work seamlessly with lazy iterables. Functions like sum(), any(), all(), and max() consume iterators just as easily as lists. Likewise, the itertools module in the standard library provides a toolkit for chaining, filtering, and slicing lazy sequences without materializing them.

import itertools

# Infinite lazy sequence (never stored in memory)
counter = itertools.count(start=1)

# Take only the first 5 squares
first_five = (n * n for n in itertools.islice(counter, 5))
print(list(first_five)) # [1, 4, 9, 16, 25]

Lazy evaluation is not always the right choice. If you need to reuse data multiple times, random-access it, or serialize it, you must first convert it into a concrete structure such as a list or tuple. Otherwise, it will be exhausted after one pass.

⚠️ Generators and iterators are one-shot by design. Once they have been consumed, they cannot be reset. To use the same sequence again, you must recreate it or store the results in a list.

Python’s iterator and generator model provides a balance between readability and efficiency. Whether you are streaming data from files, combining filters, or generating values in mathematical series, lazy evaluation allows you to scale smoothly without changing your code structure. It is a subtle but powerful idea; compute just enough, just in time.