Python Iterators and Generators: Lazy Data Processing with yield

Python course module guide: Functions, modules and data pipelines

Python iterators and generators let a program process values one at a time instead of building every result in memory first. They are useful for files, API pages, event streams and reusable data pipelines.

Iterable, iterator and generator

An iterable can produce an iterator. Lists, tuples, strings and many file objects are iterable. An iterator remembers its current position and returns the next value when next() is called. When no values remain, it raises StopIteration. A generator function is a convenient way to create an iterator by using yield.

names = ["Asha", "Ravi", "Meena"]
iterator = iter(names)

print(next(iterator))
print(next(iterator))

A for loop performs this protocol for you. Understanding it explains why a file can be processed line by line and why an exhausted generator does not restart automatically.

How yield changes a function

def countdown(start: int):
    current = start
    while current > 0:
        yield current
        current -= 1

for value in countdown(3):
    print(value)

Calling countdown(3) returns a generator object without running the full body. Each request for the next item resumes execution after the previous yield. Local variables keep their values between resumptions. When the function finishes, iteration stops.

Generators still benefit from small, explicit functions. The Python functions guide explains parameters, return values and scope that support this design.

Lazy processing for large files

from pathlib import Path

def error_lines(path: Path):
    with path.open(encoding="utf-8") as handle:
        for line_number, line in enumerate(handle, start=1):
            if "ERROR" in line:
                yield line_number, line.rstrip()

for number, message in error_lines(Path("app.log")):
    print(number, message)

The function holds one current line instead of a separate list containing the complete file. It also owns the file resource, so the with block remains active while iteration proceeds. For file encodings, CSV, JSON and error cases, review the Python file handling guide.

Compose generators into a pipeline

def non_empty(lines):
    for line in lines:
        cleaned = line.strip()
        if cleaned:
            yield cleaned

def parse_scores(lines):
    for line in lines:
        name, score = line.split(",")
        yield name, int(score)

source = ["Asha,82", "", "Ravi,74"]
passing = (
    (name, score)
    for name, score in parse_scores(non_empty(source))
    if score >= 50
)

for item in passing:
    print(item)

Each stage does one job: clean, parse, filter and consume. This is easier to test than one generator containing every rule. When invalid input should carry a precise domain error, use the later custom exceptions guide.

Generator expressions

A generator expression uses parentheses instead of the square brackets of a list comprehension. It is ideal when the consumer accepts any iterable, such as sum(), max() or a loop.

total = sum(number * number for number in range(1_000_000))

The program does not first create a million-element list of squares. If you need to revisit those squares or access item 500 directly, create a list instead. Compare both forms in the Python list comprehensions guide.

Use yield from for delegation

def all_lines(paths):
    for path in paths:
        with path.open(encoding="utf-8") as handle:
            yield from handle

yield from delegates iteration to another iterable. It is useful when combining child sequences, but a normal loop may be preferable when each item needs transformation, logging or error handling.

Common generator mistakes

• Trying to iterate twice: generator objects are normally one-shot.
• Hiding errors until consumption: generator work happens later, so exceptions may appear far from creation.
• Leaking resources: make resource ownership explicit and close a partially consumed generator when necessary.
• Using a generator when random access is required: choose a list for indexing and repeated traversal.
• Adding unclear side effects: generators are easiest to reason about when they yield data rather than modify unrelated state.

Practice project: streaming report filter

Create a generator that reads a CSV export one row at a time, validates the column count, converts an amount, filters records above a threshold and yields clean dictionaries. Add counters for rejected rows and write tests for empty input, invalid numbers and early termination. The exercise connects functions, file handling, exceptions and lazy processing without requiring a large framework.

Iterator and generator FAQs

Can a generator be restarted?

A generator object is normally one-shot. Call the generator function again to create a new iterator, provided the underlying source can also be reopened or recreated.

Does a generator always save memory?

It avoids materializing every yielded item, but downstream code can remove that advantage by calling list() or retaining all results.

When should I return a list?

Return a list when callers reasonably need indexing, length, repeated traversal or a stable snapshot. Use an iterator when one-pass streaming is part of the contract.

Official reference: Python data model reference for generator functions.