Iterators & Generators in Python

Deep dive into Python iterators and generators: iteration protocol, custom iterators, generator functions, lazy evaluation, infinite sequences, generator expressions, coroutines, async generators, itertools, and real-world backend use cases.

Why Iterators & Generators Matter

Iterators and generators allow memory-efficient, lazy data processing. They are essential for handling large datasets, streams, and infinite sequences.

1. Pull-Based Data Model

Consumer controls execution.

The Iteration Protocol

Python iteration relies on __iter__() and __next__().

1. Iterator Structure

Contract methods.

Generator Functions

Generators simplify iterator creation using yield.

1. Basic Generator

Using yield.

2. Manual next() Calls

Control execution manually.

yield from

yield from delegates to another iterable.

1. Delegating Generators

Flatten nested iteration.

Lazy Evaluation & Infinite Generators

Generators compute values only when requested.

1. Infinite Generator

Unbounded sequence.

Generator Expressions

A generator expression is a lightweight and memory-efficient way to create iterators in Python. It looks very similar to a list comprehension but uses parentheses instead of square brackets. However, unlike a list comprehension, it does not build the entire result in memory immediately. Instead, it produces values one at a time only when they are requested during iteration. This lazy evaluation makes generator expressions ideal for large datasets, streaming data, and backend systems where loading everything into memory would be inefficient or unnecessary.

1. Basic Generator Expression

Memory-efficient comprehension.

Advanced Generator Control

Generators support send(), throw(), and close().

1. send() Method

Two-way communication.

Coroutines (Pre-async Era)

Generators were used to build coroutines before async/await.

1. Simple Coroutine

Pause and resume.

Async Generators

Async generators combine async and yield.

1. Async Generator Function

async def + yield.

itertools Module

Python provides powerful iterator utilities.

1. itertools.count()

Infinite counter.

2. itertools.chain()

Combine iterables.

Best Practices

Generators are powerful but must be used carefully.

1. Recommended Guidelines

Production rules.