Python Docs

Generators

Generators are lazy iterables defined with the yield keyword. They produce values on demand instead of storing everything in memory, making them ideal for large datasets and infinite sequences.

Why Generators?

  • Memory-efficient: yields items one-by-one
  • Lazy evaluation: compute only when needed
  • Composable for pipelines (map, filter, itertools)
  • Avoids writing complex iterator classes

Yield Basics

def count_up_to(n): i = 1 while i <= n: yield i i += 1 for x in count_up_to(5): print(x)

Each yield pauses the function and resumes from the same point on the next iteration.

Generator Expressions

# List comprehension (eager) squares_list = [x*x for x in range(10)] # Generator expression (lazy) squares_gen = (x*x for x in range(10)) print(next(squares_gen)) # 0 print(next(squares_gen)) # 1

Advanced Generator Methods

def echo(): received = None try: while True: received = yield received except GeneratorExit: print("Generator closed") g = echo() print(next(g)) # Start -> None print(g.send("hi")) # Send value in g.close() # Triggers GeneratorExit

Chaining with itertools

import itertools as it def read_lines(paths): # Lazily read from many files files = (open(p, 'r', encoding='utf-8') for p in paths) with it.chain.from_iterable(files) as all_lines: for line in all_lines: yield line.rstrip('\n')

Use Cases

  • Streaming large logs / CSV data
  • Producer–consumer pipelines
  • Infinite sequences
  • Tree & graph traversals
def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b f = fibonacci() for _ in range(6): print(next(f)) # 0 1 1 2 3 5

Best Practices

  • Prefer generator expressions for simple transforms
  • Use yield from to delegate subgenerators
  • Use contextlib.ExitStack for many file handles
  • Document that generators are single-use