Intermediate18 min

Iterators, iterables, and zip

How for-loops really work, building your own iterator, and the pairing tools zip, enumerate, and itertools.

Why this matters in AI / ML / GenAI

PyTorch DataLoaders, Hugging Face datasets, streaming API responses, and file readers are all iterators. Understanding the protocol explains why a dataset can be consumed only once and why a generator uses no memory.

The iteration protocol

An iterable can produce an iterator: it implements __iter__. Lists, strings, dicts, sets, and files are iterables.

An iterator produces values one at a time: it implements __next__ and raises StopIteration when exhausted.

for x in items: is shorthand for: call iter(items) to get an iterator, then call next() repeatedly until StopIteration.

The key consequence: an iterator is consumed. A list can be looped many times; a generator or a zip object cannot. Looping it a second time yields nothing — a silent bug that shows up as an empty second epoch or an empty validation set.

If you need to iterate twice, materialise with list() first, or rebuild the iterator.

zip, enumerate, and unpacking

zip(a, b) pairs elements positionally and stops at the shortest input. That silent truncation is the classic bug when features and labels have different lengths — pass strict=True (Python 3.10+) to raise instead.

enumerate(seq, start=1) gives index and value together, which is cleaner than range(len(seq)).

Unzip with zip(*pairs). Tuple unpacking in the loop header (for name, score in pairs:) keeps loops readable, and for i, (name, score) in enumerate(pairs): combines both.

Writing your own iterator

Implement __iter__ (returning self) and __next__ (returning the next value or raising StopIteration).

In practice a generator function is almost always simpler — yield gives you the same protocol for free. Write a class only when the iterator needs to carry substantial state or extra methods.

itertools covers the standard patterns: islice (take n from an infinite stream), chain (join iterables), cycle, count, groupby, product (hyperparameter grids), and combinations.

Copy-paste examples

Copy into your own editor, or load one into the compiler below and press Run.

What a for-loop actually does

Manual iteration makes StopIteration visible.

items = ["a", "b", "c"]
it = iter(items)

print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration -> the for-loop would end here")

print("list is re-iterable:", list(items), list(items))

Iterators are consumed once

Run this — the second loop over the generator prints nothing.

def batches(n):
    for i in range(n):
        yield f"batch-{i}"

gen = batches(3)
print("first pass :", list(gen))
print("second pass:", list(gen), "<- empty, already consumed")

pairs = zip([1, 2, 3], ["a", "b", "c"])
print("zip first  :", list(pairs))
print("zip second :", list(pairs), "<- also empty")

materialised = list(batches(3))
print("materialised twice:", materialised, materialised)

zip truncation and strict mode

Mismatched feature and label lengths fail silently without strict=True.

features = [[1, 2], [3, 4], [5, 6]]
labels = [0, 1]

print("silently truncated:", list(zip(features, labels)))

try:
    list(zip(features, labels, strict=True))
except ValueError as err:
    print("strict caught it:", err)

names = ["mlops", "rag", "agents"]
scores = [0.9, 0.8, 0.7]
for i, (name, score) in enumerate(zip(names, scores), start=1):
    print(f"{i}. {name:8} {score:.2f}")

unzipped_names, unzipped_scores = zip(*zip(names, scores))
print("unzipped:", unzipped_names, unzipped_scores)

A custom iterator class vs a generator

Both satisfy the protocol; the generator is four lines shorter.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

def countdown(start):
    while start > 0:
        yield start
        start -= 1

print("class    :", list(Countdown(4)))
print("generator:", list(countdown(4)))

itertools for real tasks

product builds hyperparameter grids; islice safely samples an infinite stream.

import itertools

grid = list(itertools.product([1e-5, 2e-5], [16, 32], ["adam", "adamw"]))
print("grid size:", len(grid))
for lr, batch, opt in grid[:4]:
    print(f"  lr={lr} batch={batch} optimizer={opt}")

infinite = itertools.count(start=100, step=50)
print("\nfirst 4 of an infinite counter:", list(itertools.islice(infinite, 4)))
print("chained:", list(itertools.chain([1, 2], [3, 4])))
print("pairs:", list(itertools.combinations(["a", "b", "c"], 2)))

Pair predictions with labels safely

Try it — in-browser Python

Remove a label from the list and watch strict=True catch the mismatch.

Output

Python runs in your browser. First run downloads the runtime.

Press Run (or Ctrl+Enter) to execute.

CPython in WebAssembly. Stdlib works. NumPy, pandas, scikit-learn and Matplotlib load on demand, and charts render below. No input(), no GPU, no network installs.

Takeaways

for-loops call iter() then next() until StopIteration.
Generators and zip objects are consumed once — materialise with list() to reuse.
zip truncates to the shortest input unless you pass strict=True.