Intermediate24 min

pandas for datasets

Load, filter, group, and clean tabular data — the step before every model and the place most data bugs live.

Why this matters in AI / ML / GenAI

Training data, eval results, feature tables, and LLM cost logs are tables. pandas is how you inspect them, find nulls and duplicates, and compute per-segment metrics. Bad data beats a good model every time.

DataFrame and Series

A DataFrame is a table; a Series is one column. df["score"] gives a Series; df[["id", "score"]] gives a smaller DataFrame.

First four commands on any new dataset:

df.head() — look at actual rows
df.shape — rows and columns
df.dtypes — is that numeric column secretly a string?
df.describe() — ranges, means, and obvious outliers

Numeric columns loaded as object dtype means the CSV has stray text — fix that before training, not after.

Selecting, filtering, and adding columns

Boolean masks filter rows: df[df["score"] >= 0.7]. Combine with & and | and wrap each condition in parentheses — and/or do not work on Series.

.loc[rows, cols] selects by label; .iloc[] by position.

Add a column by assignment: df["passed"] = df["score"] >= 0.7.

Chained assignment (df[df.a > 1]["b"] = 0) may silently do nothing. Use .loc for assignment: df.loc[df["a"] > 1, "b"] = 0.

groupby and missing data

df.groupby("model")["latency_ms"].mean() is the split-apply-combine pattern behind every metric breakdown. .agg() computes several statistics at once.

Missing data: df.isna().sum() counts nulls per column. Then decide deliberately — dropna() when rows are unusable, fillna(value) when a default is meaningful. Silently filling with 0 corrupts metrics.

Duplicates: df.duplicated().sum(), then drop_duplicates(). Duplicate training rows leak between train and test splits and inflate scores.

Copy-paste examples

Copy into your own editor, or load one into the compiler below and press Run.

Build and inspect a DataFrame

First run loads pandas into the browser — allow a few seconds.

import pandas as pd

df = pd.DataFrame({
    "request_id": ["r1", "r2", "r3", "r4", "r5"],
    "model": ["gpt-4.1-mini", "gpt-4.1-mini", "llama-3", "llama-3", "gpt-4.1-mini"],
    "latency_ms": [820, 640, 1500, 1320, 700],
    "tokens": [512, 340, 900, 810, 410],
    "score": [0.91, 0.72, 0.65, 0.80, 0.55],
})

print(df.head())
print("shape:", df.shape)
print(df.dtypes)
print(df[["latency_ms", "score"]].describe().round(2))

Filter, derive, and sort

Parentheses around each condition are required when combining with &.

import pandas as pd

df = pd.DataFrame({
    "model": ["a", "a", "b", "b"],
    "latency_ms": [400, 1200, 300, 900],
    "score": [0.9, 0.6, 0.85, 0.45],
})

df["passed"] = df["score"] >= 0.7
fast_and_good = df[(df["latency_ms"] < 1000) & (df["score"] >= 0.7)]

print(df)
print("---")
print(fast_and_good.sort_values("score", ascending=False))

groupby with multiple aggregates

This is how you produce a per-model quality and cost report.

import pandas as pd

df = pd.DataFrame({
    "model": ["gpt-4.1-mini", "gpt-4.1-mini", "llama-3", "llama-3", "llama-3"],
    "latency_ms": [820, 640, 1500, 1320, 1410],
    "tokens": [512, 340, 900, 810, 850],
    "score": [0.91, 0.72, 0.65, 0.80, 0.70],
})

report = df.groupby("model").agg(
    n=("model", "size"),
    avg_latency=("latency_ms", "mean"),
    p95_latency=("latency_ms", lambda s: s.quantile(0.95)),
    avg_score=("score", "mean"),
    total_tokens=("tokens", "sum"),
).round(2)

print(report)

Find and handle dirty data

Always count nulls and duplicates before you train on anything.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "id": ["a", "b", "c", "c", "e"],
    "text": ["hello", None, "world", "world", "  spaced  "],
    "score": [0.9, 0.5, np.nan, np.nan, 0.7],
})

print("nulls per column:\n", df.isna().sum())
print("duplicate rows:", int(df.duplicated().sum()))

clean = (
    df.drop_duplicates()
      .dropna(subset=["text"])
      .assign(text=lambda d: d["text"].str.strip())
)
clean["score"] = clean["score"].fillna(clean["score"].median())
print("---")
print(clean)

Cost report per model

Try it — in-browser Python

Packages: pandas, numpy

Change the price per 1K tokens and see the cost column update.

Output

Python runs in your browser. First run downloads the runtime.

Press Run (or Ctrl+Enter) to execute.

CPython in WebAssembly. Stdlib works. NumPy and pandas load on demand. No input(), no GPU, no network installs.

Takeaways

head, shape, dtypes, describe — run these before anything else.
Filter with boolean masks and parentheses; assign with .loc.
groupby().agg() produces the per-segment metrics that reports need.