Advanced20 min

From NumPy to PyTorch tensors

Tensors, autograd, and a training loop — the Python patterns behind every model you will fine-tune.

Why this matters in AI / ML / GenAI

PyTorch is the default framework for training and fine-tuning. A tensor is a NumPy array that also tracks gradients and can live on a GPU. Once you can read a training loop, model code stops being intimidating.

Tensors are arrays with two extras

A PyTorch tensor behaves like a NumPy array — same shapes, same broadcasting, same indexing — plus two capabilities:

  1. Device placement: tensor.to("cuda") moves data to a GPU.
  2. Autograd: with requires_grad=True, PyTorch records operations and computes gradients by calling .backward().

Conversion is cheap: torch.from_numpy(arr) and tensor.numpy(). On CPU they can share memory, so modifying one changes the other.

The error you will meet most: "Expected all tensors to be on the same device". Your model is on the GPU and your batch is still on the CPU. Move both.

The training loop, every time

Five steps repeated for each batch:

  1. optimizer.zero_grad() — clear old gradients (forget this and gradients accumulate, which silently ruins training)
  2. outputs = model(inputs) — forward pass
  3. loss = criterion(outputs, targets) — measure the error
  4. loss.backward() — compute gradients
  5. optimizer.step() — update weights

At inference wrap the forward pass in with torch.no_grad(): and call model.eval(). Skipping no_grad wastes memory tracking gradients you never use; skipping eval() leaves dropout and batch-norm in training mode and gives wrong predictions.

Gradient descent, demystified

Training is: guess parameters, measure the error, nudge parameters in the direction that reduces the error, repeat.

The gradient is the slope of the loss with respect to each parameter. The learning rate controls the step size — too large and the loss diverges, too small and training crawls.

PyTorch is not available in this browser sandbox, so the runnable example below implements gradient descent in pure NumPy. The mechanics are identical; PyTorch just computes the derivatives for you.

Copy-paste examples

Copy into your own editor, or load one into the compiler below and press Run.

Gradient descent in NumPy

Runs here. Fits y = 3x + 2 from scratch — this is what .backward() automates.

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, size=(200, 1)).astype(np.float32)
y = 3.0 * x + 2.0 + rng.normal(0, 0.05, size=(200, 1)).astype(np.float32)

w = np.zeros((1, 1), dtype=np.float32)
b = np.zeros((1,), dtype=np.float32)
lr = 0.5

for epoch in range(1, 61):
    pred = x @ w + b
    error = pred - y
    loss = float((error ** 2).mean())

    grad_w = 2.0 * (x.T @ error) / len(x)
    grad_b = 2.0 * error.mean(axis=0)

    w -= lr * grad_w
    b -= lr * grad_b

    if epoch % 15 == 0:
        print(f"epoch {epoch:3d} loss={loss:.5f} w={w[0, 0]:.3f} b={b[0]:.3f}")

print("learned: y =", round(float(w[0, 0]), 2), "* x +", round(float(b[0]), 2))

The PyTorch equivalent (run locally)

Same maths, autograd handles the derivatives. pip install torch.

# pip install torch
import torch
import torch.nn as nn

torch.manual_seed(0)
x = torch.rand(200, 1) * 2 - 1
y = 3.0 * x + 2.0 + torch.randn(200, 1) * 0.05

model = nn.Linear(1, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)

for epoch in range(1, 61):
    optimizer.zero_grad()
    pred = model(x)
    loss = criterion(pred, y)
    loss.backward()
    optimizer.step()
    if epoch % 15 == 0:
        print(f"epoch {epoch:3d} loss={loss.item():.5f}")

print("weight:", model.weight.item(), "bias:", model.bias.item())

model.eval()
with torch.no_grad():
    print("prediction at x=0.5:", model(torch.tensor([[0.5]])).item())

A model class and a device-safe loop (run locally)

nn.Module subclass plus the .to(device) pattern that avoids device mismatch errors.

# pip install torch
import torch
import torch.nn as nn

class Classifier(nn.Module):
    def __init__(self, in_features=768, hidden=256, n_classes=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_features, hidden),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(hidden, n_classes),
        )

    def forward(self, x):
        return self.net(x)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Classifier().to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
criterion = nn.CrossEntropyLoss()

def train_one_epoch(loader):
    model.train()
    total = 0.0
    for inputs, targets in loader:
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()
        loss = criterion(model(inputs), targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        total += loss.item()
    return total / max(1, len(loader))

Tune the learning rate

Try it — in-browser Python

Packages: numpy

Set lr to 2.5 and watch the loss diverge; set it to 0.01 and watch it crawl.

Output

Python runs in your browser. First run downloads the runtime.

Press Run (or Ctrl+Enter) to execute.

CPython in WebAssembly. Stdlib works. NumPy and pandas load on demand. No input(), no GPU, no network installs.

Takeaways

  • A tensor is a NumPy array plus device placement and autograd.
  • zero_grad, forward, loss, backward, step — the five lines of every training loop.
  • Use model.eval() and torch.no_grad() for inference.