Advanced26 min

scikit-learn: classification

Logistic regression, decision trees, KNN, and the confusion matrix — plus why accuracy lies on imbalanced data.

Why this matters in AI / ML / GenAI

Classification covers spam detection, sentiment, intent routing, and fraud. The confusion matrix and the precision/recall trade-off are asked about in almost every ML interview, and getting them wrong in production means shipping a model that looks great and helps nobody.

The classifiers

Logistic regression — despite the name it classifies. Fast, interpretable coefficients, a strong baseline. Always try it first.
Decision tree — splits on feature thresholds, easy to visualise, but overfits badly unless you cap max_depth.
Random forest — many trees voting. Robust, strong on tabular data, less interpretable.
KNN — labels a point by its nearest neighbours. No training step; slow at prediction time; requires scaled features.

For tabular problems, start with logistic regression as a baseline, then try a random forest or gradient boosting. Reach for a neural network only when the data is text, images, or audio.

Distance-based models (KNN, SVM) and regularised linear models require feature scaling. Tree-based models do not care.

The confusion matrix

For binary classification with a positive class:

	Predicted negative	Predicted positive
Actually negative	True negative	False positive
Actually positive	False negative	True positive

From those four numbers:

Accuracy = correct / total. Misleading when classes are imbalanced.
Precision = TP / (TP + FP). Of everything flagged, how much was right? Matters when a false positive is expensive — blocking a legitimate transaction.
Recall = TP / (TP + FN). Of everything that mattered, how much did we catch? Matters when a false negative is expensive — missing a fraud or a tumour.
F1 = harmonic mean of precision and recall. One number when both matter.

The accuracy trap: with 99% negatives, a model that always predicts "negative" scores 99% accuracy and catches zero positives. Always look at the confusion matrix, never accuracy alone.

Thresholds and ROC AUC

Classifiers output a probability. The default cutoff of 0.5 is a choice, not a law. predict_proba gives the probability so you can pick your own.

Lower the threshold to catch more positives (higher recall, lower precision). Raise it to be more certain when you do flag something (higher precision, lower recall). Set it from the cost of each error type in your domain, not from convention.

ROC AUC summarises performance across every threshold: 0.5 is random, 1.0 is perfect. For heavily imbalanced data, precision-recall AUC is more informative than ROC AUC.

classification_report prints precision, recall, and F1 per class and is the fastest way to see what a model is actually doing.

Copy-paste examples

Copy into your own editor, or load one into the compiler below and press Run.

Logistic regression with a full report

classification_report is the first thing to print after training.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score

rng = np.random.default_rng(42)
n = 600
length = rng.normal(50, 20, n)
links = rng.integers(0, 6, n).astype(float)
score = -4 + 0.05 * length + 0.9 * links + rng.normal(0, 1, n)
is_spam = (score > 0).astype(int)

X = np.column_stack([length, links])
X_train, X_test, y_train, y_test = train_test_split(X, is_spam, test_size=0.25, random_state=42, stratify=is_spam)

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
pred = model.predict(X_test)

print("accuracy:", round(accuracy_score(y_test, pred), 4))
print("\nconfusion matrix [[TN FP] [FN TP]]:")
print(confusion_matrix(y_test, pred))
print("\n" + classification_report(y_test, pred, target_names=["ham", "spam"]))

The accuracy trap on imbalanced data

99% accuracy while catching zero fraud cases. Run it.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, recall_score, precision_score, confusion_matrix

rng = np.random.default_rng(0)
n = 2000
X = rng.normal(0, 1, (n, 3))
y = np.zeros(n, dtype=int)
fraud_idx = rng.choice(n, size=20, replace=False)      # 1% positives
y[fraud_idx] = 1
X[fraud_idx] += 2.2

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1, stratify=y)

for name, clf in [
    ("always predicts 'not fraud'", DummyClassifier(strategy="most_frequent")),
    ("logistic regression", LogisticRegression()),
    ("logistic + class_weight", LogisticRegression(class_weight="balanced")),
]:
    clf.fit(X_train, y_train)
    pred = clf.predict(X_test)
    print(f"{name:30} accuracy={accuracy_score(y_test, pred):.3f} "
          f"recall={recall_score(y_test, pred, zero_division=0):.3f} "
          f"precision={precision_score(y_test, pred, zero_division=0):.3f}")

print("\nthe first model is 99% accurate and catches nothing")

Compare three classifiers

Same interface, different trade-offs. Note the tree overfitting without a depth cap.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import f1_score, accuracy_score

rng = np.random.default_rng(5)
n = 800
X = rng.normal(0, 1, (n, 4))
y = ((X[:, 0] + X[:, 1] ** 2 - X[:, 2]) > 1).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

models = {
    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression()),
    "decision tree (deep)": DecisionTreeClassifier(random_state=0),
    "decision tree (depth 4)": DecisionTreeClassifier(max_depth=4, random_state=0),
    "random forest": RandomForestClassifier(n_estimators=100, random_state=0),
    "knn (k=5)": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5)),
}

print(f"{'model':<26}{'train acc':>11}{'test acc':>10}{'test F1':>10}")
print("-" * 57)
for name, model in models.items():
    model.fit(X_train, y_train)
    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_pred = model.predict(X_test)
    print(f"{name:<26}{train_acc:>11.3f}{accuracy_score(y_test, test_pred):>10.3f}{f1_score(y_test, test_pred):>10.3f}")

Tuning the decision threshold

Precision and recall move in opposite directions — pick the point your domain needs.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score

rng = np.random.default_rng(2)
n = 1000
X = rng.normal(0, 1, (n, 2))
y = ((X[:, 0] + X[:, 1] + rng.normal(0, 0.6, n)) > 0.8).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = LogisticRegression().fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]

print("ROC AUC (threshold independent):", round(roc_auc_score(y_test, probs), 4))
print(f"\n{'threshold':>10}{'precision':>11}{'recall':>9}{'F1':>8}{'flagged':>9}")
for t in [0.2, 0.35, 0.5, 0.65, 0.8]:
    pred = (probs >= t).astype(int)
    print(f"{t:>10.2f}{precision_score(y_test, pred, zero_division=0):>11.3f}"
          f"{recall_score(y_test, pred, zero_division=0):>9.3f}"
          f"{f1_score(y_test, pred, zero_division=0):>8.3f}{pred.sum():>9}")

print("\nlow threshold = catch more, be wrong more often")

Plot the confusion matrix

A labelled heatmap is easier to read in a report than raw numbers.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

rng = np.random.default_rng(42)
n = 700
X = rng.normal(0, 1, (n, 2))
y = ((X[:, 0] + X[:, 1] + rng.normal(0, 0.7, n)) > 0.5).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
pred = LogisticRegression().fit(X_train, y_train).predict(X_test)
cm = confusion_matrix(y_test, pred)

fig, ax = plt.subplots(figsize=(4.5, 4))
ax.imshow(cm, cmap="Blues")
labels = ["negative", "positive"]
ax.set_xticks([0, 1], labels=[f"predicted\n{l}" for l in labels])
ax.set_yticks([0, 1], labels=[f"actual\n{l}" for l in labels])
for i in range(2):
    for j in range(2):
        ax.text(j, i, cm[i, j], ha="center", va="center", fontsize=16,
                color="white" if cm[i, j] > cm.max() / 2 else "#0f172a", fontweight="bold")
ax.set_title("Confusion matrix")

tn, fp, fn, tp = cm.ravel()
print(f"true negatives {tn} | false positives {fp}")
print(f"false negatives {fn} | true positives {tp}")
print(f"\nprecision {tp / (tp + fp):.3f}  recall {tp / (tp + fn):.3f}")

Pick a threshold from business cost

Try it — in-browser Python

Packages: numpy, scikit-learn, matplotlib

Change the cost of a missed fraud to 5000 and see the optimal threshold drop.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

COST_FALSE_POSITIVE = 20     # annoying a good customer
COST_FALSE_NEGATIVE = 500    # a fraud we missed

rng = np.random.default_rng(3)
n = 1500
X = rng.normal(0, 1, (n, 3))
y = ((X[:, 0] * 1.5 + X[:, 1] + rng.normal(0, 0.8, n)) > 1.6).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
probs = LogisticRegression().fit(X_train, y_train).predict_proba(X_test)[:, 1]

print(f"positives in test set: {y_test.sum()} of {len(y_test)}")
print(f"\n{'threshold':>10}{'FP':>6}{'FN':>6}{'total cost':>13}")

best = (None, float("inf"))
for t in np.arange(0.1, 0.95, 0.05):
    tn, fp, fn, tp = confusion_matrix(y_test, (probs >= t).astype(int)).ravel()
    cost = fp * COST_FALSE_POSITIVE + fn * COST_FALSE_NEGATIVE
    if cost < best[1]:
        best = (round(float(t), 2), cost)
    print(f"{t:>10.2f}{fp:>6}{fn:>6}{cost:>13,}")

print(f"\ncheapest threshold: {best[0]} at a cost of {best[1]:,}")

Output

Python runs in your browser. First run downloads the runtime.

Press Run (or Ctrl+Enter) to execute.

CPython in WebAssembly. Stdlib works. NumPy, pandas, scikit-learn and Matplotlib load on demand, and charts render below. No input(), no GPU, no network installs.

Takeaways

Accuracy hides failure on imbalanced data — always read the confusion matrix.
Precision matters when false positives cost; recall matters when false negatives cost.
The 0.5 threshold is a choice: use predict_proba and set it from real business cost.