Foundations of Machine Learning: Vectorized Gradient Descent, Regularization & Neural Nets

Understanding the mathematical mechanics beneath modern machine learning frameworks—vectorized matrix calculus, L2 weight decay, bias-variance diagnostics, and backpropagation—separates engineers who can debug model convergence failures from those who treat ML libraries as black boxes.

1. Vectorized Cost & Gradient Computation

Replacing explicit for-loops over training examples with vectorized matrix multiplications (X @ theta) leverages SIMD and BLAS hardware instructions, reducing iteration latency by two orders of magnitude:

import numpy as np

def logistic_loss_and_grad(X: np.ndarray, y: np.ndarray, w: np.ndarray, l2_lambda: float):
    m = X.shape[0]
    logits = X @ w
    probs = 1.0 / (1.0 + np.exp(-np.clip(logits, -250, 250)))
    eps = 1e-12
    cost = -(1.0 / m) * np.sum(y * np.log(probs + eps) + (1 - y) * np.log(1 - probs + eps))
    reg_cost = (l2_lambda / (2.0 * m)) * np.sum(w[1:] ** 2)

    grad = (1.0 / m) * (X.T @ (probs - y))
    grad[1:] += (l2_lambda / m) * w[1:]
    return cost + reg_cost, grad

2. Diagnosing High Bias vs. High Variance in Production

• High Bias (Underfitting): Training and validation errors plateau at a high level; remediate by expanding feature representations, increasing model capacity, or reducing regularization strength.
• High Variance (Overfitting): Training error approaches zero while validation error diverges; remediate via data augmentation, L2 regularization, dropout, or early stopping.