Deep Learning with PyTorch: Custom Neural Networks, Transfer Learning & GPU Training

Building production computer vision and natural language classifiers in PyTorch requires disciplined tensor pipeline design, mixed-precision GPU training, and deterministic validation loops. This guide summarizes core architectural patterns from training multi-class image and sequence models in PyTorch.

1. Transfer Learning & Classifier Head Replacement

Rather than training convolutional or Vision Transformer backbones from scratch on small domain datasets, freeze the pre-trained feature extractor layers and replace the final classification head with a custom nn.Sequential block:

import torch
import torch.nn as nn
from torchvision import models

def build_fine_tuned_resnet(num_classes: int) -> nn.Module:
    model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
    for param in model.parameters():
        param.requires_grad = False

    in_features = model.fc.in_features
    model.fc = nn.Sequential(
        nn.Linear(in_features, 512),
        nn.LayerNorm(512),
        nn.GELU(),
        nn.Dropout(p=0.3),
        nn.Linear(512, num_classes),
    )
    return model

2. Mixed-Precision Training (`torch.amp`) & Cosine Annealing

• Automatic Mixed Precision (AMP): Wrapping forward passes in torch.cuda.amp.autocast() with GradScaler halves VRAM consumption and accelerates throughput on modern NVIDIA Tensor Core GPUs.
• Differential Learning Rates: After warming up the classification head, unfreeze the top backbone blocks with a 10x lower learning rate governed by CosineAnnealingWarmRestarts.