AWS ECR & Multi-Model SageMaker Endpoints: Deploying Deep Learning Predictors

Running a dedicated GPU or CPU instance for every microservice machine learning model quickly burns thousands of dollars per month in idle cloud compute. By combining a single custom Amazon ECR Docker image with AWS SageMaker Multi-Model Endpoints (MME) or multi-service inference workers, you can serve dozens of PyTorch/ONNX models from a shared fleet.

1. Designing a Unified Inference Container for Amazon ECR

SageMaker invokes containers using two standard HTTP endpoints on port 8080:

• GET /ping — Health check verifying that model weights and shared memory workers are ready.
• POST /invocations — Executes inference on the request payload, routing to the target sub-model via header or JSON envelope.

2. Dynamic Model Loading & Memory LRU Caching

When hosting multiple specialized classifiers or embedding models behind a single ECR image:

1. Store serialized TorchScript or quantized ONNX model artifacts in versioned S3 prefixes (s3://ml-artifacts/models/v2/).
2. Pre-warm high-traffic models during container startup and lazy-load long-tail tenant models into an LRU memory cache guarded by a thread-safe lock.
3. Use uvicorn / gunicorn with shared memory tensors so multiple worker processes do not duplicate model weights in RAM.

3. Automated CI/CD Push to Amazon ECR

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com
docker build -t sagemaker-multi-predictor:latest .
docker tag sagemaker-multi-predictor:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/sagemaker-multi-predictor:latest
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/sagemaker-multi-predictor:latest