Training a model gets most of the attention. Deploying it is the part that matters. A model that lives in a notebook is a science project; a model in production, serving real requests and updating as the world changes, is the point.
The gap between the two is larger than most tutorials admit.
The deployment landscape
Before choosing an approach, be clear about what you’re deploying:
- Latency: real-time inference (< 100ms), batch predictions (minutes to hours), or offline scoring (daily)
- Throughput: requests per second at peak load
- Model size: a BERT-base is 400MB, and a large vision model might be 10GB
- Hardware: CPU inference is cheaper, GPU inference is faster for large models
- Update frequency: how often you retrain and redeploy
The answers determine your architecture before you write a line of deployment code.
Serving approaches
A REST API with FastAPI is the right starting point for most use cases:
from fastapi import FastAPI
from pydantic import BaseModel
import joblib
app = FastAPI()
model = joblib.load("model.pkl")
class PredictRequest(BaseModel):
features: list[float]
@app.post("/predict")
def predict(req: PredictRequest):
prediction = model.predict([req.features])[0]
return {"prediction": float(prediction)}Add Docker, expose a port, and you have a deployable service. This works for sklearn models, XGBoost, LightGBM, and anything that fits in memory and runs on CPU.
For deep learning models, TorchServe and TF Serving handle batching, GPU memory management, and model versioning, which a generic web framework doesn’t do well.
Serverless (AWS Lambda, Google Cloud Functions) works for models that fit within the memory and cold-start limits, typically under 512MB and no GPU. It suits low-traffic, bursty workloads where you don’t want to pay for idle compute.
ONNX Runtime is worth a mention. Exporting your PyTorch or TensorFlow model to ONNX lets you run inference with a single dependency that optimises for the target hardware, often 2 to 4x faster than the native framework on CPU.
Packaging and reproducibility
The model artifact alone isn’t enough. You need:
- The model weights or pickle
- The exact version of every library used for training and inference
- Preprocessing code: the same transformations used at training time, applied at inference time
- A feature schema: which inputs are expected, and their types and ranges
Use mlflow or DVC to track experiments and associate artifacts with the code and data that produced them. A model you can’t reproduce is a liability.
Docker solves the “works on my machine” problem:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY model.pkl ./
COPY app.py ./
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]The training-serving skew problem
This is the most common cause of production model failures. Training-serving skew happens when the data your model sees in production differs from the data it trained on, usually because preprocessing is implemented in two different places.
The fix is to implement preprocessing once, in a form that runs identically at training and inference time. A sklearn Pipeline that includes both preprocessing and the model is one pattern; a shared library imported by both the training and serving code is another. Either way, the rule is one implementation used in both places.
Monitoring
Once deployed, a model needs monitoring at two levels.
Infrastructure monitoring is standard DevOps: request latency (p50, p95, p99), error rate, CPU/memory/GPU utilisation, and throughput.
Model monitoring is specific to ML:
- Input distribution drift: are the features arriving at inference time still distributed like the training data? Large shifts mean the world has changed and the model may be stale.
- Prediction distribution drift: is the model producing noticeably different predictions than before? That can signal drift or a bug.
- Ground truth comparison: where you can collect labels (click-through rate, fraud confirmation, diagnosis outcome), compare predictions against actual outcomes to measure real-world performance.
Evidently, WhyLogs, and Arize are commonly used for ML-specific monitoring.
Model versioning and rollout
Never deploy a new model to 100% of traffic at once. Use a canary deployment:
- Deploy v2 alongside v1
- Route 5% of traffic to v2
- Monitor metrics for 24 to 48 hours
- Increase to 20%, 50%, then 100% if metrics stay stable
- Decommission v1
This requires your serving infrastructure to support traffic splitting, which most Kubernetes setups and managed services (SageMaker, Vertex AI, Azure ML) provide out of the box.
Version your models with semantic versioning and keep a registry (MLflow Model Registry, Weights & Biases, or even a simple database table). You should be able to answer “which model version served this prediction?” at any time.
Retraining pipelines
Models decay. The world changes, user behaviour shifts, and the training data stops being representative. Build retraining into the system from the start:
- Scheduled retraining: retrain weekly or monthly on a rolling window of recent data
- Triggered retraining: retrain when drift monitoring detects a significant distribution shift
- Online learning: update model weights continuously as new data arrives, which is complex and often overkill
The retraining pipeline should be automated, tested, and fast enough to deploy a new version within hours of a trigger. A model you can’t update quickly is a liability when something goes wrong.
The checklist
Before promoting a model to production:
- Preprocessing is implemented once, shared between training and serving
- Model artifact and code are versioned together
- Inference latency meets SLA at expected load (load test first)
- Monitoring is set up for both infrastructure and model metrics
- Rollback procedure is documented and tested
- Retraining pipeline exists and is automated
- Model card documents performance across demographic subgroups
The gap between a notebook model and a production system is real work, but it’s tractable, and the patterns above cover most of the deployments you’ll run into.