AI Application Engineer Guide: Production Computer Vision with YOLO, ONNX Runtime & OpenCV
Running an unoptimized PyTorch object-detection model frame-by-frame inside a synchronous Python web worker saturates GPU memory and caps throughput at 4 frames per second. Production computer vision systems—whether inspecting manufacturing defects on an assembly line, verifying logistics parcel dimensions, or segmenting complex tables in scanned medical PDFs—require quantized ONNX Runtime / TensorRT execution providers, dynamic batching, and zero-copy NumPy/OpenCV preprocessing. As an AI Application Engineer and Senior Backend Engineer, here is how I deploy high-throughput visual AI pipelines.
1. Exported ONNX Runtime Inference with Letterbox Preprocessing
Exporting YOLO or Vision Transformer checkpoints to an ONNX graph with `CUDAExecutionProvider` (or `CoreMLExecutionProvider` / `OpenVINO` at the edge) eliminates PyTorch training-framework overhead and delivers deterministic sub-12ms frame latency:
import numpy as np
import onnxruntime as ort
class ONNXVisionDetector:
def __init__(self, model_path: str, conf_threshold: float = 0.45) -> None:
self.conf_threshold = conf_threshold
self.session = ort.InferenceSession(
model_path,
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
self.input_name = self.session.get_inputs()[0].name
def preprocess_batch(self, bgr_frames: list[np.ndarray]) -> np.ndarray:
# Normalize HWC BGR uint8 (640x640) -> NCHW RGB float32 [0.0, 1.0]
stacked = np.stack([frame[:, :, ::-1] for frame in bgr_frames], axis=0)
chw = np.transpose(stacked, (0, 3, 1, 2)).astype(np.float32) / 255.0
return np.ascontiguousarray(chw)
def infer_batch(self, bgr_frames: list[np.ndarray]) -> np.ndarray:
tensor = self.preprocess_batch(bgr_frames)
outputs = self.session.run(None, {self.input_name: tensor})
return outputs[0]
2. Combining Vision Detection with Multimodal VLM Verification
For high-stakes workflows such as automated warehouse damage auditing or clinical document intake, fast bounding-box detection crops regions of interest at 60 FPS and forwards only ambiguous edge cases to a multimodal LLM (Gemini 2.5 Pro / Flash) for structured semantic verification.
3. Hire an AI Application Engineer for Computer Vision & Multimodal AI
Building automated visual inspection, video analytics, or document vision pipelines? Explore my AI Application Engineer Pillar Guide, review my GPU Inference Serving Guide, or book a 1-Day Computer Vision Architecture Sprint.