How to optimize real-time video stream processing pipelines with YOLO and OpenCV?

Use a GPU‑accelerated pipeline that decodes frames with cv2.VideoCapture, batches them for YOLOv8 inference via ONNX Runtime, and post‑processes detections with OpenCV’s vectorized operations.

1. Capture – Open the video source with cv2.VideoCapture(source, cv2.CAP_FFMPEG) and set CAP_PROP_BUFFERSIZE=2 to limit latency. Read frames in a dedicated thread and push them into a queue.Queue(maxsize=4).

import cv2, queue, threading

frame_q = queue.Queue(maxsize=4)
def producer(src):
    cap = cv2.VideoCapture(src, cv2.CAP_FFMPEG)
    cap.set(cv2.CAP_PROP_BUFFERSIZE, 2)
    while True:
        ret, frm = cap.read()
        if not ret: break
        frame_q.put(frm)
threading.Thread(target=producer, args=('rtsp://camera/stream',), daemon=True).start()

2. Pre‑process – Resize each frame to the model input (e.g., 640×640) using cv2.resize(..., interpolation=cv2.INTER_LINEAR), convert BGR→RGB, normalize to [0,1], and stack into a NumPy batch of size N (typically 4–8).

3. Inference – Load the exported YOLOv8 ONNX model with ort.InferenceSession('yolov8n.onnx', providers=['CUDAExecutionProvider']). Call session.run(None, {'images': batch}) and obtain raw boxes, scores, and class IDs.

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession('yolov8n.onnx',
                               providers=['CUDAExecutionProvider'])
def infer(batch):
    return session.run(None, {'images': batch.astype(np.float32)})[0]

4. Post‑process – Apply per‑class NMS with cv2.dnn.NMSBoxes using score_thr=0.25 and nms_thr=0.45. Vectorize mask creation to avoid Python loops.

def nms(boxes, scores, iou_thr=0.45, score_thr=0.25):
    idxs = cv2.dnn.NMSBoxes(boxes.tolist(), scores.tolist(),
                           score_thr, iou_thr)
    return idxs

5. Render – Draw boxes with cv2.rectangle and labels with cv2.putText. Blend detections using cv2.addWeighted(frame, 0.6, overlay, 0.4, 0) to keep latency low.

6. Output – Encode the annotated frame with cv2.imencode('.jpg', frame, [int(cv2.IMWRITE_JPEG_QUALITY), 85]) and pipe to an RTMP server via ffmpeg.

Model vs. latency (1080p, batch 4, RTX 4090)

| Model | Params (M) | FPS |
|-------|------------|-----|
| YOLOv8n | 3.2 | ~120 |
| YOLOv7‑tiny | 4.0 | ~105 |
| YOLOv5s | 7.2 | ~95 |

Gotcha: if the capture thread and the ONNX session share the same CUDA stream, the GPU may stall and frames will be dropped; assign each thread its own stream or use a thread‑safe queue to decouple them.