CSE445Computer Vision
Tutorials/Object Detection/OD 9A: Evaluation, Error Analysis & Deployment with YOLO
Previous tutorialOD 8B: RF-DETR Transformer DetectionNext tutorialOD 9B: Evaluation, Error Analysis & Deployment with RF-DETR
OD-09AIntermediate 90 min

OD 9A: Evaluation, Error Analysis & Deployment with YOLO

Run standardised COCO evaluation, diagnose failure modes with per-category and per-size breakdowns, draw Precision-Recall and F1 curves, and package the model for edge deployment.

pycocotoolsPyTorchMatplotlibONNX Runtime

Code Structure Note

These notes are provided only to help students understand the code structure. Students should review, optimize, and adapt the code before using it in assignments, projects, or experiments.

Topics Covered
  1. 1COCO API: pycocotools eval, mAP@50, mAP@50-95, APs/APm/APl
  2. 2Per-category AP breakdown: identifying weak classes
  3. 3Error analysis taxonomy: missed detections, false positives, localisation errors
  4. 4Confidence calibration: F1-confidence curve and optimal threshold selection
  5. 5Deployment pipeline: ONNX export, TensorRT, edge benchmarking on Jetson/RPi

Learning Objectives

  • Run pycocotools COCO evaluation on a detector results JSON and interpret every metric row
  • Identify the three weakest-performing categories and hypothesise causes from dataset statistics
  • Plot the F1-confidence curve and select the operating threshold for a precision-constrained task
  • Export a model to ONNX and measure end-to-end latency (resize → infer → NMS) on CPU

Key Takeaways

  • APs < 30 for most detectors — small objects are consistently the hardest regime
  • Per-category AP breakdown reveals dataset imbalance, annotation noise, and missing augmentations
  • Pre/post-processing often exceeds model forward-pass time on CPU — profile the full pipeline
  • ONNX export decouples the model from PyTorch and unlocks TensorRT, CoreML, and ONNX Runtime

Tutorial Infographic

Notebook Resources

Jupyter Notebook

Student action area

Use this section to practice the tutorial code. Kaggle opens the prepared notebook, Colab lets you experiment in your own runtime, and the .ipynb download is best for local editing or submission.

Open in Kaggle Open in Colab Download .ipynb
Kaggle: follow the official tutorial notebook.
Colab: modify and rerun quickly.
IPYNB: keep a local copy.

Hands-On Exercises

  1. 1Run YOLOv8n, YOLOv8s, and YOLOv8m on COCO val2017 and compare their full eval tables
  2. 2Find the 5 categories with the lowest AP and visualise 3 failure examples per category
  3. 3Plot the F1-confidence curve for YOLOv8s and mark the threshold that maximises F1
  4. 4Export YOLOv8s to ONNX and benchmark latency vs. native PyTorch on both CPU and GPU

Tools & Libraries

pycocotoolsPyTorchMatplotlibONNX Runtime
Previous tutorialOD 8B: RF-DETR Transformer DetectionNext tutorialOD 9B: Evaluation, Error Analysis & Deployment with RF-DETR