Tutorials/Object Tracking

CSE445 · Temporal computer vision

Object Tracking and Multi-Object Tracking

Learn how computer-vision systems follow objects across video frames using motion estimation, state prediction, data association, appearance matching, identity management, and tracking evaluation.

This four-part tutorial series develops the foundations of single-object and multi-object tracking. Students learn how detections are connected across time, how object states are predicted, how identities are preserved through occlusion, and how tracking systems are evaluated. YOLO26 is used as the object detector in practical demonstrations, while the tracking concepts remain applicable to any detector or tracking framework.

4 TutorialsBeginner–Intermediate to AdvancedApproximately 9–11 HoursObject TrackingMulti-Object TrackingMotion EstimationData AssociationRe-IdentificationPython, OpenCV, and PyTorchYOLO26 Demonstrations

Mathematical core

Kalman Filtering and Hungarian Assignment

Two algorithms carry almost every tracking-by-detection system: a Kalman filter predicts where each track should appear, and the Hungarian algorithm decides which detection belongs to which track. Both labs below run the real algorithms live — press Play, scrub frames, drag boxes, and change the noise to see how the numbers respond.

Interactive tracking lab

Kalman Filter: Predict → Measure → Update

A constant-velocity Kalman filter follows a curving object through noisy detections and a full detector dropout. Every number below comes from the live filter — adjust R and Q to change how much it trusts the detector. Use Step one phase to walk predict → measure → update by hand: each phase draws the one vector it is responsible for, so the gain stops being a coefficient and becomes a distance.

detail×5

ν 0.19 · K 0.63 · Kν 0.12

k = 32/32 · update: x̂ = x̂⁻ + K (z − H x̂⁻)

Frame scrubberk = 32Measurement noise σ (R = σ²I)0.28Process noise q (scales Q)0.020Animation speed1.00×

Uncertainty, every frame

click to scrub

One tooth per frame: predict pushes σ up by Q, a fused detection pulls it back down. Across the shaded dropout there is nothing to pull it down, so the teeth become a staircase — and that rising σ is exactly what a tracker gates matches against when the object reappears.

Raise σ and the filter distrusts detections: the gain drops, the green Kν arrow becomes a stub next to the amber innovation, and the estimate smooths through noise but lags the curve. Raise q and it distrusts its own model: the gain rises, Kν covers nearly the whole innovation, the estimate hugs detections, and the dropout hurts more.

x̂ₖ = x̂ₖ⁻ + Kₖ(zₖ − Hx̂ₖ⁻)

Detection fused

frame k: 32 / 32
x̂ position: (4.71, -1.48)
x̂ velocity: (0.33, -0.24)
gain K₁₁: 0.630
innovation ‖z − Hx̂⁻‖: 0.189
σₓ predicted → updated: 0.366 → 0.222

Update walks a fraction K = 0.63 of the way along ν and stops. Everything past the green arrow is the correction the filter deliberately declined to make.

truth×detectionsprediction + 2σestimate

The recursion behind the lab

Step 1 · Predict

\hat{\mathbf{x}}_k^- = F\,\hat{\mathbf{x}}_{k-1} \qquad P_k^- = F P_{k-1} F^\top + Q

The constant-velocity model projects the previous estimate one frame forward. Uncertainty grows by the process noise Q.

Step 2 · Measure

\mathbf{z}_k = H\,\mathbf{x}_k + \mathbf{v}_k, \qquad \mathbf{v}_k \sim \mathcal{N}(0, R)

The detector supplies a noisy observation of the true state, with measurement noise covariance R.

Step 3 · Update

K_k = P_k^- H^\top \bigl(H P_k^- H^\top + R\bigr)^{-1} \qquad \hat{\mathbf{x}}_k = \hat{\mathbf{x}}_k^- + K_k\bigl(\mathbf{z}_k - H\hat{\mathbf{x}}_k^-\bigr)

The Kalman gain blends prediction and measurement in proportion to their uncertainties; the corrected estimate lands between them.

Constant-velocity state model

\mathbf{x} = \begin{bmatrix} u \\ v \\ \dot{u} \\ \dot{v} \end{bmatrix}, \qquad F = \begin{bmatrix} 1 & 0 & \Delta t & 0 \\ 0 & 1 & 0 & \Delta t \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}

This is exactly the state the lab above estimates; SORT extends it with box scale and aspect ratio, but the recursion is identical.

Worked 1-D example

$u_k^- = 42 + 6 = 48,\;\; P_k^- = 4 + 5 = 9,\;\; K = \tfrac{9}{9 + 6} = 0.6$

$u_k = 48 + 0.6\,(51 - 48) = 49.8,\;\; P_k = (1 - 0.6) \cdot 9 = 3.6$

Interactive tracking lab

Hungarian Assignment: Detections → Tracks

Dashed blue boxes are Kalman predictions; amber boxes are detections — drag them anywhere. The cost matrix, greedy baseline, and Hungarian optimum are re-solved live on every change. Run solver then walks the Kuhn–Munkres tableau one matrix operation at a time, so the optimum stops being a number the page asserts and becomes something you watch being built.

Greedy pays 3.40, Hungarian pays 3.10 — greedy wastes +0.29

Animation speed1.00×

The solver minimizes Σᵢⱼ cᵢⱼ xᵢⱼ over permutation matrices — each track keeps at most one detection and vice versa. Greedy commits to the cheapest pair first — the #1…#4 badges on its edges are the order it commits, and the damage always shows up in the last one, where a track is left with whatever nobody else wanted. The Hungarian algorithm (Kuhn–Munkres, O(n³)) never commits early: it rewrites the matrix until enough zeros exist to assign everything at once. That is what SORT, ByteTrack, and DeepSORT run every frame.

Reading the tape. Emerald cells are zeros — candidate pairs that cost nothing beyond the baselines already removed. Sky cells are covered by a line. A step that covers every zero with fewer than four lines is the algorithm reporting that the zeros are too bunched to assign, so it shifts the smallest uncovered value out of the uncovered block and back in at the crossings, which is guaranteed to create a new zero without changing which assignment is cheapest.

Kalman predictiondetection (drag me)Hungarian matchgreedy match

cᵢⱼ = 0.55(1 − IoU) + 0.45·d̄ᵢⱼ

Hungarian saves 0.29

1/6

The cost matrix

D1D2D3D4T10.640.881.001.00T20.651.000.991.00T31.001.000.801.00T41.001.000.750.77

Every prediction paired with every detection: 4×4 numbers, and the job is to pick 4 of them, one per row and one per column, with the smallest total.

Greedy total: 3.40Hungarian total: 3.10

Assignment objective

X^{*} = \arg\min_{X \in \mathcal{P}_n} \sum_{i=1}^{n} \sum_{j=1}^{n} C_{ij}\, X_{ij}, \qquad X_{ij} \in \{0, 1\},\;\; \textstyle\sum_i X_{ij} = \sum_j X_{ij} = 1

X ranges over permutation matrices: each track receives at most one detection and vice versa. Pairs whose cost exceeds a gating threshold are removed before solving, so unlikely matches can never be forced. The lab above solves this exact problem live with Kuhn–Munkres in O(n³).

IoU cost (SORT, ByteTrack)

\mathrm{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|}, \qquad C_{ij} = 1 - \mathrm{IoU}(\hat{b}_i, d_j)

Pure geometric overlap between the predicted box and the detection. Fast, and sufficient when frame rate is high and motion is smooth.

Mahalanobis gate (DeepSORT)

d^{2}_{ij} = (\mathbf{z}_j - H\hat{\mathbf{x}}_i)^\top S_i^{-1} (\mathbf{z}_j - H\hat{\mathbf{x}}_i)

Distance measured in units of predicted uncertainty S. Matches are rejected when d² exceeds the χ² threshold 9.4877 (95%, 4 DOF).

Appearance fusion (DeepSORT, BoT-SORT)

d^{\text{app}}_{ij} = 1 - \frac{\mathbf{f}_i^\top \mathbf{f}_j}{\lVert \mathbf{f}_i \rVert\, \lVert \mathbf{f}_j \rVert}, \qquad c_{ij} = \lambda\, d^{\text{mot}}_{ij} + (1 - \lambda)\, d^{\text{app}}_{ij}

Cosine distance between ReID embeddings, blended with the motion cost so identity survives occlusion and missed frames.

Learning outcomes

What Students Will Learn

Ten measurable outcomes spanning motion modelling, data association, identity management, and evaluation.

Foundations and association

Outcomes 1–5

1Explain the difference between object detection, single-object tracking, and multi-object tracking.
2Represent an object using position, velocity, size, and identity state.
3Predict object motion using constant-velocity models and Kalman filtering.
4Build detection-to-track cost matrices using IoU, distance, and appearance similarity.
5Apply greedy matching and Hungarian assignment.

Identity, evaluation, and applications

Outcomes 6–10

6Explain track creation, confirmation, loss, recovery, and termination.
7Compare SORT, ByteTrack, DeepSORT, BoT-SORT, and OC-SORT.
8Understand identity switches, fragmentation, occlusion, and duplicate tracks.
9Evaluate trackers using MOTA, IDF1, HOTA, and runtime metrics.
10Build counting, trajectory, dwell-time, and speed-analysis applications.

Four-part notebook sequence

Student Tracking Notebooks

Complete the notebooks in order. Each one clearly separates what to learn, what to build, what to submit, and the full reference syllabus.

NOTEBOOK 01Beginner–Intermediate2 Hours

Tracking Fundamentals

Foundations of Object Tracking and Motion Prediction

Introduce object tracking as the estimation of an object’s state over time. Students learn how positions, velocities, bounding boxes, trajectories, and uncertainty are represented. The tutorial develops motion prediction from frame differencing and optical flow to constant-velocity models and Kalman filtering.

PythonNumPyOpenCVFilterPyMatplotlib

Notebook workflow

Video framesYOLO26 observationsState predictionMeasurement updateCorrected trackTrajectory

Learn

Learning objectives

Differentiate detection, single-object tracking, and multi-object tracking
Represent position, size, velocity, identity, and uncertainty as track state
Explain prediction, observation, innovation, correction, and covariance
Apply a Kalman filter across missed and noisy observations

Build

Notebook activities

Detect one object in a video and extract its center position
Estimate velocity and predict the next position
Apply a Kalman filter to noisy observations
Simulate missed detections and compare raw with filtered trajectories

Submit

Expected outcome

Students create a motion-prediction notebook that visualizes noisy detections, Kalman predictions, corrected states, uncertainty, and object trajectories.

View full notebook syllabusConcepts, algorithms, mathematics, and practice

Core concepts

Video as a temporal signal
Frames and frame rate
Object state
Bounding-box center, width, and height
Velocity and acceleration
Trajectory and observation
State prediction and measurement update
Process and measurement noise
Uncertainty
Track initialization and lifecycle
Single-object tracking
Multi-object tracking

Key algorithms

Frame differencing
Background subtraction
Sparse and dense optical flow
Constant-position model
Constant-velocity model
Constant-acceleration model
Kalman filter prediction–update cycle

Mathematical topics

State vector: xₜ = [cₓ, cᵧ, w, h, vₓ, vᵧ]ᵀPrediction: x̂ₜ|ₜ₋₁ = F x̂ₜ₋₁|ₜ₋₁Covariance prediction: Pₜ|ₜ₋₁ = F Pₜ₋₁|ₜ₋₁ Fᵀ + QKalman gain and innovationCorrected state and covariance ellipse

Practice checklist

Visualize true, observed, predicted, and corrected positions
Adjust process noise Q and measurement noise R
Plot the covariance ellipse during an observation gap
Document how the track recovers after detections return

Student guidance: complete the notebook in sequence and keep the generated figures, metrics, and written observations.

Open Notebook

NOTEBOOK 02Intermediate2.5 Hours

Data Association

Detection-to-Track Association: From SORT to ByteTrack

Teach how detections from the current frame are assigned to existing tracks. Students construct similarity and cost matrices, apply gating, solve assignment problems, manage track lifecycles, and study the progression from SORT to ByteTrack.

PythonNumPySciPyFilterPyOpenCV

Notebook workflow

Predicted tracksCurrent detectionsCost matrixGatingHungarian assignmentTrack update

Learn

Learning objectives

Build IoU, distance, and motion-gated association costs
Compare greedy matching with Hungarian assignment
Compose detector, Kalman filter, assignment, and lifecycle logic into SORT
Explain how ByteTrack uses low-confidence observations for recovery

Build

Notebook activities

Run detection without tracking and inspect repeated observations
Construct a numerical cost matrix and solve it manually
Compare greedy, SORT-style, and ByteTrack association
Visualize unmatched detections, lost tracks, trajectories, and ID switches

Submit

Expected outcome

Students build a complete tracking-by-detection pipeline and understand how SORT and ByteTrack convert frame-level detections into persistent identities.

View full notebook syllabusConcepts, algorithms, mathematics, and practice

Core concepts

Tracking by detection
Existing tracks and current detections
Predicted bounding boxes
Intersection over Union
Center and Mahalanobis distance
Similarity and cost matrices
Gating
Matched and unmatched states
Track confirmation, deletion, and buffer

Key algorithms

Greedy matching
Hungarian assignment
SORT
ByteTrack two-stage matching
Confidence-aware recovery
Track lifecycle state machine

Mathematical topics

IoU(A, B) = |A ∩ B| / |A ∪ B|Association cost: Cᵢⱼ = 1 − IoU(Tᵢ, Dⱼ)Three-track by four-detection cost matrixGating threshold and invalid assignmentsOne-to-one minimum-cost assignment

Practice checklist

Change the association threshold and track buffer
Compare greedy and Hungarian assignment on the same matrix
Inspect ID switches during partial occlusion
Explain the role of persist=True in a manual video frame loop

Student guidance: complete the notebook in sequence and keep the generated figures, metrics, and written observations.

Open Notebook

NOTEBOOK 03Intermediate–Advanced3 Hours

Robust Multi-Object Tracking

Appearance, Occlusion, and Robust Identity Tracking

Extend motion-only tracking with appearance representations, re-identification, camera-motion compensation, and observation-centric motion modeling. Students compare DeepSORT, BoT-SORT, OC-SORT, and related robust tracking strategies.

PythonPyTorchOpenCVscikit-learnYOLO26 detections

Notebook workflow

Motion predictionAppearance embeddingFused associationOcclusion bufferRe-identificationPersistent identity

Learn

Learning objectives

Explain when motion-only association becomes ambiguous
Use appearance embeddings for re-identification and recovery
Compare DeepSORT, BoT-SORT, and OC-SORT conceptually
Analyze camera motion, occlusion, fragmentation, and duplicate identities

Build

Notebook activities

Compare motion-only and appearance-assisted association
Test static-camera, moving-camera, crowded, and abrupt-motion scenes
Enable or disable re-identification and camera compensation
Measure identity switches, fragmentation, recovery, duplicates, and runtime

Submit

Expected outcome

Students learn when motion-only tracking is insufficient and how appearance, camera compensation, and observation-centric updates improve identity preservation.

View full notebook syllabusConcepts, algorithms, mathematics, and practice

Core concepts

Appearance embedding and re-identification
Cosine similarity
Motion–appearance fusion
Occlusion and identity switch
Track fragmentation and lost-track recovery
Camera motion and homography
Global motion compensation
Observation-centric motion estimation
Duplicate-track suppression

Key algorithms

DeepSORT matching cascade
BoT-SORT motion and appearance association
Camera-motion compensation
OC-SORT observation-centric updates
Short-term and long-term re-association

Mathematical topics

Cosine similarity between appearance embeddingsMotion and appearance cost fusionHomography-based global motion estimationObserved motion = object motion + camera motionDirection-consistency constraints

Practice checklist

Visualize appearance embeddings before and after occlusion
Estimate camera motion from background features
Compare track recovery across multiple robust trackers
Write a failure analysis for one identity switch

Student guidance: complete the notebook in sequence and keep the generated figures, metrics, and written observations.

Open Notebook

NOTEBOOK 04Advanced / Research3 Hours

Evaluation and Applications

Tracking Evaluation and Intelligent Video Analytics

Evaluate tracking systems using detection, identity, association, trajectory, and runtime metrics. Students conduct controlled tracker comparisons and transform tracked trajectories into practical video-analytics applications.

Pythonpy-motmetricsTrackEvalPandasOpenCV

Notebook workflow

Ground truth + predictionsMOT event matchingMOTA · IDF1 · HOTARuntime analysisTrajectory exportVideo analytics

Learn

Learning objectives

Interpret MOTA, MOTP, IDF1, HOTA, FPS, and latency
Design a controlled comparison with fixed detector and input settings
Diagnose identity switches, fragmentation, and localization errors
Build counting, trajectory, dwell-time, and speed-analysis applications

Build

Notebook activities

Compare SORT-style, ByteTrack, appearance-assisted, BoT-SORT, and OC-SORT systems
Build a metrics table without fabricated values
Export identities and trajectories to CSV
Implement at least three temporal video-analytics applications

Submit

Expected outcome

Students produce a complete evaluated tracking system, a comparative tracker report, trajectory exports, and at least three video-analytics applications.

View full notebook syllabusConcepts, algorithms, mathematics, and practice

Core concepts

Ground-truth trajectories
False positives and false negatives
Identity switches and fragmentation
Mostly tracked and mostly lost
MOTA, MOTP, IDF1, and HOTA
Detection and association accuracy
Runtime, FPS, latency, and memory
Trajectory and event analytics

Key algorithms

MOT event matching
Controlled tracker comparison
Unique-object counting
Line-crossing detection
Region dwell-time analysis
Trajectory heatmaps and calibrated speed estimation

Mathematical topics

MOTA = 1 − (FN + FP + IDSW) / GTIDP, IDR, and harmonic-mean IDF1HOTA as balanced detection and association qualityPixel displacement versus calibrated world speedRuntime, throughput, and per-frame latency

Practice checklist

Mark false positives, false negatives, and identity switches on a short sequence
Compute a small MOTA and IDF1 example by hand
Build a line-crossing counter with duplicate-event prevention
Write a research-style tracker recommendation grounded in metrics and scene conditions

Student guidance: complete the notebook in sequence and keep the generated figures, metrics, and written observations.

Open Notebook

Reference

Tracker Architecture and Comparison

The detector-independent architecture, the main tracker families, and how each evaluation metric should be read.

Anatomy of a Multi-Object Tracker

Detector-independent tracking-by-detection architecture

Video frameExample detector: YOLO26ObservationsTrack-state predictionMotion + appearance featuresCost matrix + gatingAssignmentTrack lifecyclePersistent identities + trajectories

Tracker comparison matrix

SORT, ByteTrack, DeepSORT, BoT-SORT, and OC-SORT

Tracker	Motion model	Appearance	Camera compensation	Main strength
SORT	Linear prediction	No	No	Simplicity and speed
ByteTrack	Linear prediction	No	No	Low-confidence recovery
DeepSORT	Linear prediction	Yes	No	Appearance-based identity
BoT-SORT	Motion + association	Optional	Yes	Moving cameras and ReID
OC-SORT	Observation-centric	No	No	Abrupt and nonlinear motion

Metric interpretation

MOTA, MOTP, IDF1, HOTA, FPS, and latency

Metric	Main question
Precision	How many reported detections were correct?
Recall	How many true objects were detected?
MOTA	How many overall tracking errors occurred?
MOTP	How accurately were boxes localized?
IDF1	How consistently were identities preserved?
HOTA	How balanced are detection and association quality?
FPS	Can the tracker operate in real time?
Latency	How long does one frame require?

Reusable implementation

Common Tracking Workflow

The workflow stays detector-independent; YOLO26 is the example observation source in the notebooks.

Twelve-step tracking workflow

From video input to temporal analytics

1Load or train an object detector.
2Read frames from a video source.
3Detect objects in each frame.
4Predict existing track states.
5Calculate detection-to-track similarity.
6Solve the assignment problem.
7Update matched tracks.
8Initialize unmatched detections.
9Retain or remove unmatched tracks.
10Save identities and trajectories.
11Evaluate tracking performance.
12Build temporal analytics.

Project folder structure

Notebooks, trackers, outputs, and reusable source modules

tracking-course/
├── data/
│   ├── videos/
│   ├── detections/
│   └── annotations/
├── models/
│   └── yolo26n.pt
├── trackers/
│   ├── sort/
│   ├── bytetrack/
│   ├── deepsort/
│   ├── botsort/
│   └── ocsort/
├── notebooks/
│   ├── TR-01-Tracking-Fundamentals-and-Kalman-Filtering.ipynb
│   ├── TR-02-Data-Association-SORT-and-ByteTrack.ipynb
│   ├── TR-03-Appearance-Occlusion-and-Robust-Tracking.ipynb
│   └── TR-04-MOT-Evaluation-and-Video-Analytics.ipynb
├── outputs/
│   ├── videos/
│   ├── trajectories/
│   ├── metrics/
│   └── analytics/
└── src/
    ├── detector.py
    ├── motion.py
    ├── association.py
    ├── track.py
    ├── visualization.py
    ├── evaluation.py
    └── analytics.py

Prerequisites

Knowledge expected before starting the series

Python
NumPy
Basic OpenCV
Bounding boxes
Intersection over Union
Object detection
Matrix multiplication
Probability fundamentals
Basic linear algebra

Kalman filtering, Hungarian assignment, and re-identification are taught inside this series—not required beforehand.

Series Assessment

Nine required practical and analytical deliverables

One Kalman-filter motion-prediction experiment
One numerical association-matrix exercise
One SORT or ByteTrack implementation
One motion-versus-appearance tracker comparison
One occlusion and identity-switch analysis
One MOT evaluation table
One trajectory visualization
One line-crossing counter
One research-style tracker recommendation

Previous seriesSelf-Supervised Learning for Computer Vision

Object Tracking and Multi-Object Tracking

Tracking Through Time

Detection Is Not Tracking

Object Detection → Multi-Object Tracking

Object detection

Object tracking

Kalman Filtering and Hungarian Assignment

Kalman Filter: Predict → Measure → Update

Hungarian Assignment: Detections → Tracks

What Students Will Learn

Student Tracking Notebooks

Foundations of Object Tracking and Motion Prediction

Detection-to-Track Association: From SORT to ByteTrack

Appearance, Occlusion, and Robust Identity Tracking

Tracking Evaluation and Intelligent Video Analytics

Evolution of Tracking-by-Detection

Tracker Architecture and Comparison

Common Tracking Workflow