Vision & GPU

The pipeline

capture → detect → track → count

A FrameStream opens a USB webcam or an IP/RTSP camera and hands frames to the rest of the pipeline; a single-slot, latest-wins buffer decouples the capture thread from inference, so the loop always works on the freshest frame and latency stays bounded. The detector runs only every N-th frame; the tracker advances every frame, carrying boxes between detections.

Detection

YoloPersonDetector wraps the OpenCV-Zoo YOLOX model (COCO, Apache-2.0, downloaded on first use with a checksum guard) and decodes its grid output into every person box, with NMS and IoU extracted as shared utilities. A face detector is available as a head-count cross-check, and an in-house path exists for environments that cannot download weights: a HOG-plus-linear-SVM detector behind the same interface.

Tracking and counting

• SortTracker — SORT-style: a lightweight constant-velocity filter per track, greedy IoU association, and a lifecycle (tentative → confirmed → lost) that suppresses one-frame false positives.
• Current occupancy — the number of confirmed tracks in this frame, smoothed by a sliding-window median so the number does not flicker.
• Cumulative unique count — every distinct track ID ever confirmed.
• LineCounter — directional in/out crossings of a virtual tripwire, from segment intersection with each track's path.
• ZoneCounter — per-zone occupancy and dwell time via point-in-polygon on the track's feet.
• Re-identification — each newly confirmed track is embedded and matched against a gallery, so the same person returning after an occlusion reuses their ID instead of inflating the unique count.

The whole loop is covered by unit tests and has been verified on still images and a pedestrian clip: the classic two-image checks return the expected counts, and a 100-frame segment of the standard test video reports about seven people concurrent and 21 distinct.

The no-ONNX path

A companion tool, DetectorDistiller, uses the ONNX teachers once, offline, to auto-label frames and train the in-house students: YOLOX distils into the HOG person detector, RetinaFace into a HOG face detector, and ArcFace into a distilled face embedder. After distillation the detectors run with no ONNX runtime and no downloads — lower accuracy than the teachers, in exchange for transparency and zero external weights at inference.

The device layer

Under everything sits PredictiveCoding.Device: device tensors that pair a host buffer with a device pointer, a bucketed CUDA allocator with pinned host memory, per-thread streams, and a GEMM provider. The design rule is that models never reference a backend — they use provider interfaces, and a backend registers itself:

DeviceRegistry.EnableCuda();                 // route matmuls to cuBLAS if a GPU is present
var g = host.ToDevice(Device.Cuda());        // upload (CUDA→CPU fallback automatic)
using var c = TensorOps.MatMul(g, b);        // cuBLAS GEMM
var result = c.ToHostTensor();               // download

There is no custom native library to build: the CUDA path P/Invokes the standard toolkit libraries (cudart, cublas), with a resolver mapping logical names to their version-suffixed files. With no GPU, everything falls back to the CPU — SIMD and multithreaded — and the code is unchanged. New accelerators (ROCm, or a real NCCL binding) are a matter of implementing the provider interfaces.

Licensing, on purpose

The documentation is explicit about detector weights: Ultralytics YOLO models are AGPL-3.0 and are avoided; YOLOX (Apache-2.0) is the recommended permissive default, with RT-DETR and torchvision detectors as permissive fallbacks. The same care runs through the repo — dependency choices are argued in the docs, not just made.