State Of The Art Object Detection with Isaac Robinson

In this video I sat down with Isaac to discuss RF-DETR, a new state-of-the-art family of real-time object detection and segmentation models from Roboflow. We cover the motivation for building models that are not just accurate but also fast, cost-efficient, and deployable across diverse hardware and data regimes, and why moving beyond fixed architectures is key to achieving that. Isaac explains how RF-DETR combines strong foundation backbones like DINOv2 with efficient neural architecture search to unlock novel speed–accuracy trade-offs, including dropping decoder layers and queries after training. We also discuss the model’s strong transfer performance on domains far from COCO, the introduction of a memory-efficient instance segmentation head, and the team’s unusually rigorous benchmarking approach, before closing on the challenges of open-source research and upcoming improvements to inference and platform integration.

* https://www.linkedin.com/in/robinsonish/
* https://github.com/roboflow/rf-detr
* https://arxiv.org/abs/2511.09554

🚀 TIMELINE
0:18 – Isaac’s Role: ML research engineer at Roboflow building new CV models for diverse user use cases.
0:40 – What Is RF-DETR: Presented as SOTA for real-time object detection (COCO + downstream transfer), beating much slower/larger models.
1:20 – ICLR Acceptance: RF-DETR paper accepted to ICLR; now formally peer-reviewed.
2:20 – How They Did It: Combine strong pretrained vision backbones (e.g., DINOv2) with architecture optimization rather than a single fixed design.
4:12 – Neural Architecture Search: Efficient weight-sharing NAS trains thousands of variants and selects the best architecture for a dataset; NAS engine planned as a platform feature.
6:19 – Benchmarking at Scale: Team trains huge numbers of models quickly (e.g., hundreds of YOLO variants overnight) to evaluate transfer thoroughly.
9:06 – Speed Tricks From NAS: Can drop queries post-training (dataset-dependent object-count prior) and even remove all decoder layers (ViT-forward style), improving latency.
10:49 – Segmentation + Memory Wins: Adds a real-time instance segmentation head and cuts segmentation training VRAM ~3–4× (≈40–50GB → ≈10GB) without performance loss.
13:37 – Domain Transfer: RF-DETR performs especially well on “non-COCO-like” domains (notably medical, aerial, and documents) on Roboflow’s RF100-VL benchmark dataset.
20:18 – High-End Result: DINOv2-base RF-DETR reaches greater than 60 mAP on COCO at ~17 ms, framed as the first published real-time detector past 60 mAP.
24:12 – Open-Source Ramp-Up: A new open-source engineer (ex–PyTorch Lightning) is increasing repo velocity; more dev time planned.
26:05 – Deployment Reality + Fix: “Single artefact benchmarking” + upcoming inference stack improvements to make advertised latency reproducible and easier across hardware backends.

Bio: Isaac Robinson is a Machine Learning Research Engineer at Roboflow. He’s worked across the field of computer vision, from real-time stereo depth estimation on household robots to biomedical research at the NIH to founding a zero shot computer vision infrastructure startup. Isaac focusses on the intersection of low latency and high performance, with the goal of helping people unlock new capabilities through vision.

Видео State Of The Art Object Detection with Isaac Robinson канала Robin Cole