Imagine a robot watching parts move along a conveyor. Its detector must find each part accurately, but it must also respond before the part has moved away. RF-DETR is designed to help engineers find that balance between detection accuracy and speed.
1. What is RF-DETR?
Return to the conveyor example. If the camera captures 30 frames each second but detection takes half a second, the robot is always reacting to the past. A smaller model may respond on time but miss difficult parts. The useful detector is not simply the most accurate one—it is the most accurate one that meets the system’s timing budget.
RF-DETR is a real-time specialist detection transformer built around that trade-off. It looks at an image and produces a set of labeled boxes. Specialist means the model is adapted to the classes needed by a target task. Real-time means latency is treated as a design objective rather than checked only after training.
RF-DETR is therefore different from an open-vocabulary detector. It does not accept any new phrase and immediately understand a new class. Instead, it asks a deployment-focused question: once a detector has learned the target domain, which model configuration gives the best accuracy on the available hardware without responding too slowly?
1.1 The Complete Path From an Image to a Prediction Set
Image + object queries → prediction set
Weight-sharing search varies the backbone and decoder configuration, then selects a point on the accuracy–latency frontier.
RF-DETR first needs to turn millions of pixel values into visual evidence that can be searched efficiently. A pre-trained DINOv2 vision transformer does this by representing image regions as feature tokens. Its pre-training gives the detector a broad visual starting point before object-detection training begins.
Objects in a robot image can be tiny, large, nearby, or distant. The multi-scale projector reshapes the backbone output into several feature maps with different spatial resolutions. Detailed maps help preserve small objects; coarser maps provide wider context for large ones.
The deformable DETR decoder then introduces a fixed set of learned object queries. Think of each query as an empty prediction slot. It samples a small number of useful locations from the multi-scale maps, gathers evidence, and gradually becomes a class prediction and a box—or concludes that its slot contains no object.
All queries are refined together into one prediction set. During training, one-to-one matching assigns each real object to one prediction, which removes the need for a separate NMS stage. The paper also supervises every decoder layer, allowing later layers to be dropped when a deployment needs lower latency. The following sections examine this path one stage at a time.
1.2 The Backbone Turns the Image Into Multi-Scale Features
The diagram begins at Image. A pre-trained DINOv2 backbone converts it into visual tokens. RF-DETR interleaves windowed attention, which limits computation to local regions, with non-windowed attention, which allows global information exchange. The multi-scale projector then builds the feature hierarchy consumed by the decoder.
1.3 Object Queries Ask the Decoder What Is in the Image
The projector has prepared the image features, but RF-DETR still needs a mechanism for turning those features into individual objects. That mechanism begins with object queries. A query is a learned prediction slot: an initially empty vector that will try to claim one object from the image. Following the set-prediction idea introduced by DETR, the model uses a fixed collection of these slots rather than a dense list of overlapping candidate boxes.
Picture the queries as a row of empty inspection forms. After the decoder studies the image, one form may contain “worker” and a box, another may contain “forklift” and a box, and the remaining forms may say “no object.” A query is not permanently tied to one class; it is a reusable slot that competes to explain one object in the current image.
The object-query analogy becomes more concrete by separating two kinds of attention. Self-attention lets queries communicate with one another. If two queries are converging on the same forklift, their representations can become different enough that one keeps the object while the other becomes “no object.” Cross-attention lets every query read from image features and gather the evidence needed for its class and box.
For one query and image keys , a simplified cross-attention update is
Here, is object query , and are key and value matrices derived from image features, and is the key dimension. Softmax converts query–key similarity into weights over image evidence.
RF-DETR uses deformable cross-attention, so a query samples a small collection of relevant positions instead of forming a dense weight over the entire feature map. Across decoder layers, its reference point and box are refined. Prediction heads after each layer convert the current query state into a class distribution and normalized box coordinates.
After the final retained decoder layer, the useful query results form the prediction set. We can now explain how training decides which query should be responsible for which annotated object.
1.4 One-to-One Matching Assigns Each Object to One Query
Set prediction creates one important training problem. Predictions have no natural order. The first query is not guaranteed to describe the first annotated object, so the loss cannot simply compare row one with row one.
DETR solves this using the assignment method introduced by Kuhn. It finds a one-to-one pairing between predicted slots and annotated objects that minimizes a matching cost. Conceptually, that cost is
Here, is the cost of matching prediction to target ; is the predicted box and the annotated box. is classification cost, is coordinate distance, and is generalized-IoU cost. The values weight those three terms.
The three terms ask complementary questions. Does the predicted class match? Are the box coordinates close? Do the boxes overlap geometrically? Once the lowest-cost pairing is found, every matched query learns from one object and every unmatched query learns “no object.” The one-to-one structure discourages several queries from claiming the same target.
The overlap term begins with intersection over union:
Here, and are two box regions; is their intersection, their union, and vertical bars denote region area.
IoU is one when boxes match perfectly and zero when they do not overlap. The zero case is awkward during learning because it provides little guidance about how the prediction should move. Generalized IoU also considers the smallest box enclosing both regions, providing a useful signal even before the boxes touch.
Interactive set matching
Assign each object to exactly one query
Hungarian matching considers the assignment as a whole. Two queries cannot both claim the same training object; unused queries learn the no-object class.
1.5 The Detection Loss Improves Every Decoder Layer
Matching and loss are related but different. Matching selects which query is responsible for each target. The detection loss then measures how wrong the matched predictions are and supplies the gradients that update the network.
A compact DETR-style view is
Here, is the total detection loss; penalizes an incorrect object class or an incorrect “no object” prediction; penalizes localization error for matched boxes; and and control their relative weight. In DETR-family implementations, the box term is commonly built from coordinate and overlap losses such as L1 and generalized IoU.
The RF-DETR paper’s important explicit choice is to attach detection—and, for RF-DETR-Seg, segmentation—losses to every decoder layer. An early layer must therefore produce a usable prediction instead of relying on later layers to repair everything. That is what makes decoder dropout possible at inference: an engineer can remove later layers for speed because the earlier outputs were directly supervised during training.
The paper does not enumerate a new named detection-loss formula as its primary contribution. Its contribution is the way these losses supervise a weight-sharing family of detector configurations and every possible decoder exit. Keeping that distinction avoids attributing a new loss to RF-DETR that the paper does not claim.
1.6 Weight-Sharing Search Finds Accuracy–Latency Trade-offs
So far, this explains how RF-DETR detects objects. Its distinctive contribution is how it searches for a deployable version of that detector.
Training every possible architecture independently would be prohibitively expensive. RF-DETR instead trains a weight-sharing super-network. Think of it as one large network that contains many smaller valid subnetworks. Those subnetworks reuse learned weights, allowing thousands of candidate configurations to be evaluated without starting training from zero each time.
The search varies choices such as image resolution, patch size, windowed attention, decoder depth, and query count. Increasing one of these may improve accuracy but also add computation. The goal is not to find one universally best configuration. It is to recover an accuracy–latency curve for the target dataset and platform.
Along that curve, the Pareto frontier contains configurations for which no alternative is both faster and more accurate. An engineer can select the point that satisfies the robot’s timing budget. Because runtime depends on the accelerator, numerical precision, batch size, and inference engine, that choice must be measured on the actual deployment stack.
1.7 Pre-training and Detection Data Give the Model Its Vocabulary
RF-DETR combines two training histories. Its DINOv2 vision backbone begins with internet-scale self-supervised visual pre-training, which supplies general-purpose image features before detection training. The RF-DETR super-network is then trained for object detection on target datasets such as COCO; the paper also evaluates architecture search and transfer across the 100 diverse datasets in RF100-VL. The segmentation variant is pre-trained on Objects365 with instance masks pseudo-labeled by SAM2.
During weight-sharing neural architecture search, each optimization step samples a configuration with a particular resolution, patch size, attention-window count, decoder depth, and number of queries. The shared weights therefore learn across many subnetworks instead of one fixed architecture. For a new specialist domain, the selected architecture can be fine-tuned on that domain’s labeled boxes; the paper finds this especially useful for the smaller RF100-VL datasets.
This history matters in deployment. DINOv2 contributes broad visual priors, but the final detector’s class vocabulary and decision boundaries still come from detection annotations and any domain fine-tuning. A proprietary component absent from those labels does not become a supported class merely because the backbone once saw broad internet imagery.
1.8 Set Prediction Usually Removes the Need for NMS
Dense detectors often produce many overlapping candidates for the same object and then use non-maximum suppression to remove duplicates. DETR-style training attacks duplication earlier. Because one-to-one matching rewards only one query for each annotated object, duplicate queries are trained toward “no object.”
That does not guarantee perfect uniqueness. Difficult scenes can still contain duplicate or missed predictions, and filtering low-confidence queries remains necessary. The architectural difference is that suppression is not the central mechanism used to turn thousands of local candidates into a final list. The model learns to emit a set directly.
For robotics, this removes one sensitive overlap threshold but introduces a different question: how many query slots and decoder layers are worth their cost? Too few queries can cap the number of detectable instances. More queries and deeper decoding may improve crowded scenes while increasing latency.
1.9 “Real Time” Must Be Defined by the Robot
“Real time” has no useful meaning without the motion and control loop. Imagine a conveyor moving at m/s. If capture, transfer, inference, and planning together take 120 ms, the part moves
The first value is conveyor velocity, the second is total system latency, and the result is the physical displacement during that delay.
or 72 mm before the robot acts. A detector running at an impressive benchmark frame rate can still be unusable if that displacement exceeds the grasp tolerance.
Measure the complete perception path: camera exposure, image transport, decoding, model execution, output serialization, network time, and downstream geometry. Record median latency and a tail percentile such as p95 or p99. A rare 300 ms delay can matter more than a one-point AP improvement when a fast conveyor has no recovery space.
The Pareto frontier is therefore a system-design tool. Select a configuration only after translating its measured latency into physical scene change and checking that its accuracy is adequate for the failure modes that carry real cost.
1.10 Where RF-DETR Can Fail
- Domain shift: deployment imagery can differ from the detector’s fine-tuning distribution.
- Small objects: limited pixels and feature resolution reduce class and box evidence.
- Occlusion and crowds: nearby instances may be missed or localized poorly.
- Look-alike classes: visually similar categories can exchange confidence.
- Motion blur and exposure: an otherwise familiar object loses usable texture.
- Box insufficiency: a correct rectangle still includes background and says nothing about graspable geometry.
These limitations complete the architecture story. Neural architecture search can help a conveyor detector respond faster without giving up unnecessary accuracy, but it cannot restore detail lost to blur, reveal a fully hidden part, or teach a class absent from the training data. A correct box also remains only a 2D region; manipulation still requires depth, shape, calibration, and safety checks.
2. How to Use RF-DETR in the Telekinesis Agentic OS
Telekinesis provides RF-DETR as the Retina Skill detect_objects_using_rfdetr. The code below loads an image, applies a score threshold, resolves the COCO category IDs, and converts one returned box from width–height form to corner coordinates.
from telekinesis import constants, datatypes, retina
image = datatypes.Image.from_url(
url="https://assets.telekinesis.ai/examples/v1/images/warehouse_1.jpg"
)
detections = retina.detect_objects_using_rfdetr(
image=image,
score_threshold=0.50,
)
categories = constants.get_coco_categories(model="rfdetr")
for detection in detections:
category = categories[detection.category_id]
x, y, width, height = detection.bbox
print(category.name, detection.score, (x, y, width, height))
# Convert Retina's [x, y, width, height] box to corner coordinates.
x_min, y_min, width, height = detection.bbox
x_max = x_min + width
y_max = y_min + height
The output image shows the final prediction set after thresholding. Use it to check class identity and box coverage before connecting a detection to depth, tracking, or manipulation logic.

A box localizes image extent; it does not provide depth, 6D pose, free space, or a safe grasp.
3. Benchmarking
3.1 Results Reported in the RF-DETR Paper
The RF-DETR paper evaluates COCO using TensorRT 10.4 on an NVIDIA T4 and reports latency with the same model artifact used for accuracy. The paper also evaluates transfer across the 100 datasets in RF100-VL and reports instance-segmentation results.
COCO Object Detection
| Model | Size | Parameters | Latency | COCO AP | AP50 |
|---|---|---|---|---|---|
| D-FINE | Nano | 3.8M | 2.1 ms | 42.7 | 60.2 |
| RF-DETR | Nano | 30.5M | 2.3 ms | 48.0 | 67.0 |
| D-FINE | Small | 10.2M | 3.5 ms | 50.6 | 67.6 |
| RF-DETR | Small | 32.1M | 3.5 ms | 52.9 | 71.9 |
| RF-DETR | Medium | 33.7M | 4.4 ms | 54.7 | 73.5 |
| RF-DETR | 2XL | 126.9M | 17.2 ms | 60.1 | 78.5 |
The full COCO breakdown explains which object sizes drive that AP:
| RF-DETR size | AP | AP50 | AP75 | APS | APM | APL |
|---|---|---|---|---|---|---|
| Nano | 48.0 | 67.0 | 51.4 | 25.2 | 53.5 | 70.0 |
| Small | 52.9 | 71.9 | 57.0 | 32.0 | 58.3 | 73.0 |
| Medium | 54.7 | 73.5 | 59.2 | 36.1 | 59.7 | 73.8 |
| 2XL | 60.1 | 78.5 | 65.5 | 43.2 | 64.9 | 76.2 |
Transfer Across RF100-VL
RF100-VL averages results over 100 varied detection datasets. This is important because COCO accuracy alone can reward designs and training schedules that do not transfer well.
| Model | Size | Latency | AP | AP50 | AP75 |
|---|---|---|---|---|---|
| YOLOv11 | Medium | 5.1 ms | 57.0 | 82.5 | 61.9 |
| D-FINE | Medium | 5.6 ms | 60.6 | 85.5 | 65.8 |
| Grounding DINO | Tiny | 309.9 ms | 62.3 | 88.8 | 67.8 |
| RF-DETR | Nano, fine-tuned | 2.5 ms | 58.7 | 85.6 | 63.5 |
| RF-DETR | Small, fine-tuned | 3.7 ms | 61.0 | 87.2 | 66.4 |
| RF-DETR | Medium, fine-tuned | 4.6 ms | 61.9 | 87.9 | 67.3 |
| RF-DETR | 2XL, fine-tuned | 15.6 ms | 63.5 | 89.0 | 69.2 |
COCO Instance Segmentation
The segmentation extension is not the detector exposed by the Retina RF-DETR Skill, but it is a key result of the paper’s shared architecture-search approach.
| Model | Size | Latency | Mask AP | AP50 | AP75 |
|---|---|---|---|---|---|
| YOLOv11-Seg | Medium | 6.9 ms | 38.5 | 60.0 | 40.9 |
| FastInst | R50 | 39.6 ms | 34.9 | 56.0 | 36.2 |
| MaskDINO | R50 | 242 ms | 46.3 | 69.0 | 50.7 |
| RF-DETR-Seg | Nano | 3.4 ms | 40.3 | 63.0 | 42.6 |
| RF-DETR-Seg | Small | 4.4 ms | 43.1 | 66.2 | 45.9 |
| RF-DETR-Seg | Medium | 5.9 ms | 45.3 | 68.4 | 48.8 |
| RF-DETR-Seg | 2XL | 21.8 ms | 49.9 | 73.1 | 54.5 |
The paper also shows that replacing the CAEv2 backbone with DINOv2 recovers the accuracy lost by gentler, scheduler-free training, while weight-sharing architecture search adds roughly another two AP without increasing the selected operating point’s latency. These figures are not Retina service latency. Network transport, serialization, input resolution, service artifact, and concurrency must be measured end to end in your Agentic OS deployment.
3.2 Comparison with Other Telekinesis Skills
| Property | RF-DETR | YOLOX |
|---|---|---|
| Core formulation | Transformer set prediction | Dense, one-stage anchor-free prediction |
| Global context | Native attention-based reasoning | Primarily convolutional feature hierarchy |
| Duplicate handling | Trained toward unique set predictions | Explicit NMS exposed by Retina |
| Retina tuning | Score threshold | Score and NMS thresholds |
| Label vocabulary | Fixed COCO classes | Fixed COCO classes |
Neither architecture wins every scene. RF-DETR may benefit from global relationships and the DETR formulation; YOLOX offers direct control over duplicate suppression and a mature one-stage design. Evaluate accuracy, tail latency, memory, and failure consistency on the same recorded frames.
4. Where to Go Next?
5. References
- Robinson, I., Robicheaux, P., Popov, M., Ramanan, D., and Peri, N. “RF-DETR: Neural Architecture Search for Real-Time Detection Transformers”, 2025.
- Carion, N. et al. “End-to-End Object Detection with Transformers”, ECCV, 2020.
- Rezatofighi, H. et al. “Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression”, CVPR, 2019.
- Lin, T.-Y. et al. “Microsoft COCO: Common Objects in Context”, ECCV, 2014.


