Imagine telling a robot, “Find the red emergency-stop button.” With Grounding DINO, you can describe what the robot should look for in ordinary words—even when that exact object was not one of the detector’s original class labels.
1. What is Grounding DINO?
Imagine commissioning a warehouse robot with a detector trained to recognize person, forklift, and box. A new task arrives: find a carton with torn wrapping. The camera can see it, and a human can describe it, but the detector has no class called “carton with torn wrapping.” Adding that class normally means collecting images, drawing labels, and training again.
Grounding DINO changes the interface. Instead of asking only, “Which trained class is this?”, it also asks, “Which region matches these words?” You supply an image and a phrase such as carton with torn wrapping, and the model returns boxes aligned with that phrase.
This is why Grounding DINO is called an open-set, language-conditioned object detector. Open-set means the request is not limited to one small list of class IDs. Language-conditioned means the text changes what the detector searches for.
That is useful in mixed-SKU logistics, flexible assembly, inspection, laboratory automation, and human-directed manipulation—settings where the robot’s vocabulary changes faster than a dedicated detector can be retrained.
The result is still only a 2D hypothesis. A box aligned with red emergency-stop button does not provide depth, a surface mask, a 6D pose, or a safe grasp. Grounding DINO helps the robot decide where to look next; later perception and planning stages decide what the robot can safely do there.
1.1 Overview of the Grounding DINO Architecture
Image + language → grounded boxes
Language enters the detector at feature fusion, query initialization, and decoding.
Grounding DINO begins with two separate inputs because they answer different questions. The image contains possible objects and regions. The text prompt—perhaps carton with torn wrapping—specifies which visual concept matters for this request.
The paper describes the model as a dual-encoder, single-decoder architecture. A visual encoder turns the image into multi-scale features, while a text encoder turns the words into contextual language features. At this point, each stream has a useful representation, but neither has yet solved the grounding task alone.
The feature enhancer lets image and text features exchange information, so visual locations can be interpreted in relation to the prompt. Language-guided query selection then chooses the image locations that appear most relevant. This gives the decoder a focused set of candidates rather than asking it to investigate every location equally.
The single cross-modality decoder repeatedly refines those candidates while consulting both streams. Its final outputs are boxes paired with the words or phrases they match. The name summarizes the design: DINO supplies the query-based detector, while grounding connects its object candidates to language. The next section follows one prompt through each of these stages.
1.2 The Two Encoders Build Visual and Text Features
Follow the prompt red emergency-stop button through the architecture.
The image enters a Swin Transformer, the first encoder. Rather than producing a class name, it produces feature maps at several scales. Fine-scale maps preserve small details, while coarse maps carry wider scene context.
A feature map is a grid of learned vectors. Each vector describes what the network sees around one image location: perhaps an edge, a texture, or a more complete object part. It is not yet a detection. It is the visual evidence from which later stages will build detections.
At the same time, the prompt enters BERT, the second encoder. BERT produces a contextual feature for each token. The representation of “red” is influenced by “button,” so the model can represent the whole request rather than searching independently for redness and button-like objects.
The encoders have now translated pixels and words into compatible numerical representations. The remaining stages use those representations to decide where the phrase appears.
1.3 The Feature Enhancer Lets Vision and Language Meet
Attention is a learned way for one feature to read a weighted mixture of other features. Self-attention stays within one stream: image locations exchange visual context, while text tokens exchange linguistic context. Cross-attention connects streams: an image location can read relevant words, and a word can read relevant image locations.
The feature enhancer packages those operations together. As described by Liu et al., it uses deformable self-attention for the image, ordinary self-attention for the text, and cross-attention in both directions. Deformable means that an image feature samples a small set of useful spatial locations instead of comparing itself densely with every position, reducing computation on large feature maps.
After this exchange, a possible button region is no longer represented only as a visual pattern; it is represented in the context of the supplied phrase. This is the first of the paper’s three fusion points.
1.4 Language-Guided Query Selection Chooses Candidates
Grounding DINO next needs to turn thousands of enhanced image locations into a manageable set of object candidates. In a transformer detector, each candidate is represented by an object query: a vector that will carry one possible object’s content and approximate location through the decoder.
Language-guided query selection scores enhanced image locations against enhanced text features. It selects the top 900 locations as decoder candidates in the architecture reported by the paper. In simplified form, the compatibility between image feature and text feature can be written as
Here, is the compatibility between image location and text token ; and are their feature vectors, and denotes transpose so the expression forms a dot product.
The paper takes the maximum score over text tokens for each image location and retains the strongest image positions. Those positions initialize the positional part of the object queries, while the query content is learned. This is the second fusion point: the language decides which visual candidates enter the decoder.
1.5 The Cross-Modality Decoder Produces Grounded Boxes
A decoder is a stack of repeated refinement blocks. It does not decode a compressed file; it turns the selected queries into final predictions. Each Grounding DINO decoder layer applies query self-attention, cross-attention to image features, cross-attention to text features, and a feed-forward network. A query therefore updates its internal representation and box while repeatedly consulting both the scene and the phrase. This is the third fusion point.
The stages form one chain:
- The feature enhancer makes the image representations aware of the text.
- Language-guided query selection promotes the most relevant image locations.
- The decoder refines those candidates into boxes and token alignments.
The last box in the diagram is Grounded detections. Each result joins the refined image box with the phrase or tokens that best match it.
Together, those stages answer two questions at once: “Is there an object-like region here?” and “Does it match the supplied words?”
This differs from a fixed classifier. A fixed detector compares a region against a stored weight vector for each trained class. Grounding DINO compares regions with the text supplied for the current image, then returns token-level alignments alongside its boxes.
1.6 Keep Several Prompted Categories Separate
The key innovation is not simply attaching a text encoder to the end of a detector. Grounding DINO introduces language during feature enhancement, query selection, and box decoding. The paper calls this tight fusion because language participates throughout the detection process.
The model also needs to handle prompts containing several categories. If button, pallet, and carton are placed in one sequence, they should not blur into one compound concept. Sub-sentence attention masks limit unwanted attention between separate category phrases.
The sub-sentence mask solves the text-side problem; training must now teach the visual queries to align with those separated phrases and with the correct boxes.
1.7 The Loss Function Teaches Language Alignment and Box Geometry
At the start of training, the 900 decoder queries are merely predictions. The annotated objects are targets. Because neither list has a meaningful order, Grounding DINO first uses bipartite matching to decide which prediction should learn from which target. Think of this as pairing each annotated object with the prediction that currently offers the best combination of phrase match and box geometry.
The matching cost combines three terms:
Here, is the cost of matching prediction to target ; is the predicted box and is the annotated box. measures phrase alignment, is coordinate distance, and is generalized-IoU cost. The three terms weight their relative influence.
Once matching is complete, the paper applies the corresponding training losses. Focal contrastive classification loss compares each query with text-token features and teaches the correct phrase alignment. L1 loss directly penalizes differences between predicted and target box coordinates. Generalized IoU loss judges the boxes as regions, supplying a useful geometric signal even when they do not yet overlap. Auxiliary versions of these losses are applied after intermediate decoder layers and encoder outputs, so earlier stages receive a direct learning signal rather than waiting for the final decoder output.
The important distinction is that matching chooses which prediction is responsible for an object; the loss says how that prediction should change. Neither step runs during inference.
1.8 The Training Data Teaches What Words Look Like
The architecture creates a path between vision and language, and the loss tells the parameters how to improve. The data supplies the experience that makes those signals meaningful. Each example may contain an image, phrases, and regions associated with those phrases. Together, these examples teach objectness, box geometry, and region–phrase alignment.
This joint supervision is important. A model trained only on captions might know that emergency-stop buttons exist without learning their precise extent. A model trained only on closed-set boxes might localize a button but remain tied to a fixed class ID. Grounded data connects phrases directly to regions.
The paper trains different checkpoints on different mixtures. The principal sources include Objects365 (O365) for detection boxes, GoldG for grounded region–phrase pairs, Cap4M for image captions, and—depending on the experiment—OpenImages, COCO, and the RefCOCO family. For example, the reported Swin-T variants progress from O365 alone to O365 + GoldG and then O365 + GoldG + Cap4M; the strongest zero-shot Swin-L result uses O365, OpenImages, and GoldG.
These sources teach complementary behavior. Detection data teaches objectness and box geometry. Grounding data teaches which phrase belongs to which region. Caption data broadens the visual vocabulary but provides weaker localization supervision. Consequently, “open vocabulary” means transfer from broad grounded pre-training—not that the model has direct visual experience with every possible object or workcell.
1.9 How Prompt Wording Changes the Search
Grounding DINO is open-vocabulary, but a prompt is not a magic database query. Words are converted into contextual token embeddings, so wording affects the representation against which regions are compared. Button, red button, and red emergency-stop button provide different amounts of constraint. The most detailed phrase is helpful when the visual evidence supports those attributes; it can be harmful when the object is tiny and its colour or type is not visible.
Prompts containing several categories are usually separated into sub-sentences. The attention mask keeps those category phrases from becoming one accidental compound request. Even so, semantically overlapping names such as carton, box, and package may compete for the same region. A robot application should define a small, testable vocabulary for each task rather than sending every possible synonym at once.
Negative scenes are just as important as successful examples. If the requested object is absent, the safe result is no usable detection—not the most visually similar region. Evaluate paraphrases, distractors, partial occlusion, small targets, and images with no target. That is how the appealing idea of “find anything by name” becomes a measured perception capability.
Interactive language grounding
Change the phrase and see which regions become relevant
1.10 When Grounding DINO Is the Right Detector
| Detector type | Inference query | Best fit | Main constraint |
|---|---|---|---|
| Classical geometry | Radius, contour, colour | Controlled geometry | No semantic vocabulary |
| Fixed detector | Trained class IDs | Fast repeated classes | Retraining for new classes |
| Grounding DINO | Names or phrases | Open-vocabulary localization | Prompt and threshold sensitivity |
| Qwen detector | Natural-language object list | Semantic flexibility | Generative behavior and latency |
Grounding DINO offers far more vocabulary flexibility than a fixed detector while remaining more detection-focused than a general vision-language model. Use it when the robot’s requested concepts change at inference time and a phrase-aligned box is the desired output. Prefer a fixed detector when the vocabulary is stable and predictable latency matters more than linguistic flexibility.
2. How to Use Grounding DINO in the Telekinesis Agentic OS
Telekinesis provides Grounding DINO as the Retina Skill detect_objects_using_grounding_dino. The code below loads an image, searches for cartons and pallets, applies separate box and text thresholds, and prints the typed detections.
from telekinesis import datatypes, retina
image = datatypes.Image.from_url(
url="https://assets.telekinesis.ai/examples/v1/images/palletizing.jpg"
)
detections, categories = retina.detect_objects_using_grounding_dino(
image=image,
objects=["carton", "wooden pallet"],
box_threshold=0.45,
text_threshold=0.50,
)
for detection in detections:
category = categories[detection.category_id]
print(category.name, detection.score, detection.bbox)
print("boxes [x, y, w, h]:", detections.bboxes)
print("scores:", detections.scores)
print("category IDs:", detections.category_ids)
if len(detections) == 0:
print("No target: stop, reobserve, or request clarification")
The Skill returns detections alongside the prompt-derived category table. The rendered result below makes the contract tangible: each phrase is resolved into a category, score, and image-space bounding box that downstream code can inspect.

Grounding DINO converts the supplied object vocabulary into localized image regions. The visualization is evidence about 2D grounding—not yet about depth, reachability, or grasp safety.
3. Benchmarking
3.1 Results Reported in the Grounding DINO Paper
The Grounding DINO paper evaluates several kinds of generalization. COCO tests common categories, LVIS stresses a long-tail vocabulary, and ODinW measures transfer across many real-world detection datasets. Reading them together is more informative than quoting only the best COCO AP.
Zero-Shot COCO Detection
These are COCO 2017 validation results from Table 2. “Zero-shot” means the COCO training split was not used for that row; it does not mean the model saw no detection or caption data.
| Model | Backbone | Pre-training data | COCO AP |
|---|---|---|---|
| GLIP-T (C) | Swin-T | O365, GoldG | 46.7 |
| Grounding DINO-T | Swin-T | O365, GoldG | 48.1 |
| Grounding DINO-T | Swin-T | O365, GoldG, Cap4M | 48.4 |
| GLIP-L | Swin-L | FourODs, GoldG, Cap24M | 49.8 |
| Grounding DINO-L | Swin-L | O365, OpenImages, GoldG | 52.5 |
Long-Tail Detection on LVIS
LVIS reports overall mask AP and separate AP for rare, common, and frequent categories. The split exposes a limitation hidden by the overall number: the paper notes that DETR-like models remain weaker on rare categories.
| Model | Setting | LVIS AP | APr | APc | APf |
|---|---|---|---|---|---|
| GLIP-T | Zero-shot | 24.9 | 17.7 | 19.5 | 31.0 |
| DetCLIPv2-T | Zero-shot | 40.4 | 36.0 | 41.7 | 40.0 |
| Grounding DINO-T | Zero-shot | 25.6 | 14.4 | 19.6 | 32.2 |
| Grounding DINO-T + Cap4M | Zero-shot | 27.4 | 18.1 | 23.3 | 32.7 |
| Grounding DINO-L | Zero-shot | 33.9 | 22.2 | 30.7 | 38.8 |
| Mask R-CNN R101 | Fine-tuned | 33.3 | 26.3 | 34.0 | 33.9 |
| DetCLIPv2-T | Fine-tuned | 50.7 | 44.3 | 52.4 | 50.3 |
| Grounding DINO-T | Fine-tuned | 52.1 | 35.4 | 51.3 | 55.7 |
Transfer Across ODinW
ODinW aggregates many out-of-distribution detection datasets. The paper reports the mean and median AP because a model can score well on a few datasets while failing on many others.
| Model | Parameters | Pre-training data | Mean AP | Median AP |
|---|---|---|---|---|
| MDETR | 169M | GoldG, RefC | 10.7 | 3.0 |
| OWL-ViT L/14 | >1,243M | O365, VG | 18.8 | 9.8 |
| GLIPv2-T | 232M | O365, GoldG, Cap4M | 22.3 | 8.9 |
| DetCLIP-L | 267M | O365, GoldG, YFCC1M | 24.9 | 18.3 |
| Grounding DINO-T | 172M | O365, GoldG | 20.0 | 9.5 |
| Grounding DINO-T | 172M | O365, GoldG, Cap4M | 22.3 | 11.9 |
| Grounding DINO-L | 341M | O365, OI, GoldG, Cap4M, COCO, RefC | 26.1 | 18.4 |
What the Data Ablation Reveals
Table 6 changes the pre-training mixture while holding the Swin-T architecture fixed.
| Pre-training data | COCO zero-shot AP | COCO fine-tuned AP | LVIS zero-shot AP | ODinW mean AP |
|---|---|---|---|---|
| O365, GoldG | 48.1 | 57.1 | 25.6 | 20.0 |
| + RefCOCO/+/g | 48.5 | 57.3 | 21.9 | 17.7 |
| + COCO | 56.1 | 57.5 | 22.3 | 17.4 |
Adding target-like data greatly improves COCO, but it does not automatically improve open-domain transfer. Paper AP is evidence about the evaluated checkpoints—not a benchmark of the Telekinesis-hosted artifact, network path, or your robot images. Measure end-to-end latency and task accuracy in the deployed Agentic OS pipeline.
3.2 Comparison with Other Telekinesis Skills
| Skill | Vocabulary | Main controls | Prefer it when |
|---|---|---|---|
| Grounding DINO | Open text phrases | Box and text thresholds | New categories and referring phrases |
| Qwen | Requested object names | Object list | Broad semantic interpretation matters |
| RF-DETR | Fixed COCO-80 | Score threshold | Fixed common classes and transformer detection |
| YOLOX | Fixed COCO-80 | Score and NMS thresholds | Fast dense detection and explicit suppression control |
| Contours/Hough | Geometric priors | Shape-specific parameters | The scene is controlled and geometry is the signal |
4. Where to Go Next?
5. References
- Liu, S. et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection”, 2023.
- Zhang, H. et al. “DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection”, ICLR 2023.
- Carion, N. et al. “End-to-End Object Detection with Transformers”, ECCV 2020.
- Telekinesis. “Detect Objects Using Grounding DINO”, Retina Skill documentation.


