What is SAM 3?

Imagine asking a robot to pick every damaged carton from a mixed pallet. A bounding box tells the robot roughly where each carton is, but the gripper and depth pipeline need the exact pixels that belong to it. SAM 3 connects both parts of the request: words identify the concept, and masks trace every matching instance.

1. What is SAM 3?

SAM 3 is a promptable segmentation model that detects, segments, and—in video—tracks every object matching a visual concept. A concept can be supplied as a short noun phrase such as yellow safety helmet, as one or more image exemplars, or as a combination of the two.

This task is called Promptable Concept Segmentation (PCS). It fills a gap between two familiar tools. An open-vocabulary detector can find all objects named by text, but usually returns coarse rectangles. Earlier promptable segmenters can trace one object from a point or box, but they do not necessarily search the whole image for every instance of a named concept. SAM 3 is built to do both.

The distinction matters in robotics. A prompt such as red cable can return separate masks for all matching cables. Those masks support pixel-level measurements, depth filtering, collision checking, and grasp-region selection. They are still image-space hypotheses: the model does not supply calibrated depth or guarantee that a selected region is safe to manipulate.

1.1 The Complete Image Segmentation Path

Image + concept prompt → instance masks

ImageRGB scene

Concept promptNoun phrase or exemplar box

Aligned encodersPerception Encoder image and text features

Fusion encoderImage features attend to prompt tokens

DETR-style decoderObject queries localize matching instances

Presence + mask headsConcept existence, boxes, scores, and masks

Prompted instancesOne mask per matching object

The presence head separates “is this concept present?” from “where are its instances?”, helping the model learn from hard negative prompts.

SAM 3 begins with two different kinds of input. The image contains everything the camera can see, while a short prompt such as yellow safety helmet says which concept should become a mask. The model must connect those words with the correct image regions before it can trace their boundaries.

A vision encoder turns the image into a grid of visual features, and a text encoder turns the prompt into language features. These encoders are trained so related visual and textual concepts can be compared. A fusion encoder then lets the prompt influence the image representation, emphasizing evidence relevant to the requested concept.

Learned object queries enter a DETR-style decoder and gather prompt-conditioned evidence from that fused representation. Prediction heads turn the resulting query states into confidence scores, boxes, and pixel masks. The output can contain several instances when the prompted concept appears more than once.

The paper also describes a memory-based tracker for carrying identities through video. The Telekinesis Skill discussed here processes a single image, so the diagram follows only the detector path. The next sections explain how aligned encoders and prompt-conditioned fusion make that path work.

1.2 Aligned Encoders Put Images and Words in a Shared Space

The Perception Encoder produces image features and text features that are already aligned. The image encoder divides the scene into a grid of learned visual representations. The text encoder turns a noun phrase into prompt tokens. If exemplar boxes are supplied, an exemplar encoder represents the visual examples as prompt tokens too.

Alignment is important because the phrase yellow safety helmet must be comparable with image regions. A conventional closed-set segmenter stores a fixed output channel for every trained class. SAM 3 instead computes its target from the prompt used for the current image.

The paper deliberately limits direct concept prompts to short noun phrases with optional modifiers. That covers forklift, red apple, and striped cat. It is not designed to interpret an arbitrarily long instruction such as “the carton that arrived after the forklift stopped.” More complex reasoning belongs in a language model or agent that reduces the request to suitable visual concepts.

1.3 The Fusion Encoder Conditions the Image on the Prompt

The image features initially describe the whole scene without knowing which concept will be requested. The fusion encoder changes that. Image representations cross-attend to the prompt tokens, so features relevant to the supplied concept become easier for later queries to recover.

A simplified cross-attention operation is

$$\operatorname{Attention}(Q,K,V)= \operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt d}\right)V.$$

Here, $Q$ contains query vectors derived from image features, while $K$ and $V$ contain keys and values derived from prompt tokens; $d$ is the key dimension. The resulting image features are no longer merely “what is visible?” features. They are “what is visible in relation to this prompt?” features.

This conditioning explains why the same camera frame can produce different masks for person, yellow helmet, and forklift. The pixels never change; the prompt changes the evidence the detector is asked to organize.

1.4 Object Queries Turn Conditioned Features Into Instances

After fusion, a DETR-style decoder uses object queries as prediction slots. Each query gathers evidence from the conditioned image features and tries to explain one matching instance. One query may settle on the first helmet, another on the second helmet, while unused queries predict no object.

This is instance segmentation rather than one binary union mask. Separate masks preserve object identity within the image, which is useful when a robot must count targets or select one accessible instance.

The decoder predicts localization and mask information, but SAM 3 adds a crucial presence head. The presence head answers a global question: does this concept exist anywhere in the image? The object queries answer the local question: where are its instances? Decoupling recognition from localization improves learning from hard negatives—plausible phrases that do not occur in the image.

1.5 Mask Prediction Converts Each Instance Into Pixels

For every retained instance, the model predicts a box, score, and mask. A mask can be understood as a probability $p_i\in[0,1]$ for every image pixel $i$. A threshold $\tau$ creates the binary region

$$m_i=\begin{cases} 1,&p_i\geq\tau,\\ 0,&p_i<\tau. \end{cases}$$

Here, $m_i$ is the final binary label at pixel $i$, $p_i$ is the predicted foreground probability, and $\tau$ is the mask threshold. Raising $\tau$ usually contracts a mask toward its most confident core; lowering it includes uncertain boundary pixels.

This last step is why segmentation is more useful than a box when geometry matters. A box includes background between a worker’s arm and torso or between the spokes of a wheel. A mask preserves which pixels the model believes belong to the object.

1.6 The Training Objective Teaches Recognition, Localization, and Masks

SAM 3 must learn several related tasks. Its detector uses one-to-one, DETR-style assignment so that one prediction query becomes responsible for one annotated instance. The overall training objective can be understood as a weighted combination

$$\mathcal L= \lambda_{cls}\mathcal L_{cls}+ \lambda_{box}\mathcal L_{box}+ \lambda_{mask}\mathcal L_{mask}+ \lambda_{pres}\mathcal L_{pres}.$$

Here, $\mathcal L_{cls}$ trains concept-conditioned instance classification; $\mathcal L_{box}$ trains localization; $\mathcal L_{mask}$ compares predicted and annotated masks; and $\mathcal L_{pres}$ trains the global presence decision. The $\lambda$ terms control their relative influence. This expression is a teaching summary of the objective families, not a claim that the paper reduces its full training recipe to this single displayed equation.

Hard negative concepts are especially important. If an image contains a yellow bus but the prompt says red bus, localization alone can tempt the model to return the visually similar object. Presence supervision teaches it that a plausible location is not enough—the requested concept itself must be present.

1.7 What SAM 3 Was Trained On

The paper’s data engine builds high-quality and synthetic media–phrase–mask supervision across images and video. It reports 4 million unique noun phrases and 52 million high-quality masks, plus a synthetic collection with 38 million phrases and 1.4 billion masks. Human annotators and model-based verifiers focus effort on hard cases, missed instances, incorrect masks, and negative concepts.

This scale gives SAM 3 broad visual vocabulary, but “open vocabulary” does not mean unlimited reliability. Proprietary parts, thermal imagery, transparent materials, ambiguous phrases, blur, and severe occlusion can fall outside its effective training distribution. A deployment test set must include absent concepts as well as obvious positives.

1.8 When Should You Use SAM 3?

Method	Input	Output	Best fit
RGB/HSV/LAB threshold	Numeric color range	One binary mask	Controlled color and lighting
BiRefNet	Image only	Salient foreground mask	Automatic background removal
Grounding DINO	Image + text	Boxes	Open-vocabulary localization
SAM 3	Image + text and/or boxes	Per-instance masks	Named concepts needing precise boundaries

Use SAM 3 when the target can be named or boxed and individual masks matter. Prefer a color threshold when the scene is engineered and speed, transparency, and deterministic behavior matter more than semantic flexibility. Prefer BiRefNet when there is one obvious foreground subject and no prompt should be required.

1.9 Prompt Meaning Is Part of the Measurement

Open vocabulary introduces ambiguity that fixed class IDs hide. Does mirror include its frame? Does worker include a carried box? Is a distant, partly occluded helmet still an instance? The paper handles this by focusing on atomic noun phrases and by acknowledging multiple valid interpretations in evaluation.

For robotics, define a small prompt contract and test it. Record exact prompts with outputs, include negative scenes, and avoid silently swapping synonyms in a validated pipeline. If an instruction requires relations or history, let an upstream reasoning system reduce it to a visual concept or a box, then use SAM 3 for the segmentation task it was designed to solve.

Interactive prompt semantics

Ask for a different concept and receive different instances

—

Concept promptall cartonsdamaged cartonblue container

The image features stay fixed. The prompt changes which concept the fusion and detection stages ask the masks to represent.

1.10 From Masks to Safe Actions

Instance masks can filter depth more cleanly than boxes, but they do not resolve occluded geometry. Eroding a mask before depth sampling may reduce background contamination at the cost of discarding boundary points. The best choice depends on whether the robot needs a robust central surface or the complete silhouette.

Validate mask confidence, minimum area, image-boundary contact, depth support, reachability, and collision state before motion. If several masks match, the selection policy—nearest, largest, least occluded, or explicitly confirmed—is a robotics decision outside the segmentation model.

2. How to Use SAM 3 in the Telekinesis Agentic OS

Telekinesis provides SAM 3 as the Cornea Skill segment_image_using_sam3. The code below loads an image, requests every pedestrian, and inspects the score, box, category, and encoded segmentation returned for each instance.

from telekinesis import cornea, datatypes

image = datatypes.Image.from_url(
    url="https://assets.telekinesis.ai/examples/v1/images/pedestrians.jpg"
)

objects = ["pedestrian"]
results = cornea.segment_image_using_sam3(
    image=image,
    objects=objects,
    threshold=0.5,
    mask_threshold=0.5,
)

for result in results:
    label = objects[result.category_id - 1]
    print(label, result.score, result.bbox, result.segmentation)

The output contains one result per matching instance. Text-prompt categories use i + 1, where i is the prompt’s index; category 0 is reserved for a box-only prompt.

The mask follows the named object more closely than a rectangle, but it remains a 2D prediction that must be connected to depth and calibrated frames before robot motion.

Try it out

Build with SAM 3 in Cornea

See the complete Skill signature, prompt formats, thresholds, output schema, and current examples.

Open SAM 3 docs

Runnable example

Run the SAM 3 segmentation example

Open the exact Python script used to segment prompted objects with SAM 3 and Cornea.

View Python example

3. Benchmarking

3.1 Results Reported in the SAM 3 Paper

The paper evaluates three distinct abilities: finding every instance of a text concept, following visual prompts, and tracking an object through video. The metrics are not interchangeable, so the tables keep each task separate.

Text-Prompted Video Segmentation

The paper’s Table 4 evaluates open-vocabulary video instance segmentation. cgF1 measures concept-grounded segmentation quality and pHOTA adds tracking consistency. The public datasets use their official mAP or HOTA metric.

Model	SA-V cgF1	SA-V pHOTA	YT-Temporal cgF1	YT-Temporal pHOTA	SmartGlasses cgF1	SmartGlasses pHOTA	LVVIS mAP	BURST HOTA	YTVIS21 mAP	OVIS mAP
GLEE, all phrases	0.1	8.7	1.6	16.7	0.0	4.7	20.8	28.4	62.2	38.7
LLMDet + SAM 3 tracker	2.3	30.1	8.0	37.9	0.3	18.6	15.2	33.3	31.3	20.4
SAM 3 detector + tracking-by-detection	25.7	55.7	47.6	68.2	29.7	60.0	35.9	39.7	56.5	55.1
SAM 3	30.3	58.0	50.8	69.9	36.4	63.6	36.3	44.5	57.4	60.5

Visual Object Segmentation

Table 5 starts from a visual mask prompt and measures how well that object is followed. Higher values are better; the exact official metric varies by dataset.

Model	MOSEv1	DAVIS17	LVOSv2	SA-V val	SA-V test	YTVOS19	MOSEv2
SAMURAI	72.6	89.9	84.2	79.8	80.0	88.3	51.1
SAM2Long	75.2	91.4	85.9	81.1	81.2	88.7	51.5
SeC	75.3	91.3	86.5	82.7	81.7	88.6	53.8
SAM 2.1 L	77.9	90.7	79.6	77.9	78.4	89.3	47.9
SAM 3	78.4	92.2	88.5	83.5	84.4	89.7	60.3

Interactive Image Segmentation and Counting

Model	1-click mIoU	3-click mIoU	5-click mIoU	FPS
SAM 1 H	58.5	77.0	82.1	41.0
SAM 2.1 L	66.4	80.3	84.3	93.0
SAM 3	66.1	81.3	85.1	43.5

Model	CountBench MAE ↓	CountBench accuracy ↑	PixMo-Count MAE ↓	PixMo-Count accuracy ↑
DINO-X	0.62	82.9	0.21	85.0
Molmo-72B	0.27	92.4	0.17	88.8
Gemini 2.5 Pro	0.24	92.4	0.38	78.2
SAM 3	0.12	93.8	0.21	86.2

The results show why one headline AP is insufficient: SAM 3 can be strong at text-prompted segmentation, visual propagation, and counting, but its relative advantage changes by task. These are paper-checkpoint results, not Telekinesis service latency. For a robot, also measure prompt recall, mask IoU, absent-concept false positives, boundary error, transfer time, and downstream task success.

4. Where to Go Next?

5. References

• Carion, N. et al. “SAM 3: Segment Anything with Concepts”, 2025.
• Kirillov, A. et al. “Segment Anything”, ICCV 2023.
• Carion, N. et al. “End-to-End Object Detection with Transformers”, ECCV 2020.
• Telekinesis. “Segment Image Using SAM3”, Cornea Skill documentation.