Imagine asking a warehouse robot to find “the damaged carton beside the blue pallet.” The request contains an object, a visual property, and a relationship. Qwen3-VL-4B-Instruct can use all three because it processes the camera image and the instruction inside one multimodal model. This tutorial follows the exact path from pixels to patches, visual tokens, language-conditioned reasoning, and grounded coordinates.
1. What is Qwen3-VL-4B-Instruct?
Qwen3-VL-4B-Instruct is a dense, four-billion-parameter vision-language model trained to follow instructions about images, text, and video. The easiest way to understand it is to use the same division as the technical report: a vision encoder reads the visual input, an MLP vision–language merger converts visual features into the language model’s representation, and a Qwen3 language model interprets those visual tokens together with the instruction.
Its name identifies the Qwen3 generation, a vision-language (VL) interface, a compact dense 4B parameter tier, and an Instruct checkpoint post-trained for direct conversational requests rather than the separate Thinking edition’s extended reasoning behavior.
For the Telekinesis detection Skill, the practical task is visual grounding. Classification asks whether a pallet exists. Detection asks where all pallets from a known class vocabulary are. Grounding asks which image region corresponds to a phrase such as the pallet behind the forklift. The text determines which visual evidence matters.
Unlike a detector with a dedicated box head, Qwen ultimately expresses its answer through the language model. It can generate words, points, or box coordinates in response to an instruction. That provides semantic flexibility, but it also means the result is generated rather than emitted by a conventional detection head and must be validated accordingly.
1.1 The Complete Path From an Image to an Answer
The paper’s three-module architecture
DeepStack is an extra connection between the vision encoder and early LLM layers; it is not a fourth top-level module.
The paper organizes Qwen3-VL into three top-level modules. Following those three modules gives us a simple mental model before we examine patch sizes, positional encodings, and generation.
First, the vision encoder reads the image. It divides the image into patches and turns them into visual features. Self-attention then gives each patch context from other parts of the scene, so a brown region can be interpreted together with its edges, label, and surrounding pallet rather than as an isolated patch of color.
Second, the MLP vision–language merger prepares those visual features for the language model. It groups nearby patch features to shorten the sequence and projects them to the numerical width expected by Qwen3. The result is a sequence of visual tokens that the language model can accept alongside ordinary text tokens.
Third, the Qwen3 language model receives those visual tokens together with the user’s instruction. It processes both in one multimodal context and generates an answer token by token. If the instruction requests localization, that answer can contain labels and coordinate tokens rather than only prose.
Patchification belongs inside the vision encoder’s path. DeepStack supplies an additional route from intermediate vision features into early language-model layers, and interleaved M-RoPE preserves spatial and temporal position. These mechanisms enrich the three-module flow rather than forming separate top-level networks. The rest of Section 1 now follows the image through that flow in detail.
The 4B model combines useful language understanding with a much smaller serving footprint than the 32B dense or 235B mixture-of-experts variants.
Its published configuration gives a concrete picture of what “4B” contains:
| Component | Qwen3-VL-4B-Instruct configuration | Why it matters |
|---|---|---|
| Vision encoder | 24 transformer layers, width 1,024, 16 attention heads | Builds contextual visual features |
| Image patch | pixels | Sets the initial spatial sampling unit |
| Spatial merger | patch groups | Reduces four visual features to one token |
| Language decoder | 36 layers, hidden width 2,560 | Combines prompt and visual evidence |
| Language attention | 32 query heads, 8 key/value heads | Grouped-query attention reduces KV-cache cost |
| Native context | 262,144 tokens | Supports long interleaved text, images, and video |
| Positional encoding | Interleaved M-RoPE | Represents text, spatial, and temporal position |
These are architectural limits, not robot-performance guarantees; long contexts increase latency and can retain stale observations.
1.2 The Vision Encoder Turns Image Patches Into Visual Features
The first module begins with preprocessing. The image processor decodes and normalizes the RGB image, then resizes it within an allowed pixel budget while preserving its aspect ratio. Qwen3-VL calls this dynamic resolution: a wide conveyor image can remain wide instead of being squeezed into one fixed square.
A transformer cannot consume a two-dimensional pixel array directly. The image is therefore divided into small square regions called patches. Each patch is converted into a vector—a list of learned numerical features—and that vector becomes a visual token. A token is simply one item in the sequence processed by a transformer.
Dynamic resolution means computational cost follows image area. For a processed image of height and width , a simplified patch count is
Here, is the number of patch positions, while and are the processed image height and width in pixels. The divisor 16 is Qwen3-VL’s patch size along each axis.
Each region is projected into a vector:
contains patch ; flattens it, is the learned projection, its bias, and the resulting embedding. It represents a local region and position—not yet an object such as “carton.”
At this point, the vectors still describe local patches rather than complete objects. The rest of the vision encoder supplies context. Qwen3-VL uses a SigLIP-2-based Vision Transformer; for the 2B and 4B models, the report specifies SigLIP2-Large rather than the larger visual backbone used by bigger family members.
Inside every vision block, self-attention lets one patch gather information from other patches:
Here, , , and are the query, key, and value matrices derived from patch features; is the key dimension used to scale the dot products; and softmax converts the query–key scores into normalized attention weights.
A brown patch alone could be cardboard, timber, or floor. Self-attention lets it gather evidence from nearby edges, printed labels, and the wider scene. Across 24 transformer layers, isolated patch vectors become contextual visual features. This completes the main job of the first module.
Patch size and image resolution therefore form the first engineering trade-off. More pixels preserve small details, but they also create more patches. No language model can recover a defect that occupies too few input pixels.
1.3 The MLP Merger Makes the Visual Sequence Smaller
The vision encoder and Qwen3 language model do not use the same feature width, and sending every patch feature to the language model would be expensive. The paper’s second module—the MLP vision–language merger—solves both problems. MLP means multilayer perceptron: a small learned network that transforms each group of input features into a new vector. Here, it groups each neighbouring set of visual features, compresses that group into one token, and projects the result into the language model’s 2,560-dimensional hidden space.
The approximate token count after spatial merging is
Here, is the number of merged visual tokens. and again denote processed image dimensions, while 32 is the effective spatial step produced by a 16-pixel patch followed by a merger.
A image therefore produces roughly 1,024 merged positions; a image produces about 256. Resolution is also a token-budget setting.
The merger performs two jobs. It compresses the sequence so the language decoder has less work, and it projects visual features from width 1,024 into the decoder’s width 2,560. Unlike the fixed 256-query adapter in the original Qwen-VL model, the Qwen3-VL token count changes with image resolution.
Compression can weaken very small boundaries. Test the smallest object, narrowest gap, and finest printed text required by the task.
Interactive visual token budget
Change image resolution and patch merging
1.4 The Qwen3 Language Model Combines Vision With the Instruction
Visual tokens + prompt → grounded response
The prompt conditions which visual evidence matters because generated text states can attend back to the preceding visual tokens.
The third module is the 36-layer Qwen3 causal language model. The merged visual tokens are placed in its context, and the text instruction joins them there. Conceptually, the decoder receives
Here, is visual token and is the number of visual tokens; is text token and is the instruction length. The angle-bracket markers identify the visual portion of the multimodal sequence.
The image features do not change when the wording changes, but the language model’s use of them does. The instructions find every carton and find the damaged carton beside the pallet direct attention toward different evidence in the same visual representation. This is the core mechanism behind language-conditioned visual understanding.
The 4B model uses grouped-query attention: 32 query heads share 8 key/value heads. Sharing fewer key/value projections reduces KV-cache memory during long-context generation, which matters when several images, video frames, or a long robot history occupy the context.
1.5 DeepStack Adds Earlier Visual Features to the Language Model
Multi-level vision → DeepStack fusion
DeepStack gives early LLM layers fresh intermediate visual features without lengthening the token sequence.
We can now see the limitation of the main three-module path. The merger normally receives the final vision-encoder features. Those features are semantically strong—they are good at representing that a region is a carton—but some fine local detail may be clearer in earlier vision layers.
DeepStack creates extra connections from those intermediate vision layers into the language model. For the 4B configuration, it taps vision layers 5, 11, and 17. Dedicated mergers compress and project each feature level, then add the resulting vectors to the corresponding hidden states of the first three language-model layers.
The vectors are added to existing states rather than appended as new context tokens, so the sequence does not become longer. The main merger supplies the final semantic representation; DeepStack supplements it with earlier visual detail. In a warehouse example, fine features may preserve a torn cardboard edge while deeper features establish that the region is a carton.
1.6 Positional Encoding Preserves Where and When Things Appear
A language model must distinguish token order. A vision-language model must additionally preserve image height, image width, and—for video—time. Qwen3-VL uses interleaved Multimodal Rotary Position Embeddings, or interleaved M-RoPE.
Qwen3-VL interleaves temporal, horizontal, and vertical axes across low- and high-frequency bands. Consequently, similar tokens at different image positions receive different phase information, allowing the decoder to learn above, inside, and left of.
M-RoPE represents image layout, not calibrated geometry. Camera intrinsics, metric depth, and robot-frame transforms still come from the surrounding system.
1.7 Autoregressive Generation Produces Words and Coordinates
Qwen3-VL generates output autoregressively:
Here, is the complete generated response of length ; is the encoded image, is the prompt, is the generation step, and denotes conditional probability.
Every token depends on the image, prompt, and previous answer tokens. For grounding, Qwen3-VL is trained to produce boxes and points with relative coordinates normalized to . A structured response can represent a box as {"bbox_2d": [x_1, y_1, x_2, y_2]}.
For an image with width and height , the corresponding pixel coordinates are approximately
Here, and are generated coordinates on the normalized 0–1000 scale; and are pixel coordinates; and and are the original image width and height.
Coordinates are generated as text tokens learned from grounding supervision; there is no final box-regression head or NMS. The output must therefore be parsed and validated.
A box remains a 2D hypothesis. It contains no verified depth, surface normal, collision state, or grasp pose. Before motion, associate it with synchronized depth or segmentation, transform the resulting geometry through calibrated frames, and apply reachability and collision checks.
1.8 The Loss Function Teaches Qwen to Predict the Next Token
Qwen3-VL is trained as an autoregressive model. During training, it is shown the correct response and asked to predict every next token from the preceding image and text context. If the correct next token is , the token-level cross-entropy loss is
Here, is the loss at output position ; is the encoded visual input; is the instruction; denotes all correct response tokens before position ; and is the probability the model assigned to the correct token.
The model minimizes a weighted aggregate of these token losses. This one objective can teach many behaviours because the target sequence can contain ordinary words, structured markup, point coordinates, or box coordinates. For grounding examples, predicting the wrong coordinate token produces loss just as predicting the wrong noun does. Qwen therefore learns localization through the language-generation objective rather than through a separate box-regression head.
The Qwen3-VL report also changes how losses from examples of different lengths are balanced. A simple per-sample average can give a short caption and a very long multimodal document similar overall influence, even though their token counts differ greatly. The report uses a square-root-normalized per-token loss to balance text-only and multimodal data more carefully. The exact weighting is an optimization choice; the mental model remains next-token prediction across a diverse multimodal mixture.
Post-training adds further objectives. Supervised fine-tuning continues to train on desired responses, teacher distillation transfers behaviour from stronger models, and reinforcement learning rewards outputs that satisfy reasoning and instruction-following criteria. These stages shape the Instruct checkpoint, but they do not add a separate detection head.
1.9 The Training Stages Align Vision and Language
Qwen3-VL begins with a 67-billion-token alignment stage that trains only the merger on captions, visual knowledge, and OCR while freezing both backbones. Full-parameter pre-training then uses about one trillion tokens at 8K context, another trillion at 32K, and 100 billion tokens for 256K-context adaptation.
Its mixture covers captions, interleaved documents, OCR, VQA, 2D/3D grounding, counting, spatial reasoning, STEM, code, GUI agents, and video. Grounding sources include COCO, Objects365, OpenImages, RefCOCO/+/g, and filtered synthetic annotations. Supervised instructions, teacher distillation, and reinforcement learning then produce the Instruct behavior.
Training breadth explains both the model’s strength and its limit. It can connect unfamiliar wording to broad visual concepts, but it may still fail on proprietary parts, unusual camera spectra, rare defects, or workcell views absent from that distribution.
1.10 How Qwen3-VL Differs From a Conventional Detector
| Question | Conventional detector | Qwen3-VL-4B-Instruct |
|---|---|---|
| Input | Image | Image plus natural-language instruction |
| Vocabulary | Usually fixed during training | Concepts and relationships expressed in text |
| Visual interface | Detector feature maps | Dynamic visual tokens projected into an LLM |
| Localization | Dedicated box head | Generated relative coordinate tokens |
| Duplicate handling | Often NMS or set prediction | Learned structured generation |
| Best fit | Repeated known classes at high rate | Attributes, relations, OCR, flexible requests |
| Main risk | Missed trained objects | Plausible but unsupported or malformed output |
A detector is usually better when the robot repeatedly tracks the same known categories and latency must be predictable. Qwen3-VL becomes valuable when the instruction contains information that a class ID loses: the open drawer, the warning label reading corrosive, or the tool beneath the red cable.
The two architectures can cooperate. Qwen can interpret the instruction and nominate a target; a detector can track it at frame rate; a segmenter can refine its boundary; and calibrated depth can recover physical geometry.
1.11 How the Qwen-VL Family Evolved
| Generation | Representative models | Main architectural change | Practical meaning |
|---|---|---|---|
| Qwen-VL | 7B, 7B-Chat | ViT-bigG plus a fixed 256-query adapter | Established unified language, OCR, and grounding |
| Qwen2-VL | 2B, 7B, 72B | Dynamic resolution, 2×2 patch merging, M-RoPE | Visual token count follows image detail |
| Qwen2.5-VL | 3B, 7B, 32B, 72B | Improved temporal modeling, localization, and structured outputs | Stronger documents, video, and agent use |
| Qwen3-VL | 2B–235B-A22B | Interleaved M-RoPE, DeepStack, explicit video timestamps | Multi-level visual fusion and long multimodal context |
The important discontinuity is between original Qwen-VL and the current model. Qwen3-VL-4B-Instruct does not use the original fixed 256-token position-aware adapter. It uses a dynamic SigLIP-2 vision encoder, spatial patch merging, and DeepStack.
1.12 Choosing Among the Qwen3-VL Model Sizes
Every Qwen3-VL size is released in an Instruct edition for direct answers and a Thinking edition for more deliberate reasoning. Dense models activate all parameters for every token. MoE models store many expert parameters but activate only a subset, written after A in the name.
| Family member | Architecture | Editions | Relative deployment position |
|---|---|---|---|
| Qwen3-VL-2B | Dense, 2B | Instruct, Thinking | Lowest resource demand; edge-oriented experiments |
| Qwen3-VL-4B | Dense, 4B | Instruct, Thinking | Telekinesis focus: compact semantic grounding |
| Qwen3-VL-8B | Dense, 8B | Instruct, Thinking | More capacity with higher memory and latency |
| Qwen3-VL-32B | Dense, 32B | Instruct, Thinking | High-quality dense deployment |
| Qwen3-VL-30B-A3B | MoE, 30B total / 3B active | Instruct, Thinking | Lower active compute with MoE serving complexity |
| Qwen3-VL-235B-A22B | MoE, 235B total / 22B active | Instruct, Thinking | Flagship quality; distributed cloud-scale serving |
Parameter count alone does not choose the best robotics model. Compare task accuracy, time to first token, total generation latency, memory, image-token budget, and failure behavior on the target hardware. The 4B Instruct edition is attractive when direct structured output and practical serving matter more than extended visible reasoning.
1.13 Engineering Trade-offs for Robotics
- Resolution versus latency: more pixels create more visual tokens and more LLM work. Set an explicit pixel budget for each camera mode.
- Semantic flexibility versus repeatability: natural language handles attributes and relationships, but paraphrases can change the result. Test a controlled prompt set.
- Long context versus fresh state: 256K tokens support rich histories, but moving robots should not reason from stale frames. Timestamp and retire old observations.
- Generation versus rigid schemas: generated coordinates are flexible, but require JSON parsing, range checks, duplicate handling, and a safe no-result path.
- 4B efficiency versus difficult reasoning: the compact model is easier to serve; larger or Thinking variants may solve harder tasks but raise latency and memory.
- Image position versus physical position: normalized 2D coordinates are not metric robot coordinates. Calibration and depth remain mandatory.
The paper’s three-module story now gives us a compact mental model. The vision encoder converts a dynamic image grid into contextual features. The MLP merger compresses and projects those features into visual tokens. The Qwen3 language model combines those tokens with the instruction and generates an answer. DeepStack strengthens the connection between the first and third modules, while interleaved M-RoPE preserves spatial and temporal position. Every implementation detail fits somewhere inside that flow.
2. How to Use Qwen3-VL-4B-Instruct in the Telekinesis Agentic OS
Telekinesis provides Qwen3-VL-4B-Instruct grounding as the Retina Skill detect_objects_using_qwen. The code below loads an image, requests three object names, and prints the typed categories, scores, and boxes returned by the Skill.
from telekinesis import datatypes, retina
image = datatypes.Image.from_url(
url="https://assets.telekinesis.ai/examples/v1/images/warehouse_1.jpg"
)
detections, categories = retina.detect_objects_using_qwen(
image=image,
objects=["person", "forklift", "pallet"],
)
for detection in detections:
label = categories[detection.category_id]
print(label.name, detection.score, detection.bbox)
The visualization shows the structured grounding result rather than free-form model prose. Check that every returned box is supported by visible evidence, especially when the requested object is absent or several similar instances are present.

Qwen3-VL-4B-Instruct uses multimodal understanding to localize requested concepts. Treat every box as a hypothesis until downstream geometry validates it.
3. Benchmarking
3.1 Results Reported for Qwen3-VL-4B
The Qwen3-VL technical report evaluates a broad multimodal model, so no single score describes it. The tables below extract the 4B results most relevant to robot perception and keep Instruct separate from Thinking. The Telekinesis Skill uses the Instruct edition.
Visual Reasoning and General VQA
| Benchmark | 4B Thinking | 4B Instruct | What it probes |
|---|---|---|---|
| MMMU | 70.8 | 67.4 | Multi-discipline image reasoning |
| MMMU-Pro | 57.0 | 53.2 | Harder, less shortcut-prone MMMU |
| MathVista mini | 79.5 | 73.7 | Visual mathematical reasoning |
| MathVision | 60.0 | 51.6 | Diagram and competition-style math |
| MMBench-EN | 84.6 | 83.9 | General visual question answering |
| RealWorldQA | 73.2 | 70.9 | Questions about real-world imagery |
| HallusionBench | 64.1 | 57.6 | Resistance to visually unsupported answers |
Thinking usually improves difficult reasoning, but it also generates more tokens and is not the edition used by the Skill.
2D/3D Grounding, Counting, and Spatial Understanding
| Benchmark | 4B Thinking | 4B Instruct | Metric / interpretation |
|---|---|---|---|
| RefCOCO average | 88.2 | 89.0 | Referring-expression grounding accuracy |
| CountBench | 89.4 | 84.9 | Counting accuracy |
| ODinW-13 | 39.4 | 48.2 | Open-vocabulary detection mAP |
| ARKitScenes | 46.3 | 56.6 | 3D grounding mAP@0.15 |
| Hypersim | 11.9 | 12.2 | 3D grounding mAP@0.15 |
| SUN RGB-D | 28.0 | 34.7 | 3D grounding mAP@0.15 |
| ERQA | 47.3 | 41.3 | Embodied reasoning |
| VSI-Bench | 55.2 | 59.3 | Visual-spatial intelligence |
| EmbSpatialBench | 80.7 | 79.6 | Embodied spatial relations |
| RefSpatialBench | 45.3 | 46.6 | Spatial referring expressions |
| RoboSpatialHome | 63.2 | 61.7 | Home-robot spatial understanding |
The mixed pattern matters. Thinking is not universally better: the 4B Instruct checkpoint scores higher on ODinW-13 and all three reported 3D grounding datasets. Benchmark the exact checkpoint and prompting mode used in production.
OCR and Document Understanding
| Benchmark | 4B Thinking | 4B Instruct |
|---|---|---|
| DocVQA test | 94.2 | 95.3 |
| InfoVQA test | 83.0 | 80.3 |
| ChartQA test | 88.8 | 84.6 |
| OCRBench | 808 | 881 |
| OCRBench v2 English | 61.8 | 63.7 |
| CC-OCR | 73.8 | 76.2 |
Video Understanding
| Benchmark | 4B Thinking | 4B Instruct |
|---|---|---|
| MVBench | 69.3 | 68.9 |
| Video-MME without subtitles | 68.9 | 69.3 |
| MLVU M-Avg | 75.7 | 75.3 |
| LVBench | 53.5 | 56.2 |
| Charades-STA mIoU | 59.0 | 55.5 |
| VideoMMMU | 69.4 | 56.2 |
These paper results establish capability, not safe robot behavior or Telekinesis service performance. Build a prompt-by-scene test set and report box success at an IoU tolerance, negative-scene hallucination rate, malformed-output rate, duplicate rate, time to first token, total latency, and final robot-task success.
3.2 Comparison with Other Telekinesis Skills
| Skill | Query | Threshold control | Strongest reason to choose it |
|---|---|---|---|
| Qwen3-VL-4B-Instruct | Requested names or concepts | No exposed tuning threshold | Attributes, relationships, OCR, and semantic flexibility |
| Grounding DINO | Names or phrases | Box and text thresholds | Detection-focused open-vocabulary grounding |
| RF-DETR | COCO class set | Score threshold | End-to-end fixed-vocabulary set prediction |
| YOLOX | COCO class set | Score and NMS thresholds | Efficient dense detection with explicit suppression control |
4. Where to Go Next?
5. References
- Qwen Team. “Qwen3-VL Technical Report”, 2025.
- Qwen. “Qwen3-VL-4B-Instruct Model Card”, 2025.
- Bai, S. et al. “Qwen2.5-VL Technical Report”, 2025.
- Wang, P. et al. “Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution”, 2024.
- Bai, J. et al. “Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond”, 2023.
- AMD. “Qwen-VL on AMD Instinct GPUs”, 2026.
- Telekinesis. “Detect Objects Using QWEN”, Retina Skill documentation.


