Object Detection · Retina

What is YOLOX?

YOLOX solves the problem of detecting and localizing known object classes quickly by using a one-stage, anchor-free object detection model. Learn the architecture of YOLOX, the loss functions it was trained with, and how to use it in the Telekinesis Agentic OS.

Imagine a mobile robot moving through a warehouse. It must notice people, boxes, and vehicles quickly enough to react. YOLOX is an object detector built for this kind of fast visual recognition, and its design is a useful way to learn how modern real-time detection works.

1. What is YOLOX?

Suppose a mobile robot enters a warehouse aisle. One camera frame may contain a person nearby, several cartons at different distances, and a forklift partly hidden at the far end. The robot needs both identity and location: what is present and where it is in the image.

YOLOX is a real-time object detector that returns boxes, class labels, and confidence scores. It belongs to the YOLO family, whose central goal is to perform detection in one fast network pipeline.

YOLOX is described as a one-stage, anchor-free detector. One-stage means it predicts candidates directly from image features instead of using one learned stage to propose regions and another to classify them. Anchor-free means it predicts box geometry without starting from a catalogue of predefined rectangle shapes. Neither term changes the vocabulary: YOLOX still recognizes the classes it learned during training.

The paper develops this pipeline through three connected ideas: anchor-free prediction simplifies where boxes come from, a decoupled head separates the jobs of classification and localization, and SimOTA decides which predictions should learn from each annotated object.

1.1 The Complete Path From an Image to Final Detections

Image → filtered detections

ImageDense RGB input
Backbone + SPPHierarchical visual features
PAN feature pyramidFuses several image scales
Anchor-free decoupled headSeparate class and box branches
Score filter + NMSRemove weak and duplicate boxes
DetectionsCOCO classes and boxes

SimOTA assigns positive locations during training; NMS resolves duplicates during inference.

YOLOX starts from a YOLOv3-style baseline and combines multi-scale features with an anchor-free, decoupled head; SimOTA governs training assignment and NMS removes duplicates at inference. Adapted as an explanatory diagram fromGe et al., “YOLOX”.

YOLOX begins by passing the camera image through a backbone. The backbone replaces pixels with feature maps that respond to useful visual patterns. An SPP block helps those features include wider scene context, while the PAN feature pyramid combines several resolutions so small and large objects can both be represented.

The detection head examines many positions on those feature maps. It is anchor-free, so it does not begin with a catalogue of preset box shapes. At each candidate position, separate branches predict whether an object exists, which trained class it belongs to, and where its box lies.

Dense prediction means neighboring positions may all describe the same physical object. Score filtering first removes weak candidates. Non-maximum suppression then keeps the strongest box and removes highly overlapping duplicates, leaving the final detections used by the robot.

SimOTA appears in the training story rather than this inference path. It decides which candidate locations should learn from each labeled object. YOLOX is therefore a set of connected design choices: multi-scale features, anchor-free prediction, separated output branches, dynamic training assignment, and explicit duplicate removal. The next sections unpack them in the same order.

1.2 The Backbone and Feature Pyramid Preserve Several Scales

Follow one camera frame from the first diagram box. The image enters the backbone and spatial-pyramid-pooling path, which turns pixels into a hierarchy of feature maps. The paper begins its experiments from YOLOv3-SPP with DarkNet53, then uses a CSPNet backbone and PAN head for the scaled YOLOX variants. Stating both matters: “YOLOX” names the method and model family, not one immutable backbone configuration.

The PAN feature pyramid combines several resolutions. This matters because cartons near the robot and a forklift at the far end occupy very different numbers of pixels. Feature-pyramid representations let small objects use detailed high-resolution features while larger objects use features with wider context.

The Decoupled head then examines many positions across those multi-scale feature maps. At each candidate position it predicts box geometry, objectness, and class evidence. “Dense” simply means that the network checks a regular grid of possible locations rather than beginning with a short list of proposed objects.

1.3 Anchor-Free Prediction Defines Where Boxes Come From

To see why anchor-free prediction matters, consider an older anchor-based detector. At every grid location, it places several template rectangles with predefined widths and aspect ratios. The network learns how to shift and resize them. If the dataset changes from people and cars to long tools and tiny fasteners, those templates may need to be reconsidered.

YOLOX removes the template catalogue. It treats a feature-map location as a possible object centre and directly predicts the distances needed to describe the box. Relative to the paper’s anchor-based baseline, this reduces the predictions at each location from three to one.

The design is simpler, but it is not assumption-free. Input resolution determines how much detail exists, feature strides determine the spatial grid, and the training assignment determines which locations learn each object. Anchor-free removes one family of hand-designed choices; it does not remove the need for sound data and geometry.

1.4 The Decoupled Head Separates Three Different Questions

Once a candidate location has been chosen, the detector must answer different questions. “What is it?” depends on appearance and semantic cues. “Where exactly does it end?” depends on boundaries and geometry. Those tasks can prefer different features.

YOLOX therefore uses a decoupled head. The branches share the feature pyramid but separate near the output: one branch predicts class evidence, another predicts box geometry, and a third predicts objectness—whether any supported object is present at that location. This reduces direct interference between classification and localization.

A simplified final score can be understood as

sclass≈P(object) P(class∣object).s_{class} \approx P(\text{object})\,P(\text{class}\mid\text{object}).

Here, sclasss_{class} is the combined score for one class, P(object)P(\text{object}) is the location’s objectness estimate, and P(class∣object)P(\text{class}\mid\text{object}) is its class score conditioned on an object being present.

This product explains why a location must look both object-like and class-consistent to receive a strong final score. It is a mental model, not a promise that the score is a calibrated probability. Thresholds still need to be validated on the robot’s own camera data.

1.5 SimOTA Decides Which Locations Should Learn Each Object

Anchor-free prediction and the decoupled head define what the network can predict, but training still needs to decide which grid locations are responsible for each annotated object. Dense prediction creates an ambiguity: many nearby locations may produce boxes around the same person. If all are treated as correct, duplicates dominate. If a poor location is chosen, the model learns from a weak example.

SimOTA resolves this in stages. A centre prior first removes implausible distant candidates. For the remaining candidates, the algorithm combines classification and regression losses into a matching cost. It then chooses a dynamic number of low-cost positives for each object. An object with several good candidates can supervise more locations than one with only a few reliable candidates.

This assignment exists only during training. It teaches the dense head which locations should become responsible for which objects. During inference, the trained network directly emits its candidates.

1.6 The Loss Function Trains Classification, Objectness, and Boxes

After SimOTA has assigned positive locations, the loss tells each branch how to improve. The YOLOX paper uses binary cross-entropy for the classification and objectness branches and IoU loss for box regression. A simplified total objective is

LYOLOX=λclsLBCEcls+λobjLBCEobj+λboxLIoU.\mathcal{L}_{YOLOX}= \lambda_{cls}\mathcal{L}_{BCE}^{cls} +\lambda_{obj}\mathcal{L}_{BCE}^{obj} +\lambda_{box}\mathcal{L}_{IoU}.

Here, LYOLOX\mathcal{L}_{YOLOX} is the total training loss; LBCEcls\mathcal{L}_{BCE}^{cls} penalizes incorrect class scores; LBCEobj\mathcal{L}_{BCE}^{obj} penalizes incorrect objectness scores; LIoU\mathcal{L}_{IoU} penalizes disagreement between predicted and target boxes; and the three λ\lambda terms weight the components.

Binary cross-entropy asks whether each class or objectness target should be active. IoU loss asks whether the predicted rectangle covers the same region as the annotation. The decoupled head gives these objectives separate branches, while joint optimization encourages the branches to agree on one useful detection.

This loss is distinct from SimOTA’s matching cost. SimOTA chooses the positive training locations. The loss updates the predictions made at those locations. Both disappear at inference time.

1.7 COCO Training Gives YOLOX Its Class Vocabulary

The paper trains the main YOLOX models from scratch on COCO train2017 for 300 epochs, with five warm-up epochs, stochastic gradient descent, a cosine learning-rate schedule, and multi-scale inputs sampled from 448 to 832 pixels. After strong augmentation was introduced, the authors found ImageNet initialization no longer beneficial for these experiments.

Training images use horizontal flips, colour jitter, multi-scale resizing, Mosaic, and MixUp. Mosaic combines transformed images into one training canvas; MixUp blends examples and their targets. Both are disabled for the last 15 epochs so the model finishes on samples closer to the real COCO distribution. Small YOLOX variants use weaker augmentation because aggressive mixing can remove too much useful signal at limited capacity.

COCO supplies 80 common object categories. A COCO-trained YOLOX can therefore detect people, vehicles, and everyday objects, but it does not automatically know a factory’s proprietary fastener or defect class. Supporting those targets requires labeled domain data and retraining or fine-tuning.

These objectives also explain common errors. High objectness with confused class scores means the model sees an object-like region but cannot identify it. A correct class with poor overlap means recognition succeeded while localization failed. Low values in every branch suggest that the feature pyramid did not preserve enough evidence or the scene differs too much from training.

1.8 Score Filtering and NMS Turn Candidates Into Final Detections

Training is now complete. At inference time, the network emits a large collection of candidates. Score filtering first removes candidates whose class and objectness evidence is too weak. Even among the remaining candidates, neighbouring grid locations can predict the same physical object.

Non-maximum suppression, or NMS, solves that duplicate problem. It sorts boxes by score, keeps the strongest, and removes weaker boxes that overlap it beyond a threshold. Overlap is measured with intersection over union:

IoU(A,B)=∣A∩B∣∣A∪B∣.IoU(A,B)=\frac{|A\cap B|}{|A\cup B|}.

Here, AA and BB are candidate box regions; A∩BA\cap B is their intersection, A∪BA\cup B is their union, and vertical bars denote region area.

A lower NMS threshold suppresses duplicates more aggressively but can erase one of two nearby parts. A higher threshold preserves crowded objects but may leave several boxes around one part. NMS finishes detection within one image; it is not tracking and cannot establish identity across frames.

Interactive post-processing

See score filtering and NMS remove candidate boxes

—
candidate cluster Acandidate cluster B

Solid boxes survive; dashed boxes are removed by the score gate or by overlap with a stronger box.

1.9 Feature Stride Sets a Practical Small-Object Limit

Each feature-pyramid level has a stride: one feature cell corresponds to several input pixels. A stride-8 map retains four times as many spatial positions as a stride-16 map across the same image area. Small targets depend heavily on these fine maps because they may occupy only a few cells at coarser resolution.

This creates a concrete camera-design question. If a fastener is 6 pixels wide after resizing, no threshold can recover detail that the input pipeline removed. Increasing input resolution may help, but it raises compute and latency. Moving the camera closer, narrowing the field of view, or cropping a known workspace may provide more useful pixels at lower cost.

Before comparing model variants, calculate the apparent pixel size of the smallest required target across the robot’s working distance. Detection architecture, optics, and camera placement form one system; a faster network cannot compensate for an object that was never adequately sampled.

1.10 Where YOLOX Can Fail

The whole story now connects. YOLOX extracts multi-scale features, predicts dense anchor-free candidates through separate output branches, learns their responsibility through SimOTA, and removes duplicates with NMS. This makes it fast and practical, but not infallible.

Small or blurred objects may not leave enough visual evidence. Unseen classes remain outside the learned vocabulary. Crowded objects can interfere during suppression, and a correct 2D box may still be too coarse for grasping. A robot should treat YOLOX as the localization stage of a larger perception system, not as a complete understanding of the scene.

2. How to Use YOLOX in the Telekinesis Agentic OS

Telekinesis provides YOLOX as the Retina Skill detect_objects_using_yolox. The code below loads an image, applies score and non-maximum-suppression thresholds, resolves the COCO category names, and prints each detection.

from telekinesis import constants, datatypes, retina

image = datatypes.Image.from_url(
    url="https://assets.telekinesis.ai/examples/v1/images/warehouse_2.jpg"
)

detections = retina.detect_objects_using_yolox(
    image=image,
    score_threshold=0.80,
    nms_threshold=0.45,
)

categories = constants.get_coco_categories(model="yolox")

for detection in detections:
    label = categories[detection.category_id]
    print(label.name, detection.score, detection.bbox)

The visualization below is the post-NMS result, not the network’s much larger dense candidate set. Compare it against crowded and overlapping scenes when selecting deployment thresholds.

YOLOX detections returned by Retina

Dense predictions become a short final list only after confidence filtering and suppression of overlapping candidates.

Try it out

Build with YOLOX in Retina

See the complete Skill signature, input types, output schema, and current examples.

Open YOLOX docs

Runnable example

Run the YOLOX example

Open the exact Python script used to detect objects with YOLOX and Retina.

View Python example

3. Benchmarking

3.1 Results Reported in the YOLOX Paper

The YOLOX paper reports both the final model family and the ablations that created it. That distinction matters: the family table tells you which size fits a device, while the ablation table explains where the accuracy came from.

Model Family: Accuracy, Compute, and Latency

Table 3 evaluates 640×640 inputs with FP16 and batch size 1 on a Tesla V100.

ModelCOCO APParametersGFLOPsLatency
YOLOX-S39.69.0M26.89.8 ms
YOLOX-M46.425.3M73.812.3 ms
YOLOX-L50.054.2M155.614.5 ms
YOLOX-X51.299.1M281.917.3 ms

The compact variants in Table 4 use 416×416 inputs:

ModelCOCO APParametersGFLOPs
YOLOv4-Tiny21.76.06M6.96
YOLOX-Tiny32.85.06M6.45
NanoDet23.50.95M1.20
YOLOX-Nano25.30.91M1.08

COCO Test-Dev Accuracy Breakdown

Table 6 reports 300-epoch models at 640×640. APS, APM, and APL split performance by small, medium, and large objects.

ModelFPSAPAP50AP75APSAPMAPL
YOLOX-DarkNet5390.147.467.352.127.551.560.9
YOLOX-M81.346.465.450.626.351.059.9
YOLOX-L69.050.068.554.529.854.564.4
YOLOX-X57.851.269.655.731.256.166.1

What the Training Choices Contributed

The paper reports that SimOTA raises the authors’ strong anchor-free baseline from 45.0 AP to 47.3 AP. Its augmentation study also shows that one recipe does not fit every model size:

ModelMosaic scale jitterExtra augmentationAP
YOLOX-Nano[0.5, 1.5]None25.3
YOLOX-Nano[0.1, 2.0]MixUp24.0
YOLOX-L[0.1, 2.0]None48.6
YOLOX-L[0.1, 2.0]MixUp49.5
YOLOX-L[0.1, 2.0]Copy-Paste49.4

The result is an engineering lesson: stronger augmentation helped the large model but hurt Nano, and latency depends on the exact hardware and implementation. Retina service performance additionally includes image transfer, decoding, preprocessing, model execution, NMS, serialization, and network latency, so benchmark the complete call with representative images.

3.2 Comparison with Other Telekinesis Skills

SkillVocabularyPost-processing controlPrefer it when
YOLOXFixed COCO-80Score + NMS thresholdsDense real-time detection and NMS control
RF-DETRFixed COCO-80Score thresholdTransformer set prediction and global context
Grounding DINOOpen phrasesBox + text thresholdsCategories change at inference time
QwenRequested namesObject listBroader semantic interpretation is needed

3.3 YOLOX, RF-DETR, and Open-Vocabulary Detectors

YOLOX is a strong choice when the fixed COCO vocabulary fits, latency matters, and explicit NMS tuning is useful. RF-DETR offers transformer set prediction and stronger global interaction. Grounding DINO accepts changing text prompts, but prompt design and open-vocabulary calibration become new variables.

Benchmark them as complete pipelines. Model inference speed alone omits image transfer, decoding, preprocessing, post-processing, and network latency. Record median and tail latency, detection quality by scenario, and final robot task success.

4. Where to Go Next?

5. References