What is BiRefNet?

Imagine placing a complex assembly in front of a camera and asking for a clean cut-out. The object has thin wires, holes, and sharp edges, and nobody wants to draw a box or tune a color range for every image. BiRefNet is designed to find the main foreground and reconstruct those difficult boundaries automatically.

1. What is BiRefNet?

BiRefNet is a learned foreground-segmentation model for dividing a high-resolution image into foreground and background. The paper calls this dichotomous image segmentation (DIS): every pixel belongs to one of two sides, but the foreground boundary may be extremely detailed.

BiRefNet answers a different question from semantic or instance segmentation. It does not ask for every screw as separate objects, and it does not accept a text prompt specifying red screw. It asks which pixels form the visually salient foreground subject. The output is one binary mask.

That prompt-free behavior is convenient for product imaging, background removal, and workcells where one part is deliberately presented against a background. It becomes ambiguous when several plausible subjects compete for attention or when the intended target is not visually salient.

1.1 The Complete BiRefNet Architecture

Image → high-resolution foreground mask

ImageHigh-resolution RGB scene

Backbone hierarchyMulti-level semantic image features

Localization moduleFind the global foreground region

Bilateral referencesSource patches + target gradients

Reconstruction moduleRecover fine boundaries progressively

Foreground maskOne foreground/background label map

Global semantics answer “what is foreground?” while image patches and gradient supervision restore the boundary detail lost during downsampling.

BiRefNet has to solve two different questions: which object is the foreground, and where exactly does that object end? A network that concentrates only on whole-image meaning may identify the subject correctly but return a soft or incomplete boundary. A network that concentrates only on fine texture may preserve edges without knowing which ones belong to the subject.

The backbone first turns the image into several layers of features. Deeper features see more of the scene and carry stronger semantic meaning, while earlier features preserve finer spatial detail. The localization module uses the high-level view to establish the broad foreground region.

The reconstruction module then works back toward a full-resolution mask. BiRefNet’s bilateral reference mechanism gives this reconstruction two forms of guidance: patches from the source image help recover local detail, while target-gradient information teaches the model to pay attention to the boundary structure expected in the mask.

The output is therefore built in a deliberate order: find the subject globally, then reconstruct its edge locally. The next sections follow the feature hierarchy and show how the two reference signals help bridge semantics and detail.

1.2 The Backbone Builds a Hierarchy of Image Features

The input passes through a feature backbone, producing several resolutions. Early levels preserve local edges and fine texture. Later levels have wider receptive fields and stronger semantic information. Downsampling makes the deep representation computationally manageable, but a one-pixel wire can vanish when the spatial grid becomes coarse.

The architecture therefore does not rely only on the deepest feature map. Its decoder receives a hierarchy. Coarse features answer “where is the foreground object?”, while finer features help recover “where exactly does its edge run?”

1.3 The Localization Module Finds the Foreground Globally

The localization module works with high-level semantics. Its role is to keep reconstruction focused on the foreground rather than spending equal effort on every background edge. In a photo of a screw on a work surface, both the screw threads and wood grain contain strong local gradients. Global context distinguishes the likely subject from irrelevant texture.

This stage is why BiRefNet is more than edge detection. An edge detector responds wherever image intensity changes. The localization module predicts which region should form the positive side of the final foreground/background split.

1.4 Source Reference Brings Original Image Detail Back

Repeated downsampling discards spatial detail. The inward, or source, reference counters that loss by bringing hierarchical patches from the original image into reconstruction. Local image patches preserve evidence such as hair, thin structures, notches, and narrow gaps that may no longer be explicit in the deepest feature representation.

The model does not simply paste raw pixels into the output. It embeds aligned patches and combines them with decoder features so reconstruction can use high-frequency evidence in the context of the localized foreground.

1.5 Target Reference Teaches the Decoder About Boundaries

The second side of bilateral reference points toward the desired output. BiRefNet derives gradient maps from ground-truth masks during training. These target references emphasize where foreground labels change sharply from 0 to 1.

For a binary ground-truth mask $G$, a simple gradient magnitude can be pictured as

$$E=\sqrt{(\partial_x G)^2+(\partial_y G)^2}.$$

Here, $\partial_x G$ and $\partial_y G$ are horizontal and vertical mask derivatives, and $E$ is strong near the object boundary. Auxiliary gradient supervision pushes reconstruction features to retain boundary-sensitive information.

Together, the references are bilateral: source patches say what fine evidence exists in the image, while target gradients teach what kind of boundary evidence the output requires.

Interactive boundary reconstruction

Blend coarse semantics with fine image detail

—

Fine-detail contribution

65%

Global semantic features identify the foreground. Fine image evidence then restores indentations and thin boundary structure.

1.6 The Reconstruction Module Produces the Final Mask

The reconstruction module progressively combines localization, hierarchical backbone features, and bilateral references as it returns to higher spatial resolution. Its output is a probability map. Telekinesis binarizes that map using mask_threshold.

If the raw value at pixel $i$ is $s_i\in[0,255]$ and the threshold is $\tau$, then

$$M_i=\mathbb 1[s_i>\tau].$$

$M_i$ is the final binary label and $\mathbb 1[\cdot]$ is one when its condition is true. A higher threshold rejects weak foreground probabilities and can reduce background bleed, but it may also remove uncertain fine structures.

1.7 The Loss Function Balances Pixels, Regions, and Structure

BiRefNet uses a hybrid objective rather than relying on one error measure:

$$\mathcal L= \lambda_{BCE}\mathcal L_{BCE}+ \lambda_{IoU}\mathcal L_{IoU}+ \lambda_{SSIM}\mathcal L_{SSIM}+ \lambda_{CE}\mathcal L_{CE}.$$

$\mathcal L_{BCE}$ is binary cross-entropy over foreground labels; $\mathcal L_{IoU}$ rewards whole-region overlap; $\mathcal L_{SSIM}$ encourages local structural consistency; and $\mathcal L_{CE}$ supervises the auxiliary classification used by the localization path. Each $\lambda$ weights one objective.

These losses complement one another. Pixel-wise BCE supplies dense learning signals. IoU discourages a prediction that is locally plausible but has the wrong total shape. SSIM is sensitive to local structure and contours. Auxiliary supervision helps the network first establish what kind of foreground it is reconstructing.

1.8 What BiRefNet Was Trained On

The original work studies high-resolution DIS and evaluates generalization across four related binary segmentation tasks: dichotomous image segmentation, high-resolution salient object detection, camouflaged object detection, and general salient object detection. DIS5K is central to its fine-detail setting and organizes difficult images by structural complexity.

Telekinesis intentionally abstracts the deployed checkpoint. The research architecture and datasets explain the model family, but they should not be read as a guarantee that every hosted artifact uses exactly one paper configuration. Deployment behavior must be measured on the target imagery.

1.9 Where BiRefNet Works—and Where It Does Not

BiRefNet is strongest when an image has a meaningful foreground/background composition and boundary detail matters. It needs no prompt, box, or manually selected color range. That simplicity is also its limitation: “foreground” is inferred from learned visual salience rather than specified by the user.

If three equally prominent parts appear on a bench, they may merge into one mask or one may dominate. If the robot needs each instance separately, SAM 3 is a better conceptual fit. If a distinctive color under fixed lighting already separates the part, RGB/HSV/LAB thresholding is cheaper and easier to audit.

1.10 Foreground Segmentation Is Not Instance Segmentation

Suppose five screws touch one another. BiRefNet may correctly mark every screw pixel as foreground, yet the output remains one binary layer. Nothing in that label map says which pixels belong to screw one or screw two. Connected components can separate objects only when their foreground regions do not touch.

This distinction affects robot planning. Background removal is enough for measuring the silhouette of one presented part. Bin picking needs identities, occlusion reasoning, and often per-instance masks. A high-quality binary foreground mask should not be described as instance understanding merely because several objects happen to be visible.

1.11 Why High Resolution Creates a Memory Trade-off

Fine masks need fine feature grids, but attention and convolution on large images consume memory and time. Downsampling makes inference cheaper while erasing the narrow structures DIS cares about. BiRefNet’s reconstruction design tries to recover detail efficiently, but the input still needs enough pixels to represent the feature.

A one-pixel wire cannot be reconstructed reliably after camera blur has removed it. Choose optics, focus, distance, and resolution so the smallest required structure spans several pixels. Then benchmark inference at that resolution rather than quoting a score or latency from a different input size.

1.12 Connect the Mask to a Robotics Pipeline

A foreground mask can filter a registered depth image, producing a point cloud for the visible subject. From that cloud, the system may estimate a centroid, oriented bounding box, surface normal, or grasp candidates. Every step assumes RGB and depth are synchronized and geometrically aligned.

Mask errors have three-dimensional consequences. Background leakage can introduce a table plane into the object cloud. Missing thin structures can change principal axes. A smooth-looking 2D output should therefore be evaluated by the downstream geometry it creates.

Use a safe empty-mask path as well. If no reliable foreground exists, the system should reobserve or stop rather than calculate a centroid from zero pixels or reuse a stale result.

2. How to Use BiRefNet in the Telekinesis Agentic OS

Telekinesis provides prompt-free foreground extraction as the Cornea Skill segment_image_foreground_using_birefnet. The code loads an RGB image and returns one foreground/background label map.

from telekinesis import cornea, datatypes

image = datatypes.Image.from_url(
    url="https://assets.telekinesis.ai/examples/v1/images/screws_standing.jpg"
)

mask = cornea.segment_image_foreground_using_birefnet(
    image=image,
    mask_threshold=0,
)

print(mask.shape, mask.label_codes, mask.number_of_labels)

Start with the documented default threshold. Raise it only after inspecting whether low-confidence background pixels are leaking into the mask.

BiRefNet returns one foreground label map, not separate identities for every visible object.

Try it out

Build with BiRefNet in Cornea

See the complete Skill signature, threshold behavior, output schema, and current examples.

Open BiRefNet docs

Runnable example

Run the BiRefNet foreground example

Open the exact Python script used to extract a foreground mask with BiRefNet and Cornea.

View Python example

3. Benchmarking

The BiRefNet paper evaluates dichotomous image segmentation (DIS), salient-object detection, and concealed-object detection. Higher is better for F-measure, S-measure, and E-measure; lower is better for mean absolute error (MAE) and human-correction effort (HCE).

3.1 Multi-Stage Supervision and Component Ablations

The first ablation asks whether supervision at several decoder stages can shorten training without losing much accuracy on DIS-VD.

Multi-stage supervision	Epochs	Max F ↑	Weighted F ↑	MAE ↓	S-measure ↑	Mean E ↑	HCE ↓
No	200	.875	.848	.041	.886	.914	1207
No	400	.897	.863	.036	.905	.937	1039
Yes	200	.892	.858	.037	.901	.932	1043

The practical strategies in Table 3 are complementarity-aware feature fusion (CFF), intermediate prediction targets (IPT), and reconstruction-guided local fine-tuning (RLFT):

CFF	IPT	RLFT	Max F ↑	Weighted F ↑	MAE ↓	S-measure ↑	Mean E ↑	HCE ↓
No	No	No	.889	.851	.038	.900	.924	1065
Yes	No	No	.893	.856	.038	.904	.928	1054
No	Yes	No	.895	.857	.037	.904	.927	1051
No	No	Yes	.890	.861	.036	.899	.932	1043
Yes	Yes	Yes	.897	.863	.036	.905	.937	1039

3.2 Concealed-Object Detection

The paper’s Table 6 reports BiRefNet on three concealed-object datasets. The compact table retains the principal structure, F, E, and MAE metrics.

Dataset	S-measure ↑	Weighted F ↑	Mean F ↑	Mean E ↑	Max E ↑	MAE ↓
CAMO	.904	.890	.904	.954	.959	.030
COD10K	.913	.874	.888	.960	.967	.014
NC4K	.914	.894	.909	.953	.960	.023

3.3 Accuracy–Efficiency Trade-offs

All runtime measurements in Table 7 use one NVIDIA A100. The paper reports parameter storage in MB and multiply-accumulate operations at 1024×1024 input.

Variant	Runtime	Parameter storage	MACs	DIS-TE HCE ↓	DIS-TE weighted F ↑
BiRefNet-SwinL	83.3 ms	215 MB	1,143G	916	.858
BiRefNet-SwinB	61.4 ms	101 MB	561G	954	.855
BiRefNet-SwinT	40.9 ms	39 MB	231G	980	.822
BiRefNet-PVTv2-B2	47.8 ms	35 MB	195G	969	.838
BiRefNet-PVTv2-B0	32.9 ms	11 MB	89G	1013	.806
IS-Net	16.0 ms	44 MB	160G	1016	.726
UDUN-Res50	33.5 ms	25 MB	142G	977	.772

These scores capture different properties and should not be collapsed into one universal rank. For robotics, also report mask IoU, boundary F-score, thin-structure recall, false foreground area, end-to-end latency, and downstream measurement or grasp success. Test cluttered and multi-object scenes explicitly, because a high-quality cut-out benchmark does not prove the model will choose the operator’s intended target.

4. Where to Go Next?

5. References

• Zheng, P. et al. “Bilateral Reference for High-Resolution Dichotomous Image Segmentation”, CAAI Artificial Intelligence Research, 2024.
• Qin, X. et al. “Highly Accurate Dichotomous Image Segmentation”, ECCV 2022.
• Wang, Z. et al. “Image Quality Assessment: From Error Visibility to Structural Similarity”, IEEE Transactions on Image Processing, 2004.
• Telekinesis. “Segment Image Using Foreground BiRefNet”, Cornea Skill documentation.