Point Cloud Processing · Vitreous

What is RANSAC Plane Segmentation?

RANSAC plane segmentation finds a dominant flat surface in a noisy point cloud without letting every outlier influence the fit. Learn how plane hypotheses, consensus sets, distance thresholds, and iteration counts work, then use the method with Vitreous.

Imagine an industrial arm looking into a tote that rests on a workbench. Its depth camera sees the parts, the bench, the tote walls, and scattered sensor noise as one point cloud. Before the robot can reason about the parts, it needs a reliable way to identify the large flat work surface and either keep it or remove it. A simple least-squares plane is easily pulled toward the objects. RANSAC is designed for exactly this situation: fit a model even when many measurements do not belong to it.

Input

Output

The input contains several kinds of geometry. The output isolates the points that agree with the strongest plane hypothesis. Media: Telekinesis Skill documentation.

1. What is RANSAC Plane Segmentation?

RANSAC plane segmentation solves the problem of finding a dominant plane in a point cloud that also contains objects, missing measurements, and noise. RANSAC stands for Random Sample Consensus. Instead of fitting every point at once, it repeatedly chooses a small random sample, builds a possible plane, and asks how many other points agree with it.

That reversal is the key idea. Ordinary least squares assumes that most measurements should influence one model. RANSAC assumes that a useful model is hidden inside a mixture of inliers, which follow the model, and outliers, which do not. A tabletop may contribute thousands of mutually consistent points even though tools and parts rise above it. If three sampled points happen to lie on that table, their plane will receive a large consensus vote from the rest of the table.

The output has two parts. The segmented point cloud contains either the plane inliers or everything except those inliers, depending on keep_outliers. The plane model contains four coefficients that describe the plane mathematically. Together, they let a robot remove a floor, isolate a work surface, split space above and below a shelf, or recover a surface normal for alignment.

1.1 The Complete RANSAC Plane Pipeline

3D samples → dominant plane and inliers

Point cloudSurface samples, objects, and noise
Draw a minimal sampleChoose three non-collinear points
Fit a plane hypothesisCompute coefficients a, b, c, and d
Measure consensusCount points within the distance threshold
Keep the strongest modelRepeat and retain the largest support set
Plane segmentationInlier cloud plus plane equation

RANSAC succeeds by testing many small hypotheses, so outliers do not participate in every fit.

RANSAC repeatedly fits a model from the minimum number of samples and scores that model against the full dataset. Plane segmentation uses three non-collinear points per minimal hypothesis. Adapted as an explanatory diagram fromFischler and Bolles, “Random Sample Consensus”.

Begin with the entire point cloud. Most scenes contain far more than one surface, so the algorithm first creates a deliberately small hypothesis: it randomly selects three points. Three non-collinear points are the minimum needed to define one plane in 3D.

Those three points produce a candidate plane. The algorithm then returns to the full cloud and measures the perpendicular distance from every point to that candidate. Points closer than the chosen threshold form the candidate’s consensus set. A plane passing through three points on a carton may explain only a small patch. A plane passing through the workbench may explain thousands of points.

RANSAC repeats this experiment many times and retains the hypothesis with the strongest support. The result is not “the plane” in an absolute semantic sense; it is the largest planar structure found under the selected threshold and sampling budget. This distinction matters when a wall is larger than the table the robot actually cares about.

1.2 How Three Points Define a Plane

A plane is represented by

ax+by+cz+d=0.ax+by+cz+d=0.

Here, (x,y,z)(x,y,z) is any point on the plane; aa, bb, and cc are the components of a vector perpendicular to the plane; and dd moves the plane along that normal direction. The returned model is commonly written as [a,b,c,d][a,b,c,d].

Suppose the random sample contains points p1\mathbf p_1, p2\mathbf p_2, and p3\mathbf p_3. Two directions lying within the candidate plane are

u=p2−p1,v=p3−p1.\mathbf u=\mathbf p_2-\mathbf p_1, \qquad \mathbf v=\mathbf p_3-\mathbf p_1.

u\mathbf u and v\mathbf v are vectors between sampled points. Their cross product gives a normal:

n=u×v=[abc]T.\mathbf n=\mathbf u\times\mathbf v =\begin{bmatrix}a&b&c\end{bmatrix}^{\mathsf T}.

n\mathbf n is perpendicular to both directions and therefore to the candidate plane. The remaining coefficient is d=−nTp1d=-\mathbf n^{\mathsf T}\mathbf p_1. If the three points are collinear or almost collinear, ∥n∥\|\mathbf n\| is zero or unstable, so that sample cannot define a reliable plane.

1.3 How a Point Votes for a Hypothesis

RANSAC does not ask whether a point lies exactly on the plane because depth measurements contain noise. It measures perpendicular distance:

ri=∣axi+byi+czi+d∣a2+b2+c2.r_i=\frac{|a x_i+b y_i+c z_i+d|}{\sqrt{a^2+b^2+c^2}}.

rir_i is the residual distance of point ii; (xi,yi,zi)(x_i,y_i,z_i) is that point’s position; and aa, bb, cc, and dd describe the candidate plane. The denominator removes any arbitrary scaling of the plane coefficients.

Point ii becomes an inlier when ri≤τr_i\le \tau, where τ\tau is distance_threshold. The consensus set is therefore

I(θ)={i:ri(θ)≤τ}.\mathcal I(\theta)=\{i:r_i(\theta)\le\tau\}.

θ=[a,b,c,d]\theta=[a,b,c,d] denotes the current plane model, and I(θ)\mathcal I(\theta) is the set of indices supporting it. Basic RANSAC prefers the model with the largest ∣I∣|\mathcal I|.

Interactive plane consensus

Change the distance threshold and watch the support set change

—

This side view turns a 3D plane into a line so perpendicular residuals are easy to see. The actual Skill fits a plane in three dimensions.

The threshold expresses what “flat enough” means in the cloud’s units. If the camera reports metres, 0.005 means five millimetres. A threshold that is too small breaks one physical surface into a thin, incomplete subset. A threshold that is too large starts swallowing parts sitting on the surface. Tune it from measured depth noise and the smallest height difference the task must preserve.

1.4 Why Repetition Makes Random Sampling Reliable

One random sample is a gamble. Repetition turns it into a controlled probability. Let ww be the fraction of points that belong to the desired plane and let ss be the number of points in one minimal sample. For a plane, s=3s=3. The probability that one sample contains only inliers is wsw^s.

After NN independent trials, the probability of having drawn at least one all-inlier sample is

p=1−(1−ws)N.p=1-(1-w^s)^N.

pp is the desired success probability, ww is the expected inlier fraction, ss is the sample size, and NN is the number of trials. Rearranging gives

N≥log⁡(1−p)log⁡(1−ws).N\ge\frac{\log(1-p)}{\log(1-w^s)}.

This equation explains why max_iterations depends on the scene. If the plane occupies most of the cloud, an all-inlier sample appears quickly. If it occupies only a small fraction, the required trial count rises sharply. More iterations improve the chance of a useful sample but increase compute; they cannot compensate for a threshold that defines the wrong consensus.

1.5 What the Plane Coefficients Give the Robot

The first three coefficients define the normal direction. A unit normal is

n^=[a,b,c]Ta2+b2+c2.\hat{\mathbf n}=\frac{[a,b,c]^{\mathsf T}}{\sqrt{a^2+b^2+c^2}}.

n^\hat{\mathbf n} has length one and describes the plane’s orientation. It can help a cobot align a tool with a work surface or help an AMR distinguish horizontal ground from vertical walls. Its sign is ambiguous: [a,b,c,d][a,b,c,d] and [−a,−b,−c,−d][-a,-b,-c,-d] describe the same plane. If the application needs the normal to point toward the camera or upward in the world, orient it using that external convention.

The signed plane expression also divides space into two half-spaces. After normalization, ax+by+cz+d>0a x+b y+c z+d>0 lies on one side and a negative value lies on the other. That supports rules such as “keep points at least 10 mm above the workbench.” It remains essential to know the point cloud’s coordinate frame; an equation in the camera frame is not automatically meaningful in the robot base frame.

1.6 Failure Modes to Recognize

RANSAC is robust to outliers, not omniscient. It can return a mathematically strong but operationally wrong plane.

  • If a wall contains more points than the table, the wall may win because this Skill returns the largest plane.
  • If the surface is curved, a plane is the wrong model; a loose threshold merely hides the mismatch.
  • If several planes have similar support, random trials can select different winners across images.
  • If one plane is visible only in a narrow strip, its orientation may be poorly constrained even with many points.
  • If the cloud mixes frames or has incorrect scale, the distance threshold has no stable physical meaning.

Use priors when they are available. Crop to the expected workspace, reject plane normals that are inconsistent with gravity, and check that the plane height lies near the fixture. RANSAC supplies a geometric hypothesis; the robot system decides whether that hypothesis is plausible.

1.7 One Plane, Several Planes, or a Refined Plane?

The Skill returns the single largest plane. To recover several surfaces, use a detect–remove–repeat pipeline. Segment the strongest plane, remove its inliers, and run RANSAC again on the remaining cloud. A warehouse AMR might recover the floor first and then a wall; an industrial arm might recover a fixture base and then an angled inspection surface.

Stopping needs an explicit rule. End the loop when the remaining cloud is too small, when the next consensus set falls below a minimum support size, or when its normal and position are outside the surfaces relevant to the task. Without such a rule, random fragments can eventually become the “largest” plane in a tiny remainder.

RANSAC also separates model discovery from parameter refinement. The minimal three-point sample is good for proposing a plane without letting outliers dominate, but it is not the most precise estimate available. Once the best consensus set has been identified, a common refinement fits a plane to all of those inliers using least squares or singular value decomposition. RANSAC decides which points deserve influence; the refinement uses that cleaner set to reduce noise. When comparing implementations, check whether this final refit is performed rather than assuming the three sampled points are the final numerical model.

2. How to Use RANSAC Plane Segmentation in the Telekinesis Agentic OS

The Vitreous Skill returns the selected cloud and its plane model. Set keep_outliers=False to keep the dominant plane, or True to remove it and keep the remaining geometry.

from telekinesis import datatypes, vitreous

point_cloud = datatypes.PointCloud.from_url(
    url=(
        "https://assets.telekinesis.ai/examples/v1/point_clouds/"
        "can_vertical_3_downsampled.ply"
    ),
    use_cache=True,
)

plane_points, plane_model = vitreous.segment_point_cloud_using_plane(
    point_cloud=point_cloud,
    distance_threshold=0.01,
    num_initial_points=3,
    max_iterations=1000,
    keep_outliers=False,
)

print("plane [a, b, c, d]:", plane_model.data)
print("plane points:", plane_points.positions.shape[0])

For object isolation, call the same Skill with keep_outliers=True, then cluster what remains. Keep the parameters in the units of the input cloud and validate the output visually before using the plane in motion planning.

Try it out

Build with RANSAC plane segmentation in Vitreous

See the complete Skill signature, plane-model output, parameter guidance, and current example.

Open plane segmentation docs

Runnable example

Run the plane segmentation example

Open the exact Telekinesis Python example for dominant-plane segmentation.

View Python example

3. How to Evaluate Plane Segmentation

The original RANSAC paper introduced a general robust-estimation strategy rather than a universal point-cloud benchmark. For a robot deployment, measure the behavior that matters in the actual workspace:

MetricWhat it revealsPractical check
Inlier precisionWhether objects are being absorbed into the planeLabel plane and non-plane points on representative scans
Inlier recallWhether the physical surface is recovered completelyCheck edges, distant areas, and noisy depth regions
Normal errorWhether plane orientation is accurateCompare with a calibrated reference normal
Offset errorWhether plane position is accurateMeasure signed distance to a reference surface
StabilityWhether repeated scans return the same modelTrack normal and offset variance over time
RuntimeWhether the iteration budget meets the control rateReport median and tail latency at deployed point counts

Test empty surfaces, cluttered surfaces, occlusion, multiple competing planes, and objects whose lowest face nearly touches the plane. A segmentation is successful when it preserves the geometry needed by the next robot operation—not merely when it looks flat in a viewer.

4. Where to Go Next

5. References