Imagine an industrial arm looking into a tote that rests on a workbench. Its depth camera sees the parts, the bench, the tote walls, and scattered sensor noise as one point cloud. Before the robot can reason about the parts, it needs a reliable way to identify the large flat work surface and either keep it or remove it. A simple least-squares plane is easily pulled toward the objects. RANSAC is designed for exactly this situation: fit a model even when many measurements do not belong to it.
Input
Output
1. What is RANSAC Plane Segmentation?
RANSAC plane segmentation solves the problem of finding a dominant plane in a point cloud that also contains objects, missing measurements, and noise. RANSAC stands for Random Sample Consensus. Instead of fitting every point at once, it repeatedly chooses a small random sample, builds a possible plane, and asks how many other points agree with it.
That reversal is the key idea. Ordinary least squares assumes that most measurements should influence one model. RANSAC assumes that a useful model is hidden inside a mixture of inliers, which follow the model, and outliers, which do not. A tabletop may contribute thousands of mutually consistent points even though tools and parts rise above it. If three sampled points happen to lie on that table, their plane will receive a large consensus vote from the rest of the table.
The output has two parts. The segmented point cloud contains either the plane inliers or everything except those inliers, depending on keep_outliers. The plane model contains four coefficients that describe the plane mathematically. Together, they let a robot remove a floor, isolate a work surface, split space above and below a shelf, or recover a surface normal for alignment.
1.1 The Complete RANSAC Plane Pipeline
3D samples → dominant plane and inliers
RANSAC succeeds by testing many small hypotheses, so outliers do not participate in every fit.
Begin with the entire point cloud. Most scenes contain far more than one surface, so the algorithm first creates a deliberately small hypothesis: it randomly selects three points. Three non-collinear points are the minimum needed to define one plane in 3D.
Those three points produce a candidate plane. The algorithm then returns to the full cloud and measures the perpendicular distance from every point to that candidate. Points closer than the chosen threshold form the candidate’s consensus set. A plane passing through three points on a carton may explain only a small patch. A plane passing through the workbench may explain thousands of points.
RANSAC repeats this experiment many times and retains the hypothesis with the strongest support. The result is not “the plane” in an absolute semantic sense; it is the largest planar structure found under the selected threshold and sampling budget. This distinction matters when a wall is larger than the table the robot actually cares about.
1.2 How Three Points Define a Plane
A plane is represented by
Here, is any point on the plane; , , and are the components of a vector perpendicular to the plane; and moves the plane along that normal direction. The returned model is commonly written as .
Suppose the random sample contains points , , and . Two directions lying within the candidate plane are
and are vectors between sampled points. Their cross product gives a normal:
is perpendicular to both directions and therefore to the candidate plane. The remaining coefficient is . If the three points are collinear or almost collinear, is zero or unstable, so that sample cannot define a reliable plane.
1.3 How a Point Votes for a Hypothesis
RANSAC does not ask whether a point lies exactly on the plane because depth measurements contain noise. It measures perpendicular distance:
is the residual distance of point ; is that point’s position; and , , , and describe the candidate plane. The denominator removes any arbitrary scaling of the plane coefficients.
Point becomes an inlier when , where is distance_threshold. The consensus set is therefore
denotes the current plane model, and is the set of indices supporting it. Basic RANSAC prefers the model with the largest .
Interactive plane consensus
Change the distance threshold and watch the support set change
The threshold expresses what “flat enough” means in the cloud’s units. If the camera reports metres, 0.005 means five millimetres. A threshold that is too small breaks one physical surface into a thin, incomplete subset. A threshold that is too large starts swallowing parts sitting on the surface. Tune it from measured depth noise and the smallest height difference the task must preserve.
1.4 Why Repetition Makes Random Sampling Reliable
One random sample is a gamble. Repetition turns it into a controlled probability. Let be the fraction of points that belong to the desired plane and let be the number of points in one minimal sample. For a plane, . The probability that one sample contains only inliers is .
After independent trials, the probability of having drawn at least one all-inlier sample is
is the desired success probability, is the expected inlier fraction, is the sample size, and is the number of trials. Rearranging gives
This equation explains why max_iterations depends on the scene. If the plane occupies most of the cloud, an all-inlier sample appears quickly. If it occupies only a small fraction, the required trial count rises sharply. More iterations improve the chance of a useful sample but increase compute; they cannot compensate for a threshold that defines the wrong consensus.
1.5 What the Plane Coefficients Give the Robot
The first three coefficients define the normal direction. A unit normal is
has length one and describes the plane’s orientation. It can help a cobot align a tool with a work surface or help an AMR distinguish horizontal ground from vertical walls. Its sign is ambiguous: and describe the same plane. If the application needs the normal to point toward the camera or upward in the world, orient it using that external convention.
The signed plane expression also divides space into two half-spaces. After normalization, lies on one side and a negative value lies on the other. That supports rules such as “keep points at least 10 mm above the workbench.” It remains essential to know the point cloud’s coordinate frame; an equation in the camera frame is not automatically meaningful in the robot base frame.
1.6 Failure Modes to Recognize
RANSAC is robust to outliers, not omniscient. It can return a mathematically strong but operationally wrong plane.
- If a wall contains more points than the table, the wall may win because this Skill returns the largest plane.
- If the surface is curved, a plane is the wrong model; a loose threshold merely hides the mismatch.
- If several planes have similar support, random trials can select different winners across images.
- If one plane is visible only in a narrow strip, its orientation may be poorly constrained even with many points.
- If the cloud mixes frames or has incorrect scale, the distance threshold has no stable physical meaning.
Use priors when they are available. Crop to the expected workspace, reject plane normals that are inconsistent with gravity, and check that the plane height lies near the fixture. RANSAC supplies a geometric hypothesis; the robot system decides whether that hypothesis is plausible.
1.7 One Plane, Several Planes, or a Refined Plane?
The Skill returns the single largest plane. To recover several surfaces, use a detect–remove–repeat pipeline. Segment the strongest plane, remove its inliers, and run RANSAC again on the remaining cloud. A warehouse AMR might recover the floor first and then a wall; an industrial arm might recover a fixture base and then an angled inspection surface.
Stopping needs an explicit rule. End the loop when the remaining cloud is too small, when the next consensus set falls below a minimum support size, or when its normal and position are outside the surfaces relevant to the task. Without such a rule, random fragments can eventually become the “largest” plane in a tiny remainder.
RANSAC also separates model discovery from parameter refinement. The minimal three-point sample is good for proposing a plane without letting outliers dominate, but it is not the most precise estimate available. Once the best consensus set has been identified, a common refinement fits a plane to all of those inliers using least squares or singular value decomposition. RANSAC decides which points deserve influence; the refinement uses that cleaner set to reduce noise. When comparing implementations, check whether this final refit is performed rather than assuming the three sampled points are the final numerical model.
2. How to Use RANSAC Plane Segmentation in the Telekinesis Agentic OS
The Vitreous Skill returns the selected cloud and its plane model. Set keep_outliers=False to keep the dominant plane, or True to remove it and keep the remaining geometry.
from telekinesis import datatypes, vitreous
point_cloud = datatypes.PointCloud.from_url(
url=(
"https://assets.telekinesis.ai/examples/v1/point_clouds/"
"can_vertical_3_downsampled.ply"
),
use_cache=True,
)
plane_points, plane_model = vitreous.segment_point_cloud_using_plane(
point_cloud=point_cloud,
distance_threshold=0.01,
num_initial_points=3,
max_iterations=1000,
keep_outliers=False,
)
print("plane [a, b, c, d]:", plane_model.data)
print("plane points:", plane_points.positions.shape[0])
For object isolation, call the same Skill with keep_outliers=True, then cluster what remains. Keep the parameters in the units of the input cloud and validate the output visually before using the plane in motion planning.
3. How to Evaluate Plane Segmentation
The original RANSAC paper introduced a general robust-estimation strategy rather than a universal point-cloud benchmark. For a robot deployment, measure the behavior that matters in the actual workspace:
| Metric | What it reveals | Practical check |
|---|---|---|
| Inlier precision | Whether objects are being absorbed into the plane | Label plane and non-plane points on representative scans |
| Inlier recall | Whether the physical surface is recovered completely | Check edges, distant areas, and noisy depth regions |
| Normal error | Whether plane orientation is accurate | Compare with a calibrated reference normal |
| Offset error | Whether plane position is accurate | Measure signed distance to a reference surface |
| Stability | Whether repeated scans return the same model | Track normal and offset variance over time |
| Runtime | Whether the iteration budget meets the control rate | Report median and tail latency at deployed point counts |
Test empty surfaces, cluttered surfaces, occlusion, multiple competing planes, and objects whose lowest face nearly touches the plane. A segmentation is successful when it preserves the geometry needed by the next robot operation—not merely when it looks flat in a viewer.
4. Where to Go Next
5. References
- Fischler, M. A., and Bolles, R. C. “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography”, Communications of the ACM, 1981.
- Telekinesis. “Segment Point Cloud Using Plane”, Vitreous Skill documentation.