Image & Point Cloud Fundamentals · Fundamentals

What is a Point Cloud?

A point cloud is a collection of sampled positions in three-dimensional space. Learn how points represent visible geometry, why coordinate frames matter, and how depth images become spatial data for robots.

An image tells a robot where something appears in the camera view. A point cloud goes further: it samples where visible surfaces lie in three-dimensional space.

1. What is a Point Cloud?

A point cloud is a collection of points whose coordinates sample surfaces in 3D space. Each point usually stores a position and may also carry color, intensity, confidence, or a semantic label.

A point cloud represented in a three-dimensional coordinate frame

A basic point cloud can be written as

P={pi}i=1N,pi=(xi,yi,zi).\mathcal{P}=\{\mathbf{p}_i\}_{i=1}^{N}, \qquad \mathbf{p}_i=(x_i,y_i,z_i).

P\mathcal{P} is the complete point cloud, NN is the number of points, ii identifies one sample, and xix_i, yiy_i, and ziz_i are that point’s coordinates.

Unlike pixels, points do not have to form a rectangular grid. They can be irregularly spaced, and their density normally changes with distance and viewing angle.

1.1 Every Point Needs a Coordinate Frame

The coordinates (0.2,−0.1,1.4)(0.2,-0.1,1.4) are meaningless without a frame and units. In a camera frame, the axes are attached to the camera. In a robot-base frame, they are attached to the robot. The same physical point has different coordinates in each frame.

A rigid transform moves a point between frames:

pB=RBCpC+tBC.\mathbf{p}_{B}=\mathbf{R}_{BC}\mathbf{p}_{C}+\mathbf{t}_{BC}.

pC\mathbf{p}_{C} is a point in camera frame CC; pB\mathbf{p}_{B} is the same point in robot-base frame BB; RBC\mathbf{R}_{BC} is the rotation from CC to BB; and tBC\mathbf{t}_{BC} is the corresponding translation. Camera-to-robot calibration estimates this transform.

Interactive coordinate frame

Apply a rigid transform to the same points

—
XYgray: original frame · green: transformed frame

Coordinates change when the frame changes; the physical shape does not. Registration estimates this rotation and translation between clouds.

The view is a 2D projection of the same rigid-transform idea used in 3D.

1.2 Where Point Clouds Come From

A depth camera can produce an organized point cloud by back-projecting every valid depth pixel. Its points retain the rows and columns of the source depth map, which makes image-to-point lookup easy.

A lidar produces points by measuring distance along emitted laser directions. Its cloud may form scan rings rather than an image grid. Multiple captures can also be registered into one larger cloud, but moving objects and calibration error can create doubled or blurred geometry.

1.3 A Point Cloud Is a Sample, Not a Solid Object

Points describe observed surface locations. They do not automatically define which points belong together, what lies between them, or what the object looks like on its unseen side.

Common processing stages add that structure:

OperationPurpose
FilteringRemove noise or reduce point density
SegmentationSeparate surfaces or objects
Normal estimationApproximate local surface orientation
RegistrationAlign clouds captured from different poses
ClusteringGroup nearby points into candidate objects
Pose estimationAlign a known object model with observed points

Sparse sampling can miss thin parts. Reflective surfaces can create outliers. Occlusion means a single viewpoint only observes visible surfaces. A dense-looking cloud should therefore not be mistaken for a complete or perfectly accurate model.

2. Images, Depth Maps, and Point Clouds

RepresentationOrganized asStores
Color image2D pixel gridAppearance measurements
Depth map2D pixel gridOne surface distance per valid pixel
Point cloud3D point samplesSpatial coordinates in a stated frame

These representations are complementary. An image is convenient for appearance models, a depth map connects distance to pixels, and a point cloud makes geometry explicit for measurement, collision checking, and pose estimation.

2.1 The Telekinesis PointCloud Datatype

Telekinesis represents this data with datatypes.PointCloud. Positions are required as an array with shape (N, 3), where N is the number of points. Matching (N, 3) arrays can optionally store surface normals and RGB colors.

import numpy as np
from telekinesis import datatypes

positions = np.array([[0.2, -0.1, 1.4], [0.3, -0.1, 1.4]], dtype=np.float32)
point_cloud = datatypes.PointCloud(positions)

The datatype validates the array shape, but it cannot infer the coordinate frame or physical unit. Keep that metadata explicit in the surrounding robot system.

3. Where to Go Next