Image & Point Cloud Fundamentals · Fundamentals

What are Camera Intrinsics and Extrinsics?

Camera intrinsics describe how 3D rays map to pixels, while extrinsics place the camera in another coordinate frame. Learn why robots need both and how Telekinesis represents them.

A camera mounted on an industrial arm may detect a component at pixel (620, 340). The arm cannot move to a pixel. It needs the corresponding direction from the camera and the camera’s position relative to the robot. Intrinsics and extrinsics provide those two parts of the geometry.

1. What are Camera Intrinsics and Extrinsics?

Camera intrinsics describe how the camera projects 3D rays onto image pixels; camera extrinsics describe the camera’s position and orientation relative to another coordinate frame. Intrinsics explain the inside of the camera model. Extrinsics connect the camera to the outside world.

Intrinsic projection and an extrinsic camera-to-robot transform

1.1 Intrinsics: From a Camera Ray to a Pixel

For a pinhole camera, the intrinsic matrix is

K=[fx0cx0fycy001].\mathbf K= \begin{bmatrix} f_x & 0 & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1 \end{bmatrix}.

K\mathbf K is the intrinsic matrix, fxf_x and fyf_y are focal lengths measured in pixels, and (cx,cy)(c_x,c_y) is the principal point—the image position where the optical axis meets the sensor model.

For a 3D point (X,Y,Z)(X,Y,Z) already expressed in the camera frame, the ideal pixel is

u=fxXZ+cx,v=fyYZ+cy.u=f_x\frac{X}{Z}+c_x, \qquad v=f_y\frac{Y}{Z}+c_y.

(u,v)(u,v) is the pixel coordinate. The division by ZZ creates perspective: an object appears smaller as its depth increases. Real lenses also bend rays, so calibration normally includes a distortion model and distortion coefficients.

Intrinsics belong to a specific image geometry. If an image is resized or cropped, the focal lengths and principal point must be adjusted to match.

1.2 Extrinsics: From the Camera Frame to the Robot Frame

The camera has its own axes, while an AMR, cobot, or drone has another coordinate frame. Extrinsics are the rigid transform between them:

pR=RRCpC+tRC.\mathbf p_R=\mathbf R_{RC}\mathbf p_C+\mathbf t_{RC}.

pC\mathbf p_C is a point in camera frame CC, pR\mathbf p_R is the same physical point in robot frame RR, RRC\mathbf R_{RC} is the rotation from camera to robot, and tRC\mathbf t_{RC} is the translation between their origins.

Extrinsics answer questions such as “where is the camera mounted on the arm?” They do not describe focal length or lens distortion. If the camera moves on a wrist or drone gimbal, its transform to the world changes even though its intrinsics remain the same.

1.3 Using Both Together

Intrinsics alone turn a pixel plus depth into a point in the camera frame. Extrinsics then move that point into the robot frame. Reversing the path projects a robot-frame point into the image. This chain is what allows a detected image region to become a measured 3D target.

Calibration errors appear as physical errors. Incorrect intrinsics bend the mapping between pixels and rays; incorrect extrinsics shift or rotate the whole reconstructed scene. Validate the final camera-to-robot mapping with points spread across the actual workspace.

2. Camera Geometry in Telekinesis

Telekinesis deliberately represents the two concepts separately:

  • datatypes.CameraCalibration stores one camera’s image resolution, distortion model, distortion parameters, and 3×3 intrinsic matrix.
  • datatypes.Transform3D stores a valid 4×4 rigid transform in SE(3)SE(3) and can represent camera-to-robot extrinsics.
from telekinesis import datatypes

intrinsics = datatypes.CameraCalibration(
    width=1280,
    height=720,
    distortion_model="plumb_bob",
    distortion_parameters=[-0.2, 0.1, 0.0, 0.0, 0.0],
    intrinsic_matrix=[1000, 0, 640, 0, 1000, 360, 0, 0, 1],
)

camera_to_robot = datatypes.Transform3D([
    [1, 0, 0, 0.35],
    [0, 1, 0, 0.00],
    [0, 0, 1, 0.80],
    [0, 0, 0, 1.00],
])

The variable names state the transform direction. That convention matters: camera_to_robot and robot_to_camera are inverses, not interchangeable matrices.

3. Where to Go Next