Image Processing · Pupil

What is Image and Mask Composition?

Image and mask composition combines aligned pixel grids using logical, arithmetic, channel, or weighted operations. Learn AND, OR, XOR, NOT, difference, split/merge, and overlays, including the datatype and coordinate pitfalls.

Imagine a warehouse industrial arm asked to pick blue cartons. One vision process marks every blue pixel, calibration provides a mask of the reachable conveyor area, and a safety system marks pixels occupied by a person’s hand. The arm may pick only where all conditions agree: blue AND reachable AND NOT occupied. Image and mask composition turns these separate pieces of evidence into one explicit operating region.

1. What is Image and Mask Composition?

Image and mask composition applies the same rule to corresponding pixels in aligned images. This tutorial explains logical mask operations, absolute difference, channel splitting and merging, and weighted overlays through Pupil.

The arithmetic is simple; the assumptions are not. Inputs must refer to the same coordinates, use compatible dimensions and channels, and agree on value meaning. A mask using 255 for foreground and another using class IDs cannot be combined blindly just because both look black and white on screen.

1.1 The Composition Pipeline

Several pixel decisions → one composed region

Image or mask AFirst aligned pixel grid
Image or mask BSecond grid with identical dimensions
Align shape + typeMatch width, height, channels, and value convention
Apply a pixel ruleAND, OR, XOR, NOT, difference, or weighted blend
Preserve provenanceTrack what each input and channel represents
Composed imageIntersection, union, change, complement, or overlay

Logical composition is exact only when the masks share coordinates and agree on which value means foreground.

Composition applies the same rule independently at aligned pixel locations. The mathematics is simple; coordinate, channel, and datatype mismatches are the practical risk. Adapted as an explanatory diagram fromTelekinesis Pupil image-processing overview.

Suppose the warehouse color mask marks every blue carton and the workspace mask marks the conveyor area the arm can reach. Before combining them, both must describe the same camera coordinates. If the color mask belongs to a resized model image while the workspace belongs to the original camera image, matching array sizes alone does not mean the same pixels refer to the same physical place.

Once the grids and value conventions agree, composition becomes a simple question asked at every location. AND keeps a pixel only when both masks accept it. OR keeps it when either mask accepts it. XOR keeps disagreements, while NOT reverses one decision. The same idea can compare intensities or blend two aligned images for visualization.

Repeating that rule across the grid produces a new image or mask. The calculation does not judge whether either input is correct; it only combines the evidence it was given. For this reason, the output should retain a clear description of its origin—for example, “blue AND reachable AND NOT occupied”—rather than becoming an unexplained black-and-white array.

Use the interactive example to see the set operations geometrically. The deeper sections then connect those shapes to pixel values, class maps, confidence maps, and practical robot constraints.

Interactive mechanism

Compose two masks one pixel at a time

—
mask Amask B

Logical mask operations have exact set meanings. They combine existing evidence; they do not decide whether either mask is correct.

1.2 Think of Binary Masks as Sets

For the warehouse industrial arm, each mask can be understood as a set of allowed image coordinates: blue locations, reachable locations, or unsafe locations. This viewpoint makes the operations intuitive. AND keeps the intersection of two sets, OR keeps their union, and NOT selects everything outside one set.

Let mask A(x,y)A(x,y) be one when pixel (x,y)(x,y) belongs to set AA and zero otherwise. The main logical operations are

(A∧B)(x,y)=A(x,y)B(x,y),(A\land B)(x,y)=A(x,y)B(x,y), (A∨B)(x,y)=max⁡(A(x,y),B(x,y)),(A\lor B)(x,y)=\max(A(x,y),B(x,y)), (¬A)(x,y)=1−A(x,y).(\lnot A)(x,y)=1-A(x,y).

∧\land is intersection, ∨\lor is union, and ¬\lnot is complement. XOR is one only when the inputs differ:

(A⊕B)(x,y)=∣A(x,y)−B(x,y)∣.(A\oplus B)(x,y)=|A(x,y)-B(x,y)|.

This set interpretation makes pipelines readable. A safety-constrained target could be

Mpick=Mtarget∧Mworkspace∧¬Munsafe.M_{pick}=M_{target}\land M_{workspace}\land\lnot M_{unsafe}.

MpickM_{pick} is the final allowed region; MtargetM_{target} is visual evidence for the object; MworkspaceM_{workspace} is a reachability mask; and MunsafeM_{unsafe} identifies forbidden pixels. The formula is auditable in a way that an ambiguous sequence of image calls is not.

1.3 Bitwise Operations Are Defined on Bits

Set logic describes the desired decision, but image libraries often implement it with bitwise operations on stored integers. These behave exactly like logical masks only when the value encoding has been chosen deliberately, such as zero for background and 255 for foreground.

For an 8-bit value, bitwise AND compares every binary digit. For example,

204 AND 170=136,204\ \texttt{AND}\ 170 = 136,

because 204=110011002204=11001100_2, 170=101010102170=10101010_2, and the common set bits form 100010002=13610001000_2=136. On binary masks containing only 0 and 255, this behavior matches logical set operations cleanly. On arbitrary grayscale photographs, bitwise results are rarely an intuitive blend.

Use bitwise Skills for integer masks and deliberately encoded images. For floating-point probabilities, use numerical comparisons or multiplication with well-defined semantics instead of bit operations.

1.4 AND, OR, XOR, and NOT Answer Different Questions

With the set and storage meanings separated, we can choose an operation by the question the robot needs to answer. The same two masks produce very different evidence depending on whether we require agreement, accept either source, or want to expose disagreement.

AND asks where all conditions agree. It is useful for restricting a segmentation to a region of interest or enforcing a workspace constraint.

OR asks where any condition is true. It can merge masks from several acceptable object categories, but overlapping errors accumulate too.

XOR asks where exactly one input is true. It highlights disagreement between two masks and is useful in regression testing or change analysis. It deliberately removes their common region.

NOT swaps foreground and background. In an 8-bit binary mask, zero becomes 255 and 255 becomes zero. On other integers, every bit flips; it is not the same as arithmetic negation.

1.5 Absolute Difference Detects Change, Not Its Cause

Imagine an aerospace cobot photographing the same tool tray before and after maintenance. No mask exists yet; the first question is simply which image locations changed. Absolute difference compares the two camera images directly and makes large intensity changes bright.

For two intensity images, absolute difference is

D(x,y)=∣I1(x,y)−I2(x,y)∣.D(x,y)=|I_1(x,y)-I_2(x,y)|.

I1I_1 and I2I_2 are aligned images and DD is their per-pixel change magnitude. A high value may indicate motion, a missing part, changed illumination, camera vibration, exposure adjustment, or sensor noise. Difference supplies evidence of change but no semantic explanation.

Before using it for inspection, register the images, stabilize camera settings, and measure the normal repeatability of unchanged pixels. A one-pixel camera shift can create strong double edges around every object.

1.6 Splitting and Merging Channels

A humanoid may need to enhance kitchen brightness without shifting the apparent color of food or utensils. Splitting a converted color image exposes lightness as one image and color information as others, allowing one channel to be processed before they are merged again.

A color image is an array with a channel axis. Splitting RGB produces three single-channel images:

I(x,y)=[R(x,y),G(x,y),B(x,y)].I(x,y)=[R(x,y),G(x,y),B(x,y)].

Each channel can be inspected or processed separately. Merging stacks compatible single-channel images back into a multi-channel array. Order matters: RGB and BGR contain the same three kinds of numbers in different positions and display with swapped colors if confused.

Channel operations become powerful after color conversion. For example, convert RGB to LAB, split the channels, enhance only L∗L^*, then merge and convert back. This changes lightness while preserving opponent-color coordinates more faithfully than enhancing R, G, and B independently.

All channels must have identical height, width, and compatible datatype. Splitting does not duplicate geometry metadata automatically in every system, so retain the parent image context.

1.7 Weighted Overlays Are Arithmetic Blends

During commissioning, a manufacturing engineer needs to see whether an industrial arm’s defect mask actually lies over the scratch it claims to detect. A weighted overlay places the camera image beneath a colored or brightened mask so both sources remain visible in one debugging view.

A weighted overlay computes

O(x,y)=αA(x,y)+βB(x,y).O(x,y)=\alpha A(x,y)+\beta B(x,y).

AA and BB are aligned images, α\alpha and β\beta are weights, and OO is the blended output. When α+β=1\alpha+\beta=1, the operation is a crossfade. Larger sums may saturate an integer output unless the implementation clips or uses a wider datatype.

Overlays are excellent for debugging: blend a mask color, edge map, depth rendering, or model output with the original camera image. They should not be confused with logical union. A half-transparent green visualization is not a binary decision mask.

1.8 Alignment, Shape, and Datatype Checks

Every operation above assumes that corresponding array entries describe the same physical location and compatible kinds of value. Before combining anything, make those hidden assumptions explicit.

Verify:

  1. Width and height match.
  2. Pixels refer to the same camera image and capture time.
  3. Cropping, resizing, and padding mappings are consistent.
  4. Channel order and count match the intended operation.
  5. Datatype and value range are known.
  6. Foreground convention is explicit.

Two 640×480 masks are not aligned merely because their shapes match. One may describe a resized model input while the other describes the original camera image. Coordinate provenance is part of the data.

1.9 Binary Masks, Class Maps, and Confidence Maps Are Different

Shape and datatype checks are not enough because arrays with identical dimensions can carry different semantics. The next decision is therefore what each stored number means and which operations preserve that meaning.

Three arrays can all appear as grayscale images while carrying different meanings. A binary mask says yes or no. A class map stores an integer category ID at each pixel. A confidence map stores a continuous score such as a probability between zero and one.

Logical operations naturally fit binary masks. For a class map, class_map == pallet_id should first create a binary pallet mask. Applying NOT directly to category IDs flips their bits and destroys their meaning. For a confidence map, multiplication may express joint soft evidence, while thresholding creates the explicit set needed for AND or OR. The choice is semantic, not merely numerical.

Suppose two independent systems assign target confidences PA(x,y)P_A(x,y) and PB(x,y)P_B(x,y). Their product is sometimes used as a joint score, but it is justified as a probability only under assumptions about calibration and dependence. Two models trained on the same data may make correlated mistakes. A high product can still be confidently wrong.

For robot decisions, name the representation in code—workspace_mask, target_probability, or semantic_labels—and assert its allowed values. A variable called image tells the next engineer almost nothing about which operations are valid.

1.10 From Mask Logic to Connected Objects

After the warehouse industrial arm combines blue, reachable, and safe masks, it still needs to know how many separate cartons remain. Connected-component analysis is the bridge from one final pixel mask to a list of object-like regions that can be measured and ranked for picking.

After forming the final mask, connected-component analysis groups neighboring foreground pixels into regions. Each component can then be filtered by area, shape, position, or distance from a previous observation.

This creates a useful division of responsibility. Pixel logic expresses constraints: correct color, inside the workspace, outside the exclusion zone. Component analysis asks which surviving pixels form coherent objects. Contours or segmentation geometry can then measure those objects more precisely.

Order matters here too. Removing an unsafe strip before connected components may split one region into two. Combining masks after component labeling may retain object IDs that no longer describe connected shapes. Compose the pixel-level decision first unless the application deliberately reasons about separately tracked objects.

1.11 Choosing the Operation

The full workflow now separates three questions: what values the inputs represent, whether their coordinates align, and what relationship the output should express. The table maps common robotics goals to the final operation.

Desired resultOperationTypical robotics use
Satisfy every constraintANDTarget inside reachable area
Accept any sourceORMerge allowed categories
Find disagreementXORCompare mask versions
Exclude a regionNOT then ANDRemove safety zone
Detect appearance changeAbsolute differenceMissing-part or motion proposal
Inspect one signalSplit channelsColor or luminance analysis
Show evidence togetherWeighted overlayDebug visualization

2. How to Compose Masks With Pupil

The example translates the earlier safety expression directly into code. It first intersects the target with the reachable workspace, then removes the unsafe area. Keeping these as named intermediate masks makes the decision inspectable.

from telekinesis import pupil

target_and_workspace = pupil.bitwise_and_images(
    image_1=target_mask,
    image_2=workspace_mask,
)

safe_area = pupil.bitwise_not_image(image=unsafe_mask)

pick_mask = pupil.bitwise_and_images(
    image_1=target_and_workspace,
    image_2=safe_area,
)

All three masks must share dimensions, coordinates, and binary convention. Log the component masks as well as the final result; otherwise a missing pick region cannot be traced to target perception, reachability, or safety exclusion.

Try it out

Intersect masks with Pupil

See the exact input requirements, output behavior, and composition examples for bitwise AND.

Open bitwise AND docs

Runnable example

Run the mask-intersection example

Open the exact Telekinesis Python example for composing aligned masks.

View Python example

3. How to Test Composition Pipelines

Composition is exact enough that small synthetic cases can test it thoroughly before real camera data is involved. Use masks whose expected truth tables can be inspected manually. Include empty masks, full masks, non-overlapping regions, complete overlap, and one-pixel features. Assert shape, datatype, unique values, and foreground count after every operation.

For difference images, test static sequences under normal camera noise and intentional shifts. For overlays, test saturation and channel order. The goal is not only mathematical correctness but preservation of the physical meaning assigned to every pixel.

Add invariants that describe the intended logic. An AND result can never contain more foreground pixels than either input. An OR result can never contain fewer. XOR of a mask with itself must be empty, while XOR with an empty mask must reproduce the original. These properties catch channel-order, range, and accidental-resizing errors without requiring a large labeled dataset.

4. Where to Go Next

5. References