Image Processing · Pupil

What is Image Filtering?

Image filtering changes each pixel using nearby pixels so that useful structure becomes easier to measure. Learn how box, Gaussian, median, and bilateral filters work, what their parameters mean, and how to choose one in the Telekinesis Agentic OS.

Imagine an industrial arm inspecting a machined turbine component. Its camera image contains the boundary and any real scratches, but it also contains sensor noise, machining texture, and tiny reflections from the metal. If every small brightness change survives, the inspection system may report hundreds of false defects. Image filtering combines neighboring pixels so repeatable structure remains easier to measure than unwanted variation.

1. What is Image Filtering?

Image filtering is the process of computing a new image by applying a repeatable rule to each pixel and its neighborhood. This tutorial explains the neighborhood model, compares common smoothing filters, and shows how to use them with Pupil in the Telekinesis Agentic OS.

Filtering is often described as “removing noise,” but that phrase hides the engineering decision. The algorithm cannot know whether a tiny bright mark is electrical noise, a scratch, printed text, or the tip of a small tool. It only knows how quickly pixel values change over space. A smoothing filter suppresses variations at one scale and preserves variations at another.

That makes filtering a trade-off, not a free improvement. A stronger filter produces a calmer image, but it can also move boundaries, merge nearby features, or erase the defect the robot was meant to inspect.

1.1 The Image Filtering Pipeline

Pixel neighborhood → new pixel value

Input imageSignal plus sensor or scene noise
Choose a neighborhoodPlace a kernel around each output location
Apply a filter ruleWeighted mean, median, or edge-aware weights
Move across the imageRepeat the same local rule at every pixel
Filtered imageLess unwanted variation, with some detail traded away

A filter is not simply “cleanup”: it defines which spatial variations count as noise and which survive as useful structure.

Neighborhood filters replace each pixel using nearby samples. The neighborhood size and aggregation rule determine the balance between noise reduction and detail preservation. Adapted as an explanatory diagram fromGonzalez and Woods, Digital Image Processing.

Focus on one pixel on the smooth surface of that turbine component. Its value may be unusually bright because of sensor noise. The surrounding pixels provide context: if they all describe the same gray metal surface, the isolated value is more likely to be noise. Near a real scratch, however, the surrounding difference may be exactly the evidence the robot must preserve.

A filter turns this idea into a repeatable local rule. It places a small window around the current pixel, gathers the values inside it, and combines them. A box filter averages them equally. A Gaussian filter trusts nearby samples more than distant ones. A median filter chooses the middle value, which makes isolated bright or dark spikes much less influential.

The window then slides to the next image location and repeats the same calculation until a complete output image has been built. The size of that window controls the scale of the decision. A 3×33\times3 neighborhood only compares immediate neighbors; a 15×1515\times15 neighborhood can remove larger variation, but it can also erase a narrow cable or small defect.

So filtering is always a trade: reduce unwanted variation while preserving the smallest structure the robot still needs. The following sections make that trade concrete for each filter family.

1.2 Convolution: Sliding a Weighted Kernel

Imagine a drone inspecting a row of aircraft rivets. To smooth sensor variation consistently, it must apply the same local rule at every rivet and every background location. Convolution is the mechanism that moves one small grid of weights across the entire image. Once you understand one output-pixel calculation, Gaussian smoothing, sharpening, and many edge filters become variations of the same operation.

Start at one output location. Center a small matrix of weights—the kernel—over the corresponding input pixel. Multiply every covered pixel by the weight above it, add all the products, and store that sum in the output. Then move the kernel one location and repeat.

Interactive walkthrough

Watch one convolution output pixel being calculated

—
1. Image neighborhood
×
2. Kernel weights
→
3. Products
=
4. Add them

The kernel does not move pixels physically. It moves across the image, calculating one new output value from each local neighborhood.

The interactive example deliberately shows the intermediate products. Select the averaging kernel to see a local mean, the edge kernel to see opposite sides of the neighborhood compete, and the sharpening kernel to see the center pixel emphasized relative to its neighbors.

Written compactly, this repeated calculation is

O(x,y)=∑u=−aa∑v=−bbK(u,v)I(x−u,y−v).O(x,y)=\sum_{u=-a}^{a}\sum_{v=-b}^{b}K(u,v)I(x-u,y-v).

Here, I(x,y)I(x,y) is the input intensity, O(x,y)O(x,y) is the filtered intensity, K(u,v)K(u,v) is the weight at kernel offset (u,v)(u,v), and the kernel has width 2a+12a+1 and height 2b+12b+1. The weights usually sum to one for a smoothing filter, which keeps a constant region at the same average brightness.

A box blur gives every sample the same weight. For a 3×33\times3 box, each weight is 1/91/9. It is fast and useful for rough averaging, but a pixel across an object boundary contributes just as much as a pixel on the same surface. Edges therefore soften quickly.

Image borders require a separate decision because part of the kernel falls outside the array. A border mode may reflect the image, repeat its edge pixel, or insert a constant value. This choice can create or avoid artificial responses near the image boundary, so it belongs in the pipeline configuration.

1.3 Gaussian Blur: Nearby Pixels Matter More

Suppose a warehouse scanner is reading a slightly noisy barcode on a carton. A box filter gives a pixel at the far corner of its window the same influence as the pixel beside the barcode location being computed. Gaussian blur uses a more natural assumption for this task: nearby evidence should matter more than distant evidence.

The kernel still computes a weighted average, but its weights are largest at the center and decrease smoothly with distance. The parameter that controls how quickly they fall is the standard deviation, sigmasigma:

G(x,y)=12πσxσyexp⁡ ⁣(−x22σx2−y22σy2).G(x,y)=\frac{1}{2\pi\sigma_x\sigma_y} \exp\!\left(-\frac{x^2}{2\sigma_x^2}-\frac{y^2}{2\sigma_y^2}\right).

G(x,y)G(x,y) is the weight at offset (x,y)(x,y); σx\sigma_x and σy\sigma_y are the horizontal and vertical standard deviations; and exp⁡\exp is the exponential function. Equal standard deviations give an isotropic blur. Different values create directional smoothing.

The Gaussian has no hard mathematical boundary, so software samples it inside a finite kernel_size. Sigma determines the shape of the weighting curve, while kernel size determines how much of that curve is retained. A large kernel with a very small sigma still concentrates most weight near the center.

Gaussian smoothing is a sensible default for approximately Gaussian sensor noise and for stabilizing derivatives before edge detection. It does not preserve sharp boundaries: pixels on both sides of an edge are averaged together.

Interactive mechanism

Increase the neighborhood and watch detail disappear

—
Input
signal + high-frequency variation
Gaussian output
noise falls · boundaries and small detail soften

The scene separates large structure from fine noise and a thin valid feature. Stronger smoothing cannot distinguish the last two by meaning.

1.4 Median Filtering: Select a Neighbor Instead of Averaging

An aerospace camera exposed to electrical interference may occasionally produce isolated black or white pixels on an otherwise smooth panel. Averaging spreads each extreme value into its neighbors, turning one bad pixel into a small blur. Median filtering is designed for this impulse-like noise.

Instead of multiplying and adding the neighborhood, it sorts the observed values and selects the one in the middle. No new average intensity is created, and isolated extremes move to the ends of the sorted list where they are unlikely to be chosen.

This makes the median filter nonlinear. Its rule is

O(x,y)=median⁡{I(x+u,y+v):(u,v)∈N}.O(x,y)=\operatorname{median}\{I(x+u,y+v):(u,v)\in\mathcal N\}.

N\mathcal N is the set of offsets in the neighborhood, and median returns its middle ordered value. Because one extreme bright or dark sample does not pull an average toward itself, the method is especially effective for salt-and-pepper noise.

The selected value already occurred nearby, so strong step edges often remain sharper than after an equivalent mean filter. The cost is that sorting is more expensive and a large window can remove small legitimate structures. In Pupil, kernel_size must be odd so the window has an unambiguous center.

1.5 Bilateral Filtering: Average Only Similar Neighbors

Imagine a humanoid viewing a dark utensil on a bright kitchen counter. Gaussian blur mixes values across the utensil boundary because pixels on both surfaces are physically close in the image. If we want to smooth small variations within the counter and utensil while keeping their boundary sharp, distance alone is not enough.

The bilateral filter adds a second test. A neighbor receives a high weight only when it is both nearby in the image and similar in intensity. Pixels across a strong brightness boundary are close spatially but dissimilar in value, so their influence falls.

Mathematically, these two preferences appear as two weighting functions:

O(p)=1Wp∑q∈N(p)Gσs(∥p−q∥)Gσr(∣I(p)−I(q)∣)I(q).O(p)=\frac{1}{W_p}\sum_{q\in\mathcal N(p)} G_{\sigma_s}(\|p-q\|)G_{\sigma_r}(|I(p)-I(q)|)I(q).

pp is the output location, qq is a neighboring location, GσsG_{\sigma_s} is the spatial Gaussian, GσrG_{\sigma_r} is the range or intensity Gaussian, and WpW_p is the sum of all weights used for normalization. Neighbors far away or very different in brightness contribute less.

Across a strong edge, the range weight becomes small, so the two surfaces are not averaged as aggressively. This preserves boundaries better than Gaussian blur, but it costs more computation and can produce stylized, flattened regions when tuned too strongly.

1.6 Start by Naming the Noise You Actually Have

We have several filters available, but choosing among them requires identifying the disturbance in the real camera. Calling every unwanted pattern “noise” hides important differences in how it was created and which mechanism can remove it.

Approximately Gaussian sensor noise adds small positive and negative deviations across many pixels; averaging filters reduce its variance. Impulse noise replaces isolated samples with extreme values, making the median more suitable. Periodic interference creates repeating bands and may need frequency-selective filtering. Surface texture may be real scene structure even when the task treats it as a nuisance.

You can distinguish these cases by capturing many images while the robot and scene remain still. For every pixel, estimate the mean and variance over time. Random sensor noise changes between images; a scratch or printed texture stays fixed. Subtract the temporal mean from each image and inspect the remaining differences. If isolated values jump to the ends of the range, test a median filter. If the differences form a compact bell-shaped distribution, begin with a small Gaussian.

Lighting flicker and camera auto-exposure are global or slowly varying changes, not local pixel noise. A larger blur may hide their spatial effect without solving their cause. Stabilize exposure and illumination before tuning a spatial filter.

1.7 Choosing the Filter From the Noise

We can now compare the algorithms by the assumption each one makes about the variation just identified. The right choice depends less on which output looks smoothest and more on how the noise behaves and which real features must survive.

FilterLocal ruleBest starting pointMain cost
Box blurEqual-weight meanVery fast rough smoothingSoftens edges strongly
GaussianDistance-weighted meanGeneral noise and derivative preprocessingBlurs boundaries and fine detail
MedianMiddle ordered sampleIsolated salt-and-pepper noiseLarger windows remove small features
BilateralSpatial and intensity-weighted meanSmoothing while retaining strong edgesMore computation and parameters

Choose using evidence, not visual preference. Capture repeated images of a static scene and measure the pixel variation. Include the smallest important feature and the sharpest required boundary in the test. The best filter is the weakest one that stabilizes the downstream measurement.

1.8 The Same Filter Has a Frequency Interpretation

The neighborhood view explains how filtering is calculated. A frequency view explains more broadly what kinds of image structure it removes. Both describe the same operation, but the second view is useful when reasoning about repeating texture, blur, and sampling.

An image can be described as a sum of spatial frequencies. Slowly varying illumination occupies low frequencies; sharp edges, texture, and fine noise occupy higher frequencies. A smoothing kernel is a low-pass filter: it keeps slow variation and attenuates rapid variation.

Convolution in the image domain corresponds to multiplication in the frequency domain:

F{I∗K}=F{I} F{K}.\mathcal F\{I*K\}=\mathcal F\{I\}\,\mathcal F\{K\}.

F\mathcal F denotes the Fourier transform, II is the image, KK is the kernel, and ∗* is convolution. This explains why smoothing cannot know whether a high-frequency feature is noise or a tiny defect. Both occupy similar spatial scales and both are attenuated.

A box kernel has a frequency response with sidelobes, which can produce less graceful scale behavior. A Gaussian transforms into another Gaussian and introduces no oscillating sidelobes, one reason it is central to scale-space theory. Repeated Gaussian smoothing progressively removes fine structure in a controlled way.

After choosing a filter family, the most consequential setting is often the neighborhood size. There is no universally safe 5×55\times5 or 7×77\times7 kernel because a pixel does not have a fixed physical meaning across cameras and viewpoints.

A seven-pixel scratch might represent 0.2 mm in a close inspection view and 20 mm in a distant warehouse view. If the task requires preserving features of width wminw_{min} pixels, the kernel should remain meaningfully smaller than that feature and be validated against it.

Resolution changes also change filter meaning. Applying a 5×55\times5 kernel after halving image dimensions covers twice the scene width it covered before downsampling. For repeatable inspection, keep camera distance and resolution fixed or derive filter scale from calibrated pixel density.

Anisotropic Gaussian parameters can encode direction. Setting σx>σy\sigma_x>\sigma_y smooths more strongly along the horizontal axis. That can suppress directional sensor streaks, but it also treats horizontal and vertical object details differently. Use it only when the nuisance has a known orientation.

1.10 Filtering Changes Every Downstream Threshold

Filtering rarely exists as the final robot output. It prepares evidence for thresholding, edge detection, measurement, or a learned model. We therefore need to reason about the entire sequence rather than declaring a filtered image “clean.”

Smoothing reduces local maxima and raises local minima. A score or intensity threshold calibrated on raw pixels may no longer mean the same thing after filtering. Median filters can remove isolated extrema entirely. Bilateral filters may flatten interiors while preserving strong boundaries, changing region statistics in a different way.

Treat the operation order as part of the algorithm:

capture → color conversion → filtering → derivative/threshold → morphology → measurement

Switching filtering and thresholding produces a different result. Filtering grayscale values before thresholding lets neighboring evidence influence the decision. Filtering a binary mask afterward becomes a shape operation and may create intermediate gray values unless morphology is used instead.

Keep the raw camera image during evaluation. Log the filtered output and the final measurement together, so an apparent improvement can be traced to the exact evidence that was suppressed.

2. How to Filter an Image in the Telekinesis Agentic OS

Telekinesis exposes filtering through Pupil Skills. This example applies Gaussian and median filters to the same image so their outputs can be compared directly.

from telekinesis import datatypes, pupil

image = datatypes.Image.from_url(
    url="https://assets.telekinesis.ai/examples/v1/images/nuts_scattered_noised.jpg"
)

gaussian = pupil.filter_image_using_gaussian_blur(
    image=image,
    kernel_size=7,
    sigma_x=2.0,
    sigma_y=2.0,
    border_type="default",
)

median = pupil.filter_image_using_median_blur(
    image=image,
    kernel_size=5,
)

print("Gaussian:", gaussian.shape)
print("Median:", median.shape)

Start with a small odd kernel. Increase it only when the unwanted variation remains, and re-check the smallest useful feature after every change. A clean-looking image is not enough; compare the effect on the next measurement, threshold, or detector.

Try it out

Use Gaussian filtering with Pupil

See the complete Skill signature, parameter ranges, border modes, and runnable example.

Open Gaussian blur docs

Runnable example

Run the Gaussian filtering example

Open the exact Telekinesis Python example for this Pupil Skill.

View Python example

3. How to Evaluate a Filter

A filter succeeds only when it suppresses the nuisance while preserving the evidence needed later. Evaluation must therefore measure both sides of that trade rather than rewarding the smoothest output.

For synthetic noise, compare the filtered image with a known clean reference using mean squared error or peak signal-to-noise ratio. For robotics, the more useful evaluation is usually downstream: boundary error, threshold stability, feature repeatability, false components, or final task success.

Measure runtime at the deployed resolution. Kernel area grows approximately with the square of kernel width for a direct two-dimensional implementation, although separable Gaussian implementations reduce the work. Bilateral and median filters have different computational behavior, so profile the actual Skill call rather than assuming that equal kernel sizes have equal cost.

4. Where to Go Next

5. References