Quantization

Quantization introduces an irreversible approximation: each continuous value is rounded to the nearest representable digital level. The maximum quantization error is ±½ LSB (Least Significant Bit). For an N-bit system, the quantization step size is (full range) / 2^N. This error manifests as banding in images (posterization) and as staircase artifacts in intensity profiles.

The quantization noise floor sets a theoretical limit on measurement precision. For uniformly distributed quantization error, the RMS noise is approximately (step size) / √12.

Quantization determines the finest intensity difference that can be distinguished in your data. If two pixel values differ by less than one quantization step, a single pixel cannot show the difference. When pixel noise is larger than one step, as it is in a camera set up with one count at half to all of the read noise, averages over many pixels can still resolve differences smaller than a step; when the noise is smaller than a step, the difference is lost. This is why the ADC bit depth should match the sensor's actual dynamic range for quantitative analysis.

At 8-bit (256 levels), the minimum distinguishable intensity difference is 1/256 ≈ 0.4% of full range. At 16-bit, it's 1/65,536 ≈ 0.0015%. For ratiometric measurements like FRET efficiency, where differences of 1-2% can be biologically significant, 8-bit quantization is too coarse for dim pixels: at a value of 50, a single step is already 2%.