The mathematics. For a threshold T on an image with L gray levels, w₁(T) and w₂(T) are the probabilities of the background class (pixels ≤ T) and the foreground class (pixels > T), m₁(T) and m₂(T) are their means, and m_G is the global mean. The between-class variance is σ²_B(T) = w₁(T) × w₂(T) × [m₁(T) − m₂(T)]². Otsu proved that the threshold maximizing σ²_B minimizes the probability of classification error when the two classes have equal covariance. The search is fast: its cost grows only linearly with the number of gray levels.
Where Otsu needs help. Three conditions blur the two-peak picture the method relies on:
- Small object area. When the foreground occupies a tiny fraction of the image, such as sparse fluorescent dots on a large dark background, its peak barely influences the histogram, and Otsu may place the threshold in the middle of the background distribution.
- High noise. Noise fills the valley between the histogram peaks, so the bimodal structure is less distinct and the optimal threshold less stable.
- Non-uniform illumination. When background intensity varies across the image, no single global threshold classifies every region correctly. Adaptive (local) thresholding, which applies Otsu in small neighborhoods, addresses this case.
Otsu in the broader context. Otsu's method ignores spatial structure and classifies each pixel by intensity alone. In effect it is a Bayesian classifier with equal Gaussian variances and equal priors, working in one dimension, intensity. Approaches such as MRF-based segmentation and learned classifiers use spatial context, the tendency of neighboring pixels to belong to the same class, for better results, but Otsu's simplicity and speed keep it the standard first approach.
Otsu works well when the histogram has two clear peaks with a valley between them. It works poorly when the foreground is sparse (a tiny peak lost beside the background), when noise fills the valley, or when illumination is uneven. It remains the standard automatic thresholding method because it is fast and simple.