The Rose criterion applied to gating: Pawley's Rose criterion states that a signal must exceed 5× the noise to be reliably detected. Applied to gating: a truly positive cell must have intensity exceeding the negative population mean + 5× the negative population's standard deviation to be reliably classified as positive. Cells in the 2-5σ range above the negative mean are in the "equivocal zone" — they might be dim positives or bright negatives.
Statistical thresholding theory (Dilbilir): Setting a gate is a special case of binary classification where the feature is one-dimensional (intensity) and the decision boundary is a single threshold. The optimal threshold minimizes total classification error — the sum of false positives (negative cells above the gate) and false negatives (positive cells below the gate). When the positive and negative populations have different sizes, the optimal threshold shifts: in a sample where 90% of cells are negative and 10% are positive, a slightly lower threshold (catching more positives at the cost of some false positives) may minimize total error.
The Gaussian intersection: If the positive and negative populations have Gaussian intensity distributions with means μ₊ and μ₋ and standard deviations σ₊ and σ₋, the optimal threshold falls at or near the intersection of the two Gaussians. When σ₊ = σ₋, this is the midpoint (μ₊ + μ₋)/2. When variances differ, the intersection shifts toward the population with larger variance. This is the Bayesian optimal threshold for equal misclassification costs.
Batch effects on gates: Staining intensity varies between batches — even with the same protocol, day-to-day variation in stain concentration, incubation time, and washing steps shifts the positive and negative populations. Gates set on one batch may not apply correctly to another. Two approaches: (1) set gates per-sample by examining each sample's histogram, or (2) normalize intensity across batches before gating. Per-sample gating is more accurate but labor-intensive; normalization enables consistent gates but assumes the batch effect is the same for all cell types.
Setting a gate is finding where the positive and negative intensity populations separate. The ideal gate sits in the valley between them — where the two Gaussian distributions intersect. The Rose criterion tells you how deep the valley needs to be: the positive population must be at least 5 standard deviations above the negative mean for reliable separation. Cells in the ambiguous zone between the two peaks are where classification errors concentrate, and batch-to-batch staining variation can shift the peaks, requiring per-sample gate adjustment.