Color Separation sits in a layer's Pre-Processing section, under Add Operations (basic), and works only in brightfieldLoading... projects. Its Input Image is the Original Image by default, or another color image in the layer. You add one marker per stain with Add Marker, give each a reference color, and choose a Method.
Single Reference Shades performs an unmixing of each pixel's color against the markers' reference colors. Suppose you pick a blue shade for Hematoxylin and set its Intensity to 200. In the Hematoxylin output, a pixel with as much marker as the spot you picked reads 200, a lighter blue reads less and a darker blue more. If the spot you picked was itself a mix, enable Is mixture of, name the other marker and enter its Mixture Percent; Pure Color then shows the corrected reference.
Multiple Reference Shades is a classification instead. Each marker gets a set of shades, and the engine estimates, for every pixel, the probability that it belongs to each shade from the distance between the colors in the chosen Color Space. The closer the color, the brighter the output pixel. Each shade has its own Measure, Gain, Sigma, Gauss and Color Space settings.
Either way, the result is one grayscale imageLoading... per marker, plus a color image per marker if you tick Generate Color Outputs.
Source: StrataQuest 8.1 Engines Reference, ch. 10 (pp. 382–395), ch. 6.2 (pp. 98–99), ch. 6.11 (p. 311); User Manual §11.1 (p. 154).