From what I remember, you could choose a layer to use as a mask for quality. However, I haven't touched PS in more than ten years, and I can't even find screens

From what I remember, you could choose a layer to use as a mask for quality. However, I haven't touched PS in more than ten years, and I can't even find screenshots of versions earlier than ~CS5 from a quick search. Perhaps it was such an obscure feature than it was yanked out. I may also be mistaking this with settings for some filter or such—but I'm fairly sure it was a setting particularly for quality of something.