It's a mistake to think that generalization doesn't matter much. In this case your false positive rate can go through the roof also, to the degree the system b

It's a mistake to think that generalization doesn't matter much. In this case your false positive rate can go through the roof also, to the degree the system becomes useless. More generally it means you don't understand well how your system will work on real world data, so can end up with lots of unfortunate surprises. It's true that if you are assuming a human/algorithm team some things are easier, but that doesn't make the problem go away.