Yes! I can't stand this kind of thinking in code reviews, someone will say X is slow, and I say, "How slow is it?" they'll say something about O(X) and I'll say

Yes! I can't stand this kind of thinking in code reviews, someone will say X is slow, and I say, "How slow is it?" they'll say something about O(X) and I'll say, no really exactly how much time in seconds will be saved by changing this. Usually at this point they have no answer. In general your O(n^4) algo is perfectly fine, and no one will ever notice your O(1) replacement. When it isn't fine it's painfully clear that it's not OK.