There are really three types of crawlers: good actors, bad actors, and "grey actors". The grey actors can be just by making a mistake (or outright incompetence)

There are really three types of crawlers: good actors, bad actors, and "grey actors". The grey actors can be just by making a mistake (or outright incompetence), or they're testing where the limits are, etc. robots.txt does nothing for the bad actors, but for grey actors it's a bit more complicated as they can move from "grey" to "good". Adding something for that in robots.txt in addition to some other limits (if needed) can be useful.