robots.txt has no force of law. Google would just be embarrassed to be caught disregarding it. Less reputation-focused crawlers don't care. With AI, it would be

robots.txt has no force of law. Google would just be embarrassed to be caught disregarding it. Less reputation-focused crawlers don't care. With AI, it would be much harder to catch them red-handed since it typically transforms it's inputs enough to be hard to trace. The risk to reputation would be zero, so who would care?