If training machine learning models is Fair Use (not totally settled, but seems likely) then they aren't bound by the license of the original content. Setting

If training machine learning models is Fair Use (not totally settled, but seems likely) then they aren't bound by the license of the original content. Setting up robots.txt is probably your best choice. In the EU commercial text and data mining must respect a machine-readable opt-out. In the US they could legally ignore it, but I reckon they'll follow it anyway to stay on the safe side and avoid having their crawler blocked for misbehaving.