If you read the law it says it’s only allowed if the rights holder doesn’t disallow it (in machine readable form) - I would argue robots.txt falls under machine

If you read the law it says it’s only allowed if the rights holder doesn’t disallow it (in machine readable form) - I would argue robots.txt falls under machine readable