3 ms·Because robots.txt is a standard people can choose to follow, its not a lawby cchance 2y agoBecause robots.txt is a standard people can choose to follow, its not a law