3 ms·
I really don't think this should be handled at FS level. robots.txt didn't need to know about who the robot was or what the hosting server was.
by nolok 7y ago
I really don't think this should be handled at FS level.
robots.txt didn't need to know about who the robot was or what the hosting server was.