3 ms·
Someday I’ll have enough “karma” to downvote things like this. The agent should respect robots.txt no matter who is using the Robot.
by GuinansEyebrows 2y ago
Someday I’ll have enough “karma” to downvote things like this.
The agent should respect robots.txt no matter who is using the Robot.