3 ms·
> "it would be a fair bit of extra work for us to read robots.txt", Can someone explain what this means? After all reading a small text file is plainly easy t
by kwhitefoot 4y ago
> "it would be a fair bit of extra work for us to read robots.txt",
Can someone explain what this means? After all reading a small text file is plainly easy to do so its meaning must be something other than obvious.
- yamtaddle 4y agoAll I can figure is they're trying really, really hard to keep this very parallel on their side and also avoiding having to coordinate between nodes. It can't possibly be the reading of the robots.txt that's hard, so I think that statement has more to do with applying those policies to all the nodes—they must regard coordination between nodes to ensure the system as a whole isn't exceeding e.g. request rate maximums, to be "a fair bit of extra work". Judging just from the linked post, the issue on which this was discussed, and this thread, it's feeling a lot like this proxy was some kind of proof-of-concept that escaped its cage and got elevated to production.