2 ms·
I'm curious if the Go crawler would correctly back-off with "429 Too Many Requests" responses. Either way it could save the bandwidth.
by waych 4y ago
I'm curious if the Go crawler would correctly back-off with "429 Too Many Requests" responses. Either way it could save the bandwidth.
- PufPufPuf 4y agoOf course not, the crawler doesn't even read robots.txt because it's "too difficult to implement"