4 ms·
I see your point - however I read from it that the author was more referring to the load/level of activity on the server that your requests make should be indis
by dotBen 16y ago
I see your point - however I read from it that the author was more referring to the load/level of activity on the server that your requests make should be indistinguishable from human traffic.
IE if the server's log files have 100's of requests from the same IP address in successive lines then that doesn't look like human behavior.
What would have been nice for a 'best practice' document would be to show how to set the HTTP AGENT string for the crawler so that it had an identifier, version number and some contact method.
- helwr 16y agoThis was already asked: http://groups.google.com/group/comp.lang.java/browse_thread/thread/6923c024ed392c85/88fa10845061c8ba?pli=1 http://groups.google.com/group/comp.lang.java/browse_thread/...