2 ms·
That's a cool project. Let me know if you need any help. I'm not sure what I could do but it sounds interesting. Here a couple ideas that might help (not tha
by tocomment 17y ago
That's a cool project. Let me know if you need any help. I'm not sure what I could do but it sounds interesting.
Here a couple ideas that might help (not that I condone violating a site's TOS).
Have a random time interval between each page download so someone looking at the logs doesn't see a regular pattern.
Pick 4 or 5 common user agent strings to alternate randomly between.
Go through free proxies, or package up your program and have 5 or more friends run it on different sections of the data you want to download.