4 ms·
Pardon me for asking, but how did you crawl Google for a whole weekend. From what I know, Google blocks you if you request too many search queries in a short pe
by elorant 12y ago
Pardon me for asking, but how did you crawl Google for a whole weekend. From what I know, Google blocks you if you request too many search queries in a short period of time. Did you use proxies?
- roel_v 12y agoMaybe now, I don't remember I had do anything sneaky back then (8-10 years ago)
- wanderingstan 12y agoIn 2004 I attempted to do some automatic crawling of Google for my masters thesis and was astonished to get an unfriendly server response saying it was disallowed and "don't even bother asking for an exception for research, it won't be granted." So at least 11 years ago it was blocked. (I didn't know about spoofing a user agent back then, so it might not have been as easy as that to get around it.)