3 ms·

If you must, why not just scrape the google cache instead of the live site? http://www.google.com/search?q=cache:news.ycombinator.com/item%3Fid%3D261985 http:/

If you must, why not just scrape the google cache instead of the live site? http://www.google.com/search?q=cache:news.ycombinator.com/item%3Fid%3D261985 http://www.google.com/search?q=cache:news.ycombinator.com/it... At this moment, that's the latest cached item they have. So, it'll lag by a few days, but that wouldn't matter much here.

deleted 18y ago
[deleted]
matt1 18y ago
On second thought Google cache might not be the best idea: "We're sorry... ... but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can't process your request right now. We'll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software. If you're continually receiving this error, you may be able to resolve the problem by deleting your Google cookie and revisiting Google. For browser-specific instructions, please consult your browser's online support center. If your entire network is affected, more information is available in the Google Web Search Help Center. We apologize for the inconvenience, and hope we'll see you again on Google. To continue searching, please type the characters you see below:"
- staticshock 18y ago
  interesting. at what point do you get that message?
  - matt1 18y ago
    very quickly. when i check it in my browser it was fine, but as soon as i started scanning with the software google blocked it. they may have recognized that the requests werent coming from a standard browser, so they flagged my ip.
- bullseye 18y ago
  Include a referrer HTTP header, set the user agent, and set a random wait time between 500ms and 5s. I crawl Google frequently and that format has always worked for me... At least it did until I made this post. :)
  - Hexstream 18y ago
    If you can bypass the rules they don't count, eh?
    - fauigerzigerk 18y ago
      To be honest I can't take it seriously when a company that makes billions from crawling other people's sites makes a rule that you're not allowed to crawl theirs. Google has a lot of conflicts with media companies and it always starts out with google ignoring some "rules" expecting to either win in court or settle at some point. So I think breaking these rules is part of the process of how sensible rules are established in the first place. Yes it's recursive, but HN readers should be smart enough to understand that ;-)
      - neilc 18y ago
        To be honest I can't take it seriously when a company that makes billions from crawling other people's sites makes a rule that you're not allowed to crawl theirs. Why not? Google obeys robots.txt -- if you don't want them to crawl your site, it is trivial to arrange. I think violating Google's terms of use is pretty hard to justify as ethical.
        fauigerzigerk 18y ago
        Google obeys robots.txt but not much else. Just ask publishers and news outlets. And as I said, I don't see rules as a black and white thing. If they were, there would never be any progress. No iTunes without napster. No Microsoft being the biggest software company in the world without piracy. That doesn't mean it's ethically justified to break any rule. I just think it's not as simple as someone stating their own rules and everyone automatically obeying it.

deleted 18y ago

[deleted]

matt1 18y ago

On second thought Google cache might not be the best idea: "We're sorry... ... but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can't process your request right now. We'll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software. If you're continually receiving this error, you may be able to resolve the problem by deleting your Google cookie and revisiting Google. For browser-specific instructions, please consult your browser's online support center. If your entire network is affected, more information is available in the Google Web Search Help Center. We apologize for the inconvenience, and hope we'll see you again on Google. To continue searching, please type the characters you see below:"

staticshock 18y ago
interesting. at what point do you get that message?
- matt1 18y ago
  very quickly. when i check it in my browser it was fine, but as soon as i started scanning with the software google blocked it. they may have recognized that the requests werent coming from a standard browser, so they flagged my ip.
bullseye 18y ago
Include a referrer HTTP header, set the user agent, and set a random wait time between 500ms and 5s. I crawl Google frequently and that format has always worked for me... At least it did until I made this post. :)
- Hexstream 18y ago
  If you can bypass the rules they don't count, eh?
  - fauigerzigerk 18y ago
    To be honest I can't take it seriously when a company that makes billions from crawling other people's sites makes a rule that you're not allowed to crawl theirs. Google has a lot of conflicts with media companies and it always starts out with google ignoring some "rules" expecting to either win in court or settle at some point. So I think breaking these rules is part of the process of how sensible rules are established in the first place. Yes it's recursive, but HN readers should be smart enough to understand that ;-)
    - neilc 18y ago
      To be honest I can't take it seriously when a company that makes billions from crawling other people's sites makes a rule that you're not allowed to crawl theirs. Why not? Google obeys robots.txt -- if you don't want them to crawl your site, it is trivial to arrange. I think violating Google's terms of use is pretty hard to justify as ethical.
      - fauigerzigerk 18y ago
        Google obeys robots.txt but not much else. Just ask publishers and news outlets. And as I said, I don't see rules as a black and white thing. If they were, there would never be any progress. No iTunes without napster. No Microsoft being the biggest software company in the world without piracy. That doesn't mean it's ethically justified to break any rule. I just think it's not as simple as someone stating their own rules and everyone automatically obeying it.