I've definitely scraped by this problem on several occasions. Recently I was writing a tool to check outgoing links from my site, to see which sites are offline

I've definitely scraped by this problem on several occasions. Recently I was writing a tool to check outgoing links from my site, to see which sites are offline (it's called notfoundbot). What I found was that many sites have "DDoS Protection" that makes such an effort impossible, other sites whitelist the CuRL headers, others like it when you pretend to be a search engine. Basically writing some code that tests whether "a website is currently online or offline" is much, much harder than you think, because, yep, the only company that can do that is Google.