2 ms·

Also, isn't this illegal? Either Google respected rate limits (which I doubt) or it was a form of a DDoS attack.

by ivanjermakov 3y ago

dylan604 3y ago
It's only illegal if you poor people do it. For us rich corporations, we do what we want
aeyes 3y ago
I guess they wanted to be crawled because the Google crawler is easy to identify/block/rate limit.
nosecreek 3y ago
Are there specific laws that deal with rate limits? Honest question - I get that something too fast could be considered DDoS, but so long as it’s below a certain threshold wouldn’t it be okay (not sure how said threshold would be determined)?
- germinator 3y ago
  In the US, CFAA prohibits causing "damage", which includes "impairment to the integrity or availability" of data or systems. But as with many other things in law, it boils down to the court trying to assess your intent, whether you could've reasonably anticipated the outcome, and what that outcome ended up being. There's no law that says "you can't send more than n packets per hour".
beeboobaa3 3y ago
Google's entire business model is based on behavior that is now illegal.
alisonkisk 3y ago
Google respected rate limits. Reddit wanted the crawl.
jedberg 3y ago
It was fine because we let them.
fasa99 3y ago
Google is the kingmaker and basically a monopoly on search. If you're going to light up dollars to share with a bot, let it be with google, who on a lucky day might decide you are king (because you let them index your site) I would presume and assume google obeys the robots.txt mandate as well. But I would agree it's a very outstanding and real problem that is YC-worthy - sharing structured webpage data with trusted partners in a generic and efficient way. I've heard about various AI companies that perform such data scraping and structuring with AI, forget the name - this is many notches in sophistication above a Selenium-headless type driver. If only html were made into a model-view-controller neatly and users were let to bring their own views & controllers.

Also, isn't this illegal? Either Google respected rate limits (which I doubt) or it was a form of a DDoS attack.

dylan604 3y ago

It's only illegal if you poor people do it. For us rich corporations, we do what we want

aeyes 3y ago

I guess they wanted to be crawled because the Google crawler is easy to identify/block/rate limit.

nosecreek 3y ago

Are there specific laws that deal with rate limits? Honest question - I get that something too fast could be considered DDoS, but so long as it’s below a certain threshold wouldn’t it be okay (not sure how said threshold would be determined)?

germinator 3y ago
In the US, CFAA prohibits causing "damage", which includes "impairment to the integrity or availability" of data or systems. But as with many other things in law, it boils down to the court trying to assess your intent, whether you could've reasonably anticipated the outcome, and what that outcome ended up being. There's no law that says "you can't send more than n packets per hour".

beeboobaa3 3y ago

Google's entire business model is based on behavior that is now illegal.

alisonkisk 3y ago

Google respected rate limits. Reddit wanted the crawl.

jedberg 3y ago

It was fine because we let them.

fasa99 3y ago

Google is the kingmaker and basically a monopoly on search. If you're going to light up dollars to share with a bot, let it be with google, who on a lucky day might decide you are king (because you let them index your site) I would presume and assume google obeys the robots.txt mandate as well. But I would agree it's a very outstanding and real problem that is YC-worthy - sharing structured webpage data with trusted partners in a generic and efficient way. I've heard about various AI companies that perform such data scraping and structuring with AI, forget the name - this is many notches in sophistication above a Selenium-headless type driver. If only html were made into a model-view-controller neatly and users were let to bring their own views & controllers.