3 ms·
Google publishes how to control their bot - with robots.txt. They then obey those instructions. Google also takes some effort to not use all your bandwidth.
by bluGill 11mo ago
Google publishes how to control their bot - with robots.txt. They then obey those instructions. Google also takes some effort to not use all your bandwidth. Google isn't perfect, but they are at least making a "good faith" effort to be nice and this does count in court. Overall most will agree that in general what google does to allow people to find their website is worth the things that google is doing.
You can of course argue a lot of edge cases if you really want. For the most part I want to say "it isn't worth the argument". In some cases I will take your side if I really have to think about it, but in general the system google has been using mostly works and is mostly an acceptable compromise.
- hackerbeat 11mo agoWhat's nice about scraping all the content for their own good while killing off websites left and right? Google needs to be sued also. Along with all the other AI companies out there, the've committed the biggest theft in human history.
- deleted 11mo ago[deleted]
- pawelduda 11mo agoBut their robots are enabled by default. So it is a form of unsolicited scraping. If I spam millions of email addresses without asking for permission but provide a link to opt-out form, am I the good guy?
- bluGill 11mo agoAt this point everyone knows about robots.txt, so if you didn't opt-out that is your own fault. Opting out of everyone at once is easy, and you get fine grained control if you want it. Also most people would agree they are fine with being indexed in general. That is different from email spam where people don't want it.
- pawelduda 11mo agoLooking at SerpApi clients, looks like most companies would agree they are fine with scraping Google. That is different from having your website content stolen and summarized by AI on Google search, which people don't want.
- bluGill 11mo agoThe claim is SerApi is not honoring robots.txt, and they are getting far more data from google/more often than needed for an index operation. Or at least that is the best I can make out of the claim in court from the article - I have not read the actual complaint. People are generally fine with indexing operations so long as you don't use too much bandwidth. Using AI to summarize content is still and open question - I wouldn't be surprised if this develops to some form of "you can index but not summarize", but only time will tell.
- deleted 11mo ago[deleted]
- conartist6 11mo agoOr by Google codewiki, which is morally the equivalent to making a business out of ersatz travel guides by ripping off the authors of real ones
- khelavastr 11mo agoWho says robots.txt is legally binding? Where's the Sherman Antitrust analysis?I'm more confused than before.
- bluGill 11mo agoThe courts say. With this as a long standing tradition they are likely to agree.
- hn_acker 11mo ago> The courts say. Do you have an example of a court saying that violating robots.txt violates an existing law? In Ziff Davis v. OpenAI [1], the District Court for the Southern District of New York found that violating robots.txt does not violate DMCA section 1201(a) (formally 17 U.S. Code § 1201(a), which prohibits circumvention of technological protection measures of copyrighted content [2]). It's my understanding that robots.txt started as a socially-enforced rule and that it remains legally voluntary. [1] https://blog.ericgoldman.org/archives/2025/12/are-robots-txt-instructions-legally-binding-ziff-davis-v-openai.htm https://blog.ericgoldman.org/archives/2025/12/are-robots-txt... [2] https://www.law.cornell.edu/uscode/text/17/1201 https://www.law.cornell.edu/uscode/text/17/1201