4 ms·

> SerpApi deceptively takes content that Google licenses from others They have a different definition of "licensing" than most people I guess. Aren't site oper

by Nextgrid 10mo ago

> SerpApi deceptively takes content that Google licenses from others They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training. The pot is calling the kettle black.

skybrian 10mo ago
As far as I know, Google respects robots.txt and doesn't obfuscate their crawlers, so you can easily block them if you want. It seems like an important distinction?
- immibis 10mo ago
  There's no law that says you have to do that. It used to be a sensible thing to do, in the early internet. In the current internet, obeying robots.txt is a self-handicap and you shouldn't do it. DDoS remains illegal regardless of robots.txt.
  - skybrian 10mo ago
    It's rather odd to use words like "should" when you're advocating for disrespecting other people's wishes. There are sometimes reasons not to cooperate, but it seems like a good default.
    - immibis 10mo ago
      The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
      - skybrian 10mo ago
        Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.
        immibis 10mo ago
        "Making a search engine is unethical" is certainly one of the takes of all time. I'm sure Google is glad you believe it.
- Nextgrid 10mo ago
  Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway. SerpApi doesn't have that privilege.
  - bitpush 10mo ago
    but SerpApi is not scraping websites, it is sending malicoius requests to google.com.
    - Nextgrid 10mo ago
      SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.
  - xnx 10mo ago
    Google has respected robots.txt from the start.
  - skybrian 10mo ago
    Some domains do block Google, often partially. There are some statistics here: https://radar.cloudflare.com/ai-insights#ai-user-agents-found-in-robotstxt https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...
- throw-12-16 10mo ago
  robots.txt is not a legally binding document, nobody needs to actually respect it

skybrian 10mo ago

As far as I know, Google respects robots.txt and doesn't obfuscate their crawlers, so you can easily block them if you want. It seems like an important distinction?

immibis 10mo ago
There's no law that says you have to do that. It used to be a sensible thing to do, in the early internet. In the current internet, obeying robots.txt is a self-handicap and you shouldn't do it. DDoS remains illegal regardless of robots.txt.
- skybrian 10mo ago
  It's rather odd to use words like "should" when you're advocating for disrespecting other people's wishes. There are sometimes reasons not to cooperate, but it seems like a good default.
  - immibis 10mo ago
    The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.
    - skybrian 10mo ago
      Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.
      - immibis 10mo ago
        "Making a search engine is unethical" is certainly one of the takes of all time. I'm sure Google is glad you believe it.
Nextgrid 10mo ago
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway. SerpApi doesn't have that privilege.
- bitpush 10mo ago
  but SerpApi is not scraping websites, it is sending malicoius requests to google.com.
  - Nextgrid 10mo ago
    SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.
- xnx 10mo ago
  Google has respected robots.txt from the start.
- skybrian 10mo ago
  Some domains do block Google, often partially. There are some statistics here: https://radar.cloudflare.com/ai-insights#ai-user-agents-found-in-robotstxt https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...
throw-12-16 10mo ago
robots.txt is not a legally binding document, nobody needs to actually respect it