4 ms·

I couldn't care less about these huge tech companies stealing from one another. Let them sue themselves to extinction.

graemep 2y ago
Hos is it even stealing? its taking copies of public information, mostly under open source licences. Amazon is causing a bit of extra server load for MS to handle.
- noprocrasted 2y ago
  > its taking copies of public information Yet the same companies will be first to tell you that scraping their public information is against ToS or even illegal. See the whole drama about LinkedIn scraping, etc.
  - jsheard 2y ago
    It's simple: AI companies are allowed to scrape whatever they want, but if you scrape an AI companies data then you are a copyright terrorist and you will never see the light of heaven.
    - lupusreal 2y ago
      This double standard from Amazon particularly predates AI by decades. They scrape their e-commerce competition but don't want anybody to scrape them back.
      - graemep 2y ago
        I have clients who have done it. The last one was a service for brands who wanted to see how their products were placed and described on Amazon.
- lobsterthief 2y ago
  You do know Microsoft has also used all private repositories to train its models, right? Especially for Copilot
  - graemep 2y ago
    Yes, and I do not like that. On the other hand I do not have any private Github repos of my own and its a good reason for not having any. I am beginning to feel that if you really want repos to be private you should probably self host.
- 7373737373 2y ago
  > a bit of extra server load One of my sites has been spammed by scrapers (Bytedance's Bytespider, Googlebot, Bingbot) several thousand times within just an hour, to the point of making it break. They do this without notification or asking for consent of the users creating the content they ingest and possibly use to train AI models with, and also without credit or compensation. I think the world needs strict regulation against this kind of parasitic, likely illegal behavior.
  - ta1243 2y ago
    Did they ignore your robots.txt file?
    - 7373737373 2y ago
      I changed it a few days ago, still getting hit by over 14800 requests within 24 hours
      - ta1243 2y ago
        From google etc? Sounds like an abuse case
        7373737373 2y ago
        19010 requests from user agents "Googlebot" and "bingbot" combined within the last 24h. Bytespider has died down to 74. 17 from Claudebot. But I've heard from a HN user that Bytedance just change their user agent when they are blocked. I'm blocking all of them with Cloudflare.
  - graemep 2y ago
    I think that is very different from cloning repos from Github in accordance with the licences. I agree crawling sites to the extent it causes problems is a problem. Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates. Is it possible that these are scrapers using fake UA strings?
beardyw 2y ago
What is being scraped is not GitHub s data. It's other people's.
- threecheese 2y ago
  Exactly. I permissively license my code, but not because I want to improve mega-corp’s bottom line. I’m annoyed. I felt exactly like this when I learned that some of my Goodwill donations - the good stuff - is marked up and sold online, instead of going to low income folks at low-income prices. It might be even worse, given the capability they are building intends to compete with me directly as a developer. It’s like if Goodwill started funding domestic terrorists or the local burglars union.

Hos is it even stealing? its taking copies of public information, mostly under open source licences. Amazon is causing a bit of extra server load for MS to handle.

noprocrasted 2y ago
> its taking copies of public information Yet the same companies will be first to tell you that scraping their public information is against ToS or even illegal. See the whole drama about LinkedIn scraping, etc.
- jsheard 2y ago
  It's simple: AI companies are allowed to scrape whatever they want, but if you scrape an AI companies data then you are a copyright terrorist and you will never see the light of heaven.
  - lupusreal 2y ago
    This double standard from Amazon particularly predates AI by decades. They scrape their e-commerce competition but don't want anybody to scrape them back.
    - graemep 2y ago
      I have clients who have done it. The last one was a service for brands who wanted to see how their products were placed and described on Amazon.
lobsterthief 2y ago
You do know Microsoft has also used all private repositories to train its models, right? Especially for Copilot
- graemep 2y ago
  Yes, and I do not like that. On the other hand I do not have any private Github repos of my own and its a good reason for not having any. I am beginning to feel that if you really want repos to be private you should probably self host.
7373737373 2y ago
> a bit of extra server load One of my sites has been spammed by scrapers (Bytedance's Bytespider, Googlebot, Bingbot) several thousand times within just an hour, to the point of making it break. They do this without notification or asking for consent of the users creating the content they ingest and possibly use to train AI models with, and also without credit or compensation. I think the world needs strict regulation against this kind of parasitic, likely illegal behavior.
- ta1243 2y ago
  Did they ignore your robots.txt file?
  - 7373737373 2y ago
    I changed it a few days ago, still getting hit by over 14800 requests within 24 hours
    - ta1243 2y ago
      From google etc? Sounds like an abuse case
      - 7373737373 2y ago
        19010 requests from user agents "Googlebot" and "bingbot" combined within the last 24h. Bytespider has died down to 74. 17 from Claudebot. But I've heard from a HN user that Bytedance just change their user agent when they are blocked. I'm blocking all of them with Cloudflare.
- graemep 2y ago
  I think that is very different from cloning repos from Github in accordance with the licences. I agree crawling sites to the extent it causes problems is a problem. Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates. Is it possible that these are scrapers using fake UA strings?

beardyw 2y ago

What is being scraped is not GitHub s data. It's other people's.

threecheese 2y ago
Exactly. I permissively license my code, but not because I want to improve mega-corp’s bottom line. I’m annoyed. I felt exactly like this when I learned that some of my Goodwill donations - the good stuff - is marked up and sold online, instead of going to low income folks at low-income prices. It might be even worse, given the capability they are building intends to compete with me directly as a developer. It’s like if Goodwill started funding domestic terrorists or the local burglars union.