9 ms·
Amazon has a way to scrape GitHub and feed its AI model
- hi-v-rocknroll 2y agoMSFT's LinkedIn scraping was also a thing about 10 years ago until the magic method was taken away. :'(
- altdataseller 2y agoYou can still scrape Linkedin today, can you not?
- hi-v-rocknroll 2y agoNo, I don't think so. Not without an account and not completely as was possible in the past.
- glimshe 2y agoI couldn't care less about these huge tech companies stealing from one another. Let them sue themselves to extinction.
- graemep 2y agoHos is it even stealing? its taking copies of public information, mostly under open source licences. Amazon is causing a bit of extra server load for MS to handle.
- noprocrasted 2y ago> its taking copies of public information Yet the same companies will be first to tell you that scraping their public information is against ToS or even illegal. See the whole drama about LinkedIn scraping, etc.
- jsheard 2y agoIt's simple: AI companies are allowed to scrape whatever they want, but if you scrape an AI companies data then you are a copyright terrorist and you will never see the light of heaven.
- lupusreal 2y agoThis double standard from Amazon particularly predates AI by decades. They scrape their e-commerce competition but don't want anybody to scrape them back.
- graemep 2y agoI have clients who have done it. The last one was a service for brands who wanted to see how their products were placed and described on Amazon.
- lobsterthief 2y agoYou do know Microsoft has also used all private repositories to train its models, right? Especially for Copilot
- graemep 2y agoYes, and I do not like that. On the other hand I do not have any private Github repos of my own and its a good reason for not having any. I am beginning to feel that if you really want repos to be private you should probably self host.
- 7373737373 2y ago> a bit of extra server load One of my sites has been spammed by scrapers (Bytedance's Bytespider, Googlebot, Bingbot) several thousand times within just an hour, to the point of making it break. They do this without notification or asking for consent of the users creating the content they ingest and possibly use to train AI models with, and also without credit or compensation. I think the world needs strict regulation against this kind of parasitic, likely illegal behavior.
- ta1243 2y agoDid they ignore your robots.txt file?
- 7373737373 2y agoI changed it a few days ago, still getting hit by over 14800 requests within 24 hours
- ta1243 2y agoFrom google etc? Sounds like an abuse case
- 7373737373 2y ago19010 requests from user agents "Googlebot" and "bingbot" combined within the last 24h. Bytespider has died down to 74. 17 from Claudebot. But I've heard from a HN user that Bytedance just change their user agent when they are blocked. I'm blocking all of them with Cloudflare.
- graemep 2y agoI think that is very different from cloning repos from Github in accordance with the licences. I agree crawling sites to the extent it causes problems is a problem. Googlebot and Bingbot do follow robots.txt and respect HTTP 429 responses and usually have reasonable default crawl rates. Is it possible that these are scrapers using fake UA strings?
- beardyw 2y agoWhat is being scraped is not GitHub s data. It's other people's.
- threecheese 2y agoExactly. I permissively license my code, but not because I want to improve mega-corp’s bottom line. I’m annoyed. I felt exactly like this when I learned that some of my Goodwill donations - the good stuff - is marked up and sold online, instead of going to low income folks at low-income prices. It might be even worse, given the capability they are building intends to compete with me directly as a developer. It’s like if Goodwill started funding domestic terrorists or the local burglars union.
- Kye 2y agoIs it git pull? >> "In response, Amazon proposed a workaround: encouraging its employees to create multiple GitHub accounts and share their access credentials." Ah, no, it's git pool.
- xmodem 2y agoEthically Microsoft has about as much claim to be able to use the data for co-pilot as anyone else. On the other hand, maybe a MSFT v Amazon lawsuit over this could be the wake up call the world needs that maybe we should stop centralising critical infrastructure in the hands of a single company. Which is why I think they wouldn't do it - at most I could see Microsoft tightening request limits on accounts associated with Amazon.
- cyanydeez 2y agoWell do.that right when distributed computing finds a workable modrl
- xmodem 2y agoYeah, I guess building a distributed version control system is basically an intractable problem.
- amadeuspagel 2y agoGitHub offers more then version control. A decentralized version is about as realistic as a decentralized facebook.
- xmodem 2y agoOf course - I was being facetious. But I disagree with your assessment of how realistic it is. We already have it on a small scale today in the form of some projects using their own gitlab/gitea instances. If Github were to enshitify I expect we would see a push for this from a lot more communities. I don't think it's that much of a stretch to design a system that keeps, for instance, issue, wiki, and PR metadata in git alongside code. This could then support simple import/export between instances. You could also support cross-instance forking and PR's. The biggest problems I think you would still have are 1. third-party integrations and 2. abuse/spam prevention. Having been the system-owner for GitHub at a large engineering org before, I can say that for us, switching away would have been virtually impossible because of all the integrations we would have to replace. But, this is a consequence of the centre of gravity being Github currently and not an immutable law of nature. As for 2, well I expect that'll remain one of the hard unsolved problems of computer science for the time being.
- htrp 2y agodisappointing that large mega Corp does the exact same thing broke developers do to get around rate limits
- shermantanktop 2y agoThis is a large mega corp that prides itself on acting like it is broke.
- amelius 2y agoExcept they have access to one of the largest "botnets" on the planet.
- koolba 2y agoIs the cover image itself generated via some ML model? The old guy in the middle is missing substantial parts of his arm. The box right by him also has some artifacting in the corner.
- willwade 2y agoand the guy on the right.. umm.. what's with his face? Or is he an Alien maybe? Image credit goes to https://linkmedya.com https://linkmedya.com - it doesn't say it is AI-generating content but yep, it certainly looks like it
- amelius 2y agoNo, this depicts exactly the nightmarish nature of a job at an Amazon warehouse.
- KineticLensman 2y agoYes, the Amazon brand arrow at centre top is also broken. In fact all of the people look wrong in some way
- deleted 2y ago[deleted]
- batch12 2y agoYeah. It is likely an edited AI image. You can confirm by looking at the text on the box at the top left. "BMGOMa"
- stamourd 2y ago"Featured image credit: Eray Eliaçık/Bing"
- firtoz 2y agoAnd the "Bing" just links to Bing's Dalle3 functionality
- jsnell 2y agoI'm surprised Amazon's legal team signed off on this. It's clearly against the GitHub terms of service[0], and Amazon employees acting on the instructions from Amazon had to approve those terms. It seems pretty much identical to the LinkedIn vs. hiQ scraping case, where as I understand the fake account creation was the key point. [0] E.g. no API key sharing for the purposes of evading rate limits, only a single free account per person or organization.
- that_guy_iain 2y agoWhat if they‘re not free accounts?
- londons_explore 2y agoWhen you pay your legal teams as much as Amazons, they probably tell you "Yeah, you'd probably lose any case, but the fine will be a couple of million dollars and you won't have to pay it for a decade, and by then you'd have cemented your market leadership".
- raarts 2y agoLanguage in this article smells like it's written or rewritten by AI.
- belter 2y agoAgree. Looks like we have a good hear: https://youtu.be/zbo6SdyWGns?t=78 https://youtu.be/zbo6SdyWGns?t=78
- rty32 2y agoIf github cares enough about this, they would have already sued Amazon. I don't think the author needs to worry about any of this
- paradite 2y agoMicrosoft is probably one of the few companies that can sue Amazon without worrying about retaliation from Amazon. For example, GitLab would need to think twice before suing because they offer deployment on AWS.
- neilv 2y agoSeparate from the courts, Microsoft could send a message to the AI gold rush field, about "abuse of Microsoft's resources", via ToS: * All Amazon domain names could be banned from accounts on GitHub, or face annoying restrictions, implemented with trivial technical changes. And lawyers could send a letter to Amazon legal, about how Amazon may and may not use GitHub, including Amazon personnel having to disclose their affiliation (not hide it with GMail), and craft some language about how those employee accounts may and may not be used. * More harshly, but fear-instilling to individuals throughout industry, the individuals who let their accounts be used for the scraping could be banned from GitHub, for ToS violation. Not only those particular accounts, but any accounts the individuals might use. (This would hurt, not only for genuine open source participation, but also given how open source is sometimes used for job-hunting appearances, and all the current employers that ask for candidate's "GitHub" specifically rather than open source in general.) If banning would have undesired effects of projects GitHub wants to host being pulled, or public reaction as too harsh and questioning why GitHub has so much power, there could instead be annoying restrictions.
- rdtsc 2y ago> the individuals who let their accounts be used for the scraping could be banned from GitHub, for ToS violation. That would work, assuming GH doesn’t make mistakes and ban someone else with the same name m. That would then be embarrassing for GH. I can already see news headline “Github banned my account because my name matches that of a web scraping account from Amazon”
- amadeuspagel 2y agoThey should send make this data available for everyone on AWS.
- lokimedes 2y agoThis just rekindled my desire to self-host my git repos. The whole idea that a platform provider can use the IP I host there is obscene. That thieves steal by bounty from each other is not the story.
- foreigner 2y agoMicrosoft could sabotage Amazon's AI model by returning poisoned code to accounts registered with @amazon.com email addresses.
- xmodem 2y agoThe way git works means that you can check that you have an un-doctored clone of a repo just by checking that the commit hash matches. Which in this instance is quite unfortunate, because it would be very funny. (barring a SHA-1 collision, of course) EDIT: i suppose another approach could be to invent poisoned repos out of whole cloth and only show them to Amazon, but I susepct that'd be even easier to detect.
- deleted 2y ago[deleted]
- threecheese 2y agoCan anyone share a Fermi estimation of the size of poison-pill training data required to impact code interpreter models? (of the size that AMZN might be building with this data) I expect it would vary by language/platform popularity (size of available training code). Is it infeasible to create or generate enough code, pushed to enough repositories, to impact the correctness of a model that includes the code in its training data set?
- lofaszvanitt 2y agoMS only provides the infra, everything else is other's hard work under the trojan horse open source whatever. If they introduce limits, time to leave github. This will evolve into an elsevier vs researchers kinda situation.
- chumanak 2y agoThis article doesn’t make any sense. Why would Amazon make their employees do all this when they can easily pay for a service like crawlbase or similar and easily scrape github without having to create employee accounts?