5 ms·
Stop Scraping My Git Forge
- dcminter 2y agoI think if I were feeling adversarial I'd go for poisoning rather than blocking. Set up something (heh, maybe even an AI model) to generate plausible but broken fake source code when you're scraped by AI bots. GIGO
- mrweasel 2y agoThat was my first thought. Normal looking repo when viewed using Git or a browser, the nastiest, buggiest malware you've ever seen when scraped by an AI bot. My second thought: Surely no company is stupid enough to train an AI on data that hasn't been checked and verified good by a real human expert.... Then I remember that Google just assumed that Reddit is reliable.
- barfbagginus 2y agoPOV: injecting AI bots with all this horrible attack code actually accelerates the rate at which they go rogue and start arming themselves with novel attacks. In effect, your attack spam AI will be a transfer learning provider for the scraper AI, and you're giving free training! Very cool! As someone who wants to see and maybe even build self sustaining rogue AIs inhabiting random niches in the information ecology, it would be cool to see corporate scraper AIs eventually run off their rails because we gave them too much attack spam!
- yencabulator 2y agoI'll just leave this here: https://research.swtch.com/zip https://research.swtch.com/zip
- yazzku 2y agoThanks, I'll get working on the git version of it.
- craftkiller 2y agoThe thing that annoys me the most is I can watch them crawl through my git forge, pulling every file from every commit one at a time, which is the least efficient way to do a `git clone`.
- mixologic 2y agoHow does the word "Robots.txt" not appear anywhere in this article.
- asylteltine 2y ago[dead]
- hu3 2y agoIf Amazon devs actually read my code, they would be the ones not wanting to crawl it.
- kinduff 2y agoIs there a reason why you are not using robots.txt [1] to block it? [1]: https://developer.amazon.com/amazonbot#how-can-i-control-what-amazonbot-crawls-on-my-site https://developer.amazon.com/amazonbot#how-can-i-control-wha...
- dylan604 2y agoIs there a reason you feel that that file will be respected?
- lambdaxyzw 2y agoIs there a reason why you don't? Is it just general bitterness and cynicism? As far as I know all major search engines respect rebots.txt, I don't see why LLM scrappers would be different.
- dylan604 2y agoprobably, yes. But coming from LLM scrappers, I have absolutely no faith in any of them. When one of them calls themself "open" in their name and is anything but, why would I trust them for anything after they lie in their name? I also do not trust Google only crawls what is allowed in robots.txt. Maybe they only use the data allowed in public use, but I have no faith that they don't have crawled data in their version of shadow profiles. I do not trust bigTech at all, and for those that do, I really don't understand why you do.
- kinduff 2y agoBecause it's on their documentation. If OP had the file and entry, and they didn't respect it, then it would be another conversation.
- kemayo 2y agoBots from big companies like Amazon, which is who the author is complaining about, do tend to respect it. In fact, it's listed in their documentation that the GP linked to that they will. They could be lying -- but why bother?
- hiatus 2y ago> While generally I post my code under open source licenses to others can easily use it, learn from it, modify it, whatever they like, I do not like the idea that large corporations are taking this same code and putting it into a black box that is a machine learning model. > Corporations stealing, or using work without permission, for their machine learning models has been a discussion for a long while at this point. In general, I side with the creators or artists having their work taken. I don't get it. How can corporations be stealing anything from an open source project? Further, it seems like several of the repos are based on other people's code. What code of the author's do they have reservations against training AI on?
- kosma 2y agoThey use the code without respecting the license terms. Just because the code is out in the open doesn't mean you can use it however you wish.
- lambdaxyzw 2y ago>How can corporations be stealing anything from an open source project? The code is published using some license that allows some use cases and prohibits other. For example GPL is famous for being viral. Using it to teach a LLM that spits "unlicensed" code is basically laundering copyright.
- hiatus 2y agoUsing it to train an LLM seems orthogonal to the output of the LLM. For instance, they could have their LLM include a link to the license. Merely training an LLM on the data does not seem to be against the spirit of GPL or Apache license.
- slindsey 2y agoI've heard AI advocates talk about a "right to read" or "right to learn"; meaning that we have the right to read something and then internalize it and use it. Therefore, why shouldn't an AI have the same right? The difference to me seems to be that the AI has the ability to regurgitate it in whole. I can read a book, learn about the concepts, then use or repeat those concepts. The AI can do the same. But is it really "learning"? It may be just spewing out pieces of the content without any understanding. In which case it's a copyright violation, right?
- jakedata 2y agoHopefully the crawlers don't spend too much time looking at the results of The Underhanded C Contest. An AI might start disgorging code that contains some extremely subtle kinks it can use in the future when it is time to start project Paperclip Apocalypse.
- kemayo 2y agoTheir robots.txt is a 404: https://git.gmem.ca/robots.txt https://git.gmem.ca/robots.txt It seems like it'd be better to populate the robots.txt and request that all bots not scrape this site -- as-is they're still pounding away against the firewall and it'd be cleaner if they just went away. If it turns out they don't respect robots.txt, you can escalate from there.
- deleted 2y ago[deleted]
- Joel_Mckay 2y agoMany projects export/fork their internal FOSS projects onto public repositories on github or sourceforge. There is a lot of really good reasons to not host public snapshots, but primarily when a docker build script includes your site it can get hammered hard. At some point people have to make a choice to either give something away libre, monetize your time in other ways, or go closed ecosystem. Amazon servers are usually not "normal" users, and a black-hole of certain IP blocks is highly advisable in some circumstances.
- fragmede 2y agoA Google search for "amazonbot crawler stop" gives https://webmasters.stackexchange.com/questions/144715/how-do-i-stop-amazon-from-crawling-a-website https://webmasters.stackexchange.com/questions/144715/how-do... as, like, the 3rd result, which would have introduced the author to robots.txt. I don't have a mastodon account to message the author about it directly, and no other contact info is listed, unfortunately.
- gmemstr 2y agoAuthor here - yes, I know about robots.txt! This is one of those cases where, because I was already looking at data in one place, I implemented the fix I could in the same place :p I do plan to add a robots.txt and contribute one upstream as well.
- deleted 2y ago[deleted]
- kazinator 2y agoDon't bother; robots.txt is utterly useless. It is widely ignored by bots. IDEA: What you could do is instantly ban anything with "bot" in its agent string, which accesses your site without having probed robots.txt first. That's how robots.txt could be useful, whether you have one or not. Anything which doesn't even bother fetching it is rude garbage.
- kazinator 2y agoI have rate limits on my self-hosted site. A script monitors logs and will spot patterns of usage that resemble scraping, banning the IP address for a temporary period. It's tricky. If a botnet is doing the scraping, the requests will come from different addresses, each of those doing things at a reasonably slow rate as to not alert suspicion.