10 ms·
Amazon's AI crawler is making my Git server unstable
- knowitnone 2y agoFeed them false data. If feed by enough people(I looking at you HN), their AI will be inaccurate to the point of being useless.
- tepidsaucer 2y agoIt's already useless, and companies seem to be doubling down.
- renewiltord 2y agoLol this isn’t Amazon. This is an impersonator. They’re crawling your git repo because people accidentally push secrets to git repos.
- ChrisArchitect 2y agoEarlier: https://news.ycombinator.com/item?id=42740095 https://news.ycombinator.com/item?id=42740095
- trebor 2y agoUpvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!
- mschuster91 2y agoGlobal tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62fdded3 https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62f...
- jks 2y agoOne such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.org/code/nepenthes/ https://web.archive.org/web/20250117030633/https://zadzmo.or...
- kazinator 2y agoHow do you know their site is down? You probably just hit their tarpit. :)
- marcus0x62 2y agoQuixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content. 0 - https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
- bwfan123 2y agoi would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.
- idlewords 2y agoThis doesn't work with the kind of highly distributed crawling that is the problem now.
- seethenerdz 2y agoDon't we have intellectual property law for this tho?
- gundmc 2y agoWhat do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
- unsnap_biceps 2y agoI believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
- noman-land 2y agoThis is highly annoying and rude. Is there a complete list of all known bots and crawlers?
- jsheard 2y agohttps://darkvisitors.com/agents https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
- joecool1029 2y agoEven some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- SR2Z 2y agoIA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
- 2y ago
- Vampiero 2y ago> The consequence will almost universal blocks otherwise! Who cares? They've already scraped the content by then.
- jsheard 2y agoBold to assume that an AI scraper won't come back to download everything again, just in case there's any new scraps of data to extract. OP mentioned in the other thread that this bot had pulled 3TB so far, and I doubt their git server actually has 3TB of unique data, so the bot is probably pulling the same data over and over again.
- xena 2y agoFWIW that includes other scrapers, Amazon's is just the one that showed up the most in the logs.
- _heimdall 2y agoIf they only needed a one-time scrape we really wouldn't be seeing noticeable not traffic today.
- seethenerdz 2y agoThat's the spirit!
- herpdyderp 2y ago> The consequence will almost universal blocks otherwise! How? The difficulty of doing that is the problem, isn't it? (Otherwise we'd just be doing that already.)
- ADeerAppeared 2y ago> (Otherwise we'd just be doing that already.) Not quite what the original commenter meant but: WE ARE. A major consequence of this reckless AI scraping is that it turbocharged the move away from the web and into closed ecosystems like Discord. Away from the prying eyes of most AI scrapers ... and the search engine indexes that made the internet so useful as an information resource. Lots of old websites & forums are going offline as their hosts either cannot cope with the load or send a sizeable bill to the webmaster who then pulls the plug.
- ksec 2y agoIs there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS? Or they could at least have the curtesy to scrap during night time / off peak hours.
- jsheard 2y agoNo, because they won't pay for anything they can get for free. There's only one situation where an AI company will pay for data, and that's when it's owned by someone with scary enough lawyers to pressure them into paying up. Hence why OpenAI has struck licensing deals with a handful of companies while continuing to bulk-scrape unlicensed data from everyone else.
- tredre3 2y agoThere is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/ https://commoncrawl.org/
- seethenerdz 2y agoIs existing intellectual property law not sufficient? Why aren't companies being prosecuted for large-scale theft?
- emmelaich 2y agoIf they're AI bots it might be fun to feed them nonsense. Just send hack megabytes of "Bezos is a bozo" or something like that. Even more fun if you could cooperate with many other otherwise-unrelated websites, e.g. via time settings in a modified tarpit.
- seethenerdz 2y agoDon't worry, though, because IP law only applies to peons like you and me. :)
- awsanswers 2y agoUnacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck
- thayne 2y agoThey said that it is coming from different ip addresses every time, so fail2ban wouldn't help.
- keisborg 2y agoMonitor access logs for links that only crawlers can find. Edit: oh, I got your point now.
- jsheard 2y agoAmazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively. https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...
- xena 2y agoI'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.
- unsnap_biceps 2y agoIf you only block new inbound requests, it shouldn't impact your route 53 or DNS-01 usage.
- actuallyalys 2y agoThey list a service for each address, so maybe you could block all the non-Route 53 IP addresses. Although that assumes they aren’t using the Route 53 IPs or unlisted IPs for scraping (the page warns it’s not a comprehensive list). Regardless, it sucks that you have to deal with this. The fact that you’re a customer makes it all the more absurd.
- srameshc 2y agoHas anyone tried using Cloudflare Bot Management and how effective is it for such bots ?
- martinp 2y agoI put my personal site behind Cloudflare last year specifically to combat AI bots. It's very effective, but I hate that the web has devolved to a state where using a service like Cloudflare is practically no longer optional.
- unsnap_biceps 2y ago> About 10% of the requests do not have the amazonbot user agent. Is there any bot string in the user agent? I'd wonder if it's GPTBot as I believe they don't respect a robots.txt deny wildcard.
- deleted 2y ago[deleted]
- AznHisoka 2y agoExcuse my technical ignorance, but is it actually trying to get all the files in your git repo? Couldn’t you just have everything behind an user/pass if so?
- xena 2y agoAuthor of the article here. The behavior of the bot seems like this: while true { const page = await load_html_page(read_from_queue()); save_somewhere(page); foreach link in page { enqueue(link); } } This means that every link on every page gets enqueued and saved to do something. Naturally, this means that every file of every commit gets enqueued and scraped. Having everything behind auth defeats the point of making the repos public.
- Ndymium 2y agoI had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?
- jsheard 2y ago> Whatever happened to courtesy in scraping? Money happened. AI companies are financially incentivized to take as much data as possible, as quickly as possible, from anywhere they can get it, and for now they have so much cash to burn that they don't really need to be efficient about it.
- nicce 2y agoNeed to act fast before the copyright cases in the court gets handled.
- bwfan123 2y agonot only money, but also a culture of "all your data belong to us" because our ai going to save you and the world. the hubris reminds me of dot-com era. that bust left a huge wreckage. not sure how this one is going to land.
- __loam 2y agoIt's gonna be rough. If you can't make money charging people $200 a month for your service then something is deeply wrong.
- Analemma_ 2y agoThe same thing that happened to courtesy in every other context: it only existed in contexts where there was no profit to be made in ignoring it. The instant that stopped being true, it was ejected.
- baobun 2y ago
- aw4y 2y agojust add a forged link in the main page, pointing to a page that doesn't exist. when hit, block that ip. they will crawl only the first page in this way maybe?
- frankwiles 2y agoHave had several clients hit by bad AI robots in the last few months. Sad because it’s easy to honor robots.txt.
- keisborg 2y agoSounds like a job for nepenthes: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147
- neilv 2y agoCan demonstrable ignoring of robots.txt help the cases of copyright infringement lawsuits against the "AI" companies, their partners, and customers?
- adastra22 2y agoOn what legal basis?
- readyplayernull 2y agoTerms of use contract violation?
- bdangubic 2y agogood thought but zippy chance this holds up in Court
- hipadev23 2y agoRobots.txt is completely irrelevant. TOU/TOS are also irrelevant unless you restrict access to only those who have agreed to terms.
- dredmorbius 2y ago"By accessing this Site, you acknowledge that you have read, understand, and agree to abide by the terms described herein." <https://www.imperva.com/legal/website-terms-of-use/ https://www.imperva.com/legal/website-terms-of-use/> Many, many, many hits for this or similar language: <https://duckduckgo.com/?q=%22By+accessing+this+Site%2C+you+acknowledge%22+agree&t=ftsa&ia=web https://duckduckgo.com/?q=%22By+accessing+this+Site%2C+you+a...> Mind: just because it's written doesn't mean it's enforceable, but to argue that what you've just denied isn't a widely-used premise of online contracts of adhesion fails the simplest empirical test.
- hipadev23 2y ago
- 23B1 2y ago[flagged]
- navanchauhan 2y agoI think there’s a difference between crawling websites at a reasonable pace instead of just hammering the server to the point it’s unusable. Nobody has problems with the Google Search indexer trying to crawl websites in a responsible way
- 23B1 2y agoFor sure. I'm really just pointing out the inconsistent technocrat attitude towards labor, sovereignty, and resources.
- Analemma_ 2y agoMost of those of those artists aren’t any better though. I’m on a couple artists’ forums and outlets like Tumblr, and I saw firsthand the immediate, total 180 re: IP protection when genAI showed up. Overnight, everybody went from “copying isn’t theft, it leaves the original!” and other such mantras, to being die-hard IP maximalists. To say nothing of how they went from “anything can be art and it doesn’t matter what tools you’re using” to forming witch-hunt mobs against people suspected of using AI tooling. AI has made a hypocrite out of everybody.
- 23B1 2y agoManga nerds on Tumblr aren't the artists I'm worried about. I'm talking about people whose intellectual labor is being laundered by gigacorps and the inane defenses mounted by their techbro serfdom.
- tepidsaucer 2y agoSomething something man understand, something something salary depends on.
- discardedrefuse 2y ago
- thayne 2y agoAre you sure it isn't a DDoS masquerading as Amazon? Requests coming from residential ips is really suspicious. Edit: the motivation for such a DDoS might be targeting Amazon, by taking down smaller sites and making it look like amazon is responsible. If it is Amazon one place to start is blocking all the the ip ranges they publish. Although it sounds like there are requests outside those ranges...
- OptionOfT 2y agoYou should check your websites like grass dot io (I refuse to give them traffic). They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.
- Aurornis 2y agoYes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.
- skywhopper 2y agoIt’s not residential proxies. It’s Amazon using IPs they sublease from residential ISPs.
- voakbasda 2y agoI wonder if anyone has checked whether Alexa devices serve as a private proxy network for AmazonBot’s use.
- tepidsaucer 2y agoYes, people have probably analyzed Alexa traffic once or twice over the years.
- scubbo 2y agoI, too, hate this future. This[0] might be a satisfying way to fight back. [0] https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/
- surfingdino 2y agoReturn "402 Payment Required" and block?
- xyzal 2y agoNo. Feed them shit. Code with deliberate security vulns and so on.
- serhack_ 2y agohttps://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
- surfingdino 2y agoI approve.
- stainablesteel 2y agocrazy how what seemed like an excellent landmark case around webcrawling turned around like this so quickly due to AI
- byyll 2y agohttps://ip-ranges.amazonaws.com/ip-ranges.json https://ip-ranges.amazonaws.com/ip-ranges.json ?
- xena 2y agoI'd love it if Amazon could give me some AWS credit as a sign of good faith to make up for the egress overages their and other bots are causing, but the ads on this post are likely going to make up for it. Unblock ads and I come out even!
- Aurornis 2y agoI don’t think I’d assume this is actually Amazon. The author is seeing requests from rotating residential IPs and changing user agent strings > It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more. Impersonating crawlers from big companies is a common technique for people trying to blend in. The fact that requests are coming from residential IPs is a big red flag that something else is going on.
- paranoidrobot 2y agoI wouldn't put it past any company these days doing crawling in an aggressive manner to use proxy networks.
- smileybarry 2y agoWith the amount of "if cloud IP then block" rules in place for many things (to weed out streaming VPNs and "potential" ddos-ing) I wouldn't doubt that at all.
- cmeacham98 2y agoI work for Amazon, but not directly on web crawling. Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot https://developer.amazon.com/en/amazonbot). I've forwarded this internally just in case there is some crazy internal team I'm not aware of pulling this stunt, but I would strongly suggest the author treats this traffic as malicious and lying about its user agent.
- xena 2y agoRandomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain. Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.
- LukeShu 2y agoBefore I configured Nginx to block them: - Bytespider (59%) and Amazonbot (21%) together accounted for 80% of the total traffic to our Git server. - ClaudeBot drove more traffic through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot.
- dbaio 2y agosuffering with it as well. why can't they just `git clone` and do their stuff? =)
- s1mplicissimus 2y agoMy guess is: Because the cloning is done by a stochastic parrot and the people running the parrot don't have the slightest idea what they are doing
- rattlesnakedave 2y agoNo evidence provided that this is amazonbot or AI related. Someone is just upset that their website is getting traffic, which seems asinine.
- kazinator 2y agoWhat is the proof that a hit from a residential IP address is actually Amazon? And if you have a way to tell, why not make use of it.
- deleted 2y ago[deleted]
- trevor-e 2y agoWhat are the actual rules/laws about scraping? I have a few projects I'd like to do that involve scraping but have always been conscious about respecting the host's servers, plus whether private content is copyrighted. But sounds like AI companies don't give a shit lol. If anyone has a good resource on the subject I'd be grateful!
- lazystar 2y agoIf you go to a police station and ask them to arrest Amazon for accessing your website too often, will they arrest Amazon, or laugh at you? While facetious in nature, my point is that people walking around in real brick and mortar locations simply do not care. If you want police to enforce laws, those are the kinds of people that need to care about your problem. Until that occurs, youll have to work around the problem.
- tepidsaucer 2y agoOh, police will happily enforce IP law, if the MPAA or RIAA want them to. You only get a pass if you're a thought leader.
- deleted 2y ago[deleted]
- armchairhacker 2y agoI like the solution in this comment: https://news.ycombinator.com/item?id=42727510 https://news.ycombinator.com/item?id=42727510. Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.
- Szpadel 2y agoI had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something that all those requests have in common that would allow you to easily identify them (like very specific and usual combination of reported language and geo location, or the same outdated browser version, etc)
- aaomidi 2y agoMaybe ban ASNs /s
- koito17 2y agoThis was indeed one mitigation used by a site to prevent bots hosted on AWS from uploading CSAM and generating bogus reports to the site's hosting provider.[1] In any case, I agree with the sarcasm. Blocking data center IPs may not help the OP, because some of the bots are resorting to residential IP addresses. [1] https://news.ycombinator.com/item?id=26865236 https://news.ycombinator.com/item?id=26865236
- pixl97 2y agoYa if it's also coming from residences it's probably some kind of botnet
- 2y ago
- more_corn 2y agoCloudflare free plan has bot protection.
- deanc 2y agoWe have had the same problem at my client now for the last couple of months, but from Facebook (using their IP ranges). They don’t even respect the 429 headers and the business is hesitant to outright ban them in case it impacts open graph or Facebook advertising tooling.
- Havoc 2y agoHe seems to have a mistake in his rule? He's got "(Amazon)" while Amazon lists their useragent as "(Amazonbot/0.1;"
- evantbyrne 2y agoIt seems like git self-hosters frequently encounter DDoS issues. I know it's not typical for free software, but I wonder if gating file contents behind a login and allowing registrations could be the answer for self-hosting repositories on the cheap.
- jdjdjdjdhy 2y agoSelf hosting a public git is as cheap as it can get, since it's completely static content Sure you might get bandwidth saturated, but that can happen with any type of content
- evantbyrne 2y agoIt's a lot less bandwidth to serve one login screen than the entire contents of all of your source repositories to a bunch of crawlers. You can also cache that one login screen in memory.
- freetanga 2y agoProbably dumb question, but any enlightenment would be welcome to help me learn: Could this be prevented by having a link that when followed would serve a dynamically generated page that does all of the following: A) insert some fake content outlining the oligarcs more lurid rumours or whichever disinformation you choose to push C) embed links to assets in oligarchs companies so they get hit with some bandwith C) dynamically create new Random pages that link to itself And thus create an infinite loop, similar to a gzip bomb, which could potentially taint the model if done by enough people.
- to11mtm 2y agoNot a crawler writer but have FAFOd with data structures in the past to large career success. ... The closest you could possibly do with any meaningful influence, is option C, with the general observations of: 1. You'd need to 'randomize' the generated output link 2. You'd also want to maximize cachability of the replayed content to minimize work. 3. Add layers of obfuscation on the frontend side, for instance a 'hidden link (maybe with some prompt fuckery if you are brave) inside the HTML with a random bad link on your normal pages. 4. Randomize parts of the honeypot link pattern. At some point someone monitoring logs/etc will see that it's a loop and blacklist the path. 5. Keep up at 4 and eventually they'll hopefully stop crawling. --- On the lighter side... 1. do some combination of above but have all honeypot links contain the right words that an LLM will just nope out of for regulatory reasons. That said, all above will do is minimize pain (except, perhaps ironically, the joke response which will more likely blacklist you but potentially get you on a list or a TLA visit)... ... Most pragmatically, I'd start by suggesting the best option is a combination of nonlinear rate limiting, both on the ramp-up and the ramp-down. That is, the faster requests come in, the more you increment their 'valueToCheckAgainstLimit`. The longer it's been since last request, the more you decrement. Also pragmatically, if you can extend that to put together even semi-sloppy code to then scan when a request to a junk link that results in a ban immediately results in another IP trying to hit the same request... well ban that IP as soon as you see it, at least for a while. With the right sort of lookup table, IP Bans can be fairly simple to handle on a software level, although the 'first-time' elbow grease can be a challenge.
- vachina 2y agoI’m surprised everyone else’s servers are struggling to handle a couple of bot scrapes. I run a couple of public facing websites on a NUC and it just… chugs along? This is also amidst the constant barrage of OSINT attempts at my IP.
- cyrnel 2y agoSeems some of these bots are behaving abusively on sites with lots of links (like git forges). I have some sites receiving 200 requests per day and some receiving 1 million requests per day from these AI bots, depending on the design of the site.
- xena 2y agoGitea in particular is a worst case for this. Gitea shows details about every file at every version and every commit if you click enough. The bots click every link. This fixed cost adds up when hundreds of IPs are at different levels of clicking of every link.
- TonyTrapp 2y agoDepends on what you are hosting. I found that source code repository viewers in particular (OP mentions Gitea, but I have seen it with others as well) are really troublesome: Each and every commit that exists in your repository can potentially cause dozens if not hundres of new unique pages to exist (diff against previous version, diff against current version, show file history, show file blame, etc...). Plus many repo viewers of them take this information directly from the source repository without much caching involved, as it seems. This is different from typical blogging or forum software, which is often designed to be able to handle really huge websites and thus have strong caching support. So far, nobody expected source code viewers to be so popular that performance could be an issue, but with AI scrapers this is quickly changing.
- marginalia_nu 2y agoCrawling a git repository can be pretty resource intensive on the server side. Viewing the state of HEAD is pretty fast, but once you go deep into historical commits, it requires considerable processing to reconstruct the historical state of the repository.
- serhack_ 2y agoIndeed: https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html try not to block LLM bot traffic and start injecting spurious content for ""improving"" their data. Markov chain at its finest!
- gazchop 2y agoBack to Gopher. They'll never get us there!
- ThinkBeat 2y agoThe best way to fight this would not to block them, that does not cause Amazon/others anything. (clearly). What if instead it was possible to feed the bots clearly damaging and harmfull content? If done on a larger scale, and Amazon discovers the poisoned pills they could have to spend money rooting it out, quick like, and make attempts to stop their bots to ingest it. Of course nobody wants to have that tuff on their own site though. That is the biggest problem with this.
- ADeerAppeared 2y ago> What if instead it was possible to feed the bots clearly damaging and harmfull content? With all respect, you're completely misunderstanding the scope of AI companies' misbehaviour. These scrapers already gleefully chow down on CSAM and all other likewise horrible things. OpenAI had some of their Kenyan data-tagging subcontractors quit on them over this. (2023, Time) The current crop of AI firms do not care about data quality. Only quantity. The only thing you can do to harm them is to hand them 0 bytes. You would go directly to jail for things even a tenth as bad as Sam Altman has authorized.
- smeggysmeg 2y agoI've seen this tarpit recommended for this purpose. it creates endless nests of directories and endless garbage content, as the site is being crawled. The bot can spend hours on it. https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/
- amarcheschi 2y agoIf with damaging content you mean incorrect content, another comment in this thread has a user doing what you said https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
- tepidsaucer 2y agoWhy is it my responsibility to piss into the wind because these billionaire companies keep getting to break the law with impunity?
- ThinkBeat 2y agoHow many TB is your repo? Do they keep retrieving the same data from the same links over and over and over again, like stuck in a forever loop, that runs week after week? Or are they crawling your site at a hype aggressive way but getting more and more data? So it may tea them last say 2 days to crawl over it and then they go away?
- Animats 2y agoIt's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the Department will consider defendants from that point onward not to be authorized." So, you get a lawyer to write an "unambiguous cease and desist" letter. You have it delivered to Amazon by either registered mail or a process server, as recommended by the lawyer. Probably both, plus email. Then you wait and see if Amazon stops. If they don't stop, you can file a criminal complaint. That will get Amazon's attention. [1] https://www.justice.gov/jm/jm-9-48000-computer-fraud https://www.justice.gov/jm/jm-9-48000-computer-fraud
- xena 2y agoHonestly, I figure that being on the front page of Hacker News like this is more than shame enough to get a human from the common sense department to read and respond to the email I sent politely asking them to stop scraping my git server. If I don't get a response by next Tuesday, I'm getting a lawyer to write a formal cease and desist letter.
- DrBenCarson 2y agoLol you really think an ephemeral HN ranking will make change?
- idlewords 2y agoMy site (Pinboard) is also getting hammered by what I presume are AI crawlers. It started out this summer with Chinese and Singapore IPs, but now I can't even block by IP range, and have to resort to captchas. The level of traffic is enough to immediately crash the site, and I don't even have any interesting text for them to train on, just link. I'm curious how OP figured out it's Amazon's crawler to blame. I would love to point the finger of blame.
- advael 2y agoUnless we start chopping these tech companies down there's not much hope for the public internet. They now have an incentive to crawl anything they can and have vastly more resources than even most governments. Most resources I need to host in a way that's internet facing are behind keyauth and I'm not sure I see a way around doing that for at least a while
- tepidsaucer 2y agoThat is an unworkable solution considering most of the people here are employed by these companies.
- amanaplanacanal 2y agoI would bet only a tiny minority here are working for big tech.
- advael 2y agoCertainly most of the ones with green names for whom a reply to me criticizing a big tech company or disavowing them in aggregate is among their first ten must be
- dmwilcox 2y agoI wonder if there is a good way to copy something out of fossil scm or externalize this component for more general use. https://fossil-scm.org/home/doc/trunk/www/antibot.wiki https://fossil-scm.org/home/doc/trunk/www/antibot.wiki I ran into this weeks ago and was super impressed to solve a self-hosted captcha and login as "anonymous". I use cgit currently but have dabbled with fossil previously and if bots were a problem I'd absolutely consider this
- Aloisius 2y agoUsing status code 418 (I'm a teapot), while cute, actually works against you since even well behaved bots don't know how to handle it and thus might not treat it as a permanent status causing them to try to recrawl again later. Plus you'll want to allow access to /robots.txt. Of course, if they're hammering new connections, then automatically adding temporary firewall rules if the user agent requests anything but /robots.txt might be the easiest solution. Well or just stick Cloudflare in front of everything.
- dizhn 2y agoIt's been pounding one of my sites too. Here's the url it's trying to get to. I wonder if someone will ever figure it out and stop it. > /wp-content/uploads/2014/09/contact-us/referanslar/petrofac/wp-content/uploads/2014/09/products_and_services/products_and_services/catalogue/references/capabilities/about-company/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/themes/domain/images/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/themes/domain/images/wp-content/uploads/2014/09/wp-content/themes/domain/images/about-company/corporate-philosophy/index.htm "User-Agent":["Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36"
- orf 2y agoI bet that URL (or a predecessor) resolves to a 404 page that has a broken link on it. technically your problem.
- anothernewdude 2y agoIf blacklisting user-agents doesn't work, whitelist the clients you actually want to accept.
- andrethegiant 2y agoI’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a convenient option to crawler authors that makes their bots play nice with the open web. [1] https://crawlspace.dev https://crawlspace.dev
- subarctic 2y agoHow does it compare to common crawl?
- andrethegiant 2y agoCommonCrawl’s data is up to a month old. Here you can write a custom crawler, and have a REST/RAG API that you can use with your apps and agents for the data that your crawler finds. All the while, it builds up a platform-wide cache, so duplicate/redundant requests don’t reach their origin servers.
- philipwhiuk 2y agoAre you sure you're not just encouraging more people to run them?
- andrethegiant 2y agoEven so, traffic funnels through the same cache, so website owners would see the same amount of hits whether there was 1 crawler or 1000 crawlers on the platform
- philipwhiuk 2y agoSo you're forcing me and OP to sign-up for the protection racket.... Thanks
- jsheard 2y ago> respecting robots.txt What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.
- layer8 2y agoRecent related discussion: https://news.ycombinator.com/item?id=42549624 https://news.ycombinator.com/item?id=42549624
- benreesman 2y agoHas any group of tech companies ever behaved so badly in so many ways so uniformly? There was 90s Microsoft, that was bad enough to get the DOJ involved, but that was one actor. It’s like the friggin tobacco companies or something. Is anyone being the “good guys” on this?
- A1kmm 2y agoIf you use nginx to front it, consider something like this in the `http` block of your config: map $http_user_agent $bottype { default ""; "~.*Amazonbot.*" "amazon"; "~.*ImagesiftBot.*" "imagesift"; "~.*Googlebot.*" "google"; "~.*ClaudeBot.*" "claude"; "~.*gptbot.*" "gpt"; "~.*semrush.*" "semrush"; "~.*mj12.*" "mj12"; "~.*Bytespider.*" "bytedance"; "~.*facebook.*" "facebook"; } limit_req_zone $bottype zone=bots:10m rate=6r/m; limit_req zone=bots burst=10 nodelay; limit_req_status 429; You can still have other limits by IPs. 429s tends to slow the scrapers, and it means you are spending a lot less on bandwidth and compute when they get too aggressive. Monitor and adjust the regex list over time as needed. Note that if SEO is a goal, this does make you vulnerable to blackhat SEO by someone faking a UA of a search engine you care about and eating their 6 req/minute quota with fake bots. You could treat Google differently. This approach won't solve for the case where the UA is dishonest and pretends to be a browser - that's an especially hard problem if they have a large pool of residential IPs and emulate / are headless browsers, but that's a whole different problem that needs different solutions.
- lobsterthief 2y agoFor Google, just read their publicly-published list of crawler IPs. They’re broken down into 3 JSON files by category. One set of IPs is for GoogleBot (the web crawler), one is for special requests like things from Google Search Console, and one is special crawlers related to things like Google Ads. You can ingest this IP list periodically and set rules based on those IPs instead. Makes you not prone to the blackhat SEO tactic you mentioned. In fact, you could completely block GoogleBot UA strings that don’t match the IPs, without harming SEO, since those UA strings are being spoofed ;)
- Groxx 2y ago>I'm working on a proof of work reverse proxy to protect my server from bots in the future. Honestly I think this might end up being the mid-term solution. For legitimate traffic it's not too onerous, and recognized users can easily have bypasses. For bulk traffic it's extremely costly, and can be scaled to make it more costly as abuse happens. Hashcash is a near-ideal corporate-bot combat system, and layers nicely with other techniques too.
- xena 2y agoI'm gonna publish more about this tomorrow. I didn't expect this to blow up so much.
- Groxx 2y agoThank you for the issue-awareness-building and the tool! Hopefully not too much hn-drama went your way, I always enjoy your content :)
- xena 2y agoNo prob! I wrote more about my defense tool here in case you're interested: https://xeiaso.net/blog/2025/anubis/ https://xeiaso.net/blog/2025/anubis/
- spike_protein 2y agoMaybe you can use https://www.cloudflare.com/lp/ppc/bot-management https://www.cloudflare.com/lp/ppc/bot-management
- Timber-6539 2y agoOne time had an AWS customer (Nokia Deepfield) spamming my shadowsocks server logs with unauthenticated attempts. Emailed AWS to curtail the abuse and they sided with their customer ofc.
- qwaxys 2y agoNothing wrong with putting a link to a zip bomb in your robots.txt and disallowing them. Also, just put a limit on requests per IP? https://nginx.org/en/docs/http/ngx_http_limit_req_module.html https://nginx.org/en/docs/http/ngx_http_limit_req_module.htm...
- jdjdjdjdhy 2y agoHow would that help you? Bots generally don't care about zip contents
- Daril 2y agoI recently experienced a similar situation on a client's small ecommerce site. After struggling for 4 days to find the cause of a strange excessive server load every afternoon, I discovered that a Woocommerce site was being hammered by several bots at the same time : AHrefs, Bitespider, MJ12Bot, SemrushBot, Claude, GPT, ... I set up a proper robots.txt file (the content doesn't change frequently, so indexing it every day doesn't make sense), but also set up proper rules in .htaccess to block all these user-agents. I also banned 20 or more of the IP addresses of these bots. The next step is to set up fail2ban to automatically ban them when they change IP or add additional IP addresses.
- jdjdjdjdhy 2y agoHaving a public server an crying about a little bit of traffic seems pathetic Just cache static content and throttle by IP. This isn't a DDoS, just rotating IPs
- lucasauton01 2y ago[dead]