10 ms·
Devs say AI crawlers dominate traffic, forcing blocks on entire countries
- throwaway81523 2y agoSaaay, what happened to that assistant US attorney who prosecuted Aaron Swartz for doing exactly this? Oh wait, Aaron didn't have billions in VC backing. That's the difference.
- ggm 2y agoEntire country blocks are lazy, and pragmatic. The US armed forces at one point blocked AU/NZ on 202/8 and 203/8 on a misunderstanding about packets from China, also from these blocks. Not so useful for military staff seconded into the region seeking to use public internet to get back to base. People need to find better methods. And, crawlers need to pay a stupidity tax or be regulated (dirty word in the tech sector)
- noirscape 2y agoThey can absolutely work if you aren't expecting any traffic from those countries whatsoever. I don't expect any international calls... ever, so I block international calling numbers on my phone (since they are always spam calls) and it cuts down on the overwhelming majority of them. Don't see why that couldn't apply to websites either.
- kragen 2y agoIt depends on whether the information on the website is supposed to be publicly available or not. "This information is publicly available except to people from Israel" sends a really terrible message.
- Retric 2y agoIt sends a great message to crack down on these companies, as long as you mention why it’s blocked.
- kragen 2y ago"You're cut off from access to knowledge because you live in the same country as AI researchers"?
- cyanydeez 2y ago"researchers" is like ignoring the whole Capitalists. We don't call them script researchers. We call them script kiddies. There's the whole other side of these AI researchers, and thats just slop artisans.
- Retric 2y agoA DoS attack is a DoS attack even if someone is pretending to be a “Researcher.” People in Iran, Russia, etc get annoyed with sanctions but that’s kind of the point. If your government isn’t responding appropriately, yes you’ll get shafted it’s what you do after that which solves the problem.
- kragen 2y agoMe, I prefer to relate to people as individuals rather than, as you are advocating, interchangeable representatives of their area of residence. If what you want is World War III, this is how you get it. In particular, universal access to knowledge is a fundamental principle of liberalism.
- Retric 2y agoI’d personally love for someone to hand me a billion dollars no strings attached. That’s got nothing to do with solving the issues created by these people, but if you’re going to toss out meaningless non sequitur’s then I figure I might as well join in on the fun.
- ggm 2y agoSure. Absolutely works. Right up until it doesn't. I think the MIL was the wrong people to assume "we will never need packets from these network blocks" The other thing is that phone numbers follow a numbering scheme where +1 is north america and +64 is NZ. Its easy to know the longterm geographic consequence of your block, modulo faked out CLID. IP packets don't follow this logic and Amazon can deploy AWS nodes with IPs acquired in Asia, in any DC they like. The smaller hosting companies don't say the IP range they route for banks have no pornographers on them. It's really not sensible to use IP blocks except for the very specific cases like yours: "I never terminate international calls" is the NAT of firewalls: "I don't want incoming packets from strangers" sure the cheapest path is to block entire swathes of IPv4 and IPv6. But if you are in general service delivery, that rarely works. If you ran a business doing trade in China, you'd remove that block immediately.
- koito17 2y agoAlthough it's a very lazy practice, this is exactly how many Japanese sites (and internet services) fight against bad actors. In short, they block non-Japanese traffic and data center IPs. I expect these measures to become insufficient as consumers adopt IoT devices and provide ample amounts of residential IPs for botnets. As for phone numbers, businesses and individuals employ a similar strategy. Most "legitimate" phone numbers begin with 060 or 070. Due to lack of supply, telcos are gradually rolling out 080 numbers. 080 numbers currently have a bad reputation because they look unfamiliar to the majority of Japanese. Similarly, VoIP numbers all begin with 050, and many services refuse such numbers. Most people instinctively refuse to answer any call that is not from a 060 or 070 number.
- alabastervlog 2y agoCountry or region blocks based on IPs used to (c. ~2000) be pretty standard. Blackhole the blocks associated with China, Russia, and maybe Africa, and your failed-login logs drop from scrolling so fast you can't read them, to a handful of lines per minute. Almost all the traffic was from those blocks, and was malicious. Meanwhile, for many sites, your odds (especially back then) of getting legitimate traffic from, say, China, was nearly zero, so the cost of blocking them was effectively nothing. Cloudflare is basically still just this, but with more steps.
- grotorea 2y agoIs this stuff only affecting the not for profit web? What are the for profit sites doing? I haven't seen Anubis around the web elsewhere. Are we just going to get more and tighter login walls and send everything into the deep web?
- surfingdino 2y agoI think we killed the old web. We'll see new ways of communicating, publishing, and gathering over the internet. It's sad, but it's also exciting.
- epolanski 2y agoLiterally nothing in this data driven world (sports, technology, entertainment, everything is data driven and maximized) is exciting, nothing.
- burkaman 2y agoFor profit sites are making deals directly with the AI companies so they can get some more of that profit.
- _xtrimsky 2y agoI think what they mean is that most not for profit small sites don't have expensive hardware or DDOS blocking mechanisms. A small 256mb ram vps might be enough for 1000 users per month traffic, but not enough for 200,000 users a day traffic.
- xena 2y agoWow it is so surreal to see a project of mine on Ars Technica! It's such an honor!
- true_blue 2y agoOn the few sites I've seen using it so far, it's been a more pleasant (and cuter) experience for me than the captchas I'd probably get otherwise. good work!
- xena 2y agoThanks! The artist I'm contracting and I are in discussions on how to make the mascot better. It will be improved. And more Canadian.
- techjamie 2y agoThe ffmpeg website is also using it. First time I actually saw it in the wild.
- seafoamteal 2y agoI've seen Anubis a couple times irl, mostly on Sourcehut, and the first time I saw it I was like, "Hey, I remember that blog post!" Congratulations on making something both useful and usable!
- shric 2y agoYou're well into https://refactoringenglish.com/tools/hn-popularity/ https://refactoringenglish.com/tools/hn-popularity/ so enjoy the fame!
- Figs 2y agoHmm. Instead of requiring JS on the client, why don't you add a delay on the server side (e.g. 1 second default, adjustable by server admin) for requests that don't have a session cookie? For each session keep a counter and a timestamp. Every time you get a request from a session, look up the tracked entry, increment the counter (or initialize it if not found) and update the timestamp. If the counter is greater than a configured threshold, slow-walk the response (e.g. add a delay before forwarding the request to the shielded web server -- or transfer the response back out at reduced bytes/second, etc.) You can periodically remove tracking data for entries older than a threshold -- e.g. once a minute or so (adjustable) remove tracked entries that haven't made a request in the past minute to keep memory usage down. That'd effectively rate limit the worst offenders with minimal impact on most well-behaved edge-case users (like me running NoScript for security) while also wasting less energy globally on unnecessary computation through the proof-of-work scheme, wouldn't it? Is there some reason I'm not thinking of that would prevent that from working?
- edoloughlin 2y agoI'm being trite, but if you can detect an AI bot, why not just serve them random data? At least they'll be sharing some of the pain they inflict.
- DecentShoes 2y agoRandom data? Why not "recipes" that just say "Bezos is a pedo" over and over ?
- xena 2y agoYou can detect the patterns in aggregate. You can't detect it easily at an individual request level.
- bluGill 2y agoIn short if you get several million requests and expect to only get 100 you won't know which are the real requests and which are the AI ones - but it is obvious that the vast majority are AI.
- noirscape 2y agoBandwidth isn't free, not at the volume these crawlers scrape at; serving them random data (for example by leading them down an endless tarpit of links that no human would end up visiting) would still incur bandwidth fees. Also it's not identifiable AI bot traffic that's detected (they mask themselves as regular browsers and hop between domestic IP addresses when blocked), it's just really obviously AI scraper traffic in aggregate: other mass crawlers have no benefit from bringing down their host sites, except for AI. A search engine has nothing if it brings down the site they're scraping (and has everything to gain from identifying itself as a search engine to try and get favorable request speeds - the only thing they'd need to check is if the site in question isn't serving different data, but that's much cheaper), same with an archive scraper and those two are pretty much the main examples I can think of for most scraping traffic.
- charcircuit 2y ago
- tedunangst 2y ago> It remains unclear why these companies don't adopt more collaborative approaches and, at a minimum, rate-limit their data harvesting runs so they don't overwhelm source websites. If the target goes down after you scrape it, that's a feature.
- MathMonkeyMan 2y agoWhy? What is the goal of a scraper, and how does disabling the source of the data benefit them?
- TuxMark5 2y agoI guess one could make a point that competition will no longer have the access to the scraped data.
- randmeerkat 2y ago> Why? What is the goal of a scraper, and how does disabling the source of the data benefit them? The next scraper doesn’t get the data. People don’t realize we’re not compute limited for ai, we’re data limited. What we’re watching is the “data war”.
- cyanydeez 2y agoat this point we're _good data_ limited, which has little to do with scraping.
- XorNot 2y agoHonestly it's hard to tell how much more value the LLM people are going to get out of another copy of the internet. It feels a lot like they're stuck for improvements but management doesn't want to hear it.
- Davidzheng 2y agoIt's a bit strange to talk about stuck when the most recent breakthrough is less than a year old.
- ipaddr 2y agoI've had a number of content sites I've shut down a few sites in the last few days because of the toll these aggressive AI bots. Alexa seems like the worst. These were created 20 years ago and updated over the years. I use to get traffic but that's been slowed to 1,000 or less legitimate visitors over the last year. But now I have to deal with server down emails caused by these aggressive bots that don't respect the robots file.
- Aurornis 2y ago> Alexa seems like the worst. Many of the bots disguise themselves as coming from Amazon or other big company. Amazon has a page where you can check some details to see if it’s really their crawler or someone imitating it.
- deleted 2y ago[deleted]
- spookie 2y agoyup, actually most that I've seen are impersonating amazon
- svelle 2y agoIt's funny how every time this topic comes up someone says "I've had this happen and x is the worst" With x being any of the big AI providers. Just a couple minutes ago I read the same in another thread and it was Anthropic. A couple weeks back it was Meta. My conclusion is that they're all equally terrible then.
- Aurornis 2y agoAll of the crawlers present themselves as being from one of the major companies, even if they’re not. Setting user-agent headers is easy.
- cyanydeez 2y agoat the same time, all the AI providers have some kind of web based AI agent, so let snot pretend they're crafting their services in care of other peoples websites.
- hoaxminion 2y ago[dead]
- egypturnash 2y agoThis is 100% off-topic but: Is it just me or does Ars keep on serving videos about the sound design of Callisto Protocol in the middle of everything? Why do they keep on promoting these videos about a game from 2022? They've been doing this for months now.
- misnome 2y agoI also get this for 1+ years, and just assume that it’s a symptom of my adblockers/pihole working and screwing up something in their auto ad targeting.
- egypturnash 2y agoI’m glad to know it’s not just me! It’s such a weird and specific failure state.
- 101008 2y agoI have a blog with content about a non tech topic and I had any problem. I am all against AI scrappers, but I didn't notice any change being behind Cloudflare (funny enough, if I ask GPT and Claude about my website, they know it)
- WesolyKubeczek 2y agoThey could hit Cloudflare’s caches and run with that. The problems are if your site is dynamic and your CDN has to hit the origin every time.
- hbcondo714 2y ago> many AI companies engage in web crawling Individuals do too. Tools like https://github.com/unclecode/crawl4ai https://github.com/unclecode/crawl4ai make it simple to obtain public content but also paywalled content including ebooks, forums and more. I doubt these folks are trying to do a DDoS though.
- CyberDildonics 2y agoHow do they manage to get 'paywalled' content?
- hbcondo714 2y agoMaybe 'paywalled' is not the best word but using their Identity Based Crawling feature with Managed Browsers[1], you can use an existing account and scrape content that requires authentication. This may not sound like anything new but IMHO, crawl4ai's workflow is easy to follow. [1] https://docs.crawl4ai.com/advanced/identity-based-crawling https://docs.crawl4ai.com/advanced/identity-based-crawling
- superkuh 2y agoWhile the motivations may be AI related the cause of the problem is the first and original type of non-human person: corporations. Corporations are doing this, not human persons, not AI.
- deleted 2y ago[deleted]
- j45 2y agoSurprised at how crawling as a whole seems to have taken a sustained step backwards with old best practices that are solved being new to devs today? I can't help but wonder if big AI crawlers belonging to the LLMs wouldn't be doing some amount of local caching with Squid or something. Maybe it's beneficial somehow to let the websites tarpit them or slow down requests to use more tokens.
- cyanydeez 2y agoI'm guessing its the rise of the AI agents that just go out and download "research" and not necessarily building LLMs.
- eevilspock 2y agoThe irony is that the people and employees of the AI companies will vehemently defend the morality of capitalism, private property and free markets. Their robber baron behavior reveals their true values and the reality of capitalism.
- randmeerkat 2y ago> Their robber baron behavior reveals their true values and the reality of capitalism. This is rather reductionist… By your same logic I could say that Stalin and Mao revealed the true values and reality of communism. Let’s not elaborate on it further though and just leave this as a simple argument. Free market capitalism has led us to the most prosperous, peaceful, and advanced society humanity has ever ventured to create. Communism threatened that prosperity and peace with atrocities on a scale that exists beyond human comprehension. Capitalism, even with all of its faults, is the obvious choice.
- CyberDildonics 2y agoInternet comments are inherently reductionist.
- johnnyanmac 2y agoIt's rather strawman to bring up communism in a conversation that talked nothing about it, except that Capitalism is clearly flawed. Capitalism without law ends up with the same kind of authoritariasm as communism without law. Some Rich Guy ends up telling everyone what to do as a ruler with loose rules that no longer resemble the economic model. That's what people complain about when they bring up terms like "late stage capitalism".
- johnnyanmac 2y agoInsert the Sinclair quote here. Anything to drive up the stock, no matter how immoral or illegal.
- deleted 2y ago[deleted]
- haswell 2y agoLately I’ve been thinking a lot about what it would take to create a sort of “friends and family” Internet using some combination of Headscale/Tailscale. I want to feel free and open about what I’m publishing again, and the modern internet is making that increasingly difficult.
- DrFalkyn 2y agoChances are one your friends/ family will eventually be compromised in some way. There’s always VPNs, though you can only be on at one at a time per device
- rco8786 2y agoI got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.
- throwaway2037 2y agorobots.txt did not work?
- lemper 2y agobro, since when vc funded ai companies have the courtesy to respect robots.txt?
- seabird 2y agoOf course it didn't work. At best, the dorks doing this think there's a gamechanging LLM application to justify the insane valuations right around the corner if they just scrape every backwater site they can find. At worst, they're doing it because it's paying good money. Either way, they don't care, they're just going to ignore robots.txt.
- hsbauauvhabzb 2y agoI have not monitored traffic in this way, but I imagine most AI companies would explicitly follow links listed in robots, even if not mentioned elsewhere on the site.
- epc 2y agoI’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it. I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network.
- Neil44 2y ago
- CyberDildonics 2y agoCan't you just rate limit beyond what a person would ever notice and do it by slowing the response?
- xena 2y agoYou'd think, but no :(
- ordersofmag 2y agoNot if they hop to a different IP address every few requests. And they generally aren't bothered slow responses. It's not like they have to wait for one request to finish before they make another one (especially if they are making requests from thousands of machines).
- CyberDildonics 2y agoYou're saying that large companies are hitting individual websites with thousands of unrelated IP addresses?
- jaggederest 2y agoHow do you think they do crawling if not like that? They'd be IP banned instantly if they used any kind of predictable IP regime for more than a few minutes.
- CyberDildonics 2y agoI don't know what is actually happening, that's why I'm asking. Also you're implying that the only way to crawl is to essentially DDOS a website by blasting them from thousands of IP addresses. There is no reason crawlers can't do more sites in parallel and avoid hitting individual sites so hard. There are plenty of crawlers for the last few decades that don't cause problems, these are just stories about the ones that do.
- arlcode 2y agoYep we've been seeing that on our random small scale site that used to be open (and mostly relevant to a very limited number of people). It was nice for interested guests to get an impression of what we are doing. First the AI crawlers came in from foreign countries that could be blocked. Then they beat down the small server by being very distributed, calling from thousands of ips one or two requests each. We finally put a stop to it by requiring a login with a message informing people to physically show up to gain access. Worked fine for over 15 years but AI finally killed it.
- ANarrativeApe 2y agoExcuse my ignorance, but is it time to update the open source licenses in the light of this behavior? If so, what should the evolved license wording be? I appreciate that this could be easily circumvented by a 'bad actor', but it would make this abuse overt...
- johnnyanmac 2y agoFrom my little understanding, we have a sort of agreement in place with an item called robot.txt that's more or less a hanshake with such scrapers. Of course, the issue is these scrapers are blatantly ignoring robots.txt A license can help as well, but what's a license without enforcement? These companies are simply treating the courts as a cost to do business.
- CaptainFever 2y agoClose, robots.txt was originally for web crawlers, to reduce accidental denial-of-service attacks. It had nothing to do with the scraping (i.e. downloading content and parsing the HTML tags in a programmatic manner).
- WesolyKubeczek 2y agoWhat do you think a search engine’s crawler bot is doing exactly? I could sure be wrong, but I have a hunch that “downloading content and paraing the HTML tags in a programmatic manner” describes it.
- CaptainFever 2y agoYes, but the difference is that the term "scraping" also targets things like automatically generating RSS feeds from HTML pages, which is not covered by robots.txt.
- WesolyKubeczek 2y agoI thought robots.txt covered all automated, programmatic access by third parties where a bot slurps stuff and follows links, without splitting hairs about it. But what do I know, the young whippersnappers will just word lawyer me to death, so I better shut up and go away.
- nashashmi 2y agoAnd this is why the Internet has become a maze of captcha’s.
- fsckboy 2y agoyeah but the HN community would be ground zero for "automating scraping tasks" "We have met the enemy and he is us." -- Walt Kelly
- kazinator 2y agoI've been seeing crawlers which report an Agent string like this: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3405.80 Safari/537.36 Everything but the Chrome/ is the same. They come from different IP addresses and make two hit-and-run requests. The different IPs always use a different Chrome string. Always some two digit main version like 69, 70. Then a .0. and then some funny minor and build numbers: typically a four digit minor. When I was hit with a lot of these a couple of weeks ago, I put in a custom rewrite rule to redirect them to the honeypot. The attack quickly abated.
- userbinator 2y agoAll these JS-heavy "anti bot" measures do is further entrench the browser monopoly, making it much harder for the minority of independents, while those who pay big $$$ can still bypass them. Instead I recommend a simple HTML form that asks questions with answers that LLMs cannot yet figure out or get consistently wrong. The more related to the site's content the questions are, the better; I remember some electronics forums would have similar "skill-testing" questions on their registration forms, and while some of them may be LLM'able now, I suspect many of them are still really CAPTCHAs that only humans can solve. IMHO the fact that this shows up at a time when the Ladybird browser is just starting to become a serious contender is suspicious.
- mvdtnz 2y agoHow does JS entrench a browser monopoly? If you're not using vendor-specific JS extensions or non-standard APIs any browser should be able to execute your JS. Like most web developers I don't have a lot of patience for the people who refuse to run JS on their clients.
- userbinator 2y agoThe effort required to implement a JS engine and keep trendchasing the latest changes with it is a huge barrier to entry, not to mention the insane amount of fingerprinting and other privacy-hostile, anti-user techniques it enables. Seeing what used to be simple HTML forms turned into bloated invasive webapps to accomplish the exact same thing seriously angers me; and everyone else who wanted an easily accessible and freedom-preserving Internet.
- deleted 2y ago[deleted]
- Terr_ 2y agoOr require every fresh "unique" visitor to run some JS that takes X seconds to compute. It's not nice for visitors using a very old smartphone, but it's arguably less-exclusionary than some of the tests and third-party gatekeepers that exist now. In many cases we don't actually care about telling if someone is truly a human alone, as much as ensuring that they aren't a throwaway sockpuppet of a larger automated system that doesn't care about good behavior because a replacement is so easy to make.
- jrvarela56 2y agoThis is going to start happening to brick and mortar businesses through their customer support channels
- cdolan 2y agoAnd enterprise call centers
- sean_lynch 2y agoI’d love to hear more. What are they seeing?
- jrvarela56 2y agoUnfounded. Assuming automated/voice-enabled assistants will spam for opening times, price & description of products/services, scheduling/booking, etc. In the long run it'll be an arms race but the transition will be rough for businesses as consumers can adopt these tools faster than SMBs or enterprises can integrate them.
- imtringued 2y agoKitboga already deployed a counter-scam LLM that calls scammers to waste their time. The goal is to keep them on the line as long as possible.
- instagib 2y agoCan we force the bots to mine cryptocurrency?
- userbinator 2y agoAnger someone enough and you'll get a real DDoS instead.
- Nckpz 2y agoI recently started a side-project with a "code everything in prod" approach for fun. I've done this many times over the past 20 years and the bot traffic is usually harmless, but this has been different. I haven't advertised the hostname anywhere, and in less than 24 hours I had a bunch of spam form submissions. I've always expected this after minor publicity, but not "start server, instantly get raided by bots performing interactions"
- ndiddy 2y agoThis is yet another great example of the innovation that the AI industry is delivering. Why just limit your scraper bots to GET requests when there might be some juicy data to train on hidden behind that form? There's a reason why the a16z funded cracked vibe coder ninjas are taking over software engineering, they're full of wonderful ideas like this.
- ohgr 2y agoI work with one of those guys. Morally bankrupt at every level of his existence.
- bendangelo 2y agoThere are bots that scrape https registration sites thats how they usually find you.
- deleted 2y ago[deleted]
- internet101010 2y agoI block all traffic except that which comes from the country of Cloudflare.
- rfurmani 2y agoAfter I opened up https://sugaku.net https://sugaku.net to be usable without login, it was astounding how quickly the crawlers started. I'd like the site to be accessible to all, but I've had to restrict most of the dynamic features to logged in users, restrict robots.txt, use cloudflare to block AI crawlers and bad bots, and I'm still getting ~1M automated requests per day (compared to ~1K organic), so I think I'll need to restrict the site to logged in users soon.
- karlgkk 2y agoOne thing that worked well for me was layering obstacles It really sucks that this is the way things are, but what I did was 10 requests for pages in a minute, you get captchad (with a little apology and the option to bypass it by logging in). asset loads don’t count After a captcha pass, 100 requests in an hour gets you auth walled It’s really shitty but my industry is used to content scraping. This allows legit users to get what they need. Although my users maybe don’t need prolonged access ahem.
- nomel 2y agoWhat happens if you use the proper rate limiting status of 429? It includes a next retry time [1]. I'm curious what (probably small) fraction would respect it. [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429 https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
- karlgkk 2y agoProbably makes sense for a b2b app where you publish status codes as part of the api Bad actors don’t care and annoying actors would make fun of you for it on twitter
- LPisGood 2y agoWhat is your website?
- rfurmani 2y agoI've wanted to but wasn't sure how to keep track of individuals. What works for you? IP Addresses, cookies, something else?
- yieldcrv 2y agonice, a free way to keep our IPFS pins alive
- i5heu 2y agoThis is not how it works. You have to actively pin data for it to be distributed.
- boyter 2y agoCrawling, incidentally, I think is the biggest issue with making a new search engine these days. Websites flat out refuse to support any crawler [other] than Google, and Cloudflare and other protection services and CDN's flat out deny access to incumbents. It is not a level playing field. I wrote the above some time ago. I think its even more true today. Its practically impossible to crawl the way the bigger players do and with the increased focus on legislation in this area its going to lock out smaller teams even faster. The old web is dead really. There really needs to be a move to more independent websites. Thankfully we are starting to see more of this like the linked searchmysite discussed earlier today https://news.ycombinator.com/item?id=43467541 https://news.ycombinator.com/item?id=43467541
- zlagen 2y agothat's a good point, in search we have google as a monopoly and since a big percentage of sites only want to be crawled by them it reinforces the monopoly. So a lot of people complain about bots not following robots.txt but if you follow them to the letter it's impossible to make anything useful. Also AFAIK robots.txt doesn't have any legal standing
- zzo38computer 2y agoMy issue is not to prevent others from obtaining copies of the files, using Lynx or curl, disabling JavaScripts and CSS and pictures, etc. It is to prevent others from overloading the server due to badly behaved software. I had briefly set up port knocking for the HTTP server (and only for HTTP; other protocols are accessible without port knocking), but due to a kernel panic I removed it and now the HTTP server is not accessible. (I may later put it back on once I can fix this problem.) As far as I can tell, the LLM scrapers do not attempt to be "smart" about it at this time; if they do in future, you might try to take advantage of that somehow. However, even if they don't, there are probably things that can be done. For example, check that if the declared user-agent declares things that it isn't doing, and display an error message if so (users who use Lynx will then remain unaffected and will still be able to access it). Another possibility is to try to confuse the scrapers however they are working, e.g. invalid redirects, valid redirects (e.g. to internal API functions of the companies that made them), invalid UTF-8, invalid compressed data, ZIP bombs (you can use the compression functions of HTTP to serve a small file that is too big when decompressed), EICAR test files, reverse pings (if you know who they really are), etc. What will work and what doesn't work depends on what software they are using.
- xyzal 2y agoI think the proper answer is to aim for the bots to get _negative_ utility value from visiting our sites, that is poisoning their well, not just zero value, that is to block them. Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach. Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent misalignment" for more info). A request rate too inhuman? Here, take those generated articles about positive effect of catching measles on performance in bed. And so on, and so forth ... Nepenthes is nice, but word salad can be detected easily. It needs a feature that pre-generates linguistically plausible but factically garbage text via open models
- gdcbe 2y agoThat’s vaguely what https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ is about
- agilob 2y agoAlso, lower upload rate to 5kb/s
- seper8 2y agoI love this idea haha
- usrnm 2y agoWhere are you going to get all that content? If it's static, it will get filtered out very fast, if it's dynamic and autogenerated, it might cost even more than just letting the crawler through
- hec126 2y agoGenerate it once every few weeks with LLaMa and then serve as static content?
- sigmoid10 2y agoThis relies a lot on being able to detect bots. Everything you said could be easily bypassed with a small to moderate amount of effort on the side of crawler's creators. Distinguishing genuine traffic has always been hard and it will not get easier in the age of AI.
- EVa5I7bHFq9mnYK 2y agoWhy can't Cloudflare do it? They know in real time which IPs spew millions of scrape requests to various sites, so they can classify them as AI bots and allow site owners to block them.
- miohtama 2y agoCloudflare already offers this feature https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...
- BiteCode_dev 2y agoSome people have issues with the idea that the entire web will eventually depend of a single point of failure and private for provide entity like cloudflare.
- EVa5I7bHFq9mnYK 2y agoWell, should choose from two evils then
- navane 2y agoHow do they know which square contains a bicycle?
- time4tea 2y agoYeah my small site got 170k requests from one bot in a few mins. Of course it was rate limited, but didn't seem to know 429 or 444 (drop connection) so kept on coming back for more. I do have ipset drop too but at the moment takes a person to enable it... just so much effort to stop these **s. Exhausting!
- miyuru 2y agoIsn't this by problem solved by using commoncrawl data. I wonder what changed to AI companies to do mass crawling individually. https://commoncrawl.org/ https://commoncrawl.org/
- JCharante 2y agoI believe we should have microtransactions to access resources. Pay a server a tiny amount and it'll return the content. This way if crawlers dominate traffic it just means they're paying a bunch for it
- RamblingCTO 2y agoSure, overly spamming websites is shitty behaviour. But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run?
- entropi 2y ago>But blocking AI crawlers hurts you in the end. Guess what will replace SEO in the long run? Maybe. But even if that turns out to be true, what good is it for the source website? The "AI" will surely not share any money (or anything else that may help the source website) with the source anyways. Why would they, they already got the content and trained on it.
- RamblingCTO 2y agoWhat good is it? If "AI" doesn't know about you down the line, you won't be discovered. Be it in LLM weights or via crawling (perplexity, jina reader etc.), you won't get any organic traffic. It's not about sharing profits.
- entropi 2y agoAgain, the "AI" doesn't care about the website. It doesn't even link to it in vast majority of the cases. Even if it did, the "AI" derives a lot of its business value from the fact that it is providing what the client requests while removing the need to visit potentially dozens of these pages. So the clients, in most cases, would not even click them (as they already got what they wanted).
- xena 2y agoSo you're willing to pay my hosting bills?
- RamblingCTO 2y agoRe-read what I posted and stop projecting.
- temp008 2y agoI wonder if a service exists where IPs known for crawling are reported and reputation of such IPs is tracked for others to use and ban / ratelimit by default
- mrweasel 2y agoThe problem with that approach is that you'll quickly add large swaths of IPs belonging to cloud service provides, such as AWS. We already know that AWS, Azure, GCP and Alibaba are part of the problem, so we can technically just rate-limit them already. I believe that all of them publish their IP ranges. Google also publish the IP ranges for GoogleBot I believe, and Bing probably does the same, so we can then whitelist those IPs and still have sites appear in searches. My issue is that the burden is again placed on everyone else, not the people/companies who are causing the problem. It's crazy to me to think about how much needless capacity is built into the internet to deal with crawlers. The resource waste is just insane.
- rambambram 2y agoMake AI 'pay' by delaying every request and presenting the content with a warning on top, like: AI bots are (sc)raping the internet, that's why we do this. Or something like: AI is making your experience worse, complain here (link to OpenAI). Maybe not the most technical solution, but this at least gets the signal across to regular human beings who want to browse a site. Puts all this AI bs in a bad spotlight.
- everdrive 2y agoOne more reason we're moving away from privacy. Didn't load all the javascript domains? You're probably a bot. Not signed in? You're probably a bot. The web we knew is dying step by step. One interesting thought: do we know if these AI crawlers intentionally avoid certain topics? Is pornography totally left unscathed by these bots? How about extreme political opinions?
- BiteCode_dev 2y agoThat's an interesting thing: put things about all dictators known currently in power that they don't want to hear, and maybe they will back off.
- GoblinSlayer 2y agoNo, https://www.heise.de/en/news/Poisoning-training-data-Russian-propaganda-for-AI-models-10317581.html https://www.heise.de/en/news/Poisoning-training-data-Russian...
- pdw 2y agoI wonder how long until we'll see DNS blocklists to blackhole IP addresses associated with scrapers. It seems like the logical evolution.
- kh_hk 2y agoThere are ip blocklist. Ranging from free, to paid, from text files to apis. Some of the sites offering IP blacklists also need to protect themselves from automated crawls too. It all goes full circle
- HermanMartinus 2y agoI literally just published a post on this and how it affects Bear Blog. https://herman.bearblog.dev/the-great-scrape https://herman.bearblog.dev/the-great-scrape
- kh_hk 2y agoProof of work is sufficient (although easy to bypass on targeted crawls) for protecting endpoints that are accessed via browsers, but plain public APIs have to resort to other more primitive methods like rate limiting. Blocking by UA is stupid, an by country kind of wrong. I am currently exploring ja4 fingerprints, that together with other metrics (country, Arn, block list), might give me a good tool to stop malicious usage. My point is, this is a lot of work, and it takes time off the budget you give to side projects.
- banq 2y agoI have blocked these ip from the country: 190.0.0.0/8 207.248.0.0/16 177.0.0.0/8 200.0.0.0/8 201.0.0.0/8 145.0.0.0/8 168.0.0.0/8 187.0.0.0/8 186.0.0.0/8 45.0.0.0/8 131.0.0.0/16 191.0.0.0/8 160.238.0.0/16 179.0.0.0/8 186.192.0.0/10 187.0.0.0/8 189.0.0.0/8
- zkmon 2y agoAn opensource repo asking who is responsible for this AI invasions? Well, it is you, who is responsible for all this. What did you think when you helped tech to advance so rapidly, over-pacing the needs of humans? Read about panchatantra story of 4 brothers who got a dead tiger alive, just to boast of their skill and greatness.
- myzie 2y agoAn aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week. This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actual site. Even if each company that operates a crawler cached pages across multiple runs, I'd expect a large improvement in the situation. For more dynamic pages, this obviously doesn't help. But a lot of the web's content is more static and is being crawled thousands of times. I built something for my own company that crawls using Playwright and caches in S3/Postgres with a TTL for this purpose. Does this make sense to anyone else? I'm not sure if I'm missing something that makes this harder than it seems on the surface. (Actual question!)
- xena 2y agoWhat does the user agent oook like for if you wanted to crawl xeiaso.net?
- jaccola 2y agoI have considered this before, but then if the content can be cached why wouldn't the website just do this themselves? They have the incentive, it is relatively easy and I don't think there's a huge benefit to centralisation (especially since it will basically be centralised to one of the big providers of caching anyways)
- myzie 2y agoI'm definitely with you that sites should be leveraging CDNs and similar. But I get that many don't want to do any work to support bots that they don't want to exist in the first place. To me it seems like the companies actually doing the crawling have an incentive to leverage centralized caching. It makes their own crawling faster (since hitting the cache is much faster than using Playwright etc to load the page) and it reduces the impact on all these sites. Which would then also decrease the impact of this whole bot situation overall.
- brookst 2y ago
- kittikitti 2y agoOne of the main peddlers for residential proxies is Israel, see BrightData.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- zlagen 2y agoin the perfect world we would have a very high trust internet in which everyone follows the rules, checks robots.txt, rate limits, etc. But we don't live in such a world. Getting angry at these ai bots is useless. People should start considering what and how they host their data. If you're worried about bandwidth costs you have many alternatives to host your data for free or at a very little cost. i.e: github/gitlab/cloudflare. If you're worried about your data getting scraped and used then maybe you can consider putting it behind a login or do some proof of work/soft captcha. Yeah, this isn't perfect but it will keep most dumb bots away. Some people are hosting their sites like we're still in 1995 and times have changed.
- _DeadFred_ 2y agoSounds like every request should require crypto processing be fed back to the host on the client side as quid pro quo.
- aorth 2y agoI think it's strange to focus on "FOSS sites" in this article. I have regular corporate sites that are getting slammed by ChatGPT, Perplexity, ByteDance, and others too.
- nektro 2y agothose crawlers and models are a scourge on the internet