9 ms·

The internet is no longer a safe haven

253 points by akyuu 11mo ago

skybrian 11mo ago
Isn’t this problem why Cloudflare is popular? You can write your own server, but outsource protecting it from bots. Perhaps there are better alternatives?
BinaryIgor 11mo ago
I wonder why is it that we get an increase in these automated scrapers and attacks as of late (some few years); is there better (open-source?) technology that allows it? Is it because hosting infrastructure is cheaper also for the attackers? Both? Something else? Maybe the long-term solution for such attacks is to hide most of the internet behind some kind of Proof of Work system/network, so that mostly humans get to access to our websites, not machines.
- trenchpilgrim 11mo ago
  Using AI you can write a naive scraper in minutes and there's now a market demand for cleaned up and structured data.
- marginalia_nu 11mo ago
  What's missing is effective international law enforcement. This is a legal problem first and foremost. As long as it's as easy as it is to get away with this stuff by just routing the traffic through a Russian or Singaporean node, it's going to keep happening. With international diplomacy going the way it has been, odds of that changing aren't fantastic. The web is really stuck between a rock and a hard place when it comes to this. Proof of work helps website owners, but makes life harder for all discovery tools and search engines. An independent standard for request signing and building some sort of reputation database for verified crawlers could be part of a solution, though that causes problems with websites feeding crawlers different content than users, an does nothing to fix the Sybil attack problem.
  - luckylion 11mo ago
    It's not necessarily going through a Russian or Singaporean node though, on the sites I'm responsible for, AWS, GCP, Azure are in the top 5 for attackers. It's just that they don't care _at all_ about that happening. I don't think you need world-wide law-enforcement, it'll be a big step ahead if you make owners & operators liable. You can limit exposure so nobody gets absolutely ruined, but anyone running wordpress 4.2 and getting their VPS abused for attacks currently has 0 incentive to change anything unless their website goes down. Give them a penalty of a few hundred dollars and suddenly they do. To keep things simple, collect from the hosters, they can then charge their customers, and suddenly they'll be interested in it as well, because they don't want to deal with that. The criminals are not held liable, and neither are their enablers. There's very little chance anything will change that way.
    - mrweasel 11mo ago
      The big cloud provides needs to step up and take responsibility. I understand that it can't be to easy to do, but we really do need a way to contact e.g. AWS and tell them to shut of a costumer. I have no problem with someone scraping our websites, but I care that they don't do so responsibly, slow down when we start responding slower, don't assume that you can just go full throttle, crash our site, wait, and then do it again once we start responding again. You're absolutely right: AWS, GCP, Azure and others, they do not care and especially AWS and GCP are massive enablers.
      - ctoth 11mo ago
        > we really do need a way to contact e.g. AWS and tell them to shut of a costumer. You realize you just described the infrastructure for far worse abuse than a misconfigured scraper, right?
        mrweasel 11mo ago
        I'm very aware of that, yes. There needs to be a good process, the current situation where AWS simply does not care, or doesn't know also isn't particularly good. One solution could be for victims to notify AWS that a number of specified IP are generating an excessive amount of traffic. An operator could then verify with AWS traffic logs, notify the customer that they are causing issue and only after a failure to respond could the customer be shut down. You're not wrong that abuse would be a massive issue, but I'm on the other side of this and need Amazon to do something, anything.
- rkagerer 11mo ago
  long-term solution How about a reputation system? Attached to IP address is easiest to grok, but wouldn't work well since addresses lack affinity. OK, so we introduce an identifier that's persistent, and maybe a user can even port it between devices. Now it's bad for privacy. How about a way a client could prove their reputation is above some threshold without leaking any identifying information? And a decentralized way for the rest of the internet to influence their reputation (like when my server feels you're hammering it)? Do anti-DDoS intermediaries like Cloudflare basically catalog a spectrum of reputation at the ASN level (pushing anti-abuse onus to ISP's)? This is basically what happened to email/SMTP, for better or worse :-S.
  - JimDabell 11mo ago
    Reputation plus privacy is probably unsolvable; the whole point of reputation is knowing what people are doing elsewhere. You don’t need reputation, you need persistence. You don’t need to know if they are behaving themselves elsewhere on the Internet as long as you can ban them once and not have them come back. Services need the ability to obtain an identifier that: - Belongs to exactly one real person. - That a person cannot own more than one of. - That is unique per-service. - That cannot be tied to a real-world identity. - That can be used by the person to optionally disclose attributes like whether they are an adult or not. Services generally don’t care about knowing your exact identity but being able to ban a person and not have them simply register a new account, and being able to stop people from registering thousands of accounts would go a long way towards wiping out inauthentic and abusive behaviour. The ability to “reset” your identity is the underlying hole that enables a vast amount of abuse. It’s possible to have persistent, pseudonymous access to the Internet without disclosing real-world identity. Being able to permanently ban abusers from a service would have a hugely positive effect on the Internet.
    - jasonjayr 11mo ago
      A digital "Death penalty" is not a win for society, without considering a fair way to atone for "crimes against your digital identity". It would be way to easy for the current regime (whomever that happens to be) to criminalize random behaviors (Trans People? Atheists? Random nationality?) to ban their identity, and then they can't apply for jobs, get bus fare, purchase anything online, communicate with their lawyers, etc.
- hnthrowaway0315 11mo ago
  I guess it is just because 1) They can, and 2) Everyone wants some data. I think it would be interesting if every website out there starts to push out BS pages just for scrappers. Not sure how much extra cost it's going to take if a website puts up say 50% BS pages that only scrappers can reach, or BS material with extremely small fonts hidden in regular pages that ordinary people cannot see.
  - inerte 11mo ago
    Something like https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ ?
    - hnthrowaway0315 11mo ago
      Yeah something like this, would be nice if it actually feeds bad data that requires human to double confirm, too. Not something seriously wrong but something subtle, like changing a couple of letters in a name of a country, or randomize the National day. Once a lot of websites start to use it AI might actually get confused, I think? But humans never read these pages so should be largely fine -- unless they are reading AI summaries.
- Vegenoid 11mo ago
  I'm pretty sure it is the commercial demand for data from AI companies. It is certainly the popular conception among sysadmins that it is AI companies who are responsible for the wave of scrapers over the past few years, and I see no compelling alternative.
  - embedding-shape 11mo ago
    > and I see no compelling alternative. Another potential cause: It's way easier for pretty much any person connected to the internet to "create" their own automation software by using LLMs. I could wager even the less smart LLMs could handle "Create a program that checks this website every second for any product updates on all pages" and give enough instructions for the average computer user to be able to run it without thinking or considering much. Multiply this by every person with access to an LLM who wants to "do X with website Y" and you'll get an magnitude increase in traffic across the internet. This been possible since what, 2023 sometime? Not sure if the patterns would line up, but just another guess for the cause(s).
- EGreg 11mo ago
  Why? It’s because of AI. It enables attacks at scale. It enables more people to attack, who previously couldn’t. And so on. It’s very explainable. And somehow, like clockwork, there are always comments to say “there is nothing new, the Internet has always been like this since the 80s”. You know, part of me wants to see AI proliferate into more and more areas, just so these people will finally wake up eventually and understand there is a huge difference when AI does it. When they are relentlessly bombarded with realistic phone calls from random numbers, with friends and family members calling about the latest hoax and deepfake, when their own specific reputation is constantly attacked and destroyed by 1000 cuts not just online but in their own trusted circles, and they have to put out fires and play whack-a-mole with an advanced persistent threat that only grows larger and always comes from new sources, anonymous and not. And this is all before bot swarms that can coordinate and plan long-term, targeting specific communities and individuals. And this is all before humanoid robots and drones proliferate. Just try to fast-forward to when human communities online and offline are constantly infiltrated by bots and drones and sleeper agents, playing nice for a long time and amassing karma / reputation / connections / trust / whatever until finally doing a coordinated attack. Honestly, people just don’t seem to get it until it’s too late. Same with ecosystem destruction — tons of people keep strawmanning it as mere temperature shifts, even while ecosystems around the world get destroyed. Kelp forests. Rainforests. Coral reefs. Fish. Insects. And they’re like “haha global warming by 3 degrees big deal. Temperature has always changed on the planet.” (Sound familiar?) Look, I don’t actually want any of this to happen. But if they could somehow experience the movie It’s a Wonderful Life or meet the Ghost of Christmas Yet to Come, I’d wholeheartedly want every denier to have that experience. (In fact, a dedicated attacker can already give them a taste of this with current technology. I am sure it will become a decentralized service soon :-( )
  - hshdhdhj4444 11mo ago
    Our tech overlords understand AI, especially any form of AGI, will basically be the end of humanity. That’s why they’re entirely focused on being the first and amassing as much wealth in the meanwhile, giving up on any sort of consideration whether they’re doing good for people or not.
tonetheman 11mo ago
[dead]
jchw 11mo ago
Anubis is definitely playing the cat-and-mouse game to some extent, but I like what it does because it forces bots to either identify themselves as such or face challenges. That said, we can likely do better. Cloudflare does good in part because Cloudflare runs so much traffic, so they have a lot of data across the internet. Smaller operators just don't get enough traffic to really deal with banning abusive IPs without banning entire ranges indefinitely, not ideal. I hope to see a solution like Crowdsec where reputation data can be crowdsourced to block known bad bots (at least for a while since they are likely borrowing IPs) while using low complexity (potentially JS-free) challenges for IPs with no bad reputation. It's probably too much to ask for Anubis upstream which is probably already too busy dealing with the challenges of what it already does at the scale it is operating, but it does leave some room for further innovation for whoever wants to go for it. In my opinion there is at least no reason why it is not plausible to have a drop-in solution that can mostly resolve these problems and make it easier for hobbyists to run services again.
gjsman-1000 11mo ago
The problem with anything, anything, without a centralized authority, is that friction overwhelms inertia. Bad actors exist and have no mercy, while good people downplay them until it’s too late. Entropy always wins. Misguided people assume the problem is powerful people, when the problem is actually what the powerful people use their authority to do, as powerful people will always exist. Accepting that and maintaining oversight is the historically successful norm; abolishing them has always failed. As such, I don’t identify with the author of this post, about trying to resist CloudFlare for moral reasons. A decentralized system where everyone plays nice and mostly cooperates, does not exist any more than a country without a government where everyone plays nice and mostly cooperates. It’s wishful thinking. We already tried this with Email, and we’re back to gatekeepers. Pretending the web will be different is ahistorical.
- pixl97 11mo ago
  The internet has made the world small and that's a problem. Nation states typically had a limited range of broadcasting their authority in the more distant past. A bad ruler couldn't rule the entire world, nor could they cause trouble with the entire world. From nukes to the interconnected web the worst of us with power can effect everyone else.
- martin-t 11mo ago
  Power is a spectrum. Power differentials will always exist but we can absolutely strive to make them smaller and smaller. 1) Most of the civilized world no longer has hereditary dictators (such as "kings"). Because they were removed from power by the people and the power was distributed among many individuals. It works because malicious (anti-social) individuals have trouble working together. And yes, oversight helps. But it's a spectrum and we absolutely can and should move the needle towards more oversight and more power distribution. 2) Corporate power structures are still authoritarian. We can change that too.
time4tea 11mo ago
Had to ban RU, CN, SG, and KR just cos of the volume of spam. The random referer headers has recently become a problem. This is particularly annoying as knowing where people come from is important. Its just another reason to give up making stuff, and give in to the FAANG and the AI enshittification. :-(
- mrweasel 11mo ago
  If you only care about regular users I'd advice banning all known datacenters, Browserbase, China and Brazil.
  - worthless-trash 11mo ago
    I also banned the middle east, Logs actually reflected real users, cost dropped, and my mental health improved. Win Win.
    - mrweasel 11mo ago
      If you know that you don't have customers or users in the area, or very few, then go for it. I worked in e-commerce previously, we reduce fraud to almost zero by banning non-local cards. It affected a few customers that had international credit cards, but not enough to justify dealing with the fraud. Sometimes you just need to limit your attack surface.
mberning 11mo ago
the internet is over. If we want to recapture the magic of the earlier times we are going to have to invent something new.
- fithisux 11mo ago
  going back to Gopher?
  - itintheory 11mo ago
    Gopher still requires the Internet. I know it's pretty common to conflate "the Internet" with "the World Wide Web", but there are actually other protocols out there (like Gopher).
    - fithisux 11mo ago
      I don't see a solution then.
  - groundzeros2015 11mo ago
    Why would that help?
- pixl97 11mo ago
  There is no something new. Anything we invent will be able to be taken over by complex bots. Welcome to the futureshock where humans aren't at the top of their domain.
- 9rx 11mo ago
  The magical times in the past have always been marked with being able to be part of an "exclusive club" that takes something from nothing to changing the world. Because of the internet, magical times can never be had again. You can invent something new, but as soon as anyone finds out about it, everyone now finds out about it. The "exclusive club" period is no more.
  - martin-t 11mo ago
bpt3 11mo ago
The internet hasn't been a safe haven since the 80s, or maybe earlier (that was before my time, and it's never been one since I got online in the early 90s). The only real solution is to implement some sort of identity management system, but that has so many issues that make it a non-starter.
- lotsofpulp 11mo ago
  > The only real solution is to implement some sort of identity management system, but that has so many issues that make it a non-starter. Apple and Alphabet seem positioned to easily enable it. https://www.apple.com/newsroom/2025/11/apple-introduces-digital-id-a-new-way-to-create-and-present-an-id-in-apple-wallet/ https://www.apple.com/newsroom/2025/11/apple-introduces-digi...
  - JSR_FDED 11mo ago
    I don’t get it. That link refers to Apple letting you put your passport and drivers license info in the wallet on your phone.
    - Astronaut3315 11mo ago
      Apple’s Wallet app presents this feature as being for “age and identity verification in apps, online and in person”.
  - pixl97 11mo ago
    Alphabet the company that bans people for opaque reasons with no recourse, good idea. Maybe tech should not be in charge of digital identification
    - lotsofpulp 11mo ago
      The governments like it that way. They want banks and tech companies to be intermediaries that are more difficult to hold accountable, because they can just say “we didn’t feel like doing business with this person”.
qwertox 11mo ago
Since I moved my DNS records to Cloudflare (that is: nameserver is now the one from Cloudflare), I get tons of odd connections, most notably SYN packets to eihter 443 or 22, which never respond back after the SYN-ACK. They ping me once a second in average, distributing the IPs over a /24 network. I really don't understand why they do this, and it's mostly some shady origins, like vps game server hoster from Brazil and so on. I'm at the point where i capture all the traffic and looks for SYN packets, check the RDAP records for them to decide if I then drop the entire subnets of that organization, whitelisting things like Google. Digital Ocean is notoriously a source of bad traffic, they just don't care at all.
- kzemek 11mo ago
  These are spoofed packets for SYNACK reflection attacks. Your response traffic goes to the victim, and since network stacks are usually configured to retry SYNACK a few times, they also get amplification out of it
  - pabs3 11mo ago
    There is a solution to that, but it requires these companies to implement source address validation. If your ISP is on the list, maybe complain about it. https://spoofer.caida.org/as_stats.php https://spoofer.caida.org/as_stats.php
- sva_ 11mo ago
  > like vps game server hoster from Brazil and so on. Probably someone DDoSing a Minecraft server or something. People in games do this where they DDoS each other. You can get access to a DDoS panel for as little as $5 a month. Some providers allow for spoofing the src ip, that's how they do these reflection attacks. So you're not actually dropping the sender of these packets, but the victims. Consider turning reverse path filter to strict as a basic anti spoofing method and see if it helps net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1
  - qwertox 11mo ago
threeducks 11mo ago
> Fail2ban was struggling to keep up: it ingests the Nginx access.log file to apply its rules but if the files keep on exploding… > [...] > But I don’t want to fiddle with even more moving components and configuration You can configure nginx to do rate-limiting directly. Blog post with more details: https://blog.nginx.org/blog/rate-limiting-nginx https://blog.nginx.org/blog/rate-limiting-nginx
embedding-shape 11mo ago
> The internet is no longer a safe haven for software hobbyists Maybe I've just had bad luck, but since I started hosting my own websites back around 2005 or so, my servers have always been attacked basically from the moment they come online. Even more so when you attach any sort of DNS name to it, especially when you use TLS and the certificates, guessing because they end up in a big index that is easily accessible (the "transparency logs"). Once you start sharing your website, it again triggers an avalanche of bad traffic, and the final boss is when you piss of some organization and (I'm assuming) they hire some bad actor to try to make you offline. Dealing with crawlers, bot nets, automation gone wrong, pissed of humans and so on have been almost a yearly thing for me since I started deploying stuff to the public internet. But again, maybe I've had bad luck? Hosted stuff across wide range of providers, and seems to happen across all of them.
- zwnow 11mo ago
  My first ever deployed project was breached on day 1 with my database dropped and a ransom note in there. Was a beginner mistake by me that allowed this, but it's pretty discouraging. Its not the internet that sucks, its people that suck.
  - mattmaroon 11mo ago
    Well I guess at least on day 1 you didn’t have much to lose!
    - zwnow 11mo ago
      Its a personal blog so even if data was lost it would've been just posts that nobody reads. Certainly not worth the 0.00054 BTC they wanted
    - timeinput 11mo ago
      more like a zero day on day zero.
- aftbit 11mo ago
  My stuff used to get popped daily. A janky PHP guestbook I wrote just to learn back in the early 2000s? No HTML injection protection & someone turned my site into spammy XSS hack within days. A WordPress installation I fell behind on patching? Turned into SEO spam in hours. A redis instance I was using just to learn some of their data structures that got accidentally exposed to the web? Used to root my computer and install a botnet RAT. This was all before 2020. I never felt this made the internet "unsafe". Instead, it just reminded me how I messed up. Every time, I learned how to do better, and I added more guardrails. I haven't gotten popped that obviously in a long time, but that's probably because I've acted to minimize my public surface area, used star-certs to avoid being in the cert logs, added basic auth whenever I can, and generally refused to _trust_ software that's exposed to the web. It's not unsafe if you take precautions, have backups, and are careful about what you install. If you want to see unsafe, look at how someone who doesn't understand tech tries to interact with it. Downloading any random driver or exe to fix a problem, installing apps when a website would do, giving Facebook or Tiktok all of their information and access without recognizing that just maybe these multi-billion-dollar companies who give away all of their services don't have your best interests in mind.
quaintdev 11mo ago
I do not have a solution for blog like this but if you are self hosting I recommend enabling mTLS on your reverse proxy. I'm doing this for a dozen services hosted at home. The reverse proxy just drops the request if user does not present a certificate. My devices which can present cert can connect seamlessly. It's a one time setup but once done you can forget about it.
- SoftTalker 11mo ago
  That's fine if you're hosting stuff just for yourself but not really practical if you're hosting stuff you want others to be able to read, such as a blog.
  - lukevp 11mo ago
    You can mTLS to CloudFlare too, if you’re not one of the anti-CloudFlare people. Then all traffic drops besides traffic that passes thru CF and the mTLS handshake prevents bypassing CF.
    - BehindTheMath 11mo ago
      You don't need mTLS for that. Just block all IPs beside for Cloudflare's ranges.
- bogwog 11mo ago
  Wireguard is much better. Not only is it easier to set up/maintain, it even works on Android and iOS. I used to use client authentication for my private git server, but getting client certs installed on every client browser or app was a pain in the ass, and not even possible for some mobile browsers. Today, my entire network of self hosted stuff exists in a personal wireguard VPN. My firewall blocks everything except the wireguard port (even SSH).
cyp0633 11mo ago
My Gitea instance also encountered aggressive scraping some days ago, but with highly distributed IP & ASN & geolocation, each of which is well below the rate of a human visitor. I assume Anubis will not stop the massively funded AI companies, so I'm considering poisoning the scrapers with garbage code, only targeting blind scrapers, of course.
- mrweasel 11mo ago
  Sadly we're now seeing services that sell proxy services that allows you to scape from a wide variety of residential IPs, some even goes so far as to labels their IPs as "ethically sources".
deleted 11mo ago
[deleted]
skopje 11mo ago
sad but hosting static content like his site in a cloud would save him a headache. i know i know, "do it yourself" and all but if that is his path he knows the price. maybe i am wrong and do not understand the problem but it seems like he is asking for a headache. edit: words
- Nifty3929 11mo ago
  I think the author would agree, and is in fact the point of his post. The only way to solve these problems is using some large hosted platform where they have the resources to constantly be managing these issues. This would solve their problem. But isn't it sad that we can't host our own websites anymore, like many of us used to? It was never easy, but it's nearly impossible now and this is only one reason.
  - skopje 11mo ago
    i think it has been a hard to host a site since about 2007. i stopped then because it is too much work to keep it safe. even worse now but it has always been extra work since search engines. maybe the OP is just getting older and wants to spend time with his kids and not play with nginx haha.
zdc1 11mo ago
I wonder if you can have a chain of "invisible" links on your site that a normal person wouldn't see or click. The links can go page A -> page B -> page C, where a request for C = instant IP ban.
- chrisweekly 11mo ago
  IP addresses from scrapers are innumerable and in constant rotation.
- SkiFire13 11mo ago
  Scrapers nowadays can use residential and mobile IPs, so banning by IP, even if actual malicious requests are coming from them, can also prevent actual unrelated people from accessing your service.
  - SoftTalker 11mo ago
    Unless you're running a very popular service, unlikely that a random residential IP would be both compromised by a malicious VPN and also trying to access your site legitimately.
    - arbol 11mo ago
      Lots of people have chrome extensions installed that use their connection like proxy so this is more common than you think
      - SoftTalker 11mo ago
        Can you provide any examples of these extensions? I'm not doubting you, just curious.
        arbol 11mo ago
        There's one mentioned here: https://www.bleepingcomputer.com/news/security/data-stealing-chrome-extensions-impersonate-fortinet-youtube-vpns/ https://www.bleepingcomputer.com/news/security/data-stealing... Anyone who owns a chrome extension with 50k+ installs is regularly asked to sell it to people (myself included). The people who buy the extensions try to monetize them any way they can, like proxying traffic for malicious scrapers / attacks.
sequoia 11mo ago
I don't know if there's a simple solution to deploy this but JA3 fingerprinting is sometimes used to identify similar clients even if they're spread across IPs: https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967/ https://engineering.salesforce.com/tls-fingerprinting-with-j...
- arbol 11mo ago
  You need to terminate the TLS connection yourself so this prevents people from using DNS proxy, e.g. Cloudflare. Then you have to run a server that has a module that computes the ja3/ja4, e.g. nginx. Even then, it's possible to set your client hello in python/curl/etc. to exactly mirror the JA4 of your chosen browser like Chrome. So ja4 stops basic bots but most seasoned scrapers already implement valid ja4s/ja3s
  - sequoia 11mo ago
    I thought it wasn't just a matter of valid but of identicalness. Multiple clients with identical JA4 which comprises if I'm not mistaken useragent but also other aspects of the host machine indicate that they are in fact a single user agent.
    - arbol 11mo ago
      I'm sorry, I don't fully understand your question. This is what I meant by invalid: Anyone using chrome 142 has the same JA4 - I've checked across OS/devices. If you use nodeJS and set your user agent to "chrome 142" then you will have an invalid/incorrect JA4 and you'll stick out from the crowd.
petermcneeley 11mo ago
The real internet has yet to be invented.
zkmon 11mo ago
Common man never had a need for internet or global connectedness. DARPA wanted to push technology to gain upper hand in the world matters. Universities pushed technology to show progress and sell research. Businesses pushed technologies to have more sales. It was kind of acid rain that was caused by the establishments and sold as scented rain.
- wartywhoa23 11mo ago
  This sentiment - along the lines of "the world became too dependent on the Internet", "Internet wasn't a good thing to begin with", "Internet is a threat to national security" etc - has been popping up on HN too often lately, emerged too abruptly and correlates with the recent initiatives to crack down on the Internet too well. If this is your own opinion and not a part of a psyop to condition people into embracing the death of the Internet as we know it, do you have any solution to propose?
  - gchamonlive 11mo ago
    You don't need to have a solution to explore a problem in my opinion. OP comment is problematic but for reasons other than not having a proposed solution.
- gchamonlive 11mo ago
  > Common man never had a need for internet or global connectedness That's not how culture evolves. You don't necessarily need to have a problem so that a solution is developed. You can very well have a technology developed for other purposes, or just for exploration sake, and then as this tech exists uses for it start to pop post hoc. You therefore ignore the immense benefit of access to information that technology has, something that wasn't necessarily a problem for the common man but once its there, the popularization of the access to information, they adapt and grow dependent on it. Just like electricity.
  - zkmon 11mo ago
    >>immense benefits ?? People with dialup telephones never asked for a smartphone connected to internet. They were just as happy back then or even more happy because phone didn't eat off their time or cause posture problems. Sure, shopping was slower without amazon website, but not less happy experience back then. Infact homes had less junk and people saved more money Messaging? sure it makes you spend time with 100 whatsapp groups, where 99% of the people don't know you personally. It helped companies to sell more of the junk more quickly. It created bloggers and content creators who lived in an imaginary world thinking that someone really consumes their content. It created karma beggers who begged globally for likes that are worth nothing. It created more concentration of wealth at some weird internet companies, which don't solve any of the world problems or basic needs of the people. And finally it created AI that pumps plastic sewage to fill the internet. There it is, your immensely useful internet. As if the plastic pollution was not enough in the real world, the internet will be filled with plastic content. What else did internet give that is immensely helpful?
xedrac 11mo ago
I run a dedicated firewall/dns box with netfilter rules to rate limit new connections per IP. It looks like I may need to change that to rate limit per /16 subnet...
bo1024 11mo ago
I wonder if a proof of work protocol is a viable solution. To GET the page, you have to spend enough electricity to solve a puzzle. The question is whether the threshold could be low enough for typical people on their phones to access the site easily, but high enough that mass scraping is significantly reduced.
- kalavan 11mo ago
  There's this paper from 2004: "Proof-of-Work Proves Not to Work": https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf The conclusion back then was that it's impossible to make a threshold that is both low enough and high enough. You need some other mechanism that can distinguish bad traffic from good (even if imperfectly), and then adjust the threshold based on it. See, for instance, "Proof of Work can Work": https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lui-pow.pdf https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lu...
  - bo1024 11mo ago
    Thanks for these references! I imagine the numbers would be entirely different in our context (20 years later and web serving, not email sending). And the idea of spammers using bot nets (therefore not paying for computer themselves) would be less relevant to LLM scraping. But I’ll try to check for forward references on these.
    - kalavan 11mo ago
      > And the idea of spammers using bot nets (therefore not paying for computer themselves) would be less relevant to LLM scraping. It's possible that the services that reward users for running proxies (or are bundled with mobile apps with a notice buried in the license) would also start rewarding/hiding compute services as well. There's currently no money in it because proof-of-work is so rare, but if it changes, their strategy might too.
  - beeflet 11mo ago
    Good links, but this is just for email and relies on some (admittedly) pretty lofty assumptions
smileson2 11mo ago
It's probably just time for the web page to die
- zelphirkalt 11mo ago
  What do you mean by that? Web pages is the central mechanism, we use to put information on the web. Of course many websites are shitty and could be much simpler to convey their information, without looking crap at all. But the web page in general? Why would we ever get rid of something so useful? And what do you suggest as an alternative?
- ksenzee 11mo ago
  ...they said, on a web page.
  - smileson2 11mo ago
    so we should settle on using web pages to do that for another 20-30 years? this is it for the rest of time?
    - ksenzee 11mo ago
      I’m suggesting we come up with a replacement for the web before we kill it off.
cosmicgadget 11mo ago
> Other things I’ve noticed is increased traffic with Referer headers coming from strange websites such as bioware.com, mcdonalds.com, and microsoft.com I've been seeing this too, I guess scrapers think they can get through some blockers with a referrer?
ModernMech 11mo ago
The Internet was a scene, and like all scenes it's done now the corpos have moved in and taken over (because at that point it's just ads and rent extraction in perpetuity). I dunno what/where/when the next tech scene will be, but I do know it's not going to come from Big Tech. See: Metaverse.
jcalvinowens 11mo ago
Scrapers have constantly been running against my cgit server for the past year, but they're bizarrely polite in my case... 2-3 requests per minute. This whole enterprise is clearly run by exceptionally dumb people, since you can just clone all the code I host there directly from upstreams... [16/Nov/2025:16:21:12 +0000] 190.92.214.144:34638 . "GET /cgit/linux/commit/drivers/vlynq?h=v5.15.76&id=59d42cd43c7335a3a8081fd6ee54ea41b0c239be HTTP/1.1" -> 200 3051b 3.42x 0.239ms [16/Nov/2025:16:22:15 +0000] 188.239.57.1:40328 . "GET /cgit/linux/commit/kernel/range.c?h=v6.12.31&id=459b37d423104f00e87d1934821bc8739979d0e4 HTTP/1.1" -> 200 2993b 3.42x 0.266ms [16/Nov/2025:16:22:56 +0000] 190.92.217.125:56580 . "GET /cgit/linux/commit/kernel?h=v5.15.92&id=f01aefe374d32c4bb1e5fd1e9f931cf77fca621a HTTP/1.1" -> 200 3091b 3.28x 0.250ms [16/Nov/2025:16:23:17 +0000] 159.138.10.64:44540 . "GET /cgit/linux/commit/drivers/mtd/mtdcore.c?h=v6.2.15&id=249858575fd3f27904d6bb775e5ab500e9ef3b0f HTTP/1.1" -> 200 3415b 3.47x 0.251ms [16/Nov/2025:16:23:58 +0000] 119.13.101.228:44342 . "GET /cgit/linux/commit/drivers/gpio?h=v6.6.93&id=bc7fe1a879fc024942bb9eff173fa619b722d09b HTTP/1.1" -> 200 3582b 3.37x 0.250ms
- deleted 11mo ago
  [deleted]
- jcalvinowens 11mo ago
  Turned out those scattered IPs were from a very small number of places, blocking these five killed 100% of my crawler spam: AS4229 Zenlayer (Singapore) PTE. LTD AS21859 ZEN-ECN, US AS45102 ALIBABA-CN-NET Alibaba US Technology Co., Ltd., CN AS132203 TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue, CN AS136907 HUAWEI INTERNATIONAL PTE. LTD. That's 4392 contiguous IP ranges.
  - deleted 11mo ago
    [deleted]
firefoxd 11mo ago
I have been using zipbombs and they were effective to some extent. Then I had the smart idea to write about it on HN [0]. The result was a flood of new types of bots that overwhelmed my $6 server. For ~100k daily request, it wasn't sustainable to serve 1 to 10MB payloads. I've updated my heuristic to only serve the worst offenders, and created honeypots to collect ips and repond with 403s. After a few months, and some other spam tricks I'll keep to myself this time, my traffic is back to something reasonable again. [0]: https://news.ycombinator.com/item?id=43826798 https://news.ycombinator.com/item?id=43826798
- tetris11 11mo ago
  There's likely a large market for this kind of thing. Maybe time to spin out a side business and deploy your heuristics to struggling IPs. Though I have to admit I dont know who your target audience would be. Self-hosting orgs don't tend to be flush with cash
AaronAPU 11mo ago
Everything good enough to become popular gets swarmed by the teeming masses and then exploited and destroyed. The only solution seems to be to constantly abandon those things and move on to new frontiers to enjoy until the cycle repeats.
svara 11mo ago
The Internet has really been an interesting case study for what happens between people when you remove a varying number of layers of social control. All the way back to the early days of Usenet really. I would hate to see it but at the same time I feel like the incentives created by the bad actors really push this towards a much more centralized model over time, e.g. one where all traffic provenance must be signed and identified and must flow through a few big networks that enforce laws around that.
- Cosi1125 11mo ago
  "Socialists"* argue for more regulations; "liberals" claim that there should be financial incentives to not do that. I'm neither. I believe that we should go back to being "tribes"/communities. At least it's a time-tested way to – maybe not prevent, but somewhat allieviate – the tragedy of the commons. (I'm aware that this is a very poor and naive theory; I'll happily ditch it for a better idea.) -- *) For the lack of a better word.
  - thrance 11mo ago
    What would prevent attacks between "tribes"? What would prevent one from taking over the others and sending us back to square one?
    - Cosi1125 11mo ago
      Little would prevent attacks by APTs and other powerful groups. (This, btw., is one of the few facets of this problem that technology could help solve.) But a trivial change: a hard requirement to sign up (=send a human-composed message to one of the moderators) to be able to participate (or, in extreme cases, to read the contents) "automagically" stops almost all spam, scrapers (in the extreme case), vandalism, etc. (from my personal experience based on a rather large sample). I think it's one of the multi-faceted problems where technology (a "moat", "palisade", etc. for your "tribe") should accompany social changes.
erickhill 11mo ago
I very much relate to the author's sour mood and frustration. I also host a small hobby forum and have experienced the same attacks constantly, and it has gotten especially bad the last couple of years with the rise of AI. In the early days I put Google Analytics on the site so I could observe traffic trends. Then, we were all forced to start adding certificates to our sites to keep them "safe". While I think we're all doomed to continue that annual practice or get blocked by browsers, I have often considered removing Google Analytics. Ever since their redesign it is essentially unusable for me now. What benefit does it bring if I can't understand the product anymore? Last year, in a fit of desperation, I added Cloudflare. This has a brute force "under attack" mode that seems to stop all bots from accessing the site. It puts up a silly "hang on a second, are you human" page before the site loads, but it does seem to work. It is great UX? No, but at least the site isn't getting hammered by various locations in Asia. Cloudflare also let me block entire countries, although that seems to be easily fooled. I also don't think a lot of the bots/AI crawlers honor the rules set in the robots.txt. It's all an honor system anyway, and they are completely lacking in it. There need to be some hard and fast rules put in place, somehow, to stop the madness.
- timpera 11mo ago
  Cloudflare does work, but it often destroys the experience for legitimate users. On the website I manage, non-technical users were often getting stuck on the Cloudflare captcha, so I ended up removing it. Then there's also the issue with dependence to US-based services, but that may not be an issue for you.
mrb 11mo ago
Unpopular opinion: the real source of the problem is not scrapers, but your unoptimized web software. Gitea and Fail2ban are resource hogs in your case, either unoptimized or poorly configured. My tiny personal web servers can whistand thousands of requests per second, barely breaking a sweat. As a result, none of the bots or scrapers are causing any issue. "The only thing that had immediate effect was sudo iptables -I INPUT -s 47.79.0.0/16 -j DROP" Well, by blocking an entire /16 range, it is this type of overzealous action that contributes to making the internet experience a bit more mediocre. This is the same thinking that lead me to, for example, not being able to browse homedepot.com from Europe. I am long-term traveling in Europe and like to frequent DIY websites with people posting links to homedepot, but no someone at HD decided that European IPs couldn't access their site, so I and millions of others are locked out. The /16 is an Alibaba AS, and you make the assumption that most of it is malicious, but in reality you don't know. Fix your software, don't blindly block.
deleted 11mo ago
[deleted]
foo-bar-bat 11mo ago
When ever was the internet a safe haven, from what exactly?
- eimrine 11mo ago
  From governments, of course. There were times when criticism of anything and everything was a common and safe practice online. There are very few places where it is possible to keep practicing this now.
m3047 11mo ago
Upvoted not because the internet has ever been a safe haven, but for simply taking a moment to document the issue. But then again, I can't even give away a feed of what's bouncing off of my walls, drowning in my moat. (An Alibaba /16? I block not just 3/8, but every AWS range I can find.)
- zokier 11mo ago
  fyi aws have been publishing (since 2014) all their IP ranges in simple json format https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/ https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/ I'd hope other major clouds would do the same
- A1kmm 11mo ago
  It might be easier to block by ASN rather than hard-coding IP ranges. Something as simple as this in cron every 24 hours will help (adjust the ASNs in the bzgrep to your taste - and couple with occasional persistence so you don't get hit every reboot): TEMPDIR=$(mktemp -d) trap 'rm -r "$TEMPDIR"' EXIT curl https://archive.routeviews.org/oix-route-views/oix-full-snapshot-latest.dat.bz2 https://archive.routeviews.org/oix-route-views/oix-full-snap... -Lo "$TEMPDIR/snapshot.bz2" bzgrep -e " (15828|213035|400377|399471|210654|46573|211252|62904|135542|132372|36352|209641|7552|36352|12876|53667|138608|150393|60781|138607) i" $TEMPDIR/snapshot.bz2 | cut -d" " -f 3 | sort | uniq > $TEMPDIR/badranges iptables -N BAD_AS || true iptables -D INPUT -j BAD_AS || true iptables -A INPUT -j BAD_AS iptables -F BAD_AS for ROUTE in $(cat "$TEMPDIR/badranges"); do iptables -A BAD_AS -s $ROUTE -j DROP; done
miyuru 11mo ago
If anyone wants the 2000s internet experience for a while, I recommend deploying a website on IPv6-only server. It will be accessible to only about 50% of the internet, but back then not many people had internet anyway.
k0tan32 11mo ago
Some HNers already mentioned that the internet has not been a safe haven for a long time. All these vulnerability scanners and parsers were pinging my localhost servers even in mid 2k. It has just become worse, and even OSS and usually captcha-free places are installing things like Anubis [1]. All of this reminds me of some of Gibson's short stories I read recently and his description of Cyberspace: small corporate islands of protected networks in a hostile sea of sapient AIs ready to burn your brain. Luckily, LLMs are not there yet, except you can still get your brain burnt from AI slop or polarizing short videos. [1] - https://anubis.techaro.lol/ https://anubis.techaro.lol/
aledalgrande 11mo ago
I wonder how much of the world's compute/electricity is wasted in malicious bots.
d4rkn0d3z 11mo ago
It's not hard to build an internet that serves the people but nobody will pay you to do it, and if you are so brazen as to do it yourself then you will be investigated, harrassed, arrested, and beaten. Having been visited with every sorrow short of death, you will beg for death.
Animats 11mo ago
I wonder how much scraper effort is being spent talking to Samsung refrigerators and such.
phendrenad2 11mo ago
I have a question about this part: > moving the entire hosting to CloudFlare that will do it for me ... nor do I want to route my visitors through tracking-enabled USA servers Isn't there some EU equivalent to CloudFlare he can use? It's hard to admit, but DDoS mitigation is an essential part of having even a simple website these days.
singpolyma3 11mo ago
Yet another story demonstrating why you should not run fail2ban
tonyhart7 11mo ago
internet "safe heaven" is polarized on multiple walled garden like small forum,channel discord,niche app etc basically small traffic site where moderations and rules is can be enforced
retired_account 11mo ago
While not particularly helpful for multi-user instances, I’ve had good enough luck putting my Gitea server behind a Unifi gateway and accessing the admin via Teleport.
bsder 11mo ago
The reason we can't just block ranges is because CGNAT means that we wind up having to block entire providers instead of the individual with the compromised machine. IPv6 would solve this and we get the end-to-end nature of the internet back. So everybody will start screaming for IPv6 when?
p3rls 11mo ago
[dead]
cadamsdotcom 11mo ago
A simple captcha would be to just make the user move their mouse or finger to 2 or 3 dots in a square. Shouldn’t take more than a second to perform the motions as a website visitor.. If the movement is suspiciously smooth, or isn’t accelerated and decelerated as a human hand would, it’s a bot - or an agentic browser.

253 points by akyuu 11mo ago

Isn’t this problem why Cloudflare is popular? You can write your own server, but outsource protecting it from bots. Perhaps there are better alternatives?

BinaryIgor 11mo ago

I wonder why is it that we get an increase in these automated scrapers and attacks as of late (some few years); is there better (open-source?) technology that allows it? Is it because hosting infrastructure is cheaper also for the attackers? Both? Something else? Maybe the long-term solution for such attacks is to hide most of the internet behind some kind of Proof of Work system/network, so that mostly humans get to access to our websites, not machines.

trenchpilgrim 11mo ago
Using AI you can write a naive scraper in minutes and there's now a market demand for cleaned up and structured data.
marginalia_nu 11mo ago
What's missing is effective international law enforcement. This is a legal problem first and foremost. As long as it's as easy as it is to get away with this stuff by just routing the traffic through a Russian or Singaporean node, it's going to keep happening. With international diplomacy going the way it has been, odds of that changing aren't fantastic. The web is really stuck between a rock and a hard place when it comes to this. Proof of work helps website owners, but makes life harder for all discovery tools and search engines. An independent standard for request signing and building some sort of reputation database for verified crawlers could be part of a solution, though that causes problems with websites feeding crawlers different content than users, an does nothing to fix the Sybil attack problem.
- luckylion 11mo ago
  It's not necessarily going through a Russian or Singaporean node though, on the sites I'm responsible for, AWS, GCP, Azure are in the top 5 for attackers. It's just that they don't care _at all_ about that happening. I don't think you need world-wide law-enforcement, it'll be a big step ahead if you make owners & operators liable. You can limit exposure so nobody gets absolutely ruined, but anyone running wordpress 4.2 and getting their VPS abused for attacks currently has 0 incentive to change anything unless their website goes down. Give them a penalty of a few hundred dollars and suddenly they do. To keep things simple, collect from the hosters, they can then charge their customers, and suddenly they'll be interested in it as well, because they don't want to deal with that. The criminals are not held liable, and neither are their enablers. There's very little chance anything will change that way.
  - mrweasel 11mo ago
    The big cloud provides needs to step up and take responsibility. I understand that it can't be to easy to do, but we really do need a way to contact e.g. AWS and tell them to shut of a costumer. I have no problem with someone scraping our websites, but I care that they don't do so responsibly, slow down when we start responding slower, don't assume that you can just go full throttle, crash our site, wait, and then do it again once we start responding again. You're absolutely right: AWS, GCP, Azure and others, they do not care and especially AWS and GCP are massive enablers.
    - ctoth 11mo ago
      > we really do need a way to contact e.g. AWS and tell them to shut of a costumer. You realize you just described the infrastructure for far worse abuse than a misconfigured scraper, right?
      - mrweasel 11mo ago
        I'm very aware of that, yes. There needs to be a good process, the current situation where AWS simply does not care, or doesn't know also isn't particularly good. One solution could be for victims to notify AWS that a number of specified IP are generating an excessive amount of traffic. An operator could then verify with AWS traffic logs, notify the customer that they are causing issue and only after a failure to respond could the customer be shut down. You're not wrong that abuse would be a massive issue, but I'm on the other side of this and need Amazon to do something, anything.
rkagerer 11mo ago
long-term solution How about a reputation system? Attached to IP address is easiest to grok, but wouldn't work well since addresses lack affinity. OK, so we introduce an identifier that's persistent, and maybe a user can even port it between devices. Now it's bad for privacy. How about a way a client could prove their reputation is above some threshold without leaking any identifying information? And a decentralized way for the rest of the internet to influence their reputation (like when my server feels you're hammering it)? Do anti-DDoS intermediaries like Cloudflare basically catalog a spectrum of reputation at the ASN level (pushing anti-abuse onus to ISP's)? This is basically what happened to email/SMTP, for better or worse :-S.
- JimDabell 11mo ago
  Reputation plus privacy is probably unsolvable; the whole point of reputation is knowing what people are doing elsewhere. You don’t need reputation, you need persistence. You don’t need to know if they are behaving themselves elsewhere on the Internet as long as you can ban them once and not have them come back. Services need the ability to obtain an identifier that: - Belongs to exactly one real person. - That a person cannot own more than one of. - That is unique per-service. - That cannot be tied to a real-world identity. - That can be used by the person to optionally disclose attributes like whether they are an adult or not. Services generally don’t care about knowing your exact identity but being able to ban a person and not have them simply register a new account, and being able to stop people from registering thousands of accounts would go a long way towards wiping out inauthentic and abusive behaviour. The ability to “reset” your identity is the underlying hole that enables a vast amount of abuse. It’s possible to have persistent, pseudonymous access to the Internet without disclosing real-world identity. Being able to permanently ban abusers from a service would have a hugely positive effect on the Internet.
  - jasonjayr 11mo ago
    A digital "Death penalty" is not a win for society, without considering a fair way to atone for "crimes against your digital identity". It would be way to easy for the current regime (whomever that happens to be) to criminalize random behaviors (Trans People? Atheists? Random nationality?) to ban their identity, and then they can't apply for jobs, get bus fare, purchase anything online, communicate with their lawyers, etc.
hnthrowaway0315 11mo ago
I guess it is just because 1) They can, and 2) Everyone wants some data. I think it would be interesting if every website out there starts to push out BS pages just for scrappers. Not sure how much extra cost it's going to take if a website puts up say 50% BS pages that only scrappers can reach, or BS material with extremely small fonts hidden in regular pages that ordinary people cannot see.
- inerte 11mo ago
  Something like https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ ?
  - hnthrowaway0315 11mo ago
    Yeah something like this, would be nice if it actually feeds bad data that requires human to double confirm, too. Not something seriously wrong but something subtle, like changing a couple of letters in a name of a country, or randomize the National day. Once a lot of websites start to use it AI might actually get confused, I think? But humans never read these pages so should be largely fine -- unless they are reading AI summaries.
Vegenoid 11mo ago
I'm pretty sure it is the commercial demand for data from AI companies. It is certainly the popular conception among sysadmins that it is AI companies who are responsible for the wave of scrapers over the past few years, and I see no compelling alternative.
- embedding-shape 11mo ago
  > and I see no compelling alternative. Another potential cause: It's way easier for pretty much any person connected to the internet to "create" their own automation software by using LLMs. I could wager even the less smart LLMs could handle "Create a program that checks this website every second for any product updates on all pages" and give enough instructions for the average computer user to be able to run it without thinking or considering much. Multiply this by every person with access to an LLM who wants to "do X with website Y" and you'll get an magnitude increase in traffic across the internet. This been possible since what, 2023 sometime? Not sure if the patterns would line up, but just another guess for the cause(s).
EGreg 11mo ago
Why? It’s because of AI. It enables attacks at scale. It enables more people to attack, who previously couldn’t. And so on. It’s very explainable. And somehow, like clockwork, there are always comments to say “there is nothing new, the Internet has always been like this since the 80s”. You know, part of me wants to see AI proliferate into more and more areas, just so these people will finally wake up eventually and understand there is a huge difference when AI does it. When they are relentlessly bombarded with realistic phone calls from random numbers, with friends and family members calling about the latest hoax and deepfake, when their own specific reputation is constantly attacked and destroyed by 1000 cuts not just online but in their own trusted circles, and they have to put out fires and play whack-a-mole with an advanced persistent threat that only grows larger and always comes from new sources, anonymous and not. And this is all before bot swarms that can coordinate and plan long-term, targeting specific communities and individuals. And this is all before humanoid robots and drones proliferate. Just try to fast-forward to when human communities online and offline are constantly infiltrated by bots and drones and sleeper agents, playing nice for a long time and amassing karma / reputation / connections / trust / whatever until finally doing a coordinated attack. Honestly, people just don’t seem to get it until it’s too late. Same with ecosystem destruction — tons of people keep strawmanning it as mere temperature shifts, even while ecosystems around the world get destroyed. Kelp forests. Rainforests. Coral reefs. Fish. Insects. And they’re like “haha global warming by 3 degrees big deal. Temperature has always changed on the planet.” (Sound familiar?) Look, I don’t actually want any of this to happen. But if they could somehow experience the movie It’s a Wonderful Life or meet the Ghost of Christmas Yet to Come, I’d wholeheartedly want every denier to have that experience. (In fact, a dedicated attacker can already give them a taste of this with current technology. I am sure it will become a decentralized service soon :-( )
- hshdhdhj4444 11mo ago
  Our tech overlords understand AI, especially any form of AGI, will basically be the end of humanity. That’s why they’re entirely focused on being the first and amassing as much wealth in the meanwhile, giving up on any sort of consideration whether they’re doing good for people or not.

tonetheman 11mo ago

[dead]

jchw 11mo ago

Anubis is definitely playing the cat-and-mouse game to some extent, but I like what it does because it forces bots to either identify themselves as such or face challenges. That said, we can likely do better. Cloudflare does good in part because Cloudflare runs so much traffic, so they have a lot of data across the internet. Smaller operators just don't get enough traffic to really deal with banning abusive IPs without banning entire ranges indefinitely, not ideal. I hope to see a solution like Crowdsec where reputation data can be crowdsourced to block known bad bots (at least for a while since they are likely borrowing IPs) while using low complexity (potentially JS-free) challenges for IPs with no bad reputation. It's probably too much to ask for Anubis upstream which is probably already too busy dealing with the challenges of what it already does at the scale it is operating, but it does leave some room for further innovation for whoever wants to go for it. In my opinion there is at least no reason why it is not plausible to have a drop-in solution that can mostly resolve these problems and make it easier for hobbyists to run services again.

gjsman-1000 11mo ago

The problem with anything, anything, without a centralized authority, is that friction overwhelms inertia. Bad actors exist and have no mercy, while good people downplay them until it’s too late. Entropy always wins. Misguided people assume the problem is powerful people, when the problem is actually what the powerful people use their authority to do, as powerful people will always exist. Accepting that and maintaining oversight is the historically successful norm; abolishing them has always failed. As such, I don’t identify with the author of this post, about trying to resist CloudFlare for moral reasons. A decentralized system where everyone plays nice and mostly cooperates, does not exist any more than a country without a government where everyone plays nice and mostly cooperates. It’s wishful thinking. We already tried this with Email, and we’re back to gatekeepers. Pretending the web will be different is ahistorical.

pixl97 11mo ago
The internet has made the world small and that's a problem. Nation states typically had a limited range of broadcasting their authority in the more distant past. A bad ruler couldn't rule the entire world, nor could they cause trouble with the entire world. From nukes to the interconnected web the worst of us with power can effect everyone else.
martin-t 11mo ago
Power is a spectrum. Power differentials will always exist but we can absolutely strive to make them smaller and smaller. 1) Most of the civilized world no longer has hereditary dictators (such as "kings"). Because they were removed from power by the people and the power was distributed among many individuals. It works because malicious (anti-social) individuals have trouble working together. And yes, oversight helps. But it's a spectrum and we absolutely can and should move the needle towards more oversight and more power distribution. 2) Corporate power structures are still authoritarian. We can change that too.

time4tea 11mo ago

Had to ban RU, CN, SG, and KR just cos of the volume of spam. The random referer headers has recently become a problem. This is particularly annoying as knowing where people come from is important. Its just another reason to give up making stuff, and give in to the FAANG and the AI enshittification. :-(

mrweasel 11mo ago
If you only care about regular users I'd advice banning all known datacenters, Browserbase, China and Brazil.
- worthless-trash 11mo ago
  I also banned the middle east, Logs actually reflected real users, cost dropped, and my mental health improved. Win Win.
  - mrweasel 11mo ago
    If you know that you don't have customers or users in the area, or very few, then go for it. I worked in e-commerce previously, we reduce fraud to almost zero by banning non-local cards. It affected a few customers that had international credit cards, but not enough to justify dealing with the fraud. Sometimes you just need to limit your attack surface.

mberning 11mo ago

the internet is over. If we want to recapture the magic of the earlier times we are going to have to invent something new.

fithisux 11mo ago
going back to Gopher?
- itintheory 11mo ago
  Gopher still requires the Internet. I know it's pretty common to conflate "the Internet" with "the World Wide Web", but there are actually other protocols out there (like Gopher).
  - fithisux 11mo ago
    I don't see a solution then.
- groundzeros2015 11mo ago
  Why would that help?
pixl97 11mo ago
There is no something new. Anything we invent will be able to be taken over by complex bots. Welcome to the futureshock where humans aren't at the top of their domain.
9rx 11mo ago
The magical times in the past have always been marked with being able to be part of an "exclusive club" that takes something from nothing to changing the world. Because of the internet, magical times can never be had again. You can invent something new, but as soon as anyone finds out about it, everyone now finds out about it. The "exclusive club" period is no more.
- martin-t 11mo ago

bpt3 11mo ago

The internet hasn't been a safe haven since the 80s, or maybe earlier (that was before my time, and it's never been one since I got online in the early 90s). The only real solution is to implement some sort of identity management system, but that has so many issues that make it a non-starter.

lotsofpulp 11mo ago
> The only real solution is to implement some sort of identity management system, but that has so many issues that make it a non-starter. Apple and Alphabet seem positioned to easily enable it. https://www.apple.com/newsroom/2025/11/apple-introduces-digital-id-a-new-way-to-create-and-present-an-id-in-apple-wallet/ https://www.apple.com/newsroom/2025/11/apple-introduces-digi...
- JSR_FDED 11mo ago
  I don’t get it. That link refers to Apple letting you put your passport and drivers license info in the wallet on your phone.
  - Astronaut3315 11mo ago
    Apple’s Wallet app presents this feature as being for “age and identity verification in apps, online and in person”.
- pixl97 11mo ago
  Alphabet the company that bans people for opaque reasons with no recourse, good idea. Maybe tech should not be in charge of digital identification
  - lotsofpulp 11mo ago
    The governments like it that way. They want banks and tech companies to be intermediaries that are more difficult to hold accountable, because they can just say “we didn’t feel like doing business with this person”.

qwertox 11mo ago

Since I moved my DNS records to Cloudflare (that is: nameserver is now the one from Cloudflare), I get tons of odd connections, most notably SYN packets to eihter 443 or 22, which never respond back after the SYN-ACK. They ping me once a second in average, distributing the IPs over a /24 network. I really don't understand why they do this, and it's mostly some shady origins, like vps game server hoster from Brazil and so on. I'm at the point where i capture all the traffic and looks for SYN packets, check the RDAP records for them to decide if I then drop the entire subnets of that organization, whitelisting things like Google. Digital Ocean is notoriously a source of bad traffic, they just don't care at all.

kzemek 11mo ago
These are spoofed packets for SYNACK reflection attacks. Your response traffic goes to the victim, and since network stacks are usually configured to retry SYNACK a few times, they also get amplification out of it
- pabs3 11mo ago
  There is a solution to that, but it requires these companies to implement source address validation. If your ISP is on the list, maybe complain about it. https://spoofer.caida.org/as_stats.php https://spoofer.caida.org/as_stats.php
sva_ 11mo ago
> like vps game server hoster from Brazil and so on. Probably someone DDoSing a Minecraft server or something. People in games do this where they DDoS each other. You can get access to a DDoS panel for as little as $5 a month. Some providers allow for spoofing the src ip, that's how they do these reflection attacks. So you're not actually dropping the sender of these packets, but the victims. Consider turning reverse path filter to strict as a basic anti spoofing method and see if it helps net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1
- qwertox 11mo ago

threeducks 11mo ago

> Fail2ban was struggling to keep up: it ingests the Nginx access.log file to apply its rules but if the files keep on exploding… > [...] > But I don’t want to fiddle with even more moving components and configuration You can configure nginx to do rate-limiting directly. Blog post with more details: https://blog.nginx.org/blog/rate-limiting-nginx https://blog.nginx.org/blog/rate-limiting-nginx

embedding-shape 11mo ago

> The internet is no longer a safe haven for software hobbyists Maybe I've just had bad luck, but since I started hosting my own websites back around 2005 or so, my servers have always been attacked basically from the moment they come online. Even more so when you attach any sort of DNS name to it, especially when you use TLS and the certificates, guessing because they end up in a big index that is easily accessible (the "transparency logs"). Once you start sharing your website, it again triggers an avalanche of bad traffic, and the final boss is when you piss of some organization and (I'm assuming) they hire some bad actor to try to make you offline. Dealing with crawlers, bot nets, automation gone wrong, pissed of humans and so on have been almost a yearly thing for me since I started deploying stuff to the public internet. But again, maybe I've had bad luck? Hosted stuff across wide range of providers, and seems to happen across all of them.

zwnow 11mo ago
My first ever deployed project was breached on day 1 with my database dropped and a ransom note in there. Was a beginner mistake by me that allowed this, but it's pretty discouraging. Its not the internet that sucks, its people that suck.
- mattmaroon 11mo ago
  Well I guess at least on day 1 you didn’t have much to lose!
  - zwnow 11mo ago
    Its a personal blog so even if data was lost it would've been just posts that nobody reads. Certainly not worth the 0.00054 BTC they wanted
  - timeinput 11mo ago
    more like a zero day on day zero.
aftbit 11mo ago
My stuff used to get popped daily. A janky PHP guestbook I wrote just to learn back in the early 2000s? No HTML injection protection & someone turned my site into spammy XSS hack within days. A WordPress installation I fell behind on patching? Turned into SEO spam in hours. A redis instance I was using just to learn some of their data structures that got accidentally exposed to the web? Used to root my computer and install a botnet RAT. This was all before 2020. I never felt this made the internet "unsafe". Instead, it just reminded me how I messed up. Every time, I learned how to do better, and I added more guardrails. I haven't gotten popped that obviously in a long time, but that's probably because I've acted to minimize my public surface area, used star-certs to avoid being in the cert logs, added basic auth whenever I can, and generally refused to _trust_ software that's exposed to the web. It's not unsafe if you take precautions, have backups, and are careful about what you install. If you want to see unsafe, look at how someone who doesn't understand tech tries to interact with it. Downloading any random driver or exe to fix a problem, installing apps when a website would do, giving Facebook or Tiktok all of their information and access without recognizing that just maybe these multi-billion-dollar companies who give away all of their services don't have your best interests in mind.

quaintdev 11mo ago

I do not have a solution for blog like this but if you are self hosting I recommend enabling mTLS on your reverse proxy. I'm doing this for a dozen services hosted at home. The reverse proxy just drops the request if user does not present a certificate. My devices which can present cert can connect seamlessly. It's a one time setup but once done you can forget about it.

SoftTalker 11mo ago
That's fine if you're hosting stuff just for yourself but not really practical if you're hosting stuff you want others to be able to read, such as a blog.
- lukevp 11mo ago
  You can mTLS to CloudFlare too, if you’re not one of the anti-CloudFlare people. Then all traffic drops besides traffic that passes thru CF and the mTLS handshake prevents bypassing CF.
  - BehindTheMath 11mo ago
    You don't need mTLS for that. Just block all IPs beside for Cloudflare's ranges.
bogwog 11mo ago
Wireguard is much better. Not only is it easier to set up/maintain, it even works on Android and iOS. I used to use client authentication for my private git server, but getting client certs installed on every client browser or app was a pain in the ass, and not even possible for some mobile browsers. Today, my entire network of self hosted stuff exists in a personal wireguard VPN. My firewall blocks everything except the wireguard port (even SSH).

cyp0633 11mo ago

My Gitea instance also encountered aggressive scraping some days ago, but with highly distributed IP & ASN & geolocation, each of which is well below the rate of a human visitor. I assume Anubis will not stop the massively funded AI companies, so I'm considering poisoning the scrapers with garbage code, only targeting blind scrapers, of course.

mrweasel 11mo ago
Sadly we're now seeing services that sell proxy services that allows you to scape from a wide variety of residential IPs, some even goes so far as to labels their IPs as "ethically sources".

deleted 11mo ago

[deleted]

skopje 11mo ago

sad but hosting static content like his site in a cloud would save him a headache. i know i know, "do it yourself" and all but if that is his path he knows the price. maybe i am wrong and do not understand the problem but it seems like he is asking for a headache. edit: words

Nifty3929 11mo ago
I think the author would agree, and is in fact the point of his post. The only way to solve these problems is using some large hosted platform where they have the resources to constantly be managing these issues. This would solve their problem. But isn't it sad that we can't host our own websites anymore, like many of us used to? It was never easy, but it's nearly impossible now and this is only one reason.
- skopje 11mo ago
  i think it has been a hard to host a site since about 2007. i stopped then because it is too much work to keep it safe. even worse now but it has always been extra work since search engines. maybe the OP is just getting older and wants to spend time with his kids and not play with nginx haha.

zdc1 11mo ago

I wonder if you can have a chain of "invisible" links on your site that a normal person wouldn't see or click. The links can go page A -> page B -> page C, where a request for C = instant IP ban.

chrisweekly 11mo ago
IP addresses from scrapers are innumerable and in constant rotation.
SkiFire13 11mo ago
Scrapers nowadays can use residential and mobile IPs, so banning by IP, even if actual malicious requests are coming from them, can also prevent actual unrelated people from accessing your service.
- SoftTalker 11mo ago
  Unless you're running a very popular service, unlikely that a random residential IP would be both compromised by a malicious VPN and also trying to access your site legitimately.
  - arbol 11mo ago
    Lots of people have chrome extensions installed that use their connection like proxy so this is more common than you think
    - SoftTalker 11mo ago
      Can you provide any examples of these extensions? I'm not doubting you, just curious.
      - arbol 11mo ago
        There's one mentioned here: https://www.bleepingcomputer.com/news/security/data-stealing-chrome-extensions-impersonate-fortinet-youtube-vpns/ https://www.bleepingcomputer.com/news/security/data-stealing... Anyone who owns a chrome extension with 50k+ installs is regularly asked to sell it to people (myself included). The people who buy the extensions try to monetize them any way they can, like proxying traffic for malicious scrapers / attacks.

sequoia 11mo ago

I don't know if there's a simple solution to deploy this but JA3 fingerprinting is sometimes used to identify similar clients even if they're spread across IPs: https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967/ https://engineering.salesforce.com/tls-fingerprinting-with-j...

arbol 11mo ago
You need to terminate the TLS connection yourself so this prevents people from using DNS proxy, e.g. Cloudflare. Then you have to run a server that has a module that computes the ja3/ja4, e.g. nginx. Even then, it's possible to set your client hello in python/curl/etc. to exactly mirror the JA4 of your chosen browser like Chrome. So ja4 stops basic bots but most seasoned scrapers already implement valid ja4s/ja3s
- sequoia 11mo ago
  I thought it wasn't just a matter of valid but of identicalness. Multiple clients with identical JA4 which comprises if I'm not mistaken useragent but also other aspects of the host machine indicate that they are in fact a single user agent.
  - arbol 11mo ago
    I'm sorry, I don't fully understand your question. This is what I meant by invalid: Anyone using chrome 142 has the same JA4 - I've checked across OS/devices. If you use nodeJS and set your user agent to "chrome 142" then you will have an invalid/incorrect JA4 and you'll stick out from the crowd.

petermcneeley 11mo ago

The real internet has yet to be invented.

zkmon 11mo ago

Common man never had a need for internet or global connectedness. DARPA wanted to push technology to gain upper hand in the world matters. Universities pushed technology to show progress and sell research. Businesses pushed technologies to have more sales. It was kind of acid rain that was caused by the establishments and sold as scented rain.

wartywhoa23 11mo ago
This sentiment - along the lines of "the world became too dependent on the Internet", "Internet wasn't a good thing to begin with", "Internet is a threat to national security" etc - has been popping up on HN too often lately, emerged too abruptly and correlates with the recent initiatives to crack down on the Internet too well. If this is your own opinion and not a part of a psyop to condition people into embracing the death of the Internet as we know it, do you have any solution to propose?
- gchamonlive 11mo ago
  You don't need to have a solution to explore a problem in my opinion. OP comment is problematic but for reasons other than not having a proposed solution.
gchamonlive 11mo ago
> Common man never had a need for internet or global connectedness That's not how culture evolves. You don't necessarily need to have a problem so that a solution is developed. You can very well have a technology developed for other purposes, or just for exploration sake, and then as this tech exists uses for it start to pop post hoc. You therefore ignore the immense benefit of access to information that technology has, something that wasn't necessarily a problem for the common man but once its there, the popularization of the access to information, they adapt and grow dependent on it. Just like electricity.
- zkmon 11mo ago
  >>immense benefits ?? People with dialup telephones never asked for a smartphone connected to internet. They were just as happy back then or even more happy because phone didn't eat off their time or cause posture problems. Sure, shopping was slower without amazon website, but not less happy experience back then. Infact homes had less junk and people saved more money Messaging? sure it makes you spend time with 100 whatsapp groups, where 99% of the people don't know you personally. It helped companies to sell more of the junk more quickly. It created bloggers and content creators who lived in an imaginary world thinking that someone really consumes their content. It created karma beggers who begged globally for likes that are worth nothing. It created more concentration of wealth at some weird internet companies, which don't solve any of the world problems or basic needs of the people. And finally it created AI that pumps plastic sewage to fill the internet. There it is, your immensely useful internet. As if the plastic pollution was not enough in the real world, the internet will be filled with plastic content. What else did internet give that is immensely helpful?

xedrac 11mo ago

I run a dedicated firewall/dns box with netfilter rules to rate limit new connections per IP. It looks like I may need to change that to rate limit per /16 subnet...

bo1024 11mo ago

I wonder if a proof of work protocol is a viable solution. To GET the page, you have to spend enough electricity to solve a puzzle. The question is whether the threshold could be low enough for typical people on their phones to access the site easily, but high enough that mass scraping is significantly reduced.

kalavan 11mo ago
There's this paper from 2004: "Proof-of-Work Proves Not to Work": https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf The conclusion back then was that it's impossible to make a threshold that is both low enough and high enough. You need some other mechanism that can distinguish bad traffic from good (even if imperfectly), and then adjust the threshold based on it. See, for instance, "Proof of Work can Work": https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lui-pow.pdf https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lu...
- bo1024 11mo ago
  Thanks for these references! I imagine the numbers would be entirely different in our context (20 years later and web serving, not email sending). And the idea of spammers using bot nets (therefore not paying for computer themselves) would be less relevant to LLM scraping. But I’ll try to check for forward references on these.
  - kalavan 11mo ago
    > And the idea of spammers using bot nets (therefore not paying for computer themselves) would be less relevant to LLM scraping. It's possible that the services that reward users for running proxies (or are bundled with mobile apps with a notice buried in the license) would also start rewarding/hiding compute services as well. There's currently no money in it because proof-of-work is so rare, but if it changes, their strategy might too.
- beeflet 11mo ago
  Good links, but this is just for email and relies on some (admittedly) pretty lofty assumptions

smileson2 11mo ago

It's probably just time for the web page to die

zelphirkalt 11mo ago
What do you mean by that? Web pages is the central mechanism, we use to put information on the web. Of course many websites are shitty and could be much simpler to convey their information, without looking crap at all. But the web page in general? Why would we ever get rid of something so useful? And what do you suggest as an alternative?
ksenzee 11mo ago
...they said, on a web page.
- smileson2 11mo ago
  so we should settle on using web pages to do that for another 20-30 years? this is it for the rest of time?
  - ksenzee 11mo ago
    I’m suggesting we come up with a replacement for the web before we kill it off.

cosmicgadget 11mo ago

> Other things I’ve noticed is increased traffic with Referer headers coming from strange websites such as bioware.com, mcdonalds.com, and microsoft.com I've been seeing this too, I guess scrapers think they can get through some blockers with a referrer?

ModernMech 11mo ago

The Internet was a scene, and like all scenes it's done now the corpos have moved in and taken over (because at that point it's just ads and rent extraction in perpetuity). I dunno what/where/when the next tech scene will be, but I do know it's not going to come from Big Tech. See: Metaverse.

jcalvinowens 11mo ago

Scrapers have constantly been running against my cgit server for the past year, but they're bizarrely polite in my case... 2-3 requests per minute. This whole enterprise is clearly run by exceptionally dumb people, since you can just clone all the code I host there directly from upstreams... [16/Nov/2025:16:21:12 +0000] 190.92.214.144:34638 . "GET /cgit/linux/commit/drivers/vlynq?h=v5.15.76&id=59d42cd43c7335a3a8081fd6ee54ea41b0c239be HTTP/1.1" -> 200 3051b 3.42x 0.239ms [16/Nov/2025:16:22:15 +0000] 188.239.57.1:40328 . "GET /cgit/linux/commit/kernel/range.c?h=v6.12.31&id=459b37d423104f00e87d1934821bc8739979d0e4 HTTP/1.1" -> 200 2993b 3.42x 0.266ms [16/Nov/2025:16:22:56 +0000] 190.92.217.125:56580 . "GET /cgit/linux/commit/kernel?h=v5.15.92&id=f01aefe374d32c4bb1e5fd1e9f931cf77fca621a HTTP/1.1" -> 200 3091b 3.28x 0.250ms [16/Nov/2025:16:23:17 +0000] 159.138.10.64:44540 . "GET /cgit/linux/commit/drivers/mtd/mtdcore.c?h=v6.2.15&id=249858575fd3f27904d6bb775e5ab500e9ef3b0f HTTP/1.1" -> 200 3415b 3.47x 0.251ms [16/Nov/2025:16:23:58 +0000] 119.13.101.228:44342 . "GET /cgit/linux/commit/drivers/gpio?h=v6.6.93&id=bc7fe1a879fc024942bb9eff173fa619b722d09b HTTP/1.1" -> 200 3582b 3.37x 0.250ms

deleted 11mo ago
[deleted]
jcalvinowens 11mo ago
Turned out those scattered IPs were from a very small number of places, blocking these five killed 100% of my crawler spam: AS4229 Zenlayer (Singapore) PTE. LTD AS21859 ZEN-ECN, US AS45102 ALIBABA-CN-NET Alibaba US Technology Co., Ltd., CN AS132203 TENCENT-NET-AP-CN Tencent Building, Kejizhongyi Avenue, CN AS136907 HUAWEI INTERNATIONAL PTE. LTD. That's 4392 contiguous IP ranges.
- deleted 11mo ago
  [deleted]

firefoxd 11mo ago

I have been using zipbombs and they were effective to some extent. Then I had the smart idea to write about it on HN [0]. The result was a flood of new types of bots that overwhelmed my $6 server. For ~100k daily request, it wasn't sustainable to serve 1 to 10MB payloads. I've updated my heuristic to only serve the worst offenders, and created honeypots to collect ips and repond with 403s. After a few months, and some other spam tricks I'll keep to myself this time, my traffic is back to something reasonable again. [0]: https://news.ycombinator.com/item?id=43826798 https://news.ycombinator.com/item?id=43826798

tetris11 11mo ago
There's likely a large market for this kind of thing. Maybe time to spin out a side business and deploy your heuristics to struggling IPs. Though I have to admit I dont know who your target audience would be. Self-hosting orgs don't tend to be flush with cash

AaronAPU 11mo ago

Everything good enough to become popular gets swarmed by the teeming masses and then exploited and destroyed. The only solution seems to be to constantly abandon those things and move on to new frontiers to enjoy until the cycle repeats.

svara 11mo ago

The Internet has really been an interesting case study for what happens between people when you remove a varying number of layers of social control. All the way back to the early days of Usenet really. I would hate to see it but at the same time I feel like the incentives created by the bad actors really push this towards a much more centralized model over time, e.g. one where all traffic provenance must be signed and identified and must flow through a few big networks that enforce laws around that.

Cosi1125 11mo ago
"Socialists"* argue for more regulations; "liberals" claim that there should be financial incentives to not do that. I'm neither. I believe that we should go back to being "tribes"/communities. At least it's a time-tested way to – maybe not prevent, but somewhat allieviate – the tragedy of the commons. (I'm aware that this is a very poor and naive theory; I'll happily ditch it for a better idea.) -- *) For the lack of a better word.
- thrance 11mo ago
  What would prevent attacks between "tribes"? What would prevent one from taking over the others and sending us back to square one?
  - Cosi1125 11mo ago
    Little would prevent attacks by APTs and other powerful groups. (This, btw., is one of the few facets of this problem that technology could help solve.) But a trivial change: a hard requirement to sign up (=send a human-composed message to one of the moderators) to be able to participate (or, in extreme cases, to read the contents) "automagically" stops almost all spam, scrapers (in the extreme case), vandalism, etc. (from my personal experience based on a rather large sample). I think it's one of the multi-faceted problems where technology (a "moat", "palisade", etc. for your "tribe") should accompany social changes.

erickhill 11mo ago

I very much relate to the author's sour mood and frustration. I also host a small hobby forum and have experienced the same attacks constantly, and it has gotten especially bad the last couple of years with the rise of AI. In the early days I put Google Analytics on the site so I could observe traffic trends. Then, we were all forced to start adding certificates to our sites to keep them "safe". While I think we're all doomed to continue that annual practice or get blocked by browsers, I have often considered removing Google Analytics. Ever since their redesign it is essentially unusable for me now. What benefit does it bring if I can't understand the product anymore? Last year, in a fit of desperation, I added Cloudflare. This has a brute force "under attack" mode that seems to stop all bots from accessing the site. It puts up a silly "hang on a second, are you human" page before the site loads, but it does seem to work. It is great UX? No, but at least the site isn't getting hammered by various locations in Asia. Cloudflare also let me block entire countries, although that seems to be easily fooled. I also don't think a lot of the bots/AI crawlers honor the rules set in the robots.txt. It's all an honor system anyway, and they are completely lacking in it. There need to be some hard and fast rules put in place, somehow, to stop the madness.

timpera 11mo ago
Cloudflare does work, but it often destroys the experience for legitimate users. On the website I manage, non-technical users were often getting stuck on the Cloudflare captcha, so I ended up removing it. Then there's also the issue with dependence to US-based services, but that may not be an issue for you.

mrb 11mo ago

Unpopular opinion: the real source of the problem is not scrapers, but your unoptimized web software. Gitea and Fail2ban are resource hogs in your case, either unoptimized or poorly configured. My tiny personal web servers can whistand thousands of requests per second, barely breaking a sweat. As a result, none of the bots or scrapers are causing any issue. "The only thing that had immediate effect was sudo iptables -I INPUT -s 47.79.0.0/16 -j DROP" Well, by blocking an entire /16 range, it is this type of overzealous action that contributes to making the internet experience a bit more mediocre. This is the same thinking that lead me to, for example, not being able to browse homedepot.com from Europe. I am long-term traveling in Europe and like to frequent DIY websites with people posting links to homedepot, but no someone at HD decided that European IPs couldn't access their site, so I and millions of others are locked out. The /16 is an Alibaba AS, and you make the assumption that most of it is malicious, but in reality you don't know. Fix your software, don't blindly block.

deleted 11mo ago

[deleted]

foo-bar-bat 11mo ago

When ever was the internet a safe haven, from what exactly?

eimrine 11mo ago
From governments, of course. There were times when criticism of anything and everything was a common and safe practice online. There are very few places where it is possible to keep practicing this now.

m3047 11mo ago

Upvoted not because the internet has ever been a safe haven, but for simply taking a moment to document the issue. But then again, I can't even give away a feed of what's bouncing off of my walls, drowning in my moat. (An Alibaba /16? I block not just 3/8, but every AWS range I can find.)

zokier 11mo ago
fyi aws have been publishing (since 2014) all their IP ranges in simple json format https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/ https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/ I'd hope other major clouds would do the same
A1kmm 11mo ago
It might be easier to block by ASN rather than hard-coding IP ranges. Something as simple as this in cron every 24 hours will help (adjust the ASNs in the bzgrep to your taste - and couple with occasional persistence so you don't get hit every reboot): TEMPDIR=$(mktemp -d) trap 'rm -r "$TEMPDIR"' EXIT curl https://archive.routeviews.org/oix-route-views/oix-full-snapshot-latest.dat.bz2 https://archive.routeviews.org/oix-route-views/oix-full-snap... -Lo "$TEMPDIR/snapshot.bz2" bzgrep -e " (15828|213035|400377|399471|210654|46573|211252|62904|135542|132372|36352|209641|7552|36352|12876|53667|138608|150393|60781|138607) i" $TEMPDIR/snapshot.bz2 | cut -d" " -f 3 | sort | uniq > $TEMPDIR/badranges iptables -N BAD_AS || true iptables -D INPUT -j BAD_AS || true iptables -A INPUT -j BAD_AS iptables -F BAD_AS for ROUTE in $(cat "$TEMPDIR/badranges"); do iptables -A BAD_AS -s $ROUTE -j DROP; done

miyuru 11mo ago

If anyone wants the 2000s internet experience for a while, I recommend deploying a website on IPv6-only server. It will be accessible to only about 50% of the internet, but back then not many people had internet anyway.

k0tan32 11mo ago

Some HNers already mentioned that the internet has not been a safe haven for a long time. All these vulnerability scanners and parsers were pinging my localhost servers even in mid 2k. It has just become worse, and even OSS and usually captcha-free places are installing things like Anubis [1]. All of this reminds me of some of Gibson's short stories I read recently and his description of Cyberspace: small corporate islands of protected networks in a hostile sea of sapient AIs ready to burn your brain. Luckily, LLMs are not there yet, except you can still get your brain burnt from AI slop or polarizing short videos. [1] - https://anubis.techaro.lol/ https://anubis.techaro.lol/

aledalgrande 11mo ago

I wonder how much of the world's compute/electricity is wasted in malicious bots.

d4rkn0d3z 11mo ago

It's not hard to build an internet that serves the people but nobody will pay you to do it, and if you are so brazen as to do it yourself then you will be investigated, harrassed, arrested, and beaten. Having been visited with every sorrow short of death, you will beg for death.

Animats 11mo ago

I wonder how much scraper effort is being spent talking to Samsung refrigerators and such.

phendrenad2 11mo ago

I have a question about this part: > moving the entire hosting to CloudFlare that will do it for me ... nor do I want to route my visitors through tracking-enabled USA servers Isn't there some EU equivalent to CloudFlare he can use? It's hard to admit, but DDoS mitigation is an essential part of having even a simple website these days.

singpolyma3 11mo ago

Yet another story demonstrating why you should not run fail2ban

tonyhart7 11mo ago

internet "safe heaven" is polarized on multiple walled garden like small forum,channel discord,niche app etc basically small traffic site where moderations and rules is can be enforced

retired_account 11mo ago

While not particularly helpful for multi-user instances, I’ve had good enough luck putting my Gitea server behind a Unifi gateway and accessing the admin via Teleport.

bsder 11mo ago

The reason we can't just block ranges is because CGNAT means that we wind up having to block entire providers instead of the individual with the compromised machine. IPv6 would solve this and we get the end-to-end nature of the internet back. So everybody will start screaming for IPv6 when?

p3rls 11mo ago

[dead]

cadamsdotcom 11mo ago

A simple captcha would be to just make the user move their mouse or finger to 2 or 3 dots in a square. Shouldn’t take more than a second to perform the motions as a website visitor.. If the movement is suspiciously smooth, or isn’t accelerated and decelerated as a human hand would, it’s a bot - or an agentic browser.