4 ms·

I wonder why is it that we get an increase in these automated scrapers and attacks as of late (some few years); is there better (open-source?) technology that a

• trenchpilgrim 11mo ago
  Using AI you can write a naive scraper in minutes and there's now a market demand for cleaned up and structured data.
• marginalia_nu 11mo ago
  What's missing is effective international law enforcement. This is a legal problem first and foremost. As long as it's as easy as it is to get away with this stuff by just routing the traffic through a Russian or Singaporean node, it's going to keep happening. With international diplomacy going the way it has been, odds of that changing aren't fantastic. The web is really stuck between a rock and a hard place when it comes to this. Proof of work helps website owners, but makes life harder for all discovery tools and search engines. An independent standard for request signing and building some sort of reputation database for verified crawlers could be part of a solution, though that causes problems with websites feeding crawlers different content than users, an does nothing to fix the Sybil attack problem.
  • luckylion 11mo ago
    It's not necessarily going through a Russian or Singaporean node though, on the sites I'm responsible for, AWS, GCP, Azure are in the top 5 for attackers. It's just that they don't care _at all_ about that happening. I don't think you need world-wide law-enforcement, it'll be a big step ahead if you make owners & operators liable. You can limit exposure so nobody gets absolutely ruined, but anyone running wordpress 4.2 and getting their VPS abused for attacks currently has 0 incentive to change anything unless their website goes down. Give them a penalty of a few hundred dollars and suddenly they do. To keep things simple, collect from the hosters, they can then charge their customers, and suddenly they'll be interested in it as well, because they don't want to deal with that. The criminals are not held liable, and neither are their enablers. There's very little chance anything will change that way.
    • mrweasel 11mo ago
      The big cloud provides needs to step up and take responsibility. I understand that it can't be to easy to do, but we really do need a way to contact e.g. AWS and tell them to shut of a costumer. I have no problem with someone scraping our websites, but I care that they don't do so responsibly, slow down when we start responding slower, don't assume that you can just go full throttle, crash our site, wait, and then do it again once we start responding again. You're absolutely right: AWS, GCP, Azure and others, they do not care and especially AWS and GCP are massive enablers.
      • ctoth 11mo ago
        > we really do need a way to contact e.g. AWS and tell them to shut of a costumer. You realize you just described the infrastructure for far worse abuse than a misconfigured scraper, right?
        • mrweasel 11mo ago
          I'm very aware of that, yes. There needs to be a good process, the current situation where AWS simply does not care, or doesn't know also isn't particularly good. One solution could be for victims to notify AWS that a number of specified IP are generating an excessive amount of traffic. An operator could then verify with AWS traffic logs, notify the customer that they are causing issue and only after a failure to respond could the customer be shut down. You're not wrong that abuse would be a massive issue, but I'm on the other side of this and need Amazon to do something, anything.
  • Aurornis 11mo ago
    > What's missing is effective international law enforcement. International law enforcement on the Internet would also subject you to the laws of other countries. It goes both ways. Having to comply with all of the speech laws and restrictions in other countries is not actually something you want.
    • ocdtrekkie 11mo ago
      This is already kind of true with every global website, the idea of a single global internet is one of those fairy tale fantasy things, that maybe happened for a little bit before enough people used it. In many cases it isn't really ideal today.
    • marginalia_nu 11mo ago
      We have historically solved this via treaties. If you want to trade with me, a country that exports software, let's agree to both criminalize software piracy. No reason why this can't be extended to DDoS attacks.
      • beeflet 11mo ago
        I don't want governments to have this level of control over the internet. It seems like you are paving over a technological problem with the way the internet is designed by giving some institution a ton of power over the internet.
        • marginalia_nu 11mo ago
          The alternative to governments stopping misbehavior is every website hiding behind Cloudflare or a small number of competitors, which is a situation that is far more susceptible to abuse than having a law that says you can't DDoS people even if you live in Singapore. It really can not be overstated how unsustainable the status quo is.
          • beeflet 11mo ago
            I think the alternative is to recreate the internet with more p2p friendly infrastructure. BitTorrent does not have this same DDoS problem. Mesh networks are designed with sybil resistance in mind
  • armchairhacker 11mo ago
    I don’t think this can solved legally without compromising anonymity. You can block unrecognized clients and punish the owners of clients that behave badly, but then, for example, an oppressive government can (physically) take over a subversive website and punish everyone who accesses it. Maybe pseudo-anonymity and “punishment” via reputation could work. Then an oppressive government with access to a subversive website (ignoring bad security, coordination with other hijacked sites, etc.) can only poison its clients’ reputations, and (if reputation is tied to sites, who have their own reputations) only temporarily.
    • ajuc 11mo ago
      > but then, for example, an oppressive government can (physically) take over a subversive website and punish everyone who accesses it. Already happens. Oppressive governments already punish people for visiting "wrong" websites. They already censor internet. There are no technological solutions to coordination problems. Ultimately, no matter what you invent, it's politics that will decide how it's used and by whom.
  • BinaryIgor 11mo ago
    Good points; I would definitely vouch for an independent standard for request signing + some kind of decentralized reputation system. With international law enforcement, I think there could be too many political issues for it not become corrupt
• rkagerer 11mo ago
  long-term solution How about a reputation system? Attached to IP address is easiest to grok, but wouldn't work well since addresses lack affinity. OK, so we introduce an identifier that's persistent, and maybe a user can even port it between devices. Now it's bad for privacy. How about a way a client could prove their reputation is above some threshold without leaking any identifying information? And a decentralized way for the rest of the internet to influence their reputation (like when my server feels you're hammering it)? Do anti-DDoS intermediaries like Cloudflare basically catalog a spectrum of reputation at the ASN level (pushing anti-abuse onus to ISP's)? This is basically what happened to email/SMTP, for better or worse :-S.
  • JimDabell 11mo ago
    Reputation plus privacy is probably unsolvable; the whole point of reputation is knowing what people are doing elsewhere. You don’t need reputation, you need persistence. You don’t need to know if they are behaving themselves elsewhere on the Internet as long as you can ban them once and not have them come back. Services need the ability to obtain an identifier that: - Belongs to exactly one real person. - That a person cannot own more than one of. - That is unique per-service. - That cannot be tied to a real-world identity. - That can be used by the person to optionally disclose attributes like whether they are an adult or not. Services generally don’t care about knowing your exact identity but being able to ban a person and not have them simply register a new account, and being able to stop people from registering thousands of accounts would go a long way towards wiping out inauthentic and abusive behaviour. The ability to “reset” your identity is the underlying hole that enables a vast amount of abuse. It’s possible to have persistent, pseudonymous access to the Internet without disclosing real-world identity. Being able to permanently ban abusers from a service would have a hugely positive effect on the Internet.
    • jasonjayr 11mo ago
      A digital "Death penalty" is not a win for society, without considering a fair way to atone for "crimes against your digital identity". It would be way to easy for the current regime (whomever that happens to be) to criminalize random behaviors (Trans People? Atheists? Random nationality?) to ban their identity, and then they can't apply for jobs, get bus fare, purchase anything online, communicate with their lawyers, etc.
• hnthrowaway0315 11mo ago
  I guess it is just because 1) They can, and 2) Everyone wants some data. I think it would be interesting if every website out there starts to push out BS pages just for scrappers. Not sure how much extra cost it's going to take if a website puts up say 50% BS pages that only scrappers can reach, or BS material with extremely small fonts hidden in regular pages that ordinary people cannot see.
  • inerte 11mo ago
    Something like https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/ ?
    • hnthrowaway0315 11mo ago
      Yeah something like this, would be nice if it actually feeds bad data that requires human to double confirm, too. Not something seriously wrong but something subtle, like changing a couple of letters in a name of a country, or randomize the National day. Once a lot of websites start to use it AI might actually get confused, I think? But humans never read these pages so should be largely fine -- unless they are reading AI summaries.
• Vegenoid 11mo ago
  I'm pretty sure it is the commercial demand for data from AI companies. It is certainly the popular conception among sysadmins that it is AI companies who are responsible for the wave of scrapers over the past few years, and I see no compelling alternative.
  • embedding-shape 11mo ago
    > and I see no compelling alternative. Another potential cause: It's way easier for pretty much any person connected to the internet to "create" their own automation software by using LLMs. I could wager even the less smart LLMs could handle "Create a program that checks this website every second for any product updates on all pages" and give enough instructions for the average computer user to be able to run it without thinking or considering much. Multiply this by every person with access to an LLM who wants to "do X with website Y" and you'll get an magnitude increase in traffic across the internet. This been possible since what, 2023 sometime? Not sure if the patterns would line up, but just another guess for the cause(s).
• EGreg 11mo ago
  Why? It’s because of AI. It enables attacks at scale. It enables more people to attack, who previously couldn’t. And so on. It’s very explainable. And somehow, like clockwork, there are always comments to say “there is nothing new, the Internet has always been like this since the 80s”. You know, part of me wants to see AI proliferate into more and more areas, just so these people will finally wake up eventually and understand there is a huge difference when AI does it. When they are relentlessly bombarded with realistic phone calls from random numbers, with friends and family members calling about the latest hoax and deepfake, when their own specific reputation is constantly attacked and destroyed by 1000 cuts not just online but in their own trusted circles, and they have to put out fires and play whack-a-mole with an advanced persistent threat that only grows larger and always comes from new sources, anonymous and not. And this is all before bot swarms that can coordinate and plan long-term, targeting specific communities and individuals. And this is all before humanoid robots and drones proliferate. Just try to fast-forward to when human communities online and offline are constantly infiltrated by bots and drones and sleeper agents, playing nice for a long time and amassing karma / reputation / connections / trust / whatever until finally doing a coordinated attack. Honestly, people just don’t seem to get it until it’s too late. Same with ecosystem destruction — tons of people keep strawmanning it as mere temperature shifts, even while ecosystems around the world get destroyed. Kelp forests. Rainforests. Coral reefs. Fish. Insects. And they’re like “haha global warming by 3 degrees big deal. Temperature has always changed on the planet.” (Sound familiar?) Look, I don’t actually want any of this to happen. But if they could somehow experience the movie It’s a Wonderful Life or meet the Ghost of Christmas Yet to Come, I’d wholeheartedly want every denier to have that experience. (In fact, a dedicated attacker can already give them a taste of this with current technology. I am sure it will become a decentralized service soon :-( )
  • hshdhdhj4444 11mo ago
    Our tech overlords understand AI, especially any form of AGI, will basically be the end of humanity. That’s why they’re entirely focused on being the first and amassing as much wealth in the meanwhile, giving up on any sort of consideration whether they’re doing good for people or not.