12 ms·

The Web Is Broken – Botnet Part 2

411 points by todsacerdoti 2y ago

kastden 2y ago
Are there any lists with known c&c servers for these services that can be added to Pihole/etc?
- udev4096 2y ago
  You can use one of the list from here: https://github.com/hagezi/dns-blocklists https://github.com/hagezi/dns-blocklists
api 2y ago
This is nasty in other ways too. What happens when someone uses these B2P residential proxies to commit crimes that get traced back to you? Anything incorporating anything like this is malware.
- reconnecting 2y ago
  Many years ago cybercriminals used to hack computers to use them as residential proxies, now they purchase them online as a service. In most cases they are used for conducting real financial crimes, but the police investigators are also aware that there is a very low chance that sophisticated fraud is committed directly from a residential IP address.
Liftyee 2y ago
I don't know if I should be surprised about what's described in this article, given the current state of the world. Certainly I didn't know about it before, and I agree with the article's conclusion. Personally, I think the "network sharing" software bundled with apps should fall into the category of potentially unwanted applications along with adware and spyware. All of the above "tag along" with something the user DID want to install, and quietly misuse the user's resources. Proxies like this definitely have an impact for metered/slow connections - I'm tempted to start Wireshark'ing my devices now to look for suspicious activity. There should be a public repository of apps known to have these shady behaviours. Having done some light web scraping for archival/automation before, it's a pity that it'll become collateral damage in the anti-AI-botfarm fight.
- zzo38computer 2y ago
  I agree, this should be called spyware, and malware. There are many other kind of software that also should, but netcat and ncat (probably) aren't malware.
- deleted 2y ago
  [deleted]
- akoboldfrying 2y ago
  I agree, but the harm done to the users is only one part of the total harm. I think it's quite plausible that many users wouldn't mind some small amount of their bandwidth being used, if it meant being able to use a handy browser extension that they would otherwise have to pay actual dollars for -- but the harm done to those running the servers remains.
arewethereyeta 2y ago
I have some success in catching most of them at https://visitorquery.com https://visitorquery.com
- lq9AJ8yrfs 2y ago
  I went to your website. Is the premise that users should not be allowed to use vpns in order to participate in ecommerce?
  - arewethereyeta 2y ago
    Nobody said that, it's your choice to take whatever action fits your scenario. I have clients where VPNs are blocked yes, it depends on the industry, fraud rate, chargeback rates etc.
- ivas 2y ago
  Checked my connection via VPN by Google/Cloudflare WARP: "Proxy/VPN not detected"
  - arewethereyeta 2y ago
    Could be, I don't claim 100% success rate. I'll have a look at one of those and see why I missed it. Thank you for letting me know.
    - nickphx 2y ago
      measuring latency between different endpoints? I see the webrtc turn relay request..
- deleted 2y ago
  [deleted]
karmanGO 2y ago
Has anyone tried to compile a list of software that uses these libraries? It would be great to know what apps to avoid
- arewethereyeta 2y ago
  No but here's the thing. Being in the industry for many years I know they are required to mention it in the TOS when using the SDKs. A crawler pulling app TOSs and parsing them could be a thing. List or not, it won't be too useful outside this tech community.
- deleted 2y ago
  [deleted]
- mzajc 2y ago
  In the case of Android, εxodus has one[1], though I couldn't find the malware library listed in TFA. Aurora Store[2], a FOSS Google Play Store client, also integrates it. [1] https://reports.exodus-privacy.eu.org/en/trackers/ https://reports.exodus-privacy.eu.org/en/trackers/ [2] https://f-droid.org/packages/com.aurora.store/ https://f-droid.org/packages/com.aurora.store/
  - takluyver 2y ago
    That seems to be looking at tracking and data collection libraries, though, for things like advertising and crash reporting. I don't see any mention of the kind of 'network sharing' libraries that this article is about. Have I missed it?
- lelanthran 2y ago
  > Has anyone tried to compile a list of software that uses these libraries? It would be great to know what apps to avoid I wouldn't mind reading a comprehensive report on SOTA with regard to bot-blocking. Sure, there's Anubis (although someone elsethread called it a half-measure, and I'd like to know why), there's captcha's, there's relying on a monopoly (cloudflare, etc) who probably also wants to run their own bots at some point, but what else is there?
- 2y ago
amiga-workbench 2y ago
What is the point of app stores holding up releases for review if they don't even catch obvious malware like this?
- SoftTalker 2y ago
  Money
- _Algernon_ 2y ago
  They pretend to do a review to justify their 30% cartel tax.
  - klabb3 2y ago
    Oh no, they review thoroughly, to make sure you don’t try to avoid the tax.
- politelemon 2y ago
  Their marketing tells you it's for protection. What they fail to omit is it's for their revenue protection - observe that as long as you do not threaten their revenue models, or the revenue models of their partners, you are allowed through. It has never been about the users or developers.
- deleted 2y ago
  [deleted]
- charcircuit 2y ago
  The definition of malware is fuzzy.
- wyck 2y ago
  This isn't obvious, 99% of apps make multiple calls to multiple services, and these SDK's are embedded into the app. How can you tell whats legit outbound/inbound? Doing a fingerprint search for the worst culprits might help catch some, but it would likely be a game of cat and mouse.
vlan121 2y ago
when the shit hits the fan, this seems like the product.
ChrisMarshallNY 2y ago
> So if you as an app developer include such a 3rd party SDK in your app to make some money — you are part of the problem and I think you should be held responsible for delivering malware to your users, making them botnet members. I suspect that this goes for many different SDKs. Personally, I am really, really sick of hearing "That's a solved problem!", whenever I mention that I tend to "roll my own," as opposed to including some dependency, recommended by some jargon-addled dependency addict. Bad actors love the dependency addiction of modern developers, and have learned to set some pretty clever traps.
- duskwuff 2y ago
  That may be true but I think you're missing the point here. The "network sharing" behavior in these SDKs is the sole purpose of the SDK. It isn't being included as a surprise along with some other desirable behavior. What needs to stop is developers including these SDKs as a secondary revenue source in free or ad-supported apps.
  - ChrisMarshallNY 2y ago
    > I think you're missing the point here Doubt it. This is just one -of many- carrots that are used to entice developers to include dodgy software into their apps. The problem is a lot bigger than these libraries. It's an endemic cultural issue. Much more difficult to quantify or fix.
- sixtyj 2y ago
  Malware, botnets… it is very similar. And people including developers are - in 80 per cent - eagier to make money, because… Is greed good? No, it isn’t. It is a plague.
  - II2II 2y ago
    You're a developer who devoted time to develop a piece of software. You discover that you are not generating any income from it: few people can even find it in the sea of similar apps, few of those are willing to pay for it, and those who are willing to pay for it are not willing to pay much. To make matters worse, you're going to lose a cut of what is paid to the middlemen who facilitate the transaction. Is that greed? I can find many reasons to be critical of that developer, things like creating a product for a market segment that is saturated, and likely doing so because it is low hanging fruit (both conceptually and in terms of complexity). I can be critical of their moral judgement for how they decided to generate income from their poor business judgment. But I don't thinks it's right to automatically label them as greedy. They may be greedy, but they may also be trying to generate income from their work.
jonplackett 2y ago
How is this not just illegal? Surely there’s something in GDPR that makes this not allowed.
- Retr0id 2y ago
  iiuc, they do actually ask the user for permission
  - fc417fc802 2y ago
    Which is ironic considering that I strongly disagree with one of the primary walled garden justifications, used particularly in the case of Apple, which amounts to "the end user is too stupid to decide on his own". Unfortunately, even if I disagree with it as a guiding principle sometimes that statement proves true.
    - klabb3 2y ago
      It’s not about stupidity, but practicality. People can’t give informed consent for 100 ToS for different companies, and keep those up to date. That’s why there are laws.
  - SoftTalker 2y ago
    No doubt in a dense wall of text that the user must accept to use the application, or worse is deemed to have accepted by using the application at all.
zahlman 2y ago
> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner. I imagine that e.g. Youtube would be happy to agree with this. Not that it would turn them against AI generally.
- BlueTemplar 2y ago
  Yeah, also this means the death of archival efforts like the Internet Archive.
  - jeroenhd 2y ago
    Welcome scrapers (IA, maybe Google and Bing) can publish their IP addresses and get whitelisted. Websites that want to prevent being on the Internet Archive can pretty much just ask for their website to be excluded (even retroactively). [Cloudflare](https://developers.cloudflare.com/cache/troubleshooting/always-online/ https://developers.cloudflare.com/cache/troubleshooting/alwa...) tags the internet archive as operating from 207.241.224.0/20 and 208.70.24.0/21 so disabling the bot-prevention framework on connections from there should be enough.
    - trinsic2 2y ago
      This sounds like it would be a good idea. Create a whitelist of IPs and block the rest.
    - realusername 2y ago
      That's basically asking to close the market in favor of the current actors. New actors have the right to emerge.
      - 0dayz 2y ago
        No they don't. There's no rule that you have to let anyone in who claims to be a web crawler.
pton_xd 2y ago
I thought the closed-garden app stores were supposed to protect us from this sort of thing?
- whstl 2y ago
  Once again this demonstrate that closed gardens only benefit the owners of the garden, and not the users. What good is all the app vetting and sandbox protection in iOS (dunno about Android) if it doesn't really protect me from those crappy apps...
  - 20after4 2y ago
    At the very least, Apple should require conspicuous disclosure of this kind of behavior that isn't just hidden in the TOS.
  - BlueTemplar 2y ago
    Also my reaction when the call is for Google, Apple, Microsoft to fix this : DDOS being illegal, shouldn't the first reaction instead to be to contact law enforcement ? If you treat platforms like they are all-powerful, then that's what they are likely to become...
  - musicale 2y ago
    Sandboxing means you can limit network access. For example, on Android you can disallow wi-fi and cellular access (not sure about bluetooth) on a per-app basis. Network access settings should really be more granular for apps that have a legitimate need. App store disclosure labels should also add network usage disclosure.
- 20after4 2y ago
  That's what they want you to think.
- kibwen 2y ago
  If you find yourself in a walled garden, understand that you're the crop being grown and harvested.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
jt2190 2y ago
I’m really struggling to understand how this is different than malware we’ve had forever. Can someone explain what’s novel about this?
- desertmonad 2y ago
  That its not being treated like malware.
  - jt2190 2y ago
    In the sense that people are voluntarily installing and running this malware on their computers, rather than being tricked into running it? Is that the only difference?
    - int_19h 2y ago
      They are still tricked into running it, since it's normally not an advertised "feature" of any app that uses such SDKs.
- downrightmike 2y ago
  I think it is funny that the mobile OS is trying to be as secure as possible, but then they allow this to run on top
rsedgwick 2y ago
I think tech can still be beautiful in a less grandiose and "omniparadisical" way than people used to dream of. "A wide open internet, free as in speech this, free as in beer that, open source wonders, open gardens..." Well, there are a lot of incentives that fight that, and game theory wins. Maybe we download software dependencies from our friends, the ones we actually trust. Maybe we write more code ourselves--more homesteading families that raise their own chickens, jar their own pickled carrots, and code their own networking utilities. Maybe we operate on servers we own, or our friends own, and we don't get blindsided by news that the platforms are selling our data and scraping it for training. Maybe it's less convenient and more expensive and onerous. Do good things require hard work? Or did we expect everyone to ignore incentives forever while the trillion-dollar hyperscalers fought for an open and noble internet and then wrapped it in affordable consumer products to our delight? It reminds me of the post here a few weeks ago about how Netflix used to be good and "maybe I want a faster horse" - we want things to be built for us, easily, cheaply, conveniently, by companies, and we want those companies not to succumb to enshittification - but somehow when the companies just follow the game theory and turn everything into a TikToky neural-networks-maximizing-engagement-infinite-scroll-experience, it's their fault, and not ours for going with the easy path while hoping the corporations would not take the easy path.
reconnecting 2y ago
Residential IP proxies have some weaknesses. One is that they ofter change IP addresses during a single web session. Second, if IP come from the same proxies provider, they are often concentrated within a sing ASN, making them easier to detect. We are working on an open‑source fraud prevention platform [1], and detecting fake users coming from residential proxies is one of its use cases. [1] https://www.github.com/tirrenotechnologies/tirreno https://www.github.com/tirrenotechnologies/tirreno
- gbcfghhjj 2y ago
  At least here in the US most residential ISPs have long leases and change infrequently, weeks or months. Trying to understand your product, where is it intended to sit in a network? Is it a standalone tool that you use to identify these IPs and feed into something else for blockage or is it intended to be integrated into your existing site or is it supposed to proxy all your web traffic? The reason I ask is it has fairly heavyweight install requirements and Apache and PHP are kind of old school at this point, especially for new projects and companies. It's not what they would commonly be using for their site.
  - reconnecting 2y ago
    Indeed, if it's a real user from a residential IP address, in most cases it will be the same network. However, if it's a proxy from residential IPs, there could be 10 requests from one network, the 11th request from a second network, and the 12th request back from the same network. This is a red flag. Thank you for your question. tirreno is a standalone app that needs to receive API events from your main web application. It can work perfectly with 512GB Postgres RAM or even lower, however, in most cases we're talking about millions of events that request resources. It's much easier to write a stable application without dependencies based on mature technologies. tirreno is fairly 'boring software'.
    - sroussey 2y ago
      My phone will be on the home network until I walk out of the house and then it will change networks. This should not be a red flag.
at0mic22 2y ago
Strange the HolaVPN e.g. Brightdata is not mentioned. They've been using user hosts for those purposes for decades, and also selling proxies en masse. Fun fact they don't have any servers for the VPN. All the VPN traffic is routed through ... other users!
- Klonoar 2y ago
  Is it really strange if the logo is right there in the article?
- arewethereyeta 2y ago
  They are even the first to do it and the most litigious of all. Trying to push patents on everything possible, even on water if they can.
- andelink 2y ago
  Hola is mentioned in the authors prior post on this topic, linked to at the top of TFA: https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/ https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
armchairhacker 2y ago
> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner. Why jump to that conclusion? If a scraper clearly advertises itself, follows robots.txt, and has reasonable backoff, it's not abusive. You can easily block such a scraper, but then you're encouraging stealth scrapers because they're still getting your data. I'd block the scrapers that try to hide and waste compute, but deliberately allow those that don't. And maybe provide a sitemap and API (which besides being easier to scrape, can be faster to handle).
panstromek 2y ago
I'd expect this to be against app store and google play rules, they are very picky.
Pesthuf 2y ago
We need a list of apps that include these libraries and any malware scanner - including Windows Defender, Play Protect and whatever Apple calls theirs - need to put infected applications into quarantine immediately. Just because it's not directly causing damage to the device running the malware is running on, that doesn't mean it's not malware.
- philippta 2y ago
  Apps should be required to ask for permission to access specific domains. Similar to the tracking protection, Apple introduced a while ago. Not sure how this could work for browsers, but the other 99% of apps I have on my phone should work fine with just a single permitted domain.
  - jay_kyburz 2y ago
    Oh, that's an interesting idea. A local DNS where I have to add every entry. A white list rather than Australia's national blacklist.
  - snackernews 2y ago
    My iPhone occasionally displays an interrupt screen to remind me that my weather app has been accessing my location in the background and to confirm continued access. It should also do something similar for apps making chatty background requests to domains not specified at app review time. The legitimate use cases for that behaviour are few.
  - zzo38computer 2y ago
    I think capability based security with proxy capabilities is the way to do it, and this would make it possible for the proxy capability to intercept the request and ask permission, or to do whatever else you want it to do (e.g. redirections, log any accesses, automatically allow or disallow based on a file, use or ignore the DNS cache, etc). The system may have some such functions built in, and asking permission might be a reasonable thing to include by default.
    - 2y ago
proxy_err 2y ago
Its a fair point but very dynamic to sort out. This needs a full research team to figure out. Or you know.. all of us combined!! It is definitely a problem. TINFOIL: Sometimes I always wondered if Azure or AWS used bots to push site traffic hits to generate money... they know you are hosted with them.. They have your info.. Send out bots to drive micro accumulation. Slow boil..
- luckylion 2y ago
  I think that's mostly that they don't care about having malicious bots on their networks as long as they pay. GCE is rare in my experience. Most bots I see are on AWS. The DDOS-adjacent hyper aggressive bots that try random URLs and scan for exploits tend to be on Azure or use VPNs. AWS is bad when you report malicious traffic. Azure has been completely unresponsive and didn't react, even for C&C servers.
aucisson_masque 2y ago
It's interesting but so far there is no definitive proof it's happening. People are jumping to conclusions a bit fast over here, yes technically it's possible but this kind of behavior would be relatively easy to spot because the app would have to make direct connections to the website it wants to scrap. Your calculator app for instance connecting to CNN.com ... iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. Android by Google doesn't have such a useful feature of course, but you can run third party firewall like pcapdroid, which I recommend highly. Macos (little snitch). Windows (fort firewall). Not everyone run these app obviously, only the most nerdy like myself but we're also the kind of people who would report on app using our device to make, what is in fact, a zombie or bot network. I'm not saying it's necessarily false but imo it remains a theory until proven otherwise.
- CharlesW 2y ago
  Botnets as a Service are absolutely happening, but as you allude to, the scope of the abuse is very different on iOS than, say, Windows.
- abaymado 2y ago
  > iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. How often is the average calculator app user checking there Privacy Report? My guess, not many!
  - gruez 2y ago
    All it takes is one person to find out and raise the alarm. The average user doesn't read the source code behind openssl or whatever either, that doesn't mean there's no gains in open sourcing it.
    - dewey 2y ago
      The average user is also not reading these raised “alarms”. And if an app has a bad name, another one will show up with a different name on the same day.
badmonster 2y ago
do you think there’s a realistic path forward for better transparency or detection—maybe at the OS level or through network-level anomaly detection?
yungporko 2y ago
it's funny, i've never heard of or thought about the possibility of this happening but actually in hindsight it seems almost too obvious to not be a thing.
jeroenhd 2y ago
> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth AKA "why do Cloudflare and Google make me fill out these CAPTCHAs all day" I don't know why Play Protect/MS Defender/whatever Apple has for antivirus don't classify apps that embed such malware as such. It's ridiculous that this is allowed to go on when detection is so easy. I don't know a more obvious example of a trojan than an SDK library making a user's device part of a botnet.
- deleted 2y ago
  [deleted]
- dx4100 2y ago
  Cloudflare and Google use CAPTCHAs to sell web scrapers? I don't get your point. I was under the impression the data is used to train models.
  - cuu508 2y ago
    Trojans in your mobile apps ruin your IP's reputation which comes back to you in the form of frequent, annoying CAPTCHAs.
  - aloha2436 2y ago
    The implication is that the users that are being constantly presented with CAPTCHAs are experiencing that because they are unwittingly proxying scrapers through their devices via malicious apps they've installed.
    - pentae 2y ago
      .. or that other people on their network/Shared public IP have installed
      - evgpbfhnr 2y ago
        or just that they don't run windows/mac OS with chome like everyone else and it's "suspicious". I get cloudflare capchas all the time with firefox on linux... (and I'm pretty sure there's no such app in my home network!)
panny 2y ago
>Apple, Microsoft and Google should act. Do nothing, win. They are the primary benefactors buying this data since they are the largest AI players.
neilv 2y ago
Couldn't Apple and Google (and, to a lesser extent, Microsoft) pretty easily shut down almost all the apps that steal bandwidth?
greesil 2y ago
How would I know if an app on my device was doing this?
- wyck 2y ago
  Install a network monitor or go even deeper and sniff packets.
  - greesil 2y ago
    I feel like this could be automated. Spin up a virtual device on a monitored network. Install one app, click on some stuff for awhile, uninstall and move onto the next. If the app reaches out to a lot of random sites then flag it Google could do this. I'm sure Apple could as well. Third parties could for a small set of apps
    - jeroenhd 2y ago
      This is being done by a couple of SDKs, it'd be much easier to just find and flag those SDK files. Finding apps becomes a matter of a single pass scan over the application contents rather than attempting to bypass the VM detection methods malware is packed full of.
matheusmoreira 2y ago
"Peer-to-business network"! Amazing. uBlock Origin gets rid of this, right?
theteapot 2y ago
Are ad blockers like AdBlock, uBlock effective against these?
- areyourllySorry 2y ago
  i don't believe extensions can modify other extensions
__MatrixMan__ 2y ago
The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the original source. Having other parties interested in your data should make your life easier (because other parties will host it for you), not harder (because now you need to work extra hard to host it for them).
- Timwi 2y ago
  Are there any systems like that, even if experimental?
  - jevogel 2y ago
    IPFS
    - alakra 2y ago
      I had high hopes for IPFS, but even it has vectors for abuse. See https://arxiv.org/abs/1905.11880 https://arxiv.org/abs/1905.11880 [Hydras and IPFS: A Decentralised Playground for Malware]
      - __MatrixMan__ 2y ago
        Can you point me at what you mean? I'm not immediately finding something that indicates that it is not fit for this use case. The fact that bad actors use it to resist those who want to shut them down is, if anything, an endorsement of its durability. There's a bit of overlap between resisting the AI scrapers and resisting the FBI. You can either have a single point of control and a single point of failure, or you can have neither. If you're after something that's both reliable and reliably censorable--I don't think that's in the cards. That's not to say that it is a ready replacement for the web as we know it. If you have hash-linked everything then you wind up with problems trying to link things together, for instance. Once two pages exist, you can't after-the-fact create a link between them because if you update them to contain that link then their hashes change so now you have to propagate the new hash to people. This makes it difficult to do things like have a comments section at the bottom of a blog post. So you've got to handle metadata like that in some kind of extra layer--a layer which isn't hash linked and which might be susceptible to all the same problems that our current web is--and then the browser can build the page from immutable pieces, but the assembly itself ends up being dynamic (and likely sensitive to the users preference, e.g. dark mode as a browser thing not a page thing). But I still think you could move maybe 95% of the data into an immutable hash-linked world (think of these as nodes in a graph), the remaining 5% just being tuples of hashes and pubic keys indicating which pages are trusted by which users, which ought to be linked to which others, which are known to be the inputs and output of various functions, and you know... structure stuff (these are our graph's edges). The edges, being smaller, might be subject to different constraints than the web as we know it. I wouldn't propose that we go all the way to a blockchain where every device caches every edge, but it might be feasible for my devices to store all of the edges for the 5% of the web I care about, and your devices to store the edges for the 5% that you care about... the nodes only being summoned when we actually want to view them. The edges can be updated when our devices contact other devices (based on trust, like you know that device's owner personally) and ask "hey, what's new?" I've sort of been freestyling on this idea in isolation, probably there's already some projects that scratch this itch. A while back I made a note to check out https://ceramic.network/ https://ceramic.network/ in this capacity, but I haven't gotten down to trying it out yet.
156287745637 2y ago
AI scrapers and "sneaker bots" are just the tip of the iceberg. Why are all these entities concentrated and metastasizing from just a few superhubs? Why do they look, smell and behave like state-level machinery? If you've researched you'll know exactly what I'm talking about. Unless complicit, tech leaders (Apple Google Microsoft) have a duty to respond swiftly and decisively. This has been going on far too long.
- deleted 2y ago
  [deleted]
_ink_ 2y ago
How can I detect such behaviour on my devices / in my home network?
gpi 2y ago
"Infatica is partnered with Bitdefender, a global leader in cybersecurity, to protect our SDK users from malicious web traffic and content, including infected URLs, untrusted web pages, fraudulent and phishing links, and more." That's not good.
- deleted 2y ago
  [deleted]
Quarrel 2y ago
FWIW, Trend Micro wrote up a decent piece on this space in 2023. It is still a pretty good lay-of-the-land. https://www.trendmicro.com/vinfo/us/security/news/vulnerabilities-and-exploits/a-closer-exploration-of-residential-proxies-and-captcha-breaking-services https://www.trendmicro.com/vinfo/us/security/news/vulnerabil...
hinkley 2y ago
When the enshitification initially hit the fan, I had little flashbacks of Phil Zimmerman talking about Web of Trust and amusing myself thinking maybe we need humans proving they're humans to other humans so we know we aren't arguing with LLMs on the internet or letting them scan our websites. But it just doesn't scale to internet size so I'm fucked if I know how we should fix it. We all have that cousin or dude in our highschool class who would do anything for a bit of money and introducing his 'friend' Paul who is in fact a bot whose owner paid for the lie. And not like enough money to make it a moral dilemma, just drinking money or enough for a new video game. So once you get past about 10,000 people you're pretty much back where we are right now.
- akoboldfrying 2y ago
  I think it should be possible to build something that generalises the idea of Web of Trust so that it's more flexible, and less prone to catastrophic breakdown past some scaling limit. Binary "X trusts Y" statements, plus transitive closure, can lead to long trust paths that we probably shouldn't actually trust the endpoints of. Could we not instead assign probabilities like "X trusts Y 95%", multiply probabilities along paths starting from our own identity, and take the max at each vertex? We could then decide whether to finally trust some Z if its percentage is more than some threshold T%. (Other ways of combining in-edges may be more suitable than max(); it's just a simple and conservative choice.) Perhaps a variant of backprop could be used to automatically update either (a) all or (b) just our own weights, given new information ("V has been discovered to be fraudulent").
  - hinkley 2y ago
    True. Perhaps a collective vote past 2 degrees of freedom out where multiple parties need to vouch for the same person before you believe they aren't a bot. Then you're using the exponential number of people to provide diminishing weight instead of increasing likelihood of malfeasance.
    - nottorp 2y ago
      But do we need an infinite and global web of trust? How about restricting them to everyone-knows-everyone sized groups, of like a couple hundred people? One can be a member of multiple groups so you're not actually limited. But the groups will be small enough to self regulate.
hubraumhugo 2y ago
We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old concept that wastes millions of human hours and billions in infra costs [0]. How can enable beneficial automation while protecting against abusive AI crawlers? [0] https://arxiv.org/abs/2311.10911 https://arxiv.org/abs/2311.10911
- udev4096 2y ago
  Blame the "AI" companies for that. I am glad the small web is pushing hard against these scrapers, with the rise of Anubis as a starting point
  - lelanthran 2y ago
    > Blame the "AI" companies for that. I am glad the small web is pushing hard towards these scrapers, with the rise of Anubis as a starting point Did you mean "against"?
    - udev4096 2y ago
      Corrected, thanks
- eastbound 2y ago
  But people don’t interact with your website anymore; they as an AI. So the AI crawler is a real user. I say we ask Google Analytics to count an AI crawler as a real view. Let’s see who’s most popular.
- CalRobert 2y ago
  I hate this but I suspect a login-only deanonymised web (made simple with chrome and WEI!) is the future. Firefox users can go to hell.
y42 2y ago
Let me get this straight: we want computers knowing everything, to solve current and future problems, but we don't want to give them access to our knowledge?
- drawfloat 2y ago
  Most people don’t want computers to know everything - ask the average person if they want more or less of their lives recorded and stored.
  - y42 2y ago
    > Most people don’t want computers to know everything. That may well be true. But how many of those people are specifically against AI companies scraping the web? That’s not really an argument—it’s an assumption based on personal perception. > Ask the average person if they want more or less of their lives recorded and stored. What exactly is the "average person"? Also, I’ll admit my earlier claim was a bit exaggerated. But let’s be clear: this isn’t about recording personal data—it’s about collecting and structuring knowledge. And beyond that: companies have been scraping the web for years. They still are. And they’re gathering far more personal data for online marketing, tracking, profiling—whatever the reason—and the so-called "average person" hasn’t raised much of a finger. People remain glued to platforms, willingly sharing their personal lives. And what do they get in return? Doomscrolling and five-second video clips.
- chairmansteve 2y ago
  Not sure we do.
- 3np 2y ago
  I don't want your computer to know everything about me, in fact.
  - y42 2y ago
    That’s not what I said. Let me rephrase it: Do you want computers to help us solve medical or scientific problems in order to improve human life?
deleted 2y ago
[deleted]
topak3000 2y ago
[flagged]
- areyourllySorry 2y ago
  the next time you copy paste an ad remove the trailing quotation mark
- nothrabannosir 2y ago
  @dang can this entire account be banned please
areyourllySorry 2y ago
further reading https://krebsonsecurity.com/?s=infatica https://krebsonsecurity.com/?s=infatica https://krebsonsecurity.com/tag/residential-proxies/ https://krebsonsecurity.com/tag/residential-proxies/ https://spur.us/blog/ https://spur.us/blog/ https://bright-sdk.com/ https://bright-sdk.com/ <- way bigger than infatica
dspillett 2y ago
> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth. This is yet another reason why we need to be wary of popular apps, add-ons, extensions, and so forth changing hands, by legitimate sale or more nefarious methods. Initially innocent utilities can be quickly coopted into being parts of this sort of scheme.
aorth 2y ago
In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all of Brazil (sorry). A few days later this same web server was on fire again. I performed the same analysis on IPs and found a similar number of unique addresses, but spread across Turkey, Russia, Argentina, Algeria and many more countries. What is going on?! Eventually I think I found a pattern to identify the requests, in that they were using ancient Chrome user agents. Chrome 40, 50, 60 and up to 90, all released 5 to 15 years ago. Then, just before I could implement a block based on these user agents, the traffic stopped. In both cases the traffic from datacenter networks was limited because I already rate limit a few dozen of the larger ones. Sysadmin life...
- rollcat 2y ago
  Try Anubis: <https://anubis.techaro.lol https://anubis.techaro.lol> It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?
  - marginalia_nu 2y ago
    Anubis is a good choice because it whitelists legitimate and well behaved crawlers based on IP + user-agent. Cloudflare works as well in that regard but then you're MITM:ing all your visitors.
  - Imustaskforhelp 2y ago
    Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis. Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and it actually made his git server down I guess??) I didn't expect it to blow up to ... UN.
jgalt212 2y ago
I blame the VCs. They don't stop, and implicitly encourage, website-crushing scrapers among their funded ventures. It's not a crime if we do it with an app https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading
reincoder 2y ago
I work for IPinfo (a commercial service). We offer a residential proxy detection service, but it costs money. If you are being bombarded by suspicious IP addresses, please consider using our free service and blocking IP addresses by ASN or Country. I think ASN is a common parameter for malicious IP addresses. If you do not have time to explore our services/tools (it is mostly just our CLI: https://github.com/ipinfo/cli https://github.com/ipinfo/cli), simply paste the IP addresses (or logs) in plain text, send it to me and I will let you know the ASNs and corresponding ranges to block.
- throwaway74663 2y ago
  Blocking countries is such a poorly disguised form of racism. Funny how it's always the brown / yellow people countries that get blocked, and never the US, despite it being one of the leading nations in malicious traffic.
  - reincoder 2y ago
    Oh, absolutely not — I have to respectfully but strongly disagree with that sentiment. In cybersecurity, decisions must be guided by objective data, not assumptions or biases. When you’re facing abuse, you analyze the IPs involved and enrich them with context — ASN, country, city, whether it’s VPN, hosting, residential, etc. That gives you the information you need to make calculated decisions: Should you block a subnet? Rate-limit it? CAPTCHA-challenge it? Here’s a small snapshot from my own SSH honeypot: Summary of 1,413 attempts - Hosting IPs: 981 (69%) - VPNs: 35 - Top ASNs: - AS204428 (SS-Net): 152 - AS136052 (PT Cloud Hosting Indonesia): 83 - AS14061 (DigitalOcean): 76 - Top Countries: - Romania: 238 (16.8%) - United States: 150 (10.6%) - China: 134 (9.5%) - Indonesia: 115 (8.1%) One single /24 from Romania accounts for over 10% of the attacks. That’s not about nationality or ethnicity — it's about IP space abuse from a specific network. If a network or country consistently shows high levels of hostile traffic and your risk tolerance justifies it, blocking or throttling it may be entirely reasonable. Security teams don’t block based on "where people come from" — they block based on where the attacks are coming from. We even offer tools to help people explore and understand these patterns better. But if someone doesn’t have the time or resources to do that, I'm more than happy to assist by analyzing logs and suggesting reasonable mitigations.

411 points by todsacerdoti 2y ago

kastden 2y ago

Are there any lists with known c&c servers for these services that can be added to Pihole/etc?

udev4096 2y ago
You can use one of the list from here: https://github.com/hagezi/dns-blocklists https://github.com/hagezi/dns-blocklists

api 2y ago

This is nasty in other ways too. What happens when someone uses these B2P residential proxies to commit crimes that get traced back to you? Anything incorporating anything like this is malware.

reconnecting 2y ago
Many years ago cybercriminals used to hack computers to use them as residential proxies, now they purchase them online as a service. In most cases they are used for conducting real financial crimes, but the police investigators are also aware that there is a very low chance that sophisticated fraud is committed directly from a residential IP address.

Liftyee 2y ago

I don't know if I should be surprised about what's described in this article, given the current state of the world. Certainly I didn't know about it before, and I agree with the article's conclusion. Personally, I think the "network sharing" software bundled with apps should fall into the category of potentially unwanted applications along with adware and spyware. All of the above "tag along" with something the user DID want to install, and quietly misuse the user's resources. Proxies like this definitely have an impact for metered/slow connections - I'm tempted to start Wireshark'ing my devices now to look for suspicious activity. There should be a public repository of apps known to have these shady behaviours. Having done some light web scraping for archival/automation before, it's a pity that it'll become collateral damage in the anti-AI-botfarm fight.

zzo38computer 2y ago
I agree, this should be called spyware, and malware. There are many other kind of software that also should, but netcat and ncat (probably) aren't malware.
deleted 2y ago
[deleted]
akoboldfrying 2y ago
I agree, but the harm done to the users is only one part of the total harm. I think it's quite plausible that many users wouldn't mind some small amount of their bandwidth being used, if it meant being able to use a handy browser extension that they would otherwise have to pay actual dollars for -- but the harm done to those running the servers remains.

arewethereyeta 2y ago

I have some success in catching most of them at https://visitorquery.com https://visitorquery.com

lq9AJ8yrfs 2y ago
I went to your website. Is the premise that users should not be allowed to use vpns in order to participate in ecommerce?
- arewethereyeta 2y ago
  Nobody said that, it's your choice to take whatever action fits your scenario. I have clients where VPNs are blocked yes, it depends on the industry, fraud rate, chargeback rates etc.
ivas 2y ago
Checked my connection via VPN by Google/Cloudflare WARP: "Proxy/VPN not detected"
- arewethereyeta 2y ago
  Could be, I don't claim 100% success rate. I'll have a look at one of those and see why I missed it. Thank you for letting me know.
  - nickphx 2y ago
    measuring latency between different endpoints? I see the webrtc turn relay request..
deleted 2y ago
[deleted]

karmanGO 2y ago

Has anyone tried to compile a list of software that uses these libraries? It would be great to know what apps to avoid

arewethereyeta 2y ago
No but here's the thing. Being in the industry for many years I know they are required to mention it in the TOS when using the SDKs. A crawler pulling app TOSs and parsing them could be a thing. List or not, it won't be too useful outside this tech community.
deleted 2y ago
[deleted]
mzajc 2y ago
In the case of Android, εxodus has one[1], though I couldn't find the malware library listed in TFA. Aurora Store[2], a FOSS Google Play Store client, also integrates it. [1] https://reports.exodus-privacy.eu.org/en/trackers/ https://reports.exodus-privacy.eu.org/en/trackers/ [2] https://f-droid.org/packages/com.aurora.store/ https://f-droid.org/packages/com.aurora.store/
- takluyver 2y ago
  That seems to be looking at tracking and data collection libraries, though, for things like advertising and crash reporting. I don't see any mention of the kind of 'network sharing' libraries that this article is about. Have I missed it?
lelanthran 2y ago
> Has anyone tried to compile a list of software that uses these libraries? It would be great to know what apps to avoid I wouldn't mind reading a comprehensive report on SOTA with regard to bot-blocking. Sure, there's Anubis (although someone elsethread called it a half-measure, and I'd like to know why), there's captcha's, there's relying on a monopoly (cloudflare, etc) who probably also wants to run their own bots at some point, but what else is there?
2y ago

amiga-workbench 2y ago

What is the point of app stores holding up releases for review if they don't even catch obvious malware like this?

SoftTalker 2y ago
Money
_Algernon_ 2y ago
They pretend to do a review to justify their 30% cartel tax.
- klabb3 2y ago
  Oh no, they review thoroughly, to make sure you don’t try to avoid the tax.
politelemon 2y ago
Their marketing tells you it's for protection. What they fail to omit is it's for their revenue protection - observe that as long as you do not threaten their revenue models, or the revenue models of their partners, you are allowed through. It has never been about the users or developers.
deleted 2y ago
[deleted]
charcircuit 2y ago
The definition of malware is fuzzy.
wyck 2y ago
This isn't obvious, 99% of apps make multiple calls to multiple services, and these SDK's are embedded into the app. How can you tell whats legit outbound/inbound? Doing a fingerprint search for the worst culprits might help catch some, but it would likely be a game of cat and mouse.

vlan121 2y ago

when the shit hits the fan, this seems like the product.

ChrisMarshallNY 2y ago

> So if you as an app developer include such a 3rd party SDK in your app to make some money — you are part of the problem and I think you should be held responsible for delivering malware to your users, making them botnet members. I suspect that this goes for many different SDKs. Personally, I am really, really sick of hearing "That's a solved problem!", whenever I mention that I tend to "roll my own," as opposed to including some dependency, recommended by some jargon-addled dependency addict. Bad actors love the dependency addiction of modern developers, and have learned to set some pretty clever traps.

duskwuff 2y ago
That may be true but I think you're missing the point here. The "network sharing" behavior in these SDKs is the sole purpose of the SDK. It isn't being included as a surprise along with some other desirable behavior. What needs to stop is developers including these SDKs as a secondary revenue source in free or ad-supported apps.
- ChrisMarshallNY 2y ago
  > I think you're missing the point here Doubt it. This is just one -of many- carrots that are used to entice developers to include dodgy software into their apps. The problem is a lot bigger than these libraries. It's an endemic cultural issue. Much more difficult to quantify or fix.
sixtyj 2y ago
Malware, botnets… it is very similar. And people including developers are - in 80 per cent - eagier to make money, because… Is greed good? No, it isn’t. It is a plague.
- II2II 2y ago
  You're a developer who devoted time to develop a piece of software. You discover that you are not generating any income from it: few people can even find it in the sea of similar apps, few of those are willing to pay for it, and those who are willing to pay for it are not willing to pay much. To make matters worse, you're going to lose a cut of what is paid to the middlemen who facilitate the transaction. Is that greed? I can find many reasons to be critical of that developer, things like creating a product for a market segment that is saturated, and likely doing so because it is low hanging fruit (both conceptually and in terms of complexity). I can be critical of their moral judgement for how they decided to generate income from their poor business judgment. But I don't thinks it's right to automatically label them as greedy. They may be greedy, but they may also be trying to generate income from their work.

jonplackett 2y ago

How is this not just illegal? Surely there’s something in GDPR that makes this not allowed.

Retr0id 2y ago
iiuc, they do actually ask the user for permission
- fc417fc802 2y ago
  Which is ironic considering that I strongly disagree with one of the primary walled garden justifications, used particularly in the case of Apple, which amounts to "the end user is too stupid to decide on his own". Unfortunately, even if I disagree with it as a guiding principle sometimes that statement proves true.
  - klabb3 2y ago
    It’s not about stupidity, but practicality. People can’t give informed consent for 100 ToS for different companies, and keep those up to date. That’s why there are laws.
- SoftTalker 2y ago
  No doubt in a dense wall of text that the user must accept to use the application, or worse is deemed to have accepted by using the application at all.

zahlman 2y ago

BlueTemplar 2y ago
Yeah, also this means the death of archival efforts like the Internet Archive.
- jeroenhd 2y ago
  Welcome scrapers (IA, maybe Google and Bing) can publish their IP addresses and get whitelisted. Websites that want to prevent being on the Internet Archive can pretty much just ask for their website to be excluded (even retroactively). [Cloudflare](https://developers.cloudflare.com/cache/troubleshooting/always-online/ https://developers.cloudflare.com/cache/troubleshooting/alwa...) tags the internet archive as operating from 207.241.224.0/20 and 208.70.24.0/21 so disabling the bot-prevention framework on connections from there should be enough.
  - trinsic2 2y ago
    This sounds like it would be a good idea. Create a whitelist of IPs and block the rest.
  - realusername 2y ago
    That's basically asking to close the market in favor of the current actors. New actors have the right to emerge.
    - 0dayz 2y ago
      No they don't. There's no rule that you have to let anyone in who claims to be a web crawler.

pton_xd 2y ago

I thought the closed-garden app stores were supposed to protect us from this sort of thing?

whstl 2y ago
Once again this demonstrate that closed gardens only benefit the owners of the garden, and not the users. What good is all the app vetting and sandbox protection in iOS (dunno about Android) if it doesn't really protect me from those crappy apps...
- 20after4 2y ago
  At the very least, Apple should require conspicuous disclosure of this kind of behavior that isn't just hidden in the TOS.
- BlueTemplar 2y ago
  Also my reaction when the call is for Google, Apple, Microsoft to fix this : DDOS being illegal, shouldn't the first reaction instead to be to contact law enforcement ? If you treat platforms like they are all-powerful, then that's what they are likely to become...
- musicale 2y ago
  Sandboxing means you can limit network access. For example, on Android you can disallow wi-fi and cellular access (not sure about bluetooth) on a per-app basis. Network access settings should really be more granular for apps that have a legitimate need. App store disclosure labels should also add network usage disclosure.
20after4 2y ago
That's what they want you to think.
kibwen 2y ago
If you find yourself in a walled garden, understand that you're the crop being grown and harvested.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

jt2190 2y ago

I’m really struggling to understand how this is different than malware we’ve had forever. Can someone explain what’s novel about this?

desertmonad 2y ago
That its not being treated like malware.
- jt2190 2y ago
  In the sense that people are voluntarily installing and running this malware on their computers, rather than being tricked into running it? Is that the only difference?
  - int_19h 2y ago
    They are still tricked into running it, since it's normally not an advertised "feature" of any app that uses such SDKs.
downrightmike 2y ago
I think it is funny that the mobile OS is trying to be as secure as possible, but then they allow this to run on top

rsedgwick 2y ago

I think tech can still be beautiful in a less grandiose and "omniparadisical" way than people used to dream of. "A wide open internet, free as in speech this, free as in beer that, open source wonders, open gardens..." Well, there are a lot of incentives that fight that, and game theory wins. Maybe we download software dependencies from our friends, the ones we actually trust. Maybe we write more code ourselves--more homesteading families that raise their own chickens, jar their own pickled carrots, and code their own networking utilities. Maybe we operate on servers we own, or our friends own, and we don't get blindsided by news that the platforms are selling our data and scraping it for training. Maybe it's less convenient and more expensive and onerous. Do good things require hard work? Or did we expect everyone to ignore incentives forever while the trillion-dollar hyperscalers fought for an open and noble internet and then wrapped it in affordable consumer products to our delight? It reminds me of the post here a few weeks ago about how Netflix used to be good and "maybe I want a faster horse" - we want things to be built for us, easily, cheaply, conveniently, by companies, and we want those companies not to succumb to enshittification - but somehow when the companies just follow the game theory and turn everything into a TikToky neural-networks-maximizing-engagement-infinite-scroll-experience, it's their fault, and not ours for going with the easy path while hoping the corporations would not take the easy path.

reconnecting 2y ago

Residential IP proxies have some weaknesses. One is that they ofter change IP addresses during a single web session. Second, if IP come from the same proxies provider, they are often concentrated within a sing ASN, making them easier to detect. We are working on an open‑source fraud prevention platform [1], and detecting fake users coming from residential proxies is one of its use cases. [1] https://www.github.com/tirrenotechnologies/tirreno https://www.github.com/tirrenotechnologies/tirreno

gbcfghhjj 2y ago
At least here in the US most residential ISPs have long leases and change infrequently, weeks or months. Trying to understand your product, where is it intended to sit in a network? Is it a standalone tool that you use to identify these IPs and feed into something else for blockage or is it intended to be integrated into your existing site or is it supposed to proxy all your web traffic? The reason I ask is it has fairly heavyweight install requirements and Apache and PHP are kind of old school at this point, especially for new projects and companies. It's not what they would commonly be using for their site.
- reconnecting 2y ago
  Indeed, if it's a real user from a residential IP address, in most cases it will be the same network. However, if it's a proxy from residential IPs, there could be 10 requests from one network, the 11th request from a second network, and the 12th request back from the same network. This is a red flag. Thank you for your question. tirreno is a standalone app that needs to receive API events from your main web application. It can work perfectly with 512GB Postgres RAM or even lower, however, in most cases we're talking about millions of events that request resources. It's much easier to write a stable application without dependencies based on mature technologies. tirreno is fairly 'boring software'.
  - sroussey 2y ago
    My phone will be on the home network until I walk out of the house and then it will change networks. This should not be a red flag.

at0mic22 2y ago

Strange the HolaVPN e.g. Brightdata is not mentioned. They've been using user hosts for those purposes for decades, and also selling proxies en masse. Fun fact they don't have any servers for the VPN. All the VPN traffic is routed through ... other users!

Klonoar 2y ago
Is it really strange if the logo is right there in the article?
arewethereyeta 2y ago
They are even the first to do it and the most litigious of all. Trying to push patents on everything possible, even on water if they can.
andelink 2y ago
Hola is mentioned in the authors prior post on this topic, linked to at the top of TFA: https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/ https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/

armchairhacker 2y ago

> I am now of the opinion that every form of web-scraping should be considered abusive behaviour and web servers should block all of them. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the “AI” hype for moving you to the bad corner. Why jump to that conclusion? If a scraper clearly advertises itself, follows robots.txt, and has reasonable backoff, it's not abusive. You can easily block such a scraper, but then you're encouraging stealth scrapers because they're still getting your data. I'd block the scrapers that try to hide and waste compute, but deliberately allow those that don't. And maybe provide a sitemap and API (which besides being easier to scrape, can be faster to handle).

panstromek 2y ago

I'd expect this to be against app store and google play rules, they are very picky.

Pesthuf 2y ago

We need a list of apps that include these libraries and any malware scanner - including Windows Defender, Play Protect and whatever Apple calls theirs - need to put infected applications into quarantine immediately. Just because it's not directly causing damage to the device running the malware is running on, that doesn't mean it's not malware.

philippta 2y ago
Apps should be required to ask for permission to access specific domains. Similar to the tracking protection, Apple introduced a while ago. Not sure how this could work for browsers, but the other 99% of apps I have on my phone should work fine with just a single permitted domain.
- jay_kyburz 2y ago
  Oh, that's an interesting idea. A local DNS where I have to add every entry. A white list rather than Australia's national blacklist.
- snackernews 2y ago
  My iPhone occasionally displays an interrupt screen to remind me that my weather app has been accessing my location in the background and to confirm continued access. It should also do something similar for apps making chatty background requests to domains not specified at app review time. The legitimate use cases for that behaviour are few.
- zzo38computer 2y ago
  I think capability based security with proxy capabilities is the way to do it, and this would make it possible for the proxy capability to intercept the request and ask permission, or to do whatever else you want it to do (e.g. redirections, log any accesses, automatically allow or disallow based on a file, use or ignore the DNS cache, etc). The system may have some such functions built in, and asking permission might be a reasonable thing to include by default.
  - 2y ago

proxy_err 2y ago

Its a fair point but very dynamic to sort out. This needs a full research team to figure out. Or you know.. all of us combined!! It is definitely a problem. TINFOIL: Sometimes I always wondered if Azure or AWS used bots to push site traffic hits to generate money... they know you are hosted with them.. They have your info.. Send out bots to drive micro accumulation. Slow boil..

luckylion 2y ago
I think that's mostly that they don't care about having malicious bots on their networks as long as they pay. GCE is rare in my experience. Most bots I see are on AWS. The DDOS-adjacent hyper aggressive bots that try random URLs and scan for exploits tend to be on Azure or use VPNs. AWS is bad when you report malicious traffic. Azure has been completely unresponsive and didn't react, even for C&C servers.

aucisson_masque 2y ago

It's interesting but so far there is no definitive proof it's happening. People are jumping to conclusions a bit fast over here, yes technically it's possible but this kind of behavior would be relatively easy to spot because the app would have to make direct connections to the website it wants to scrap. Your calculator app for instance connecting to CNN.com ... iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. Android by Google doesn't have such a useful feature of course, but you can run third party firewall like pcapdroid, which I recommend highly. Macos (little snitch). Windows (fort firewall). Not everyone run these app obviously, only the most nerdy like myself but we're also the kind of people who would report on app using our device to make, what is in fact, a zombie or bot network. I'm not saying it's necessarily false but imo it remains a theory until proven otherwise.

CharlesW 2y ago
Botnets as a Service are absolutely happening, but as you allude to, the scope of the abuse is very different on iOS than, say, Windows.
abaymado 2y ago
> iOS have app privacy report where one can check what connections are made by app, how often, last one, etc. How often is the average calculator app user checking there Privacy Report? My guess, not many!
- gruez 2y ago
  All it takes is one person to find out and raise the alarm. The average user doesn't read the source code behind openssl or whatever either, that doesn't mean there's no gains in open sourcing it.
  - dewey 2y ago
    The average user is also not reading these raised “alarms”. And if an app has a bad name, another one will show up with a different name on the same day.

badmonster 2y ago

do you think there’s a realistic path forward for better transparency or detection—maybe at the OS level or through network-level anomaly detection?

yungporko 2y ago

it's funny, i've never heard of or thought about the possibility of this happening but actually in hindsight it seems almost too obvious to not be a thing.

jeroenhd 2y ago

> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth AKA "why do Cloudflare and Google make me fill out these CAPTCHAs all day" I don't know why Play Protect/MS Defender/whatever Apple has for antivirus don't classify apps that embed such malware as such. It's ridiculous that this is allowed to go on when detection is so easy. I don't know a more obvious example of a trojan than an SDK library making a user's device part of a botnet.

deleted 2y ago
[deleted]
dx4100 2y ago
Cloudflare and Google use CAPTCHAs to sell web scrapers? I don't get your point. I was under the impression the data is used to train models.
- cuu508 2y ago
  Trojans in your mobile apps ruin your IP's reputation which comes back to you in the form of frequent, annoying CAPTCHAs.
- aloha2436 2y ago
  The implication is that the users that are being constantly presented with CAPTCHAs are experiencing that because they are unwittingly proxying scrapers through their devices via malicious apps they've installed.
  - pentae 2y ago
    .. or that other people on their network/Shared public IP have installed
    - evgpbfhnr 2y ago
      or just that they don't run windows/mac OS with chome like everyone else and it's "suspicious". I get cloudflare capchas all the time with firefox on linux... (and I'm pretty sure there's no such app in my home network!)

panny 2y ago

>Apple, Microsoft and Google should act. Do nothing, win. They are the primary benefactors buying this data since they are the largest AI players.

neilv 2y ago

Couldn't Apple and Google (and, to a lesser extent, Microsoft) pretty easily shut down almost all the apps that steal bandwidth?

greesil 2y ago

How would I know if an app on my device was doing this?

wyck 2y ago
Install a network monitor or go even deeper and sniff packets.
- greesil 2y ago
  I feel like this could be automated. Spin up a virtual device on a monitored network. Install one app, click on some stuff for awhile, uninstall and move onto the next. If the app reaches out to a lot of random sites then flag it Google could do this. I'm sure Apple could as well. Third parties could for a small set of apps
  - jeroenhd 2y ago
    This is being done by a couple of SDKs, it'd be much easier to just find and flag those SDK files. Finding apps becomes a matter of a single pass scan over the application contents rather than attempting to bypass the VM detection methods malware is packed full of.

matheusmoreira 2y ago

"Peer-to-business network"! Amazing. uBlock Origin gets rid of this, right?

theteapot 2y ago

Are ad blockers like AdBlock, uBlock effective against these?

areyourllySorry 2y ago
i don't believe extensions can modify other extensions

__MatrixMan__ 2y ago

The broken thing about the web is that in order for data to remain readable, a unique sysadmin somewhere has to keep a server running in the face of an increasingly hostile environment. If instead we had a content addressed model, we could drop the uniqueness constraint. Then these AI scrapers could be gossiping the data to one another (and incidentally serving it to the rest of us) without placing any burden on the original source. Having other parties interested in your data should make your life easier (because other parties will host it for you), not harder (because now you need to work extra hard to host it for them).

Timwi 2y ago
Are there any systems like that, even if experimental?
- jevogel 2y ago
  IPFS
  - alakra 2y ago
    I had high hopes for IPFS, but even it has vectors for abuse. See https://arxiv.org/abs/1905.11880 https://arxiv.org/abs/1905.11880 [Hydras and IPFS: A Decentralised Playground for Malware]
    - __MatrixMan__ 2y ago
      Can you point me at what you mean? I'm not immediately finding something that indicates that it is not fit for this use case. The fact that bad actors use it to resist those who want to shut them down is, if anything, an endorsement of its durability. There's a bit of overlap between resisting the AI scrapers and resisting the FBI. You can either have a single point of control and a single point of failure, or you can have neither. If you're after something that's both reliable and reliably censorable--I don't think that's in the cards. That's not to say that it is a ready replacement for the web as we know it. If you have hash-linked everything then you wind up with problems trying to link things together, for instance. Once two pages exist, you can't after-the-fact create a link between them because if you update them to contain that link then their hashes change so now you have to propagate the new hash to people. This makes it difficult to do things like have a comments section at the bottom of a blog post. So you've got to handle metadata like that in some kind of extra layer--a layer which isn't hash linked and which might be susceptible to all the same problems that our current web is--and then the browser can build the page from immutable pieces, but the assembly itself ends up being dynamic (and likely sensitive to the users preference, e.g. dark mode as a browser thing not a page thing). But I still think you could move maybe 95% of the data into an immutable hash-linked world (think of these as nodes in a graph), the remaining 5% just being tuples of hashes and pubic keys indicating which pages are trusted by which users, which ought to be linked to which others, which are known to be the inputs and output of various functions, and you know... structure stuff (these are our graph's edges). The edges, being smaller, might be subject to different constraints than the web as we know it. I wouldn't propose that we go all the way to a blockchain where every device caches every edge, but it might be feasible for my devices to store all of the edges for the 5% of the web I care about, and your devices to store the edges for the 5% that you care about... the nodes only being summoned when we actually want to view them. The edges can be updated when our devices contact other devices (based on trust, like you know that device's owner personally) and ask "hey, what's new?" I've sort of been freestyling on this idea in isolation, probably there's already some projects that scratch this itch. A while back I made a note to check out https://ceramic.network/ https://ceramic.network/ in this capacity, but I haven't gotten down to trying it out yet.

156287745637 2y ago

AI scrapers and "sneaker bots" are just the tip of the iceberg. Why are all these entities concentrated and metastasizing from just a few superhubs? Why do they look, smell and behave like state-level machinery? If you've researched you'll know exactly what I'm talking about. Unless complicit, tech leaders (Apple Google Microsoft) have a duty to respond swiftly and decisively. This has been going on far too long.

deleted 2y ago
[deleted]

_ink_ 2y ago

How can I detect such behaviour on my devices / in my home network?

gpi 2y ago

"Infatica is partnered with Bitdefender, a global leader in cybersecurity, to protect our SDK users from malicious web traffic and content, including infected URLs, untrusted web pages, fraudulent and phishing links, and more." That's not good.

deleted 2y ago
[deleted]

Quarrel 2y ago

FWIW, Trend Micro wrote up a decent piece on this space in 2023. It is still a pretty good lay-of-the-land. https://www.trendmicro.com/vinfo/us/security/news/vulnerabilities-and-exploits/a-closer-exploration-of-residential-proxies-and-captcha-breaking-services https://www.trendmicro.com/vinfo/us/security/news/vulnerabil...

hinkley 2y ago

When the enshitification initially hit the fan, I had little flashbacks of Phil Zimmerman talking about Web of Trust and amusing myself thinking maybe we need humans proving they're humans to other humans so we know we aren't arguing with LLMs on the internet or letting them scan our websites. But it just doesn't scale to internet size so I'm fucked if I know how we should fix it. We all have that cousin or dude in our highschool class who would do anything for a bit of money and introducing his 'friend' Paul who is in fact a bot whose owner paid for the lie. And not like enough money to make it a moral dilemma, just drinking money or enough for a new video game. So once you get past about 10,000 people you're pretty much back where we are right now.

akoboldfrying 2y ago
I think it should be possible to build something that generalises the idea of Web of Trust so that it's more flexible, and less prone to catastrophic breakdown past some scaling limit. Binary "X trusts Y" statements, plus transitive closure, can lead to long trust paths that we probably shouldn't actually trust the endpoints of. Could we not instead assign probabilities like "X trusts Y 95%", multiply probabilities along paths starting from our own identity, and take the max at each vertex? We could then decide whether to finally trust some Z if its percentage is more than some threshold T%. (Other ways of combining in-edges may be more suitable than max(); it's just a simple and conservative choice.) Perhaps a variant of backprop could be used to automatically update either (a) all or (b) just our own weights, given new information ("V has been discovered to be fraudulent").
- hinkley 2y ago
  True. Perhaps a collective vote past 2 degrees of freedom out where multiple parties need to vouch for the same person before you believe they aren't a bot. Then you're using the exponential number of people to provide diminishing weight instead of increasing likelihood of malfeasance.
  - nottorp 2y ago
    But do we need an infinite and global web of trust? How about restricting them to everyone-knows-everyone sized groups, of like a couple hundred people? One can be a member of multiple groups so you're not actually limited. But the groups will be small enough to self regulate.

hubraumhugo 2y ago

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space). So what are potential solutions? We're somehow still stuck with CAPTCHAS, a 25 years old concept that wastes millions of human hours and billions in infra costs [0]. How can enable beneficial automation while protecting against abusive AI crawlers? [0] https://arxiv.org/abs/2311.10911 https://arxiv.org/abs/2311.10911

udev4096 2y ago
Blame the "AI" companies for that. I am glad the small web is pushing hard against these scrapers, with the rise of Anubis as a starting point
- lelanthran 2y ago
  > Blame the "AI" companies for that. I am glad the small web is pushing hard towards these scrapers, with the rise of Anubis as a starting point Did you mean "against"?
  - udev4096 2y ago
    Corrected, thanks
eastbound 2y ago
But people don’t interact with your website anymore; they as an AI. So the AI crawler is a real user. I say we ask Google Analytics to count an AI crawler as a real view. Let’s see who’s most popular.
CalRobert 2y ago
I hate this but I suspect a login-only deanonymised web (made simple with chrome and WEI!) is the future. Firefox users can go to hell.

y42 2y ago

Let me get this straight: we want computers knowing everything, to solve current and future problems, but we don't want to give them access to our knowledge?

drawfloat 2y ago
Most people don’t want computers to know everything - ask the average person if they want more or less of their lives recorded and stored.
- y42 2y ago
  > Most people don’t want computers to know everything. That may well be true. But how many of those people are specifically against AI companies scraping the web? That’s not really an argument—it’s an assumption based on personal perception. > Ask the average person if they want more or less of their lives recorded and stored. What exactly is the "average person"? Also, I’ll admit my earlier claim was a bit exaggerated. But let’s be clear: this isn’t about recording personal data—it’s about collecting and structuring knowledge. And beyond that: companies have been scraping the web for years. They still are. And they’re gathering far more personal data for online marketing, tracking, profiling—whatever the reason—and the so-called "average person" hasn’t raised much of a finger. People remain glued to platforms, willingly sharing their personal lives. And what do they get in return? Doomscrolling and five-second video clips.
chairmansteve 2y ago
Not sure we do.
3np 2y ago
I don't want your computer to know everything about me, in fact.
- y42 2y ago
  That’s not what I said. Let me rephrase it: Do you want computers to help us solve medical or scientific problems in order to improve human life?

deleted 2y ago

[deleted]

topak3000 2y ago

[flagged]

areyourllySorry 2y ago
the next time you copy paste an ad remove the trailing quotation mark
nothrabannosir 2y ago
@dang can this entire account be banned please

areyourllySorry 2y ago

further reading https://krebsonsecurity.com/?s=infatica https://krebsonsecurity.com/?s=infatica https://krebsonsecurity.com/tag/residential-proxies/ https://krebsonsecurity.com/tag/residential-proxies/ https://spur.us/blog/ https://spur.us/blog/ https://bright-sdk.com/ https://bright-sdk.com/ <- way bigger than infatica

dspillett 2y ago

> So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library into their apps that sells users network bandwidth. This is yet another reason why we need to be wary of popular apps, add-ons, extensions, and so forth changing hands, by legitimate sale or more nefarious methods. Initially innocent utilities can be quickly coopted into being parts of this sort of scheme.

aorth 2y ago

In the last week I've had to deal with two large-scale influxes of traffic on one particular web server in our organization. The first involved requests from 300,000 unique IPs in a span of a few hours. I analyzed them and found that ~250,000 were from Brazil. I'm used to using ASNs to block network ranges sending this kind of traffic, but in this case they were spread thinly over 6,000+ ASNs! I ended up blocking all of Brazil (sorry). A few days later this same web server was on fire again. I performed the same analysis on IPs and found a similar number of unique addresses, but spread across Turkey, Russia, Argentina, Algeria and many more countries. What is going on?! Eventually I think I found a pattern to identify the requests, in that they were using ancient Chrome user agents. Chrome 40, 50, 60 and up to 90, all released 5 to 15 years ago. Then, just before I could implement a block based on these user agents, the traffic stopped. In both cases the traffic from datacenter networks was limited because I already rate limit a few dozen of the larger ones. Sysadmin life...

rollcat 2y ago
Try Anubis: <https://anubis.techaro.lol https://anubis.techaro.lol> It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?
- marginalia_nu 2y ago
  Anubis is a good choice because it whitelists legitimate and well behaved crawlers based on IP + user-agent. Cloudflare works as well in that regard but then you're MITM:ing all your visitors.
- Imustaskforhelp 2y ago
  Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis. Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and it actually made his git server down I guess??) I didn't expect it to blow up to ... UN.

jgalt212 2y ago

I blame the VCs. They don't stop, and implicitly encourage, website-crushing scrapers among their funded ventures. It's not a crime if we do it with an app https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading https://pluralistic.net/2025/01/25/potatotrac/#carbo-loading

reincoder 2y ago

I work for IPinfo (a commercial service). We offer a residential proxy detection service, but it costs money. If you are being bombarded by suspicious IP addresses, please consider using our free service and blocking IP addresses by ASN or Country. I think ASN is a common parameter for malicious IP addresses. If you do not have time to explore our services/tools (it is mostly just our CLI: https://github.com/ipinfo/cli https://github.com/ipinfo/cli), simply paste the IP addresses (or logs) in plain text, send it to me and I will let you know the ASNs and corresponding ranges to block.

throwaway74663 2y ago
Blocking countries is such a poorly disguised form of racism. Funny how it's always the brown / yellow people countries that get blocked, and never the US, despite it being one of the leading nations in malicious traffic.
- reincoder 2y ago
  Oh, absolutely not — I have to respectfully but strongly disagree with that sentiment. In cybersecurity, decisions must be guided by objective data, not assumptions or biases. When you’re facing abuse, you analyze the IPs involved and enrich them with context — ASN, country, city, whether it’s VPN, hosting, residential, etc. That gives you the information you need to make calculated decisions: Should you block a subnet? Rate-limit it? CAPTCHA-challenge it? Here’s a small snapshot from my own SSH honeypot: Summary of 1,413 attempts - Hosting IPs: 981 (69%) - VPNs: 35 - Top ASNs: - AS204428 (SS-Net): 152 - AS136052 (PT Cloud Hosting Indonesia): 83 - AS14061 (DigitalOcean): 76 - Top Countries: - Romania: 238 (16.8%) - United States: 150 (10.6%) - China: 134 (9.5%) - Indonesia: 115 (8.1%) One single /24 from Romania accounts for over 10% of the attacks. That’s not about nationality or ethnicity — it's about IP space abuse from a specific network. If a network or country consistently shows high levels of hostile traffic and your risk tolerance justifies it, blocking or throttling it may be entirely reasonable. Security teams don’t block based on "where people come from" — they block based on where the attacks are coming from. We even offer tools to help people explore and understand these patterns better. But if someone doesn’t have the time or resources to do that, I'm more than happy to assist by analyzing logs and suggesting reasonable mitigations.