7 ms·

Aggressive bots ruined my weekend

209 points by shaunpud 1y ago

_qtiw 1y ago
[flagged]
asplake 1y ago
> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed, and potentially horrific for the owners of the affected devices also! Any corroboration for that out there?
- VladVladikoff 1y ago
  This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.
  - dataviz1000 1y ago
    They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
    - VladVladikoff 1y ago
      WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?
      - arethuza 1y ago
        That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?
        alamortsubite 1y ago
        My take is it's mostly irrelevant, but read the lobsters post mentioned elsewhere.
      - alamortsubite 1y ago
        Yes, and it doesn't matter if they do read the terms- to the average user they sound totally innocuous, especially placed next to a big shiny "GET 500 FREE COINS" button.
    - arethuza 1y ago
      I suspect most people, even when told exactly what the app using that SDK would be doing, wouldn't actually see the potential problems...
      - kijin 1y ago
        Until one day, they get swatted for accessing child porn. Actually, that might be one way to draw attention to the problem. Sign up to some of these shady "residential proxy" services, and access all sorts of nasty stuff through their IPs until your favorite three-letter agency takes notice.
    - TYPE_FASTER 1y ago
      Also see https://www.youtube.com/watch?v=AGaiVApKfmc https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worker asking for an Apple gift card. The name was indeed an employee from a different part of the large company I worked for at the time, but LinkedIn was the only possible link I could figure out that was at least somewhat publicly available information. This should probably be required in all CS curriculum: https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-responsibilities-of-computing-serc/ https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-r...
      - nerdponx 1y ago
        It should be illegal, but this stuff is propping up the appearance of a healthy economy so nobody will touch it.
      - VladVladikoff 1y ago
        That scam definitely uses linked in as the source. We get a lot of those BEC emails and it’s always the people who are on LinkedIn. Also keep in mind LinkedIn has had big database leaks in the past, you might not even need to scrape them, just download a huge database from a leaks site.
    - cuu508 1y ago
      IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.
      - VladVladikoff 1y ago
        I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.
        deleted 1y ago
        [deleted]
    - seemaze 1y ago
      That's sleazy. It's slipping drugs into a kids lunchbox and letting smuggle it across the border..
  - myaccountonhn 1y ago
    One such example is brightdata, on lobsters someone did a writeup https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don_t_like_me#c_hzq0yp https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don...
    - VladVladikoff 1y ago
      Never heard of lobsters before. Cool site. Seems to be invite only though :( If you could share an invite that would be cool. torosanchez@protonmail.me Thanks!
  - corbet 1y ago
    The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.
    - nurettin 1y ago
      I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy. They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoever is paying, and they don't care what people do in there.
      - yomismoaqui 1y ago
        > ... until I figured out a way to avoid the ban hammer ... You had my curiosity ... but now you have my attention.
        walletdrainer 1y ago
        Without bothering to check on Amazon, I successfully scraped meta stuff for years at rates exceeding 20gbit/s without any proxies but just rotating IPv6 addresses on the same couple of blocks for every request There are usually silly bypasses like this that easily work even with bigco stuff
      - chritsopherh 11mo ago
        [dead]
  - wat10000 1y ago
    Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to forbid any nefarious activity, and naturally they don't actually care.
    - nemomarx 1y ago
      Salad, right? What a strange business
      - dylan604 1y ago
        Why is it strange. Of course it exists.
- curious_curios 1y ago
  If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.
  - antoniojtorres 1y ago
    This is how so many companies sell from an opaque inventory of “millions” of residential proxies.
- kaoD 1y ago
  There's crap like https://hola.org/ https://hola.org/ https://hola.org/legal/sdk https://hola.org/legal/sdk https://hola.org/legal/sla https://hola.org/legal/sla > How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula https://bright-sdk.com/eula). You may opt out by becoming a Premium user. This "VPN" is what powers these residential proxies: https://brightdata.com/ https://brightdata.com/ I'm sure there are many other companies like this.
  - piggg 1y ago
    There's also a ton of companies selling "make money off your unused internet" apps which are all over tiktok and basically turn yourself into a residential proxy/sketch VPN egress node. On top of that - lots of free tv/movie streaming stuff that also makes yourself a proxy/egress node. Sometimes you find it on tv/movie streaming devices sold online where it's already loaded on when it arrives.
- Zanfa 1y ago
  SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.
  - Cthulhu_ 1y ago
    Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection. edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, are metrics going up and metrics going brrt means profit and shareholder interest.
    - ac29 1y ago
      Its very possible they did detect it and that's why law enforcement got involved. But yes, they also might not care if they are getting paid. If the SIMs are only being used for voice/text as I suspect, it might have very minimal load on the network.
- immibis 1y ago
  You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).
  - VBprogrammer 1y ago
    If someone hasn't written a blog titled "Should we be worried about Cloudflare?" yet, I think it would be a good subject to explore. I find the idea that they could decide one day to ban you from all of their network pretty worrying. And if they did, how much fingerprinting are they doing and would the bad extend far beyond just a random IP address.
  - pjc50 1y ago
    This is one of those "ACAB" things where you might reasonably dislike Cloudflare but a world without them or an equivalent will evolve worse solutions to the same problems, which you will like even less.
  - nix0n 1y ago
    > This is not illegal Depends on what they're doing from your connection.
    - immibis 1y ago
      Strict liability by IP address is not the norm, not even in Germany any more. It's not illegal to have a botnet infect your computer either. Since they promise not to use your connection for illegal things, it's their fault if they break that.
- lucastech 1y ago
  I wrote about this back in July when this "gang" first started hitting some sites I host: https://wxp.io/blog/the-bots-that-keep-on-giving https://wxp.io/blog/the-bots-that-keep-on-giving they use a mixture of colo (M247, Datacamp, HostRoyale, Oxylabs, etc) and international residential. I suspect the latter are where those residential app proxies come into play (bright SDK, etc). Oxylabs is also a well known proxy provider, which makes me think they're the gateway into all of these IPs. Definitely interesting times to try and host a web server!
TimorousBestie 1y ago
I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.
- sudosays 1y ago
  Counterpoint: I think he should stay and fight the good fight. Indie blog businesses are great for the health of the human internet, and I don't think surrendering preemptively will help things get better.
  - add-sub-mul-div 1y ago
    That's an easy thing to say if it's someone else's time that's being wasted and not your own. But there may not be a path back to the internet under which this project was conceived. It could be like staying on Twitter and Reddit after their respective declines. You're only suffering an opportunity cost for your own time and preventing the internet from evolving better alternatives.
- flaviuspopan 1y ago
  His persistent efforts are the reason I pay for Bear Blog. I think he should fight for the chance to come out on the other side of whatever future we’re heading towards.
  - TimorousBestie 1y ago
    I pay for Bear Blog, too. But this year has been problem after problem for its sole proprietor, and I don’t think it’s going to get better.
    - josephb 1y ago
      As much as I love the service and enjoy the transparency, I do wonder about the future of smaller operator services as the Internet continues it's descent into a giant mess :-)
pingoo101010 1y ago
You may want to take a look at Pingoo (https://github.com/pingooio/pingoo https://github.com/pingooio/pingoo), a reverse proxy with automatic TLS that can also block bots with advanced rules that go beyond simple IP blocking.
kelvinjps10 1y ago
Maybe moving the blog service to completely static and letting cloudfare pages handle it, could help?
- reustle 1y ago
  Cloudflare is not a solution. Only leading to a further centralized internet.
  - uvaursi 1y ago
    I think the OP is suggesting using a caching layer at HTTP output, and suggesting CF as an option (a quick/cheap one). If you have an axe to grind with CF you can take it up with them, but it’s an option. Feel free to suggest others.
    - LilBytes 1y ago
      It seems to me that the writer, Herman, doesn't want to use the cloud short of the parts that are almost mandatory these days (e.g., CloudFlare, or a CDN of some sort). GitHub Pages, CloudFlare pages etc are a great and very simple service. But they're opposite or contrary to running your own hardware, warts and all. Herman wasn't looking for solutions IMO, I read it more as him lamenting at how hostile and insidious the internet has become. It has been for some time, but it seems to be getting exponentially worse.
      - uvaursi 1y ago
        Remember when Stormfront was a thing? Remember how everyone cheered on that CF and others banned them? I’m sure Herman was one of the loudest ones to proclaim victory that day. Hostility and insidiousness were created by you for not standing up when it was most needed and called for. And as you can see, Stormfront in hindsight was the most milquetoast website compared to landscape of politics today. All I’m saying is CF is a viable caching option. But if you’re looking for morality support - you lost that war a long time ago. “If we do not stand up for the rights of the accused, we endanger our own. For when the tide turns, who will stand up for us?”
deleted 1y ago
[deleted]
cupofjoakim 1y ago
We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots use tools like NobleTLS or JA3Cloak to go around ja3 rate limiting 5. Some valid LLM companies seem to also follow the above to gather training data. We want them to know about our company, so we don't necessarily want to block them I'm close to giving up on this front tbh. There's no longer safe methods of identifying malignant traffic at scale, and with the variations we have available we can't statically generate these. Even with a CDN cache (shoutout fastly) our catalog is simply too broad to fully saturate the cache while still allowing pages to be updated in a timely manner. I guess the solution is to just scale up the origin servers... /shrug In all seriousness, i'd love if we somehow could tell the bots about more efficient ways of fetching the data. Use our open api for fetching book informations instead of causing all that overhead by going to marketing pages please.
- FeepingCreature 1y ago
  In principle, it should be possible to identify malign IPs at scale by using a central service and reporting IPs probabilistically. That is, if you report every thousandth page hit with a simple UDP packet, the central tracker gets very low load and still enough data to publish a bloom filter of abusive IPs, say a million bits gives you pretty low false-positive. (If it's only ~10k malign IPs, tbh you can just keep a lru counter and enumerate all of them.) A billion hits per hour across the tracked sites would still only correspond to ~50KB/s inflow on the tracker service. Any individual participating site doesn't necessarily get many hits per source IP, but aggregating across a few dozen should highlight the bad actors. Then the clients just pull the bloom filter once an hour (80KB download) and drop requests that match. Any halfway modern LLM could probably code the backend for this in a day or two and it'd run on a RasPi. Some org just has to take charge and provide the infra and advertisement.
  - 01HNNWZ0MV43FF 1y ago
2OEH8eoCRo0 1y ago
Why don't we sue the abusive scrapers? Scraping is legal but DDoSing is not!
- cupofjoakim 1y ago
  Not sure if that's satire or not but how would you even identify the party to sue? What do you do if they're based in a country where you can't sue them ofer relatively trivial matters as this?
  - 2OEH8eoCRo0 1y ago
    Not satire but it's a huge problem with the internet. Everyone washes their hands and people can harm you without liability.
  - immibis 1y ago
    You could start by subpoenaing the registered holder of the IP address. In general, you can ask a lawyer for your options. Chances are good there are more than zero. But only if you can afford a lawyer. If you're getting scraped from a country where you don't do business, you can block the country. It's not good to block a country, but it works as a temporary measure. If they really want your data, they will move to a country where you do business, which means a country where you can get a lawyer. Assuming you can afford a lawyer. If they're using rotating IPs, likely some of them are from your country. You might show a judge: "Hey, look, we're getting so much traffic, from a wide variety of IP addresses but it all seems to be the same person on the other end, which would make it illegal DDoS. Can we trace back some of these?" and if you're lucky, the judge might say yes. DDoS is not a relatively trivial matter.
ItsBob 1y ago
I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger, but bots and the like will often read robots.txt (at least to figure out what you have... they'll ignore the "deny" for the most part!) and if they try and go to that fake endpoint you can be 100% sure (well, as close as possible) that it's not a human and you can ban them. So I tried that. I auto-generated a robots.txt file on the fly. It was cached for 60 seconds or so as I didn't want to expend too many resource on it. When you asked for it, you either got the cached one or I created a new one. The CPU-usage was negligible. However, I changed the "deny" endpoint each time I built the file in case the baddies cached it, however, it still went to the same ASP.NET controller method. By hitting it, I sent a 10GB zip bomb and your IP was automatically added to the FW block list. It was quite simple: anyone that hit that endpoint MUST be dodgy... I believe I even had comments for the humans that stumbled across it letting them know that if they went to this endpoint in their browser it was an automatic addition to the firewall blocklist. Anyway... at first I caught a shit load of bad guys. There were thousands at first and then the numbers dropped and dropped to only tens per day. Anyway, this is a single data point but for me, it worked... I have no regrets about the zip bomb either :) I have another site that I'm working on so I may evolve it a bit so that you are banned for a short time and if you come back to the dodgy endpoint then I know you're a bot so into the abyss with you! It's not perfect but it worked for me anyway.
- immibis 1y ago
  It's interesting to study, right? This is the Internet equivalent of background radiation. Harmless in most cases. Exploit scanners aren't new to the LLM age and shouldn't overload your server - unless you're vulnerable to the exploit. Fun fact: Some people learn about new exploits by watching their incoming requests.
  - ItsBob 1y ago
    > It's interesting to study, right? Definitely! I wasn't experiencing any issues, hell it wasn't even for public consumption at that time so no great loss to me but I found a few things fascinating (and somewhat stupid!) about it: 1. The sheer number of automated requests to scrape my content 2. That a massive number of the bots openly had "bot" or some derivative in the user agent and they were accessing a page I'd explicitly denied! :D 3. That an equally large number were faking their user agents to look like regular users and still hitting a page that a regular user couldn't possibly ever hit! Something I did notice but it was towards the end and I didn't pursue it (I should log it better the next time for analysis!) was that the endpoint was dynamically generated and only existed in the robots.txt for a short time but there were bots I caught later on, long after that auto-generated page was created (and after the IP was banned) that still went for that same page: clearly the same entities! My spidey senses are tingling. Next time, I'm going to log the shit out of these requests and publish as much as I can for others to analyse and dissect... might be interesting.
kwa32 1y ago
crazy how scraping became an industry
- npteljes 1y ago
  It's wild. Data is very valuable. This manifests in two fronts simultaneously: who has the data controls heavily on who sees it and under what circumstances, and on the other side, they scrape it as hard as they can.
r_singh 1y ago
The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.
- intended 1y ago
  What ? What do you mean ?
  - georgefrowny 1y ago
    To be fair the heyday of unshit search was driven by mostly-consensual scraping. Today there are far too many people scraping stuff that isn't intended to be scraped, for profit, and doing it in a heavy-handed way that actually does have a negative and continuous effect on the victim's capacity. Everyone from AI services too lazy or otherwise unwilling to cache to companies exfiltrating some kind of data for their own commercial purposes.
    - r_singh 1y ago
      With peering bandwidth being freely distributed to ISPs and consumers being fed media and subsidised services up until their necks makes the counter argument smell of narrative control rather than technical or financial constraints But as I’m growing older I’m learning that the tech industry is mostly politically driven and relies on truth obfuscation as explained by Peter Thiel rather than real empowerment It’s facilitating accumulation of control and power at an unparalleled pace. If anything it’s proving to be more unjust than the feudal systems it promises to replace.
      - r_singh 1y ago
        I may have been too harsh. I love capitalism, technology, and software—they’ve built a meritocratic world and given me the tools to build my own life. AI and technology feel like my best friend, but also my worst enemy when they edge toward learned helplessness. That tension exists with anything we depend on: the closer we get, the more power it holds. The relationship between user and technology is becoming deeply intimate as systems gain reach and control. It’s important to stay optimistic but skeptical—and to keep protesting everything—because the work is moving faster than our ability to register its consequences. Reading back, I realise I drifted into more of a monologue than a conversation. I get carried away when I’m trying to reason things out in public. Still, I stand by the core point about balance and transparency in how we shape the web.
uvaursi 1y ago
Do we shift over everything to le Dark Web and let the corpos use this one for selling their shit to consumers? These toys don’t want to play nice and there’s no real way to stop them without bringing in things like Real ID and other verifications that infringe on anonymity.
- Chabsff 1y ago
  The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.
  - uvaursi 1y ago
    We can set different rules on these networks however. We can choose to be choosy at the gate.
    - Cthulhu_ 1y ago
      How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.
      - embedding-shape 1y ago
        Make it P2P and content-based, instead of location-based like the current web. Content could be served from anywhere, so DDoS stops being an effective method, and shared peer quality could propagate across the network to ban bad actors quickly. I spend about 30 seconds thinking about this, so this is clearly the perfect solution with zero drawbacks or tradeoffs.
        Chabsff 1y ago
        I know this is tongue in cheek, but I'll give it a serious reply in case someone finds themselves "inspired". A CDN. What you are describing is a CDN. We have CDNs today and the problem still exists because most of today's websites refuse to operate within the constraints. There is no need for new infrastructure to deploy this solution, we just need website operators to "give up" and operate in a more static way.
y-zon128 1y ago
> Auto-restart the reverse-proxy if bandwidth usage drops to zero for more than 2 minutes It's understandable in your case as you have traffic coming in constantly, but first thing that came to my mind is a loop of contant reboots - again, very unlikely in your case. Sometimes such blanket rules hit me due to most unexpected reasons, like the proxy somehow failed to start serving traffic in the given timeframe. Though I completely appreciate and agree with the 'ship now something that works now' approach!
- marginalia_nu 1y ago
  Open port 80 and log the results. Even if you don't announce your service anywhere the baseline request rate is way above 2 rpm. Every open port of every IP is continuously scanned for exploits.
Havoc 1y ago
> monetise their apps by offering them for free, and selling tunnel access to scrapers. I bet it’s free VPN apps

209 points by shaunpud 1y ago

[flagged]

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed, and potentially horrific for the owners of the affected devices also! Any corroboration for that out there?

VladVladikoff 1y ago
This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.
- dataviz1000 1y ago
  They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
  - VladVladikoff 1y ago
    WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?
    - arethuza 1y ago
      That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?
      - alamortsubite 1y ago
        My take is it's mostly irrelevant, but read the lobsters post mentioned elsewhere.
    - alamortsubite 1y ago
      Yes, and it doesn't matter if they do read the terms- to the average user they sound totally innocuous, especially placed next to a big shiny "GET 500 FREE COINS" button.
  - arethuza 1y ago
    I suspect most people, even when told exactly what the app using that SDK would be doing, wouldn't actually see the potential problems...
    - kijin 1y ago
      Until one day, they get swatted for accessing child porn. Actually, that might be one way to draw attention to the problem. Sign up to some of these shady "residential proxy" services, and access all sorts of nasty stuff through their IPs until your favorite three-letter agency takes notice.
  - TYPE_FASTER 1y ago
    Also see https://www.youtube.com/watch?v=AGaiVApKfmc https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worker asking for an Apple gift card. The name was indeed an employee from a different part of the large company I worked for at the time, but LinkedIn was the only possible link I could figure out that was at least somewhat publicly available information. This should probably be required in all CS curriculum: https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-responsibilities-of-computing-serc/ https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-r...
    - nerdponx 1y ago
      It should be illegal, but this stuff is propping up the appearance of a healthy economy so nobody will touch it.
    - VladVladikoff 1y ago
      That scam definitely uses linked in as the source. We get a lot of those BEC emails and it’s always the people who are on LinkedIn. Also keep in mind LinkedIn has had big database leaks in the past, you might not even need to scrape them, just download a huge database from a leaks site.
  - cuu508 1y ago
    IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.
    - VladVladikoff 1y ago
      I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.
      - deleted 1y ago
        [deleted]
  - seemaze 1y ago
    That's sleazy. It's slipping drugs into a kids lunchbox and letting smuggle it across the border..
- myaccountonhn 1y ago
  One such example is brightdata, on lobsters someone did a writeup https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don_t_like_me#c_hzq0yp https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don...
  - VladVladikoff 1y ago
    Never heard of lobsters before. Cool site. Seems to be invite only though :( If you could share an invite that would be cool. torosanchez@protonmail.me Thanks!
- corbet 1y ago
  The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.
  - nurettin 1y ago
    I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy. They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoever is paying, and they don't care what people do in there.
    - yomismoaqui 1y ago
      > ... until I figured out a way to avoid the ban hammer ... You had my curiosity ... but now you have my attention.
      - walletdrainer 1y ago
        Without bothering to check on Amazon, I successfully scraped meta stuff for years at rates exceeding 20gbit/s without any proxies but just rotating IPv6 addresses on the same couple of blocks for every request There are usually silly bypasses like this that easily work even with bigco stuff
    - chritsopherh 11mo ago
      [dead]
- wat10000 1y ago
  Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to forbid any nefarious activity, and naturally they don't actually care.
  - nemomarx 1y ago
    Salad, right? What a strange business
    - dylan604 1y ago
      Why is it strange. Of course it exists.
curious_curios 1y ago
If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.
- antoniojtorres 1y ago
  This is how so many companies sell from an opaque inventory of “millions” of residential proxies.
kaoD 1y ago
There's crap like https://hola.org/ https://hola.org/ https://hola.org/legal/sdk https://hola.org/legal/sdk https://hola.org/legal/sla https://hola.org/legal/sla > How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula https://bright-sdk.com/eula). You may opt out by becoming a Premium user. This "VPN" is what powers these residential proxies: https://brightdata.com/ https://brightdata.com/ I'm sure there are many other companies like this.
- piggg 1y ago
  There's also a ton of companies selling "make money off your unused internet" apps which are all over tiktok and basically turn yourself into a residential proxy/sketch VPN egress node. On top of that - lots of free tv/movie streaming stuff that also makes yourself a proxy/egress node. Sometimes you find it on tv/movie streaming devices sold online where it's already loaded on when it arrives.
Zanfa 1y ago
SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.
- Cthulhu_ 1y ago
  Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection. edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, are metrics going up and metrics going brrt means profit and shareholder interest.
  - ac29 1y ago
    Its very possible they did detect it and that's why law enforcement got involved. But yes, they also might not care if they are getting paid. If the SIMs are only being used for voice/text as I suspect, it might have very minimal load on the network.
immibis 1y ago
You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).
- VBprogrammer 1y ago
  If someone hasn't written a blog titled "Should we be worried about Cloudflare?" yet, I think it would be a good subject to explore. I find the idea that they could decide one day to ban you from all of their network pretty worrying. And if they did, how much fingerprinting are they doing and would the bad extend far beyond just a random IP address.
- pjc50 1y ago
  This is one of those "ACAB" things where you might reasonably dislike Cloudflare but a world without them or an equivalent will evolve worse solutions to the same problems, which you will like even less.
- nix0n 1y ago
  > This is not illegal Depends on what they're doing from your connection.
  - immibis 1y ago
    Strict liability by IP address is not the norm, not even in Germany any more. It's not illegal to have a botnet infect your computer either. Since they promise not to use your connection for illegal things, it's their fault if they break that.
lucastech 1y ago
I wrote about this back in July when this "gang" first started hitting some sites I host: https://wxp.io/blog/the-bots-that-keep-on-giving https://wxp.io/blog/the-bots-that-keep-on-giving they use a mixture of colo (M247, Datacamp, HostRoyale, Oxylabs, etc) and international residential. I suspect the latter are where those residential app proxies come into play (bright SDK, etc). Oxylabs is also a well known proxy provider, which makes me think they're the gateway into all of these IPs. Definitely interesting times to try and host a web server!

TimorousBestie 1y ago

I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.

sudosays 1y ago
Counterpoint: I think he should stay and fight the good fight. Indie blog businesses are great for the health of the human internet, and I don't think surrendering preemptively will help things get better.
- add-sub-mul-div 1y ago
  That's an easy thing to say if it's someone else's time that's being wasted and not your own. But there may not be a path back to the internet under which this project was conceived. It could be like staying on Twitter and Reddit after their respective declines. You're only suffering an opportunity cost for your own time and preventing the internet from evolving better alternatives.
flaviuspopan 1y ago
His persistent efforts are the reason I pay for Bear Blog. I think he should fight for the chance to come out on the other side of whatever future we’re heading towards.
- TimorousBestie 1y ago
  I pay for Bear Blog, too. But this year has been problem after problem for its sole proprietor, and I don’t think it’s going to get better.
  - josephb 1y ago
    As much as I love the service and enjoy the transparency, I do wonder about the future of smaller operator services as the Internet continues it's descent into a giant mess :-)

pingoo101010 1y ago

You may want to take a look at Pingoo (https://github.com/pingooio/pingoo https://github.com/pingooio/pingoo), a reverse proxy with automatic TLS that can also block bots with advanced rules that go beyond simple IP blocking.

kelvinjps10 1y ago

Maybe moving the blog service to completely static and letting cloudfare pages handle it, could help?

reustle 1y ago
Cloudflare is not a solution. Only leading to a further centralized internet.
- uvaursi 1y ago
  I think the OP is suggesting using a caching layer at HTTP output, and suggesting CF as an option (a quick/cheap one). If you have an axe to grind with CF you can take it up with them, but it’s an option. Feel free to suggest others.
  - LilBytes 1y ago
    It seems to me that the writer, Herman, doesn't want to use the cloud short of the parts that are almost mandatory these days (e.g., CloudFlare, or a CDN of some sort). GitHub Pages, CloudFlare pages etc are a great and very simple service. But they're opposite or contrary to running your own hardware, warts and all. Herman wasn't looking for solutions IMO, I read it more as him lamenting at how hostile and insidious the internet has become. It has been for some time, but it seems to be getting exponentially worse.
    - uvaursi 1y ago
      Remember when Stormfront was a thing? Remember how everyone cheered on that CF and others banned them? I’m sure Herman was one of the loudest ones to proclaim victory that day. Hostility and insidiousness were created by you for not standing up when it was most needed and called for. And as you can see, Stormfront in hindsight was the most milquetoast website compared to landscape of politics today. All I’m saying is CF is a viable caching option. But if you’re looking for morality support - you lost that war a long time ago. “If we do not stand up for the rights of the accused, we endanger our own. For when the tide turns, who will stand up for us?”

deleted 1y ago

[deleted]

cupofjoakim 1y ago

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots use tools like NobleTLS or JA3Cloak to go around ja3 rate limiting 5. Some valid LLM companies seem to also follow the above to gather training data. We want them to know about our company, so we don't necessarily want to block them I'm close to giving up on this front tbh. There's no longer safe methods of identifying malignant traffic at scale, and with the variations we have available we can't statically generate these. Even with a CDN cache (shoutout fastly) our catalog is simply too broad to fully saturate the cache while still allowing pages to be updated in a timely manner. I guess the solution is to just scale up the origin servers... /shrug In all seriousness, i'd love if we somehow could tell the bots about more efficient ways of fetching the data. Use our open api for fetching book informations instead of causing all that overhead by going to marketing pages please.

FeepingCreature 1y ago
In principle, it should be possible to identify malign IPs at scale by using a central service and reporting IPs probabilistically. That is, if you report every thousandth page hit with a simple UDP packet, the central tracker gets very low load and still enough data to publish a bloom filter of abusive IPs, say a million bits gives you pretty low false-positive. (If it's only ~10k malign IPs, tbh you can just keep a lru counter and enumerate all of them.) A billion hits per hour across the tracked sites would still only correspond to ~50KB/s inflow on the tracker service. Any individual participating site doesn't necessarily get many hits per source IP, but aggregating across a few dozen should highlight the bad actors. Then the clients just pull the bloom filter once an hour (80KB download) and drop requests that match. Any halfway modern LLM could probably code the backend for this in a day or two and it'd run on a RasPi. Some org just has to take charge and provide the infra and advertisement.
- 01HNNWZ0MV43FF 1y ago

2OEH8eoCRo0 1y ago

Why don't we sue the abusive scrapers? Scraping is legal but DDoSing is not!

cupofjoakim 1y ago
Not sure if that's satire or not but how would you even identify the party to sue? What do you do if they're based in a country where you can't sue them ofer relatively trivial matters as this?
- 2OEH8eoCRo0 1y ago
  Not satire but it's a huge problem with the internet. Everyone washes their hands and people can harm you without liability.
- immibis 1y ago
  You could start by subpoenaing the registered holder of the IP address. In general, you can ask a lawyer for your options. Chances are good there are more than zero. But only if you can afford a lawyer. If you're getting scraped from a country where you don't do business, you can block the country. It's not good to block a country, but it works as a temporary measure. If they really want your data, they will move to a country where you do business, which means a country where you can get a lawyer. Assuming you can afford a lawyer. If they're using rotating IPs, likely some of them are from your country. You might show a judge: "Hey, look, we're getting so much traffic, from a wide variety of IP addresses but it all seems to be the same person on the other end, which would make it illegal DDoS. Can we trace back some of these?" and if you're lucky, the judge might say yes. DDoS is not a relatively trivial matter.

ItsBob 1y ago

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger, but bots and the like will often read robots.txt (at least to figure out what you have... they'll ignore the "deny" for the most part!) and if they try and go to that fake endpoint you can be 100% sure (well, as close as possible) that it's not a human and you can ban them. So I tried that. I auto-generated a robots.txt file on the fly. It was cached for 60 seconds or so as I didn't want to expend too many resource on it. When you asked for it, you either got the cached one or I created a new one. The CPU-usage was negligible. However, I changed the "deny" endpoint each time I built the file in case the baddies cached it, however, it still went to the same ASP.NET controller method. By hitting it, I sent a 10GB zip bomb and your IP was automatically added to the FW block list. It was quite simple: anyone that hit that endpoint MUST be dodgy... I believe I even had comments for the humans that stumbled across it letting them know that if they went to this endpoint in their browser it was an automatic addition to the firewall blocklist. Anyway... at first I caught a shit load of bad guys. There were thousands at first and then the numbers dropped and dropped to only tens per day. Anyway, this is a single data point but for me, it worked... I have no regrets about the zip bomb either :) I have another site that I'm working on so I may evolve it a bit so that you are banned for a short time and if you come back to the dodgy endpoint then I know you're a bot so into the abyss with you! It's not perfect but it worked for me anyway.

immibis 1y ago
It's interesting to study, right? This is the Internet equivalent of background radiation. Harmless in most cases. Exploit scanners aren't new to the LLM age and shouldn't overload your server - unless you're vulnerable to the exploit. Fun fact: Some people learn about new exploits by watching their incoming requests.
- ItsBob 1y ago
  > It's interesting to study, right? Definitely! I wasn't experiencing any issues, hell it wasn't even for public consumption at that time so no great loss to me but I found a few things fascinating (and somewhat stupid!) about it: 1. The sheer number of automated requests to scrape my content 2. That a massive number of the bots openly had "bot" or some derivative in the user agent and they were accessing a page I'd explicitly denied! :D 3. That an equally large number were faking their user agents to look like regular users and still hitting a page that a regular user couldn't possibly ever hit! Something I did notice but it was towards the end and I didn't pursue it (I should log it better the next time for analysis!) was that the endpoint was dynamically generated and only existed in the robots.txt for a short time but there were bots I caught later on, long after that auto-generated page was created (and after the IP was banned) that still went for that same page: clearly the same entities! My spidey senses are tingling. Next time, I'm going to log the shit out of these requests and publish as much as I can for others to analyse and dissect... might be interesting.

kwa32 1y ago

crazy how scraping became an industry

npteljes 1y ago
It's wild. Data is very valuable. This manifests in two fronts simultaneously: who has the data controls heavily on who sees it and under what circumstances, and on the other side, they scrape it as hard as they can.

r_singh 1y ago

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

intended 1y ago
What ? What do you mean ?
- georgefrowny 1y ago
  To be fair the heyday of unshit search was driven by mostly-consensual scraping. Today there are far too many people scraping stuff that isn't intended to be scraped, for profit, and doing it in a heavy-handed way that actually does have a negative and continuous effect on the victim's capacity. Everyone from AI services too lazy or otherwise unwilling to cache to companies exfiltrating some kind of data for their own commercial purposes.
  - r_singh 1y ago
    With peering bandwidth being freely distributed to ISPs and consumers being fed media and subsidised services up until their necks makes the counter argument smell of narrative control rather than technical or financial constraints But as I’m growing older I’m learning that the tech industry is mostly politically driven and relies on truth obfuscation as explained by Peter Thiel rather than real empowerment It’s facilitating accumulation of control and power at an unparalleled pace. If anything it’s proving to be more unjust than the feudal systems it promises to replace.
    - r_singh 1y ago
      I may have been too harsh. I love capitalism, technology, and software—they’ve built a meritocratic world and given me the tools to build my own life. AI and technology feel like my best friend, but also my worst enemy when they edge toward learned helplessness. That tension exists with anything we depend on: the closer we get, the more power it holds. The relationship between user and technology is becoming deeply intimate as systems gain reach and control. It’s important to stay optimistic but skeptical—and to keep protesting everything—because the work is moving faster than our ability to register its consequences. Reading back, I realise I drifted into more of a monologue than a conversation. I get carried away when I’m trying to reason things out in public. Still, I stand by the core point about balance and transparency in how we shape the web.

uvaursi 1y ago

Do we shift over everything to le Dark Web and let the corpos use this one for selling their shit to consumers? These toys don’t want to play nice and there’s no real way to stop them without bringing in things like Real ID and other verifications that infringe on anonymity.

Chabsff 1y ago
The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.
- uvaursi 1y ago
  We can set different rules on these networks however. We can choose to be choosy at the gate.
  - Cthulhu_ 1y ago
    How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.
    - embedding-shape 1y ago
      Make it P2P and content-based, instead of location-based like the current web. Content could be served from anywhere, so DDoS stops being an effective method, and shared peer quality could propagate across the network to ban bad actors quickly. I spend about 30 seconds thinking about this, so this is clearly the perfect solution with zero drawbacks or tradeoffs.
      - Chabsff 1y ago
        I know this is tongue in cheek, but I'll give it a serious reply in case someone finds themselves "inspired". A CDN. What you are describing is a CDN. We have CDNs today and the problem still exists because most of today's websites refuse to operate within the constraints. There is no need for new infrastructure to deploy this solution, we just need website operators to "give up" and operate in a more static way.

y-zon128 1y ago

> Auto-restart the reverse-proxy if bandwidth usage drops to zero for more than 2 minutes It's understandable in your case as you have traffic coming in constantly, but first thing that came to my mind is a loop of contant reboots - again, very unlikely in your case. Sometimes such blanket rules hit me due to most unexpected reasons, like the proxy somehow failed to start serving traffic in the given timeframe. Though I completely appreciate and agree with the 'ship now something that works now' approach!

marginalia_nu 1y ago
Open port 80 and log the results. Even if you don't announce your service anywhere the baseline request rate is way above 2 rpm. Every open port of every IP is continuously scanned for exploits.

Havoc 1y ago

> monetise their apps by offering them for free, and selling tunnel access to scrapers. I bet it’s free VPN apps