6 ms·

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnel

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed, and potentially horrific for the owners of the affected devices also! Any corroboration for that out there?

VladVladikoff 1y ago
This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.
- dataviz1000 1y ago
  They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
  - VladVladikoff 1y ago
    WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?
    - arethuza 1y ago
      That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?
      - alamortsubite 1y ago
        My take is it's mostly irrelevant, but read the lobsters post mentioned elsewhere.
    - alamortsubite 1y ago
      Yes, and it doesn't matter if they do read the terms- to the average user they sound totally innocuous, especially placed next to a big shiny "GET 500 FREE COINS" button.
  - arethuza 1y ago
    I suspect most people, even when told exactly what the app using that SDK would be doing, wouldn't actually see the potential problems...
    - kijin 1y ago
      Until one day, they get swatted for accessing child porn. Actually, that might be one way to draw attention to the problem. Sign up to some of these shady "residential proxy" services, and access all sorts of nasty stuff through their IPs until your favorite three-letter agency takes notice.
  - TYPE_FASTER 1y ago
    Also see https://www.youtube.com/watch?v=AGaiVApKfmc https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worker asking for an Apple gift card. The name was indeed an employee from a different part of the large company I worked for at the time, but LinkedIn was the only possible link I could figure out that was at least somewhat publicly available information. This should probably be required in all CS curriculum: https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-responsibilities-of-computing-serc/ https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-r...
    - nerdponx 1y ago
      It should be illegal, but this stuff is propping up the appearance of a healthy economy so nobody will touch it.
    - VladVladikoff 1y ago
      That scam definitely uses linked in as the source. We get a lot of those BEC emails and it’s always the people who are on LinkedIn. Also keep in mind LinkedIn has had big database leaks in the past, you might not even need to scrape them, just download a huge database from a leaks site.
  - cuu508 1y ago
    IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.
    - VladVladikoff 1y ago
      I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.
      - deleted 1y ago
        [deleted]
  - seemaze 1y ago
    That's sleazy. It's slipping drugs into a kids lunchbox and letting smuggle it across the border..
- myaccountonhn 1y ago
  One such example is brightdata, on lobsters someone did a writeup https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don_t_like_me#c_hzq0yp https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don...
  - VladVladikoff 1y ago
    Never heard of lobsters before. Cool site. Seems to be invite only though :( If you could share an invite that would be cool. torosanchez@protonmail.me Thanks!
- corbet 1y ago
  The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.
  - nurettin 1y ago
    I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy. They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoever is paying, and they don't care what people do in there.
    - yomismoaqui 1y ago
      > ... until I figured out a way to avoid the ban hammer ... You had my curiosity ... but now you have my attention.
      - walletdrainer 1y ago
        Without bothering to check on Amazon, I successfully scraped meta stuff for years at rates exceeding 20gbit/s without any proxies but just rotating IPv6 addresses on the same couple of blocks for every request There are usually silly bypasses like this that easily work even with bigco stuff
    - chritsopherh 11mo ago
      [dead]
- wat10000 1y ago
  Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to forbid any nefarious activity, and naturally they don't actually care.
  - nemomarx 1y ago
    Salad, right? What a strange business
    - dylan604 1y ago
      Why is it strange. Of course it exists.
curious_curios 1y ago
If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.
- antoniojtorres 1y ago
  This is how so many companies sell from an opaque inventory of “millions” of residential proxies.
kaoD 1y ago
There's crap like https://hola.org/ https://hola.org/ https://hola.org/legal/sdk https://hola.org/legal/sdk https://hola.org/legal/sla https://hola.org/legal/sla > How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula https://bright-sdk.com/eula). You may opt out by becoming a Premium user. This "VPN" is what powers these residential proxies: https://brightdata.com/ https://brightdata.com/ I'm sure there are many other companies like this.
- piggg 1y ago
  There's also a ton of companies selling "make money off your unused internet" apps which are all over tiktok and basically turn yourself into a residential proxy/sketch VPN egress node. On top of that - lots of free tv/movie streaming stuff that also makes yourself a proxy/egress node. Sometimes you find it on tv/movie streaming devices sold online where it's already loaded on when it arrives.
Zanfa 1y ago
SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.
- Cthulhu_ 1y ago
  Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection. edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, are metrics going up and metrics going brrt means profit and shareholder interest.
  - ac29 1y ago
    Its very possible they did detect it and that's why law enforcement got involved. But yes, they also might not care if they are getting paid. If the SIMs are only being used for voice/text as I suspect, it might have very minimal load on the network.
immibis 1y ago
You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).
- VBprogrammer 1y ago
  If someone hasn't written a blog titled "Should we be worried about Cloudflare?" yet, I think it would be a good subject to explore. I find the idea that they could decide one day to ban you from all of their network pretty worrying. And if they did, how much fingerprinting are they doing and would the bad extend far beyond just a random IP address.
- pjc50 1y ago
  This is one of those "ACAB" things where you might reasonably dislike Cloudflare but a world without them or an equivalent will evolve worse solutions to the same problems, which you will like even less.
- nix0n 1y ago
  > This is not illegal Depends on what they're doing from your connection.
  - immibis 1y ago
    Strict liability by IP address is not the norm, not even in Germany any more. It's not illegal to have a botnet infect your computer either. Since they promise not to use your connection for illegal things, it's their fault if they break that.
lucastech 1y ago
I wrote about this back in July when this "gang" first started hitting some sites I host: https://wxp.io/blog/the-bots-that-keep-on-giving https://wxp.io/blog/the-bots-that-keep-on-giving they use a mixture of colo (M247, Datacamp, HostRoyale, Oxylabs, etc) and international residential. I suspect the latter are where those residential app proxies come into play (bright SDK, etc). Oxylabs is also a well known proxy provider, which makes me think they're the gateway into all of these IPs. Definitely interesting times to try and host a web server!

VladVladikoff 1y ago

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.

dataviz1000 1y ago
They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
- VladVladikoff 1y ago
  WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?
  - arethuza 1y ago
    That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?
    - alamortsubite 1y ago
      My take is it's mostly irrelevant, but read the lobsters post mentioned elsewhere.
  - alamortsubite 1y ago
    Yes, and it doesn't matter if they do read the terms- to the average user they sound totally innocuous, especially placed next to a big shiny "GET 500 FREE COINS" button.
- arethuza 1y ago
  I suspect most people, even when told exactly what the app using that SDK would be doing, wouldn't actually see the potential problems...
  - kijin 1y ago
    Until one day, they get swatted for accessing child porn. Actually, that might be one way to draw attention to the problem. Sign up to some of these shady "residential proxy" services, and access all sorts of nasty stuff through their IPs until your favorite three-letter agency takes notice.
- TYPE_FASTER 1y ago
  Also see https://www.youtube.com/watch?v=AGaiVApKfmc https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they? Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets https://brightdata.com/lp/web-data/datasets This actually explains a phishing attack where I received a text from somebody purporting to be a co-worker asking for an Apple gift card. The name was indeed an employee from a different part of the large company I worked for at the time, but LinkedIn was the only possible link I could figure out that was at least somewhat publicly available information. This should probably be required in all CS curriculum: https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-responsibilities-of-computing-serc/ https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-r...
  - nerdponx 1y ago
    It should be illegal, but this stuff is propping up the appearance of a healthy economy so nobody will touch it.
  - VladVladikoff 1y ago
    That scam definitely uses linked in as the source. We get a lot of those BEC emails and it’s always the people who are on LinkedIn. Also keep in mind LinkedIn has had big database leaks in the past, you might not even need to scrape them, just download a huge database from a leaks site.
- cuu508 1y ago
  IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.
  - VladVladikoff 1y ago
    I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.
    - deleted 1y ago
      [deleted]
- seemaze 1y ago
  That's sleazy. It's slipping drugs into a kids lunchbox and letting smuggle it across the border..
myaccountonhn 1y ago
One such example is brightdata, on lobsters someone did a writeup https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don_t_like_me#c_hzq0yp https://lobste.rs/s/pmfuza/bro_ban_me_at_ip_level_if_you_don...
- VladVladikoff 1y ago
  Never heard of lobsters before. Cool site. Seems to be invite only though :( If you could share an invite that would be cool. torosanchez@protonmail.me Thanks!
corbet 1y ago
The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.
- nurettin 1y ago
  I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy. They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoever is paying, and they don't care what people do in there.
  - yomismoaqui 1y ago
    > ... until I figured out a way to avoid the ban hammer ... You had my curiosity ... but now you have my attention.
    - walletdrainer 1y ago
      Without bothering to check on Amazon, I successfully scraped meta stuff for years at rates exceeding 20gbit/s without any proxies but just rotating IPv6 addresses on the same couple of blocks for every request There are usually silly bypasses like this that easily work even with bigco stuff
  - chritsopherh 11mo ago
    [dead]
wat10000 1y ago
Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to forbid any nefarious activity, and naturally they don't actually care.
- nemomarx 1y ago
  Salad, right? What a strange business
  - dylan604 1y ago
    Why is it strange. Of course it exists.

curious_curios 1y ago

If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.

antoniojtorres 1y ago
This is how so many companies sell from an opaque inventory of “millions” of residential proxies.

kaoD 1y ago

There's crap like https://hola.org/ https://hola.org/ https://hola.org/legal/sdk https://hola.org/legal/sdk https://hola.org/legal/sla https://hola.org/legal/sla > How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula https://bright-sdk.com/eula). You may opt out by becoming a Premium user. This "VPN" is what powers these residential proxies: https://brightdata.com/ https://brightdata.com/ I'm sure there are many other companies like this.

piggg 1y ago
There's also a ton of companies selling "make money off your unused internet" apps which are all over tiktok and basically turn yourself into a residential proxy/sketch VPN egress node. On top of that - lots of free tv/movie streaming stuff that also makes yourself a proxy/egress node. Sometimes you find it on tv/movie streaming devices sold online where it's already loaded on when it arrives.

Zanfa 1y ago

SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.

Cthulhu_ 1y ago
Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection. edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, are metrics going up and metrics going brrt means profit and shareholder interest.
- ac29 1y ago
  Its very possible they did detect it and that's why law enforcement got involved. But yes, they also might not care if they are getting paid. If the SIMs are only being used for voice/text as I suspect, it might have very minimal load on the network.

immibis 1y ago

You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).

VBprogrammer 1y ago
If someone hasn't written a blog titled "Should we be worried about Cloudflare?" yet, I think it would be a good subject to explore. I find the idea that they could decide one day to ban you from all of their network pretty worrying. And if they did, how much fingerprinting are they doing and would the bad extend far beyond just a random IP address.
pjc50 1y ago
This is one of those "ACAB" things where you might reasonably dislike Cloudflare but a world without them or an equivalent will evolve worse solutions to the same problems, which you will like even less.
nix0n 1y ago
> This is not illegal Depends on what they're doing from your connection.
- immibis 1y ago
  Strict liability by IP address is not the norm, not even in Germany any more. It's not illegal to have a botnet infect your computer either. Since they promise not to use your connection for illegal things, it's their fault if they break that.

lucastech 1y ago

I wrote about this back in July when this "gang" first started hitting some sites I host: https://wxp.io/blog/the-bots-that-keep-on-giving https://wxp.io/blog/the-bots-that-keep-on-giving they use a mixture of colo (M247, Datacamp, HostRoyale, Oxylabs, etc) and international residential. I suspect the latter are where those residential app proxies come into play (bright SDK, etc). Oxylabs is also a well known proxy provider, which makes me think they're the gateway into all of these IPs. Definitely interesting times to try and host a web server!