9 ms·

I have a project I want to do that would involve a lot of YouTube scraping. But they'll throttle you if they detect it. This led me down the rabbit hole into th

I have a project I want to do that would involve a lot of YouTube scraping. But they'll throttle you if they detect it. This led me down the rabbit hole into the world of residential proxy services. Some of them[0] advertise up to 40 million IPs. I can't imagine many of the owners of those IPs know what they're being used for. It would be cool if there was a reputable open source project that would let people share/buy residential proxy usage, but at the end of the day there's no way to guarantee people aren't doing horrible things with your IP. [0]: https://smartproxy.com/ https://smartproxy.com/

oefrha 6y ago
> I can't imagine many of the owners of those IPs know what they're being used for. They don’t. For instance, Luminati, possibly the best known player in this market, uses HolaVPN users as exit nodes.[1] [1] https://www.trendmicro.com/vinfo/hk-en/security/news/cybercrime-and-digital-threats/shining-a-light-on-the-risks-of-holavpn-and-luminati https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...
- anderspitman 6y ago
  Thanks for the link. I figured something like that had to be the case. I can't really think of a legitimate way to get access to that many IPs.
  - MattGaiser 6y ago
    I periodically see ads on Kijiji (Craigslist for Canada) offering you 50 bucks for your internet connection. If that is the going rate, 50 million I enormously expensive.
    - ev1 6y ago
      The boxes have the ability to try to DHCP-release-renew your IP to get a new one after being banned for spam/fraud/DDoS/etc. However, this is not how these residential proxies are sold- mass market ones with millions of IPs are generally botnet/hacked/malware/extension malicious author sellout/etc./ There are not actually millions of botnet user, they just try to make themselves look big. Like they say AT&T has a /8? So they get 1 botnet hacked PC on AT&T broadband and say "wow we have a /8 worth of proxies!!"). Criminals not known for their truthfulness. The ones that ask you $20-50/m and to plug something into your network is generally used for fraud, account hijackings and maintenance (getting a stable IP for stuff like a credit card checkout or paypal account to use for more than a few days for fraud, long term social media abuse, similar) instead of mass market ban evasion.
- dessant 6y ago
  I was approached by Luminati on Twitter to turn my browser extensions into exit nodes, they are enticing developers to exploit users. https://i.imgur.com/EbT96an.png https://i.imgur.com/EbT96an.png
  - oefrha 6y ago
    This is why these days I roll my own extensions except for a few extremely popular ones developed in the open. Forget small QoL extensions, if I need it I write my own. Malware businesses routinely reach out to any remotely popular extension for acquisitions or “joint ventures”, it’s just too risky to install anything that’s not trust: ultimate, or known to be vetted by tons of people. Btw, Firefox seems to have made it impossible to run extensions uncrippled from source without uploading to their server for signing. Utter madness.
    - halfbaked99 6y ago
      I was wondering why my extensions weren't loading up even if after I messed with the settings, thank you!
  - nashashmi 6y ago
    Auto-update of extensions and apps should be set to off by default. There would be less reason for extensions to be bought by other developers for their installed user base. Secondly, how come there is no regulation about selling chrome extensions and apps?
    - acdha 6y ago
      If you turn auto-updates off, you're just reinventing early 2000s desktop security and attackers everywhere will thank you. Updates need to be automatic but there needs to be stricter rules about notifications to users about changes in ownership or functionality. A huge fraction of this would go away if corporate liability didn't allow hiding all of this in terms of service.
- bscphil 6y ago
  This is said way too often, but in this case I think it's justified - I really don't understand how this is legal. Doesn't using a non-consenting person's computer and network resources for your own purposes meet the definition of "hacking" in most jurisdictions?
  - ev1 6y ago
    I am an administrator on a forum that was attacked heavily by these "residential proxy services"; most of the time the users have agreed to a hidden EULA somewhere like at installation time of the extension buried deep within. But almost everyone "a reasonable person" would not recognise or understand the technical words used in the disclaimer, nor even read all the way down where it's buried several links in.
  - dspillett 6y ago
    It probably isn't legal. But who is going to pay for the fight to stop it? Mom & pop who have had their home connection used? A class action that no one but the lawyers really win anything from? The fact that most of the projects involved are in jurisdictions that make the legal process more complicated and more expensive, or jurisdictions that simply don't care as long as it is affecting other countries and not them (coughChinacoughRussiacoughcough*)? What the market will bare doesn't just apply to product pricing. And someone clicking through a EULA might be an argument for them having invited the use of their resources by a very similar argument to that many used to justify using other people's WiFi back when that was gloriously insecure most of the time ("but the AP broadcast its existence, effectively inviting me to join, and happily gave me an IP address to work with, how could I not believe I was welcome to use it?").
- fomine3 6y ago
  I've never heard such service exists and uses browser extension user's network. It looks horrible. Possibly random browser extension user may arrested by unknown reason.
- fowl2 6y ago
  On the other hand, if these things get common enough it's great for plausible deniability!
dylz 6y ago
These are split into either unknowing people with their computers abused like Hola, malware running proxies being sold, or otherwise. Some mobile games and chrome extensions expose proxies also as a monetisation method. Don't support them. I've been contacted by several of those "SDK companies" that basically turn phones and browsers into botnet drones for $0.01 an install.
- anderspitman 6y ago
  Do you know of any reputable ones with a sustainable source of IPs that can be used for scraping?
  - dylz 6y ago
    There are no reputable ones; they are (>99,99%+ probably) all based off of unwanted adware or malware/botnet exploit installation. There are some fake ones that also commit wire fraud, fraudulent letters of authorisation and do BGP hijacking, which is arguably significantly worse.
  - rdbell 6y ago
    https://packetstream.io https://packetstream.io is 100% opt-in and transparent
core-questions 6y ago
SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to detect people abusing their service.
- samcolvin 6y ago
  I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP. But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.
  - Fnoord 6y ago
    How does it wotk out for IPv6?
    - FreshFries 6y ago
      Complete /64's get blocked.
    - core-questions 6y ago
      IPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.
      - p1mrx 6y ago
        At what point would you consider IPv6 "widely used"? It's currently 30% of traffic: https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html
        core-questions 6y ago
        I bet you half of that is scrapers!
deleted 6y ago
[deleted]
arsome 6y ago
Do you need residential IPs for this? As far as I've experienced, YouTube doesn't block dedicated server providers, as such you may be able to use a commercial VPN provider, some of them have 1000s of IPs and offer easy or automated ways to switch frequently.

oefrha 6y ago

> I can't imagine many of the owners of those IPs know what they're being used for. They don’t. For instance, Luminati, possibly the best known player in this market, uses HolaVPN users as exit nodes.[1] [1] https://www.trendmicro.com/vinfo/hk-en/security/news/cybercrime-and-digital-threats/shining-a-light-on-the-risks-of-holavpn-and-luminati https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...

anderspitman 6y ago
Thanks for the link. I figured something like that had to be the case. I can't really think of a legitimate way to get access to that many IPs.
- MattGaiser 6y ago
  I periodically see ads on Kijiji (Craigslist for Canada) offering you 50 bucks for your internet connection. If that is the going rate, 50 million I enormously expensive.
  - ev1 6y ago
    The boxes have the ability to try to DHCP-release-renew your IP to get a new one after being banned for spam/fraud/DDoS/etc. However, this is not how these residential proxies are sold- mass market ones with millions of IPs are generally botnet/hacked/malware/extension malicious author sellout/etc./ There are not actually millions of botnet user, they just try to make themselves look big. Like they say AT&T has a /8? So they get 1 botnet hacked PC on AT&T broadband and say "wow we have a /8 worth of proxies!!"). Criminals not known for their truthfulness. The ones that ask you $20-50/m and to plug something into your network is generally used for fraud, account hijackings and maintenance (getting a stable IP for stuff like a credit card checkout or paypal account to use for more than a few days for fraud, long term social media abuse, similar) instead of mass market ban evasion.
dessant 6y ago
I was approached by Luminati on Twitter to turn my browser extensions into exit nodes, they are enticing developers to exploit users. https://i.imgur.com/EbT96an.png https://i.imgur.com/EbT96an.png
- oefrha 6y ago
  This is why these days I roll my own extensions except for a few extremely popular ones developed in the open. Forget small QoL extensions, if I need it I write my own. Malware businesses routinely reach out to any remotely popular extension for acquisitions or “joint ventures”, it’s just too risky to install anything that’s not trust: ultimate, or known to be vetted by tons of people. Btw, Firefox seems to have made it impossible to run extensions uncrippled from source without uploading to their server for signing. Utter madness.
  - halfbaked99 6y ago
    I was wondering why my extensions weren't loading up even if after I messed with the settings, thank you!
- nashashmi 6y ago
  Auto-update of extensions and apps should be set to off by default. There would be less reason for extensions to be bought by other developers for their installed user base. Secondly, how come there is no regulation about selling chrome extensions and apps?
  - acdha 6y ago
    If you turn auto-updates off, you're just reinventing early 2000s desktop security and attackers everywhere will thank you. Updates need to be automatic but there needs to be stricter rules about notifications to users about changes in ownership or functionality. A huge fraction of this would go away if corporate liability didn't allow hiding all of this in terms of service.
bscphil 6y ago
This is said way too often, but in this case I think it's justified - I really don't understand how this is legal. Doesn't using a non-consenting person's computer and network resources for your own purposes meet the definition of "hacking" in most jurisdictions?
- ev1 6y ago
  I am an administrator on a forum that was attacked heavily by these "residential proxy services"; most of the time the users have agreed to a hidden EULA somewhere like at installation time of the extension buried deep within. But almost everyone "a reasonable person" would not recognise or understand the technical words used in the disclaimer, nor even read all the way down where it's buried several links in.
- dspillett 6y ago
  It probably isn't legal. But who is going to pay for the fight to stop it? Mom & pop who have had their home connection used? A class action that no one but the lawyers really win anything from? The fact that most of the projects involved are in jurisdictions that make the legal process more complicated and more expensive, or jurisdictions that simply don't care as long as it is affecting other countries and not them (coughChinacoughRussiacoughcough*)? What the market will bare doesn't just apply to product pricing. And someone clicking through a EULA might be an argument for them having invited the use of their resources by a very similar argument to that many used to justify using other people's WiFi back when that was gloriously insecure most of the time ("but the AP broadcast its existence, effectively inviting me to join, and happily gave me an IP address to work with, how could I not believe I was welcome to use it?").
fomine3 6y ago
I've never heard such service exists and uses browser extension user's network. It looks horrible. Possibly random browser extension user may arrested by unknown reason.
fowl2 6y ago
On the other hand, if these things get common enough it's great for plausible deniability!

dylz 6y ago

These are split into either unknowing people with their computers abused like Hola, malware running proxies being sold, or otherwise. Some mobile games and chrome extensions expose proxies also as a monetisation method. Don't support them. I've been contacted by several of those "SDK companies" that basically turn phones and browsers into botnet drones for $0.01 an install.

anderspitman 6y ago
Do you know of any reputable ones with a sustainable source of IPs that can be used for scraping?
- dylz 6y ago
  There are no reputable ones; they are (>99,99%+ probably) all based off of unwanted adware or malware/botnet exploit installation. There are some fake ones that also commit wire fraud, fraudulent letters of authorisation and do BGP hijacking, which is arguably significantly worse.
- rdbell 6y ago
  https://packetstream.io https://packetstream.io is 100% opt-in and transparent

core-questions 6y ago

SEO companies are big customers of products like this. Scraping Google at scale requires a huge budget for proxies and IP blocks, and I believe they apply ML to detect people abusing their service.

samcolvin 6y ago
I can confirm that, scraping Google instantly needs huge effort and money. In our best we can scrape 2500 SERP per IP. But i must say using proxy services and other things did not helped us much. Because most of them were banned before we use.
- Fnoord 6y ago
  How does it wotk out for IPv6?
  - FreshFries 6y ago
    Complete /64's get blocked.
  - core-questions 6y ago
    IPv6 is just not widely used, so when you do use it, you stick out like a sore thumb. Think like a bayesian: for Google, it's easy to just block whole /32s of IPv6 space.
    - p1mrx 6y ago
      At what point would you consider IPv6 "widely used"? It's currently 30% of traffic: https://www.google.com/intl/en/ipv6/statistics.html https://www.google.com/intl/en/ipv6/statistics.html
      - core-questions 6y ago
        I bet you half of that is scrapers!

deleted 6y ago

[deleted]

arsome 6y ago

Do you need residential IPs for this? As far as I've experienced, YouTube doesn't block dedicated server providers, as such you may be able to use a commercial VPN provider, some of them have 1000s of IPs and offer easy or automated ways to switch frequently.