5 ms·

I don't think it's a cloud. It's more likely a residential proxy network, which are typically created by installing malware on users' machines. The operators o

by jsnell 2y ago

I don't think it's a cloud. It's more likely a residential proxy network, which are typically created by installing malware on users' machines. The operators of these proxy networks want to avoid detection by both the users whose bandwidth they're stealing, and by the companies whose data is being scraped. So they want to make the bandwidth very expensive. And that expensive bandwidth in turn means that their only clients are dodgy as well. Either people looking to scrape data without consent and monetize it, or outright criminals.

bscphil 2y ago
It's kind of surprising that a presumptively legitimate company (and YC-funded startup) would out themselves as buying black market residential proxy bandwidth, isn't it?
- jsheard 2y ago
  Their frontpage also advertises the ability to pass CAPTCHAs, whether by automation or more likely by delegating them to third-world CAPTCHA farms. If that's a major selling point for your automation service then your target market probably ranges from dubious (e.g. data scrapers trying to get around limits) to extremely dubious (e.g. ticket scalpers, spammers, click fraud, etc).
  - xp84 2y ago
    Just because something can be used for sketchy purposes doesn't mean that's the only purpose of it. there are thousands of situations where people are forced to interact with a shitty website 100x per day and the site won't provide an api. Imagine if your job was booking plane tickets all day. United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen.
    - dontlikeyoueith 2y ago
      > United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen. And you think that's NOT sketchy? I'm almost afraid to ask where you think the bar is...
      - stickfigure 2y ago
        It's exactly as sketchy as having a hypothetical robot sit down at a console and type it out. Which, IMO, is not very sketchy at all.
      - wruza 2y ago
        And why is it? A company provides you an API for a "fee" and a free web-based interface, as long as you are agile enough to use it, with some limitations per ip/cookie. You choose the second path and automate it. What's wrong with that? Limits of the free access are the public contract. You're not obliged to play along with someone's "monetary spirit". And in practice, APIs are often much more PITA than the actual interface, but you can't buy unlimited web automation. Few years ago one of my projects literally OCRed data from an android phone screen because receiving it via API took a couple minutes longer and involved email-like back and forth with polling and id matching after a convoluted authentication that fails every few requests.
        xp84 2y ago
        I really wish I was a better programmer with more time, I would install the accursed "MyQ" garage door app on a dedicated Android, and bridge it into Home Assistant using an OCR type of strategy. (they are notorious for flipping the bird to the whole open home automation community by not integrating with anything)
    - rty32 2y ago
      Imagine a legitimate travel agency cannot book 100 United tickets a day via methods outlined in business contracts and need to resort to shady practice. Dude, please provide some real solid evidence to back this up, and perhaps come up with another realistic scenario where bypassing captcha is justified.
- mrguyorama 2y ago
  How long have you been here? It's not surprising at all. HN and YC have not demonstrated an aversion to "uh, greyhat" activity. If it were 2000, people would be sharing their ad clicking startups. YC has funded a looooooot of sketchy companies.
- dewey 2y ago
  Residential proxies are not necessarily "black market".
  - _cenw 2y ago
    It's almost never done with the full understanding of the person providing the proxy, doesn't matter if they get promised some change, their browser addons betray them or they install bundleware/adware. I'd say it has about the same moral standing as a payday loan.
    - dewey 2y ago
      There’s other ways for example through mislabeled “residential” blocks, or “residential” proxies that are sold by ISPs to vendors.
    - bigfatkitten 2y ago
      The look of surprise on their faces is almost universal when the feds knock on the door with a search warrant in relation to something that came from their IP address.
iforgotpassword 2y ago
I use one. I run a bot on IRC that extracts the <title> of every link posted (or downloads the image/whatever and extracts Metadata) and announces that to the channel. It has become more and more pointless to run this on a vps. Google/YouTube block the IP range, a lot of websites return the cloudflare security check, Amazon works on some days and doesn't on others... Ever since I proxy via residential proxies it just works. I'm a smooth criminal. :>
- morkalork 2y ago
  So much for the open internet.
  - nolist_policy 2y ago
    You can thank the spammers.
    - Nextgrid 2y ago
      I’m not sure how much of this is due to spammers and how much is due to “growth & engagement” that wants to make sure a human’s time is being wasted.
    - wruza 2y ago
      To stop spammers, you implement measures before posting, not before viewing. Spam is just a minor technical nuisance. It's automated interaction that really makes their executives sweat and shiver.
    - stevenicr 2y ago
      and hackers.. install a fresh wordpress and you'll be blocking hundreds of Digital Ocean Ip addys within 24 hours.. Or you won't. If you add a failed login detector you'll see DO, OVH Hetzer and similar are non-stop password guessing your logins everyday. Maybe your admin password is so long they will never get in, maybe you don't care that your server gets slower responses for actual users as it serves pages and does DB lookups for all the bots trying to use a different password every minute. Maybe the website owner doesn't know any of these and the open internet is actually legoland where everything is awesome all the time.
floam 2y ago
It’s not necessarily malware. There are services that are pretty upfront and pay cash money for residential US bandwidth. That said, naive people might be surprised when their IP starts getting blocked. e.g. https://www.honeygain.com/ https://www.honeygain.com/ (something like 100GB = $20).
- Saris 2y ago
  >That said, naive people might be surprised when their IP starts getting blocked. Or law enforcement shows up at their door because their IP is involved in a bunch of illegal stuff.
peab 2y ago
how does expensive bandwidth equate to dodgy clients? There are lot's of valid use cases for scraping data, and it's legal to scrape publicly available data, even if the websites hosting it try to block it (try a curl request to reddit, for example)
- patmcc 2y ago
  >>>and it's legal to scrape publicly available data, even if the websites hosting it try to block it Is that something that's been fully decided? https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc. is the most relevant case I'm aware of, and it suggests it might actually be illegal (if you know you've been blocked, at least).
- suchintan 2y ago
  https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-web-scraper-bright-data-which-meta-had-used-and-then-sued/ https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-... This is another interesting example where it was allowed
morkalork 2y ago
Aren't there also some suspiciously cheap VPNs that do that in the background?
- ThePowerOfFuet 2y ago
  Yes
- duskwuff 2y ago
  An employee of one proxy network describes that exact business model here: https://news.ycombinator.com/item?id=41597315 https://news.ycombinator.com/item?id=41597315
hypeatei 2y ago
Yeah, the author confirmed it in this thread actually: https://news.ycombinator.com/item?id=41594713 https://news.ycombinator.com/item?id=41594713
miohtama 2y ago
Here more on "free VPNs” https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-services/51721/ https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s... Usually such proxy networks are outright criminal (even if users are not).
dewey 2y ago
There's many reputable residential proxy networks too, usually there's a lot of vetting involved too as they don't want people running illegal activities though their network. It's almost a necessity these days to have access to that due to how much datacenter ranges are blocked.

by jsnell 2y ago

bscphil 2y ago

It's kind of surprising that a presumptively legitimate company (and YC-funded startup) would out themselves as buying black market residential proxy bandwidth, isn't it?

jsheard 2y ago
Their frontpage also advertises the ability to pass CAPTCHAs, whether by automation or more likely by delegating them to third-world CAPTCHA farms. If that's a major selling point for your automation service then your target market probably ranges from dubious (e.g. data scrapers trying to get around limits) to extremely dubious (e.g. ticket scalpers, spammers, click fraud, etc).
- xp84 2y ago
  Just because something can be used for sketchy purposes doesn't mean that's the only purpose of it. there are thousands of situations where people are forced to interact with a shitty website 100x per day and the site won't provide an api. Imagine if your job was booking plane tickets all day. United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen.
  - dontlikeyoueith 2y ago
    > United could provide you an API key to do so via an API, but in practice they won't, only some enterprisey travel software company can get that kind of access, for a steep fee. You could build a tool which automatically puts together an itinerary based on rules and books it, through a tool like this. Perhaps a slightly contrived example but I believe things like this definitely happen. And you think that's NOT sketchy? I'm almost afraid to ask where you think the bar is...
    - stickfigure 2y ago
      It's exactly as sketchy as having a hypothetical robot sit down at a console and type it out. Which, IMO, is not very sketchy at all.
    - wruza 2y ago
      And why is it? A company provides you an API for a "fee" and a free web-based interface, as long as you are agile enough to use it, with some limitations per ip/cookie. You choose the second path and automate it. What's wrong with that? Limits of the free access are the public contract. You're not obliged to play along with someone's "monetary spirit". And in practice, APIs are often much more PITA than the actual interface, but you can't buy unlimited web automation. Few years ago one of my projects literally OCRed data from an android phone screen because receiving it via API took a couple minutes longer and involved email-like back and forth with polling and id matching after a convoluted authentication that fails every few requests.
      - xp84 2y ago
        I really wish I was a better programmer with more time, I would install the accursed "MyQ" garage door app on a dedicated Android, and bridge it into Home Assistant using an OCR type of strategy. (they are notorious for flipping the bird to the whole open home automation community by not integrating with anything)
  - rty32 2y ago
    Imagine a legitimate travel agency cannot book 100 United tickets a day via methods outlined in business contracts and need to resort to shady practice. Dude, please provide some real solid evidence to back this up, and perhaps come up with another realistic scenario where bypassing captcha is justified.
mrguyorama 2y ago
How long have you been here? It's not surprising at all. HN and YC have not demonstrated an aversion to "uh, greyhat" activity. If it were 2000, people would be sharing their ad clicking startups. YC has funded a looooooot of sketchy companies.
dewey 2y ago
Residential proxies are not necessarily "black market".
- _cenw 2y ago
  It's almost never done with the full understanding of the person providing the proxy, doesn't matter if they get promised some change, their browser addons betray them or they install bundleware/adware. I'd say it has about the same moral standing as a payday loan.
  - dewey 2y ago
    There’s other ways for example through mislabeled “residential” blocks, or “residential” proxies that are sold by ISPs to vendors.
  - bigfatkitten 2y ago
    The look of surprise on their faces is almost universal when the feds knock on the door with a search warrant in relation to something that came from their IP address.

iforgotpassword 2y ago

I use one. I run a bot on IRC that extracts the <title> of every link posted (or downloads the image/whatever and extracts Metadata) and announces that to the channel. It has become more and more pointless to run this on a vps. Google/YouTube block the IP range, a lot of websites return the cloudflare security check, Amazon works on some days and doesn't on others... Ever since I proxy via residential proxies it just works. I'm a smooth criminal. :>

morkalork 2y ago
So much for the open internet.
- nolist_policy 2y ago
  You can thank the spammers.
  - Nextgrid 2y ago
    I’m not sure how much of this is due to spammers and how much is due to “growth & engagement” that wants to make sure a human’s time is being wasted.
  - wruza 2y ago
    To stop spammers, you implement measures before posting, not before viewing. Spam is just a minor technical nuisance. It's automated interaction that really makes their executives sweat and shiver.
  - stevenicr 2y ago
    and hackers.. install a fresh wordpress and you'll be blocking hundreds of Digital Ocean Ip addys within 24 hours.. Or you won't. If you add a failed login detector you'll see DO, OVH Hetzer and similar are non-stop password guessing your logins everyday. Maybe your admin password is so long they will never get in, maybe you don't care that your server gets slower responses for actual users as it serves pages and does DB lookups for all the bots trying to use a different password every minute. Maybe the website owner doesn't know any of these and the open internet is actually legoland where everything is awesome all the time.

floam 2y ago

It’s not necessarily malware. There are services that are pretty upfront and pay cash money for residential US bandwidth. That said, naive people might be surprised when their IP starts getting blocked. e.g. https://www.honeygain.com/ https://www.honeygain.com/ (something like 100GB = $20).

Saris 2y ago
>That said, naive people might be surprised when their IP starts getting blocked. Or law enforcement shows up at their door because their IP is involved in a bunch of illegal stuff.

peab 2y ago

how does expensive bandwidth equate to dodgy clients? There are lot's of valid use cases for scraping data, and it's legal to scrape publicly available data, even if the websites hosting it try to block it (try a curl request to reddit, for example)

patmcc 2y ago
>>>and it's legal to scrape publicly available data, even if the websites hosting it try to block it Is that something that's been fully decided? https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc https://en.wikipedia.org/wiki/Craigslist_Inc._v._3Taps_Inc. is the most relevant case I'm aware of, and it suggests it might actually be illegal (if you know you've been blocked, at least).
suchintan 2y ago
https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-web-scraper-bright-data-which-meta-had-used-and-then-sued/ https://techcrunch.com/2024/01/24/court-rules-in-favor-of-a-... This is another interesting example where it was allowed

morkalork 2y ago

Aren't there also some suspiciously cheap VPNs that do that in the background?

ThePowerOfFuet 2y ago
Yes
duskwuff 2y ago
An employee of one proxy network describes that exact business model here: https://news.ycombinator.com/item?id=41597315 https://news.ycombinator.com/item?id=41597315

hypeatei 2y ago

Yeah, the author confirmed it in this thread actually: https://news.ycombinator.com/item?id=41594713 https://news.ycombinator.com/item?id=41594713

miohtama 2y ago

Here more on "free VPNs” https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-services/51721/ https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s... Usually such proxy networks are outright criminal (even if users are not).

dewey 2y ago

There's many reputable residential proxy networks too, usually there's a lot of vetting involved too as they don't want people running illegal activities though their network. It's almost a necessity these days to have access to that due to how much datacenter ranges are blocked.