5 ms·
> The crawl function reads a URL from the SQS queue, then Pyppeteer tells Chrome to navigate to that page behind a rotating residential proxy. The residential p
by cortesoft 6y ago
> The crawl function reads a URL from the SQS queue, then Pyppeteer tells Chrome to navigate to that page behind a rotating residential proxy. The residential proxy is necessary to prevent Google from blocking the IP Lambda makes requests from.
I am very interested in what a 'rotating residential proxy' is. Are they routing requests through random people's internet connections? Are these people willing participants? Where do they come from?
- kazz 6y agoIt looks like that's basically what they are (https://smartproxy.com/blog/what-is-a-residential-proxies-network https://smartproxy.com/blog/what-is-a-residential-proxies-ne...). A residential proxy is listed as an "IP address provided by an Internet Service Provider", but I still don't really understand how they get access to them. ISPs have to be selling them access, right?
- paulryanrogers 6y agoMy guess is free VPN services and browser extensions which resell your residential network in tiny chunks.
- heipei 6y agoCheck out Luminati for example. They have a huge network of true residential IPs to exit traffic from, and you have to pay a hefty premium per GB of traffic to do so ($12.50 per GB for rotating residential IPs, but requires a minimum $500 commitment per month). The reason they can offer this is because they're exiting traffic through the users of the free Hola VPN Chrome extension.
- nsgf 6y agoYes, to all your questions. https://luminati.io/ https://luminati.io/ Providers of the 'free' Hola vpn.
- Traster 6y agoI think you can make a reasonable argument that Hola VPN is largely exploiting users who don't actually understand and consent to having their IP address and connection used as a proxy.
- ac29 6y agoHow awful. "80M+ Monthly devices hosting Luminati's SDK" & "100% Peers chose to opt-in to Luminati's network" (https://luminati.io/network-details https://luminati.io/network-details) There is a 0% chance that 80M+ are agreeing "I am OK with Luminati selling access to my home internet connection to any party able to pay", which seems like an honest description of their business model. More likely Luminati is paying unscrupulous app developers to include this SDK in their apps, and some put some legalese into 10,000 word install-time agreements that no one reads.
- sdinsn 6y agoPeople who use the network also participate in the network themselves.