18 ms·
Disrupting the largest residential proxy network
- buddylw 9mo agoThis problem isn't going until we find a better solution to scraping than using your IP address as a passport.
- avastel 9mo agoSince I was also tracking this proxy network as part of my side project, I wrote a short blog post + give access to 16m+ proxy IPs IoCs that belong to this proxy network: https://deviceandbrowserinfo.com/learning_zone/articles/inside-ipidea-residential-proxy-network https://deviceandbrowserinfo.com/learning_zone/articles/insi...
- avastel 9mo agoSince I was also tracking this proxy network as part of my side project, I wrote a short blog post + give access to 16m+ proxy IPs IoCs that belong to this proxy network: https://deviceandbrowserinfo.com/learning_zone/articles/inside-ipidea-residential-proxy-network https://deviceandbrowserinfo.com/learning_zone/articles/insi... Note that even after the disruption, I'm still able to route millions of requests/day through IP IDEA's network
- nektro 9mo agoobjectively good news. thanks, google.
- nubinetwork 9mo agoI couldn't help but notice that they didn't list the affected apps, other than a couple domains for shady VPN providers...
- hafizmaaz 9mo agoHow much will it affect the other residential proxy providers?
- xyzzy_plugh 9mo ago> These efforts to help keep the broader digital ecosystem safe supplement the protections we have to safeguard Android users on certified devices. We ensured Google Play Protect, Android’s built-in security protection, automatically warns users and removes applications known to incorporate IPIDEA SDKs, and blocks any future install attempts. Nice to see Google Play Protect actually serving a purpose for once.
- trollbridge 9mo agoYeah, it serves the purpose of blocking this kind of proxy traffic that isn't in Google's personal best interests. Only Google is allowed to scrape the web.
- a456463 9mo agoYup exactly. Google must be the only one allowed to scrape the web. Google can't have any other competition. Calling it in "user's best interest" is just like their other marketing cons: "play integrity for user's security" etc
- viraptor 9mo agoHave you got any proof of Google scraping from residential proxies users don't know about, rather than from their clearly labelled AS? Otherwise you're mixing entirely different things into one claim.
- idiotsecant 9mo agoI don't think parent post is claiming that Google is using other people's networks to scrape the web only that they have a strong incentive to keep other players from doing that.
- viraptor 9mo agoNo, there are other scrapers that Google doesn't block or interact with. You can even run scraping from GCP. This has nothing to do with "only Google is allowed to scrape". They even host apps which exist for scraping data, like https://play.google.com/store/apps/details?id=com.socialleadscraper https://play.google.com/store/apps/details?id=com.sociallead...
- kotaKat 9mo agoI'm actually a little shocked seeing that there was a WebOS variant of the residential proxying SDK endpoint. Does that mean there might be a bit more unchecked malware lurking behind the scenes in the LG ecosystem? Personally I'm surprised they didn't have a Samsung option.
- wincy 9mo agoI keep my brand new LG C5 totally disconnected from the internet and use my Apple TV for movie watching. I’m not going to trust a company like LG to secure their devices.
- xnx 9mo ago> trust a company like LG to secure their devices. They have an interest in securing their devices so they can sell proxy service themselves.
- dewey 9mo agoWhy would webOS App Store be any different than the iOS or Android App Store which also have monetization frameworks for bandwidth sharing.
- whartung 9mo agoMy understanding is that routing through residential IPs is a part of the business of some VPN providers. I don't know how above board they are on this (as in notifying customers that this may happen, however buried in the usage agreement, or even allowing them to opt out). But, my main point, is that the whole business is "on the up and up" vs some dark botnet.
- Bratmon 9mo ago> I don't know how above board they are on this Saying you don't know something in the comments of an article that explains that thing is a bold strategy
- mhitza 9mo agoMullvad seems to be one of those VPN providers. [1] Though I very much doubt they would sneakily make end-users devices exit nodes. Though, as a historical side note, let's not forget Skype used to make users computers act as a relay as well during its more decentralized days. [1] Using the website mentioned by user Rasbora https://news.ycombinator.com/item?id=46837806 https://news.ycombinator.com/item?id=46837806
- nielsbot 9mo agoFTA > While operators of residential proxies often extol the privacy and freedom of expression benefits of residential proxies, Google Threat Intelligence Group’s (GTIG) research shows that these proxies are overwhelmingly misused by bad actors
- direwolf20 9mo agoGoogle's definition of a "bad actor" is someone who wants to use Google without seeing the ads. Or Kagi. Or an AI other than Gemini.
- kawsper 9mo agoOxylabs sells proxies for scrapers, I suppose you can use the socks-proxy as a VPN, and they claim to use Honeygain. Honeygain is a platform where people sell their residential internet connection and bandwidth to these companies for money. For comparison Honeygain pays someone 10 cents per GB, and Oxylabs sells it for $8/GB.
- samsullivan 9mo agoThe need for proxies in any legitimate context became obsolete with starlink being so widespread. Throw up a few terminals and you have about 500-2k cgnat IP addresses to do whatever you like.
- JDye 9mo ago2k IPs is not enough to do most enterprise scale scraping. Starlink's entire ASN doesn't seem to have enough V4 addresses to handle it even.
- chatmasta 9mo agoThe actual secret is to use IPv6 with varied source IPs in the same subnet, you get an insane number of IPs and 90% of anti-scraping software is not specialized enough to realize that any IP in a /64 is the same as a single IP in a /32 in IPv4.
- cferry 9mo ago> any IP in a /64 is the same as a single IP in a /32 in IPv4 This is very commonly true but sadly not 100%. I am suffering from a shared /64 on which a VPS is, and where other folks have sent out spam - so no more SMTP for me.
- fc417fc802 9mo agoIf they're CGNAT then unless Starlink actively provides assistance to block them it won't matter. As someone who wants the internet to maintain as much anarchy as possible I think it would be nice to see a large ISP that actively rotated its customer IPv6 assignments on a tight schedule.
- londons_explore 9mo agoWe need more residential proxies, not less. I've had enough of companies saying "you're connecting from an AWS IP address, therefore you aren't allowed in, or must buy enterprise licensing". Reddit is an example which totally blocks all data to non-residential IP's. I want exactly the same content visible no matter who you are or where you are connecting from, and a robust network of residential proxies is a stepping stone to achieving that.
- packetslave 9mo ago[flagged]
- direwolf20 9mo agoHave you tried it? Every new account will be shadowbanned and if it's shared you often get blank page 429. None of this was true before the API shutdown.
- gruez 9mo ago>Every new account will be shadowbanned That's not the same as "blocks all data to non-residential IP's"? >if it's shared you often get blank page 429. None of this was true before the API shutdown. See my other comment. I agree there's a non-zero amount of VPNs that are banned from reddit, but it's also not particularly hard to find a VPN that's not banned on reddit.
- interloxia 9mo agoProbably not hard but my poor little innocent VPS at Hetzer that I have had for years is denied and that makes me sad.
- 3rodents 9mo agoThat’s not my experience, using various VPNs, public networks, Cloudflare and Apple private relays. A captcha is common when logged out but that’s about it, I have not encountered any shadow bans. I create a new account each week.
- direwolf20 9mo agoAll of this sounds legal, so on what basis did they get them shut down?
- SOTGO 9mo agoI haven't looked at any court documents, but the WSJ article from Wednesday reported that "Last year, Google sued the anonymous operators of a network of more than 10 million internet-connected televisions, tablets and projectors, saying they had secretly pre-installed residential proxy software on them... an Ipidea spokeswoman acknowledged in an email that the company and its partners had engaged in “relatively aggressive market expansion strategies” and “conducted promotional activities in inappropriate venues (e.g., hacker forums)...”" There was also a botnet, Kimwolf, that apparently leveraged an exploit to use the residential proxy service, so it may be related to Ipidea not shutting them down.
- direwolf20 9mo agoGoogle does much worse in Google–branded devices and apps, like the wifi location data harvesting.
- toofy 9mo agoneat, so let’s stop them too. the answer is stop all the bad actors, not “well jimmy does it!”
- scirob 9mo agoso that only google and anthropic are allowed to scrape the web. No one else may have workarounds
- a456463 9mo agoExactly. This is just google building a "moat" around their shady business.
- cvalka 9mo ago100%
- mrweasel 9mo agoAnyone could scrape the net, then modern scrapes came along with their shitty code and absolutely no respect. The reason why so many of us block or throttle scrapers is because they miss behave. They don't back off, they try to by-pass caches and if they crash a site they don't adjust, they will just pound it the ground again when it's back. We managed to talk to one large AI company would didn't really want to fix anything, but told us that they'd be fine with us just rate limiting them, as if we somehow owed them anything. They just get a stupid low rps now, even if we'd let them go faster, if they'd just fix they bot. Some sites don't want you scraping, but it's their content, their rules. We don't really care, but we have to due to the number and quality of the bots we're seeing. This is in my mind a 100% self-imposed problem from the scrapers.
- htx80nerd 9mo ago[flagged]
- progbits 9mo agoI'm surprised by the negative takes... Yes, proxies are good. Ones which you pay for and which are running legitimately, with the knowledge (and compensation) of those who run them. Malware in random apps running on your device without your knowledge is bad.
- mschuster91 9mo ago> Ones which you pay for and which are running legitimately, with the knowledge (and compensation) of those who run them. The problem is, it is by default unethical to have residential users be exit nodes for VPNs - unless these users are lawyers or technical experts. No matter what you do as a "residential proxy" company - you cannot prevent your service being used by CSAM peddlers, and thus you cannot prevent that your exit nodes aren't the ones whose IP addresses show up when the FBI comes knocking.
- bettystaplesmd 9mo ago[dead]
- bdcravens 9mo agoMany are "compensated" (in the way of software they didn't pay for), so the real question is that of disclosure (in which case many software vendors check the box in the most minimal way possible by including it as fine print during the install)
- happyopossum 9mo agoNo, the question is not just disclosure. People have their bandwidth stolen, and sometimes internet access revoked due to this kind of fraud and misuse - disclosure wouldn’t solve that
- the_fall 9mo agoAlso, as a website owner, these residential proxies are a real pain. Tons and tons of abusive traffic, including people trying to exploit vulnerabilities and patently broken crawlers that send insane numbers of requests, and no real way to block it. It's just nasty stuff. Intent matters, and if you're selling a service that's used only by the bad guys, you're a bad guy too. This is not some dual-use, maybe-we-should-accept-the-risks deal that you have with Tor.
- IhateAI 9mo agoHow do you stop mobile proxies operating through similar nefarious business models... CGNAT prevents you from easily identifying the exit nodes.
- UqWBcuFx6NV4r 9mo agoWorking with network operators.
- chatmasta 9mo agoWhy are they leaving Bright Data (aka Illuminati aka Hola VPN) untouched? They are doing this exact scheme on an industrial scale.
- 7thpower 9mo agoThey have a robust KYC that appears to serve, at least in large part, as a way to stay off the shit list of companies with the resources to pursue recourse. Source: went through that process, ended up going a different route. The rep was refreshingly transparent about where they get the data, why the have the kyc process (aside from regulatory compliance). Ended up going with a different provider who has been cheaper and very reliable, so no complaints.
- chatmasta 9mo agoYeah, they make you do a Skype interview (or probably Zoom interview nowadays). You could call this KYC or collateral, depending on your view of the company. It does limit the nefariousness of their clientele but I doubt they do much, or any, monitoring of actual traffic after onboarding (not for compliance reasons, anyway).
- walletdrainer 9mo agoI’ve certainly never been asked to do KYC with Luminati after using them for hundreds of terabytes over the years. It’s not like I’m using some bigco email address or given them any other reason to skip KYC either.
- ghxst 9mo agoThey do KYC when you want to unblock certain domains.
- walletdrainer 9mo agoAlso not my experience, even though I’ve had to email them for whitelisting. It might just be because my account is very old?
- ExpertAdvisor01 9mo agoOf course brightdata doesn't get touched.
- khana 9mo ago[dead]
- walletdrainer 9mo agoIt’s interesting that when Luminati, an Israeli company, does this, it’s fine. When the Chinese do this? Very bad.
- VladVladikoff 9mo agoThey are both bad. You are showing your own bias.
- walletdrainer 9mo agoPersonally, I don’t think either of them are actually meaningfully bad. A bit naughty, maybe? I do think the disparity in attention is fascinating. These new Chinese players have been getting nonstop press while everyone ignores the established giant.
- calgoo 9mo agoNo, he is referencing Google going after the Chinese company, not the Israel based one. That does not mean there is bias with the commenter at all, just that the companies operate differently and are treated differently. The country of origin is important as Israel based companies are more integrated into the western business world, and tend to at least try to show an effort in keeping spam and other things off their platforms. Now I do agree that they are both bad companies that should not be allowed to operate the way they do. I would say the same thing about the other 1000 scrapers hitting websites everyday as well (including Google). What they did not comment directly on, is how many apps / games they might have actually removed from the Playstore with the removal of the SDKs, which would be the actual interesting data.
- JasonADrury 9mo agoFWIW a couple of years ago I was involved in a court case where there was a subpoena sent to Luminati to figure out whether or not a specific request had originated from their network, lawyers Luminati replied that they do not keep any logs whatsoever as they aren't required to do so under Israeli law. Hard to imagine any serious anti-abuse efforts by Luminati if they don't monitor what their users are doing, but this is probably a deliberate effort to avoid potential liability arising from knowing what their users are doing.
- brikym 9mo agoI'll betcha Google uses a lot of residential proxies themselves to scrape data and don't want competitors doing it.
- arewethereyeta 9mo agoI'll betcha your scraping for google simply by using Chrome
- edg5000 9mo agoResidential proxies are the only way to crawl and scrape. It's ironic for this article to come from the biggest scraping company that ever existed! If you crawl at 1Hz per crawled IP, no reasonable server would suffer from this. It's the few bad apples (impatient people who don't rate limit) who ruin the internet for both users and hosters alike. And then there's Google.
- BatteryMountain 9mo agoSaying the quiet part out loud...Shhhs
- Ronsenshi 9mo agoOne thing about Google is that many anti-scraping services explicitly allow access to Google and maybe couple of other search engines. Everybody else gets to enjoy CloudFlare captcha, even when doing crawling at reasonable speeds. Rules For Thee but Not for Me
- chii 9mo ago> many anti-scraping services explicitly allow access to Google and maybe couple of other search engines. because google (and the couple of other search engines) provide enough value that offset the crawler's resource consumption.
- JasonADrury 9mo agoThat's cool, but it's impossible for anyone to ever build a competitor that'd replace google without bypassing such services.
- ehhthing 9mo agoYou say this like robots.txt doesn't exist.
- toofy 9mo agoit almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something? implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”
- AugustoCAS 9mo agoThis was easy because it's a Chinese company. The largest companies in this space that do similar this (oxylabs, brighdata,etc) have similar tactics but are based in a different location.
- arewethereyeta 9mo agobrighdata = Israel i think oxylabs = Lithuanian, child of NordVPN
- niedbalski 9mo agoThanks google for saving us. I guess this is the equivalent of rival narcos fighting each other.
- tclancy 9mo agoBut would make for a much less interesting dramatic series. I bookmarked that link for the next time I have insomnia.
- moffkalast 9mo agoI see Google is doing their best to stamp out the competition.
- arewethereyeta 9mo agoThe big players are not taken out. This is sand thrown at our faces.
- g947o 9mo ago> attackers can mask their malicious activity by hijacking these IP addresses. Sounds like "malicious activity" == "scraping activities that don't come from Google"
- kingforaday 9mo agoGoogle shows a samaple of the IOCs but Google Trust Services have issued a number of the SSL certs for those domains that have not been revoked (yet?). Only looking at the: - a8d3b9e1f5c7024d6e0b7a2c9f1d83e5.com - af4760df2c08896a9638e26e7dd20aae.com - cfe47df26c8eaf0a7c136b50c703e173.com Looks like a standard MD5 hash domain pattern of which currently there are: user@host:/data/domains/2026/01/30$ zgrep -iE '^[a-f0-9]{32}\.com$' com.txt_domains.gz | wc -l 3005 If you look at some of the others (not listed in Google's IOC), they tend to have a pattern with their SSL certs e.g.: - 0e6f931862947ad58bf3d1a0c5a6f91f.com X509v3 Subject Alternative Name: DNS:0e6f931862947ad58bf3d1a0c5a6f91f.com, DNS:effc538138d9342c547c5df42b03d81e.com, DNS:gulfclouds.site, DNS:xinchaobccgba.net - 17e4435ad10c15887d1faea64ee7eac4.com X509v3 Subject Alternative Name: DNS:0dcbdf154c39288c91feb076795715e1.com, DNS:0e8843e8f10f20eeef59f0076e4feb83.shop, DNS:1014a1fb60e1b91404682e572ede6b4f.com, DNS:178281a79266d2faa3e578f23c8a361e.com, DNS:17e4435ad10c15887d1faea64ee7eac4.com, DNS:19f75b2642320e0606f5e38ce9fbcf17.com, DNS:1vxe.com, DNS:292893d0b31941e1c0d8eb01235be4eb.com, DNS:2b1e642f3a60130d1b2cf244891bef0d.info, DNS:354542342b7d2ddb66c97240d0c770dc.com, DNS:37d993ba8c9284bedad2a3177dfc44a6.info, DNS:3857036aaeedf670bbcca926945b50dd.com, DNS:3961f3fa3a6bacc5c4f28e81c60f4169.com, DNS:3eb4b3a3f8722b60d6ba2de7dd5f2523.org, DNS:42a17c71c0d6f2a6d7e135f8e869ab3f.com, DNS:4edd3793da3080640431430a4da57a86.org, DNS:4f5667d51451a2060067a97bcddf077f.info, DNS:5006cc38aff1ebc7d1232037fd592c60.net, DNS:54c35ec930f5b52fd9505778bb9c3f00.com, DNS:60255ec5427c2ba9a80b9c7648dd62e9.com, DNS:638d0e352728a04bb56ca102e54b8c9b.xyz, DNS:69234f9b18c0b4d572dc553dbfdb8f52.com, DNS:6934addf679d79a79f0bfc2ff090b104.com, DNS:694b64c9b41c17a229d92156d14a4ffd4.com, DNS:6eba8c4def89561e1cee02bb3c9b373d.info, DNS:7050f8c6563ff47465932e3838dc06fd.com, DNS:72ad0de0a556f763e0629c64c694df4c.com, DNS:86f7020358afaf71baeee5782b6264e4.xyz, DNS:88f2f20d26dcabeafd2f9d24e7ea4e50.com, DNS:911f4bf053ee3dadae1ca6bfdf40a817.com would there be any reason any of these would be legitimate?
- Rasbora 9mo agoI've helped multiple people remove residential proxy malware that was turning their network into a brightdata exit node and they had no idea / did not consent to it. Why is google selectively targeting one provider while letting others operate freely? You can check if your network is infected here: https://layer3intel.com/is-my-network-a-residential-proxy https://layer3intel.com/is-my-network-a-residential-proxy