3 ms·
The DB contains different types of proxies: - Residential - ISP - Data center I don't include mobile proxies since they're heavily shared, so knowing that an I
by avastel 1y ago
The DB contains different types of proxies:
- Residential
- ISP
- Data center
I don't include mobile proxies since they're heavily shared, so knowing that an IP address was used as a proxy at some point is basically useless.
Regarding your remark, indeed, there are several shared residential IPs, including IPs of legitimate users who may have a shady app that routes traffic through their device. That's why I don't recommend blocking using IP addresses as is. It's supposed to be more of a datapoint/signal to enrich your anti-fraud/anti-bot system.
However, regarding the block list, I analyze the IPs on bigger time frames, the percentage of IPs in the range that were used as proxies, and generate a confidence score to indicate whether or not it is safe to block.
- Havoc 1y agoSounds like pretty sophisticated filtering! I’m working on a scraping project at the moment so looking at this too but from the other end. Super low volume though so pretty tame - emphasis on success rate more than throughput I bought a 4G dongle for use as last resort if nothing else gets through. And also investigating ipv6
- avastel 1y agoUsing a 4G dongle makes it easier to hide in the crowd indeed. Since your traffic will go through heavily shared mobile IPs, probably with thousands of users behind them, anti-bot vendors won't/shouldn't block per IP, but per fingerprint/session cookie instead.
- Havoc 1y agoAh hadn’t realised it’s the NAT. I thought it’s because the IPs are dynamic and rotate too much. Interesting. Currently planning on doing a layered approach. Cloud IPs first etc. Interesting challenge but also trying to be somewhat respectful about it since nobody likes aggressive bots