3 ms·

Back in 2018 i crawled 200Million urls of single site by using just freely avilable proxies on the internet to evade rate limits. i do not think it can be repea

by fukd 5y ago

Back in 2018 i crawled 200Million urls of single site by using just freely avilable proxies on the internet to evade rate limits. i do not think it can be repeated today. CDNs are blocking even my genuine requests.

aj3 5y ago
Sadly, it's easier than ever. Multiple botnets create open proxies which are easily discoverable, if you know where to look. Also, cloud providers give you more IPs you might ever need.
- fukd 5y ago
  Cloud provider ip ranges are publicly known. They are the ones first get banned