3 ms·
How do you plan to keep the list up to date and especially reliably accurate? (providers change, bot nets get cleaned, ASN especially in 3rd world countries are
by bootsmann 4y ago
How do you plan to keep the list up to date and especially reliably accurate? (providers change, bot nets get cleaned, ASN especially in 3rd world countries are very unreliable, crawlers might scrap weird things etc.)
Disclaimer: I work for a company that operates in a similar space so don't go into too much secret sauce if you don't want to :)
- AndrewCopeland 4y agoSo I have a scraper that gathers the information from about 60 different sources using a wide array of techniques which is run everyday. The largest thing I try to look at is to make sure my sources are not older than 1 year. Sometimes these sources break and I update them or they go offline all together. I am always looking for new hosting providers, vpn and proxies. Also I can do further port analytics on these ips as well. A lot of the VPN services have specific ports open used by VPN services. I would say that automating every source from the start was a good way to start so I can stay up to date.