2 ms·
Cool. Thanks for the info. I'm in the middle of developing a distributed scraping (HTML to JSON API conversion) for internal use at a company I'm working for an
by malandrew 14y ago
Cool. Thanks for the info. I'm in the middle of developing a distributed scraping (HTML to JSON API conversion) for internal use at a company I'm working for and we plan on open-sourcing it eventually. However, before we do, we need to map out all the possible pathologies of such a system, which the two most concerning ones are greedy nodes and preventing usage of the system for the execution DDoS attacks. I'm going to look into "greedy routing".