RuedigerVoigt

3 ms·

1. 1.▲
   by RuedigerVoigt 6y ago

   Exoskeleton is an open-source framework for crawlers written in Python. It utilizes MariaDB to manage the queue. It is built to download a very large number of files and / or web pages. Meanwhile it avoids to overload servers and sprea
2. 2.▲
   Show HN: Exoskeleton – a framework for polite, but tenacious crawlers(github.com)

   4 points by RuedigerVoigt 6y ago|1 comments
3. 3.▲
   by RuedigerVoigt 7y ago

   Common CMS are fairly good at caching and can handle a high load, but quite often someone deems a badly programmed extension "mission critical". In that case one of your requests might trigger dozens of database calls. If multiple