3 ms·
Show HN: Exoskeleton – a framework for polite, but tenacious crawlers
- RuedigerVoigt 6y agoExoskeleton is an open-source framework for crawlers written in Python. It utilizes MariaDB to manage the queue. It is built to download a very large number of files and / or web pages. Meanwhile it avoids to overload servers and spreads the tasks - if necessary - over multiple days. Furthermore, it automatically avoids duplicates and allows you to organize file collections with labels.