3 ms·

Celery might be an option for distributing crawling jobs.