4 ms·
Would you still recommend Scrapy if the task wasn't specifically crawling?
by Bromskloss 9y ago
Would you still recommend Scrapy if the task wasn't specifically crawling?
- sharmi 9y agoNope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery are python specific. [0] : http://python-rq.org/docs/ http://python-rq.org/docs/ [1] : http://gearman.org/ http://gearman.org/ [2] : http://docs.celeryproject.org http://docs.celeryproject.org
- luckystarr 9y agoI once used it to automate the, well, scraping of statistics from an affiliate network account. So you can do pretty specific stuff, as long as it involves HTTP/HTTPS requests.
- dataslap 9y agodepends on the task. For example they have a decent file/image downloading middleware.