3 ms·

Great article, thanks for sharing. What would you recommend for people who need to operate a cluster of scrapyd instances? Are there any recommended ways of ma

by mryan 11y ago

Great article, thanks for sharing. What would you recommend for people who need to operate a cluster of scrapyd instances? Are there any recommended ways of managing the distribution of tasks to multiple scrapyd instances? I have looked at scrapyd-cluster, but I would prefer not to add a Zookeeper cluster to my stack. Currently I'm thinking of modifying scrapyd-cluster so that it uses AMQP (via Celery) to handle task distribution/retries/etc. I appreciate this conflicts with the scrapinghub business model, so any tips you can offer would be greatly appreciated :-)

asibiryakov 11y ago
Hi, mryan! I'm the core Frontera developer. The precise answer heavily depends on your use case (what "task" is? scalability requirements, data flow), so please ask your question in Frontera google groups, and we will try to address it directly. First you could try making use of Frontera, here are the different distribution models it provides out of the box http://frontera.readthedocs.org/en/latest/topics/run-modes.html http://frontera.readthedocs.org/en/latest/topics/run-modes.h.... Frontera is web crawling framework made in Scrapinghub, providing crawl frontier and scaling/distribution capabilities. Along with flexible queue and partitioning design, you will get also document metadata storage (HBase or RDBMS of your choice) with simple revisiting mechanism. Second, we have a simple redis-based solution for scaling spiders https://github.com/rolando/scrapy-redis https://github.com/rolando/scrapy-redis. It's only dependency is Redis, so it's easy to quick start, but it has only one queue shared between spiders, hard-coded partitioning, and Redis limiting scalability. You mentioned a scrapy-cluster (not scrapyd-cluster probably). It provides a more sophisticated distribution model, allowing you to separate crawls with jobs concept within the same service, maintains separate per-spider queues (allowing to crawl politely, I hope you plan to do so? :), and forcing you to use it's prioritization model. Also it allows to use spiders of different types sharing the same Redis instance, and prioritize requests on cluster level. BTW, I haven't found any dependencies on Zookeeper. None of the solutions provides provisioning out of the box. If spider was killed by OOM, or consume too much resources (open file descriptors, memory) you have to take care of it by yourself. You could use supervisord, or upstart or some custom process management solution. It all depends on you monitoring requirements. Good luck choosing the right solution! A.
- mryan 11y ago
  Sounds like I have some more research to do - thanks for the detailed response. Btw, you were right, I meant scrapy-cluster and not scrapyd-cluster: https://github.com/istresearch/scrapy-cluster https://github.com/istresearch/scrapy-cluster. There is a requirement on Zookeeper/Kafka, which is the main showstopper for me.

by mryan 11y ago

asibiryakov 11y ago

Hi, mryan! I'm the core Frontera developer. The precise answer heavily depends on your use case (what "task" is? scalability requirements, data flow), so please ask your question in Frontera google groups, and we will try to address it directly. First you could try making use of Frontera, here are the different distribution models it provides out of the box http://frontera.readthedocs.org/en/latest/topics/run-modes.html http://frontera.readthedocs.org/en/latest/topics/run-modes.h.... Frontera is web crawling framework made in Scrapinghub, providing crawl frontier and scaling/distribution capabilities. Along with flexible queue and partitioning design, you will get also document metadata storage (HBase or RDBMS of your choice) with simple revisiting mechanism. Second, we have a simple redis-based solution for scaling spiders https://github.com/rolando/scrapy-redis https://github.com/rolando/scrapy-redis. It's only dependency is Redis, so it's easy to quick start, but it has only one queue shared between spiders, hard-coded partitioning, and Redis limiting scalability. You mentioned a scrapy-cluster (not scrapyd-cluster probably). It provides a more sophisticated distribution model, allowing you to separate crawls with jobs concept within the same service, maintains separate per-spider queues (allowing to crawl politely, I hope you plan to do so? :), and forcing you to use it's prioritization model. Also it allows to use spiders of different types sharing the same Redis instance, and prioritize requests on cluster level. BTW, I haven't found any dependencies on Zookeeper. None of the solutions provides provisioning out of the box. If spider was killed by OOM, or consume too much resources (open file descriptors, memory) you have to take care of it by yourself. You could use supervisord, or upstart or some custom process management solution. It all depends on you monitoring requirements. Good luck choosing the right solution! A.

mryan 11y ago
Sounds like I have some more research to do - thanks for the detailed response. Btw, you were right, I meant scrapy-cluster and not scrapyd-cluster: https://github.com/istresearch/scrapy-cluster https://github.com/istresearch/scrapy-cluster. There is a requirement on Zookeeper/Kafka, which is the main showstopper for me.