4 ms·
How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?
by adam-_- 12y ago
How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?
- skillachie 12y ago+1 Scrapy comparison please Can you compare to scrapy as requested by other posters. Why could you not build on top of scrapy and leverage celery for scheduling etc (http://www.celeryproject.org/ http://www.celeryproject.org/) What is the immediate value add to using pyspider ?
- binux 12y agoI'm working on a benchmarking suite https://gist.github.com/binux/67b276c51e988f8e2c31 https://gist.github.com/binux/67b276c51e988f8e2c31 and meet some problem... pyspider comes from a vertical search engine project. we have two issues: - 100+ websites, they may change the template or down sometime. We need a dashboard to monitor the changes and the fails. - update in 5 minutes, when the website updated, we need follow that in 5 minutes. We are using a update time from index(list) page to tell the changed pages. And pages should been updated after about 30 days in case of we missed something. A powerful scheduler is needed. obviously, I hadn't got the right way to do so with scrapy. I'm not very familiar with scrapy. So I can't say something pyspider can do but scrapy not.