3 ms·
Check out scrapy, written in python. It also uses asynchronous io. Also, asyncio in python3 should make writing an async crawler a lot easier nowdays.
by rckclmbr 10y ago
Check out scrapy, written in python. It also uses asynchronous io. Also, asyncio in python3 should make writing an async crawler a lot easier nowdays.
- plantpark 10y agoThanks for your comment. This article is a tutorial that tell people how to build a distributed system with docker and celry easily. Scrapy is good framework for crawler. I used it before, but it doesn't have some features I want. Writing a brand new one is easier for me. Of course , I am open to compare the special performances of these two crawlers. Welcome to have a discuss more technical details here. Thanks for your comment again.