4 ms·
Instant Distributed Web Crawling with Tornado and Gearman
- aviv 14y agoGreat write-up and examples. We use a similar combination of Tornado, Gearman and Redis for our own crawling with mongodb for storage. I realized just now why your company name was familiar - we're also on the 3rd floor in the PV building.