There is ScrapyRT: http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an-api-for-scrapy-spiders/ http://blog.scrapinghub.com/2015/01/22/introducing-scr

There is ScrapyRT: http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an-api-for-scrapy-spiders/ http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-... In the project I work on we do have the usual periodic crawls and use ScrapyRT to let the frontend trigger realtime scrapes of specific items, all of this using the same spider code. Edit: Worth nothing that we trigger the realtime scrapes via AMQP.