3 ms·
what does the Realtime mean? I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme https://github.com/scrapy/scra
by mdaniel 6y ago
what does the Realtime mean?
I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else
With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR