4 ms·
Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)
by bdcravens 6y ago
Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)
- mdaniel 6y agoand async I don't get that impression from the description of the response from `POST`: https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-response https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-r... since it does not return a job-id that requires polling, it appears to block until your scrape request is completed or it times out It's likely an implementation detail about whether the scrapyrt component is the one which blocks, but otherwise uses queuing and asynchronous invocations when interacting with Scrapy, or whether -- as the custom CrawlManager implies (https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-manager https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-man...) -- scrapyrt actually takes over and makes the entire CrawlManager -> Scheduler -> Spider call stack synchronous and thus able to respond to a POST within https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-limit https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-l... seconds with the actual Items