4 ms·

I'm using scrapinghub extensively for https://pagewatch.dev https://pagewatch.dev , is this project something you can use as a self-hosted replacement? Its not

I'm using scrapinghub extensively for https://pagewatch.dev https://pagewatch.dev , is this project something you can use as a self-hosted replacement? Its not very clear how it works, what does the Realtime mean?

bdcravens 6y ago
Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)
- mdaniel 6y ago
  and async I don't get that impression from the description of the response from `POST`: https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-response https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-r... since it does not return a job-id that requires polling, it appears to block until your scrape request is completed or it times out It's likely an implementation detail about whether the scrapyrt component is the one which blocks, but otherwise uses queuing and asynchronous invocations when interacting with Scrapy, or whether -- as the custom CrawlManager implies (https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-manager https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-man...) -- scrapyrt actually takes over and makes the entire CrawlManager -> Scheduler -> Spider call stack synchronous and thus able to respond to a POST within https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-limit https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-l... seconds with the actual Items
mdaniel 6y ago
what does the Realtime mean? I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR

bdcravens 6y ago

Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)

mdaniel 6y ago
and async I don't get that impression from the description of the response from `POST`: https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-response https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-r... since it does not return a job-id that requires polling, it appears to block until your scrape request is completed or it times out It's likely an implementation detail about whether the scrapyrt component is the one which blocks, but otherwise uses queuing and asynchronous invocations when interacting with Scrapy, or whether -- as the custom CrawlManager implies (https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-manager https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-man...) -- scrapyrt actually takes over and makes the entire CrawlManager -> Scheduler -> Spider call stack synchronous and thus able to respond to a POST within https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-limit https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-l... seconds with the actual Items

mdaniel 6y ago

what does the Realtime mean? I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR