5 ms·
SpiderKeeper: Admin UI for scrapy/open source scrapinghub
- Pmop 7y agoNice. I'm stealing this concept for my scraping engine.
- r_singh 7y agoAnyone here using this in production (even for a side project)? What are some of the challenges you've faced as opposed to just using ScrapingHub's Scrapy Cloud service?
- skykery 7y agoI am using this in production inside a docker container. It does the job pretty well.
- Alex_P13 7y agoNice
- jatinshah 7y agoAll such admin tools use scrapyd which is not maintained and not good enough to be used in production.
- gregsadetsky 7y agoWhat do you recommend using instead?
- rozhok 7y agoOne of the companies I've worked for developed in-house solution (with some domain-specific stuff). I've had to write some kind of admin tool for one of my clients. And then I've used to start another project with scrapy and I feel like I need to develop another one tool for scrapyd management :)
- shiblukhan 7y agoSee also the list of contributors who participated in this project.
- oqulie 7y agoSpiderKeeper least received an update 2 years ago. Scrapyd last received an update 9 months ago. Scrapy (the actual spider) is being actively maintained.
- madc 7y agoLoads of not maintained open source projects make it to the front page lately..
- wp381640 7y agoThis project seems like a more updated fork although it's also not maintained https://github.com/fliot/ScrapyKeeper https://github.com/fliot/ScrapyKeeper
- slightwinder 7y agoHow does this compare with a general solution like Apache Airflow?
- troymc 7y agoThe following is a simplification but should suffice. Airflow is for hooking together and managing a bunch of data sources, data-processing nodes, and data sinks. This project seems like a GUI for Scrapy, which is a web crawling framework, i.e. one kind of data source.
- dzink 7y agoAre these spiders able to fetch content that loads in segments after scroll events?
- jsmith99 7y agoScrapy doesn't run javascript so it can't scrape client rendered content at all. There are projects that combine scrapy with a headless browser or javascript engine.
- cs64org 7y agoI think it can faster than the normal then a basic function. ti increase the case memory.