4 ms·
I love command-line applications for interacting with web services! They allow me to scrape things off websites that would be difficult with navigating the HTML
by vortico 6y ago
I love command-line applications for interacting with web services! They allow me to scrape things off websites that would be difficult with navigating the HTML DOM or learning the service's specific JSON API. For example, `youtube-dl` is a fantastic tool for downloading videos from almost every video hosting site such as YouTube.
I've always wanted a tool that will scrape article text (either plaintext or converted to Markdown) from news websites. I don't see it on http://weboob.org/applications/ http://weboob.org/applications/, but I think it would be a nice project addition. Sometimes I just want to read or download a news article without adapting to each specific journal's page layout and dealing with ads. Firefox's Reader View does this pretty well, but a command-line version would be more useful to me. It could be implemented by writing a heuristic for scraping arbitrary news websites, and several hand-written parsers for scraping popular ones. Cookboob (http://weboob.org/applications/cookboob http://weboob.org/applications/cookboob) is pretty close to this idea.