3 ms·
Very cool. The take-any-webpage-offline need is also common in the education space (teachers want to save a webpage and send it to their students as part of a
by ivansavz 5y ago
Very cool.
The take-any-webpage-offline need is also common in the education space (teachers want to save a webpage and send it to their students as part of a lesson and don't want to worry about availability or ads etc).
I used to work on tools for this https://github.com/learningequality/ricecooker/blob/develop/ricecooker/utils/downloader.py#L205-L502 https://github.com/learningequality/ricecooker/blob/develop/... and https://github.com/learningequality/BasicCrawler/blob/master/basiccrawler/crawler.py#L286-L382 https://github.com/learningequality/BasicCrawler/blob/master...
which worked quite well for most sites, but still very far from a general-purpose solution.
There is also more powerful/general-purpose scraper that generates a ZIM file here: https://github.com/openzim/zimit https://github.com/openzim/zimit
It would be really nice to a "common" scraper code base that takes care of scraping (possibly with a real headless browser) and outputs all assets as files + info as JSON. This common code base could then be used by all kinds of programs to package the content as standalone HTML zip files, ePub, ZIM, or even PDF for crazy people like me who like to print things ;)
- tasogare 5y agoThat's a solved problem: click right, save the page as MHTML. It can be opened later and online with a browser again. If I recall correctly the only gotcha is that the option to save in this format needs to be enable using the flags settings.
- genewitch 5y agoFirefox headless can save webpages as images or "as a complete page" can't it? I use the image functional and only sites like Twitter fail to work correctly, although it's probably my CGI gateway timing out waiting for JavaScript or whatever.