Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites wer

Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites were easier to download. A shotgun approach using something like ScraPy or via wget/curl as mentioned isn't all that useful) Since modern websites are very heavy on javascript, high res images, media, ads, comment sections, nav menus, etc - I think you just crawl portions of websites, cache the relevant data and keep a link to the original source. Instead of trying to crawl every obscure blog, I would focus on crawling the big/information dense sites really well. Many of them, like wikimedia sites, have complete data dumps available. If you systematically ignore all the garbage on the web and distill a version that is mostly valuable knowledge, there's some interesting possibilities. I have read a few times that most people browse the same,predictable subsets of websites anyway. Most of the important sites we log into provide APIs to access them. In general I'd prefer not to connect via web browser or cell phone app constantly.