4 ms·

Ask HN: Does this 'local web' tool exist yet?

• exolymph 10y ago
  Wikipedia seems to fulfill a fair amount of your requirements. Add in Project Gutenberg...
  • newman8r 10y ago
    Yeah wikipedia would be part of something like this. I already have all their data dumps and I have a lot of other cool DBs like musicBrainz, lots of stack exchange stuff, some "top" movies lists, lots of books, etc. I like to hoard data and hd space is cheap.
• cocktailpeanuts 10y ago
  Something relevant https://www.indiegogo.com/projects/lantern-one-device-free-data-from-space-forever https://www.indiegogo.com/projects/lantern-one-device-free-d... Which sources content from https://outernet.is/ https://outernet.is/
  • newman8r 10y ago
    cool concept, nice find. I want something like this that I can host, no satellites needed, I can customize, let friends connect remotely, be able to do really cool searches using regex, have some big databases, etc. I guess it's along the lines of scraping/crawling/local browsing, etc. Basically creating my own internet that I can invite people to, annotate, collaborate in, etc. I'm working on an open communications platform at the moment and am contemplating creating a microservice to enable something like this - but I don't want to reinvent the wheel if someone has already done it well.
    • cocktailpeanuts 10y ago
      I think you should definitely do it unless it takes years to ship the first version. The first in market doesn't mean the first to succeed.
      • newman8r 10y ago
        That's true. I'm not concerned about being first to market or making any profit on this type of project, would just be open source and release under GPL or whatever is appropriate. I just didn't want to spend the time or effort doing it if a similar alternative already existed. My primary motivation is essentially just wanting the tool for myself and friends. Not sure how long it would take but I might be able to release a useable proof of concept after a 3-day-weekend hackathon.
• Artlav 10y ago
  There used to be a program called Teleport Pro, which can download an entire web site. I used it rather extensively to download interesting sites for local storage, so i got a large selection of pages, a lot of which do not exist any more. Then i moved from dial-up to something faster around 2007, and the habit to download the sites gradually died out. Most of them are pre-2009, and the latest one is from 2014. I suppose making a new/free program like that might be of use today - the old one can't parse modern sites too well, i.e. Wikipedia.
  • 0942v8653 10y ago
    Try the recursive options of wget. It should be pretty much what you are talking about.
    • Artlav 10y ago
      Huh, i was going to say that wget won't fix the links in the downloaded files, but turns out they thought of it too. Nice, that means the tool does exist already.
      • newman8r 10y ago
        Check my response above, feedback appreciated. downloading entire websites via wget isn't what I was getting at although that type of tech is certainly something that would be integrated into the hypothetical system. What I am thinking of would really go beyond what you could achieve with wget. No doubt offline browsing and personal crawlers have been around for a while. Think of a 10TB+ tivo-like device that comes pre loaded with everything from wikipedia to musicBrainz. People would't need to figure out what to download, it would already be there, and they could augment it with anything. Just a thought - I am going to keep looking into it and will create a public repo for it if it makes sense.
  • newman8r 10y ago
    Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites were easier to download. A shotgun approach using something like ScraPy or via wget/curl as mentioned isn't all that useful) Since modern websites are very heavy on javascript, high res images, media, ads, comment sections, nav menus, etc - I think you just crawl portions of websites, cache the relevant data and keep a link to the original source. Instead of trying to crawl every obscure blog, I would focus on crawling the big/information dense sites really well. Many of them, like wikimedia sites, have complete data dumps available. If you systematically ignore all the garbage on the web and distill a version that is mostly valuable knowledge, there's some interesting possibilities. I have read a few times that most people browse the same,predictable subsets of websites anyway. Most of the important sites we log into provide APIs to access them. In general I'd prefer not to connect via web browser or cell phone app constantly.
• simon_acca 10y ago
  I don't know of any such curated collection, but a good tool to build what you are describing would be IPFS: https://ipfs.io https://ipfs.io
  • newman8r 10y ago
    I had no idea this even existed. This would be extremely helpful.
• leojg 10y ago
  These stuff exists, but it usefulness depends on the administrators. I was really surprised when a couple of years ago visiting Tigre, a city north of Buenos Aires, argentina: There is a huge river delta there, the Tigre delta and the people living in the small islands formed by the many branches of the river have a sort of local internet with their own news sites, wiki, etc... as well as access to the worldwide internet. Another interesting fact is that some of this island houses have electricity but no drinkable water, they have to carry it from the city docks
  • newman8r 10y ago
    This is fascinating. I've heard of these local internets in remote areas. I'd love something along those lines for my own personal enjoyment and to give access to my friends. I might go to def con this weekend and hit up the data duplication village