5 ms·

There used to be a program called Teleport Pro, which can download an entire web site. I used it rather extensively to download interesting sites for local sto

by Artlav 10y ago

There used to be a program called Teleport Pro, which can download an entire web site. I used it rather extensively to download interesting sites for local storage, so i got a large selection of pages, a lot of which do not exist any more. Then i moved from dial-up to something faster around 2007, and the habit to download the sites gradually died out. Most of them are pre-2009, and the latest one is from 2014. I suppose making a new/free program like that might be of use today - the old one can't parse modern sites too well, i.e. Wikipedia.

0942v8653 10y ago
Try the recursive options of wget. It should be pretty much what you are talking about.
- Artlav 10y ago
  Huh, i was going to say that wget won't fix the links in the downloaded files, but turns out they thought of it too. Nice, that means the tool does exist already.
  - newman8r 10y ago
    Check my response above, feedback appreciated. downloading entire websites via wget isn't what I was getting at although that type of tech is certainly something that would be integrated into the hypothetical system. What I am thinking of would really go beyond what you could achieve with wget. No doubt offline browsing and personal crawlers have been around for a while. Think of a 10TB+ tivo-like device that comes pre loaded with everything from wikipedia to musicBrainz. People would't need to figure out what to download, it would already be there, and they could augment it with anything. Just a thought - I am going to keep looking into it and will create a public repo for it if it makes sense.
newman8r 10y ago
Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites were easier to download. A shotgun approach using something like ScraPy or via wget/curl as mentioned isn't all that useful) Since modern websites are very heavy on javascript, high res images, media, ads, comment sections, nav menus, etc - I think you just crawl portions of websites, cache the relevant data and keep a link to the original source. Instead of trying to crawl every obscure blog, I would focus on crawling the big/information dense sites really well. Many of them, like wikimedia sites, have complete data dumps available. If you systematically ignore all the garbage on the web and distill a version that is mostly valuable knowledge, there's some interesting possibilities. I have read a few times that most people browse the same,predictable subsets of websites anyway. Most of the important sites we log into provide APIs to access them. In general I'd prefer not to connect via web browser or cell phone app constantly.

by Artlav 10y ago

0942v8653 10y ago

Try the recursive options of wget. It should be pretty much what you are talking about.

Artlav 10y ago
Huh, i was going to say that wget won't fix the links in the downloaded files, but turns out they thought of it too. Nice, that means the tool does exist already.
- newman8r 10y ago
  Check my response above, feedback appreciated. downloading entire websites via wget isn't what I was getting at although that type of tech is certainly something that would be integrated into the hypothetical system. What I am thinking of would really go beyond what you could achieve with wget. No doubt offline browsing and personal crawlers have been around for a while. Think of a 10TB+ tivo-like device that comes pre loaded with everything from wikipedia to musicBrainz. People would't need to figure out what to download, it would already be there, and they could augment it with anything. Just a thought - I am going to keep looking into it and will create a public repo for it if it makes sense.

newman8r 10y ago

Yeah I remember these types of offline browsers that were popular back in the day - data usage is still a big concern for some people (pre web 2.0 websites were easier to download. A shotgun approach using something like ScraPy or via wget/curl as mentioned isn't all that useful) Since modern websites are very heavy on javascript, high res images, media, ads, comment sections, nav menus, etc - I think you just crawl portions of websites, cache the relevant data and keep a link to the original source. Instead of trying to crawl every obscure blog, I would focus on crawling the big/information dense sites really well. Many of them, like wikimedia sites, have complete data dumps available. If you systematically ignore all the garbage on the web and distill a version that is mostly valuable knowledge, there's some interesting possibilities. I have read a few times that most people browse the same,predictable subsets of websites anyway. Most of the important sites we log into provide APIs to access them. In general I'd prefer not to connect via web browser or cell phone app constantly.