5 ms·

My second reply, but, I think that this is really important. We only need to look at early film history to know how easy it is to lose massive parts of our his

My second reply, but, I think that this is really important. We only need to look at early film history to know how easy it is to lose massive parts of our history. Going back to old pages, I frequently get 404 results. For politically sensitive documents, the problem is much more widespread. I would like something that not only archives pages I visit, but also versions them and tracks changes. If there was a bookmarking tool that did this, you could easily have an opt-in feature that shared content. This type of system would be a huge boost to something like the wayback machine.

lutusp 12y ago
It clearly hasn't occurred to you that private collections evaporate over time, just like public ones. > I would like something that not only archives pages I visit, but also versions them and tracks changes. That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate several terabytes per week. > This type of system would be a huge boost to something like the wayback machine. Here's the road to madness. Someone, aware of the rapidly declining cost of storage, rebuilds the Wayback machine based on your scheme, with the intent of archiving every Web page in existence, including all required resources so the pages look just as they originally did. Then, as the project approaches completion, this genius says, "For the next phase, I need to archive the Wayback machine itself." At that point, as the implications of what he's said occur to him, a strange look crosses his face and his imagination begins writing checks his intellect can't cash.
- sroerick 12y ago
  I never said I was trying to save my entire web cache locally. Just bookmarks. I don't have Flash Player installed. Instead, I use youtube-dl for flash videos across many websites. I'm well aware of the storage implications of this kind of activity.
  - idlewords 12y ago
    The storage implications are really not so bad. You would have trouble ever breaking 100 GB unless you are some kind of bookmarking mutant.
- idlewords 12y ago
  Several terabytes per week is a huge overestimate. Right now I'm getting 12 Mbps download speed (sadly typical for US broadband). If I saturated the connection, and if no one throttled me, I could download 900 gigabytes in a week. That would be some intense web surfing. My practical experience is that you need about 1.5 MB per URL for storing large numbers of web pages, if you exclude video.
- Houshalter 12y ago
  For compressed static html and maybe images, you could store your entire history on a flash drive nowdays. See this: http://memkite.com/blog/2014/04/01/technical-feasibility-of-building-hitchikers-guide-to-the-galaxy-i-e-offline-web-search-part-i/ http://memkite.com/blog/2014/04/01/technical-feasibility-of-...
walterbell 12y ago
What do you think of this generalized architecture? HARDWARE: differs depending on whether you want local search/analytics or just network storage. For mobile use, either a VPN back to your personal home/cloud server, or a hackable wifi hard drive proxy, e.g. Seagate Wirless Plus + HackGFS. For non-analytics home use, hackable router with USB3 storage and Linux software RAID, connected to a USB3 drive chassis with room for 2-4 disks. For analytics home use, a microserver like HP N54L, Dell T20 or Lenovo TS140. Up to Xeon processor with ECC memory, plus 4-6 internal disks and up to 32GB RAM. Sold without a Windows tax, supports hardware virtualization and Linux. Possibly FreeBSD with ZFS. SOFTWARE: generalized multi-tier cache AND compute. Camlistore and git-annex are tackling multi-device storage sync. For archives, we need a search interface that will query a series of caches, e.g. mobile > home > trusted friends private VPN (tinc overlay) > public paid cloud archive (pinboard et al) > public free cloud archive (archive.org). It's important for usability to have a simple, local UX that will take a search string, propagate across all private/public federated tiers of storage and compute, then aggregate the metasearch results on the client. With this approach, we can collectively pool resoures to improve on CommonCrawl.org, without locking up the 300TB index at AWS. This would turn web search engines into a secondary source, rather than a primary source. First search your archive + trusted friends, then trusted verticals (e.g. HN, StackOverflow), then a generic web search. Let's be clear: the goal is not to archive "everything" in the world, only that which is personally important to the viewer. This attention metadata has long-term value. With this architecture, it is always optional to escalate a query to a public archive or search engine. Most importantly, there is technical autonomy and low-latency compute for local queries. For web pages, wget of WARC formats (per HN advice on another thread) and wkhtmltopdf (available as Firefox plugin to print to PDF) will keep local archives. Recoll.org (xapian front-end with user-customizable python filters) on Linux will search full text and provide preview snippets, or lucene/solr can be adapted.
- sroerick 12y ago
  Holy living God. Solves my problem, then some. Also provides an alternative to search engines as the goto for internet browsing. Brilliant stuff. I can't tell you how grateful I am for your work on this. The only thing left is FLOSS version control for sound and video editing, and an effective "publish to BitTorrent" feature, and then we can pretty much put this "Web 2.0" crap to bed. EDIT: Out of curiosity, is there any reason that analytics couldn't be done with a dedicated PC, or do you think it requires server hardware to run effectively?

lutusp 12y ago

It clearly hasn't occurred to you that private collections evaporate over time, just like public ones. > I would like something that not only archives pages I visit, but also versions them and tracks changes. That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate several terabytes per week. > This type of system would be a huge boost to something like the wayback machine. Here's the road to madness. Someone, aware of the rapidly declining cost of storage, rebuilds the Wayback machine based on your scheme, with the intent of archiving every Web page in existence, including all required resources so the pages look just as they originally did. Then, as the project approaches completion, this genius says, "For the next phase, I need to archive the Wayback machine itself." At that point, as the implications of what he's said occur to him, a strange look crosses his face and his imagination begins writing checks his intellect can't cash.

sroerick 12y ago
I never said I was trying to save my entire web cache locally. Just bookmarks. I don't have Flash Player installed. Instead, I use youtube-dl for flash videos across many websites. I'm well aware of the storage implications of this kind of activity.
- idlewords 12y ago
  The storage implications are really not so bad. You would have trouble ever breaking 100 GB unless you are some kind of bookmarking mutant.
idlewords 12y ago
Several terabytes per week is a huge overestimate. Right now I'm getting 12 Mbps download speed (sadly typical for US broadband). If I saturated the connection, and if no one throttled me, I could download 900 gigabytes in a week. That would be some intense web surfing. My practical experience is that you need about 1.5 MB per URL for storing large numbers of web pages, if you exclude video.
Houshalter 12y ago
For compressed static html and maybe images, you could store your entire history on a flash drive nowdays. See this: http://memkite.com/blog/2014/04/01/technical-feasibility-of-building-hitchikers-guide-to-the-galaxy-i-e-offline-web-search-part-i/ http://memkite.com/blog/2014/04/01/technical-feasibility-of-...

walterbell 12y ago

What do you think of this generalized architecture? HARDWARE: differs depending on whether you want local search/analytics or just network storage. For mobile use, either a VPN back to your personal home/cloud server, or a hackable wifi hard drive proxy, e.g. Seagate Wirless Plus + HackGFS. For non-analytics home use, hackable router with USB3 storage and Linux software RAID, connected to a USB3 drive chassis with room for 2-4 disks. For analytics home use, a microserver like HP N54L, Dell T20 or Lenovo TS140. Up to Xeon processor with ECC memory, plus 4-6 internal disks and up to 32GB RAM. Sold without a Windows tax, supports hardware virtualization and Linux. Possibly FreeBSD with ZFS. SOFTWARE: generalized multi-tier cache AND compute. Camlistore and git-annex are tackling multi-device storage sync. For archives, we need a search interface that will query a series of caches, e.g. mobile > home > trusted friends private VPN (tinc overlay) > public paid cloud archive (pinboard et al) > public free cloud archive (archive.org). It's important for usability to have a simple, local UX that will take a search string, propagate across all private/public federated tiers of storage and compute, then aggregate the metasearch results on the client. With this approach, we can collectively pool resoures to improve on CommonCrawl.org, without locking up the 300TB index at AWS. This would turn web search engines into a secondary source, rather than a primary source. First search your archive + trusted friends, then trusted verticals (e.g. HN, StackOverflow), then a generic web search. Let's be clear: the goal is not to archive "everything" in the world, only that which is personally important to the viewer. This attention metadata has long-term value. With this architecture, it is always optional to escalate a query to a public archive or search engine. Most importantly, there is technical autonomy and low-latency compute for local queries. For web pages, wget of WARC formats (per HN advice on another thread) and wkhtmltopdf (available as Firefox plugin to print to PDF) will keep local archives. Recoll.org (xapian front-end with user-customizable python filters) on Linux will search full text and provide preview snippets, or lucene/solr can be adapted.

sroerick 12y ago
Holy living God. Solves my problem, then some. Also provides an alternative to search engines as the goto for internet browsing. Brilliant stuff. I can't tell you how grateful I am for your work on this. The only thing left is FLOSS version control for sound and video editing, and an effective "publish to BitTorrent" feature, and then we can pretty much put this "Web 2.0" crap to bed. EDIT: Out of curiosity, is there any reason that analytics couldn't be done with a dedicated PC, or do you think it requires server hardware to run effectively?