4 ms·

Small Archives

31 points by benbreen 2y ago

ravenstine 2y ago
I've seen so many things disappear off the web that I knee-jerk yt-dlp/wget/DeDRM anything I am interested in or is relevant to my personal culture. What stinks is I haven't found anything satisfactory in terms of archival software that runs locally. For now, I shove things into folders in external drives, but directory trees are very poor when it comes to categorization and retrieval.
- zellyn 2y ago
  Have you looked at Camlistore?
  - sea-gold 2y ago
    Now known as Perkeep (previously Camlistore) https://en.m.wikipedia.org/wiki/Perkeep https://en.m.wikipedia.org/wiki/Perkeep
    - breakingcups 2y ago
      Perkeep is very interesting on a fundamental level, as a concept and as a building block, but sadly hasn't received enough love to fully fulfill its vision. I wish there were more people organizing around Perkeep and breathing life into it.
- fosco 2y ago
  I’m reasonable satisfied with Shiori and its text search functionality
- packetlost 2y ago
  I too am looking for a tool that can decently mirror sites. The next best thing is something like Instapaper or Obsidian Clipper which export copies of a site (into markdown, in the latter's case).
- alkh 2y ago
  While I haven't personally tried it myself, ArchiveBox seems to be tailored specifically for archiving[1]. Mb you should give it a try? [1]https://archivebox.io/ https://archivebox.io/
- didgetmaster 2y ago
  Modern hard drives are now above 20TB in capacity. If your average file size is 100KB, then they will hold 200 million+ files. File systems with their directory or folder hierarchy are poor organizers of files. Searching for files is a huge hassle. File systems, which were designed over 50 years ago when storage was measured in MB, need to be replaced by object stores that are much better at organizing and finding data.
  - packetlost 2y ago
    In what way are object stores better than hierarchical filesystems? With links you can create arbitrary indices over the same data with different (or the same) names.
    - 3036e4 2y ago
      This is what I do. Some 5 million files (3 TB) with hard and soft links everywhere to make the files show up in places I hope to find them again. Ripgrep Anything or whatever it is called is pretty good for finding things, at least as long as I can restrict the search to a not too huge subset of my archive. I also have org-mode documents full of links to interesting files, and sometimes to locations within files. A bit messy, but I do not want to have dependencies on any special software. My oldest files have been with me since I first started hoarding on 360 kB DOS floppies.
    - didgetmaster 2y ago
      With file systems, searching for files (e.g. find all documents) requires traversing the entire directory hierarchy. With hard drives this comes at a cost of many seeks. The last time I traversed a folder tree with over 100 million files, it took more than 30 minutes to complete. A properly designed object store can find everything in less than a minute.

31 points by benbreen 2y ago

I've seen so many things disappear off the web that I knee-jerk yt-dlp/wget/DeDRM anything I am interested in or is relevant to my personal culture. What stinks is I haven't found anything satisfactory in terms of archival software that runs locally. For now, I shove things into folders in external drives, but directory trees are very poor when it comes to categorization and retrieval.

zellyn 2y ago
Have you looked at Camlistore?
- sea-gold 2y ago
  Now known as Perkeep (previously Camlistore) https://en.m.wikipedia.org/wiki/Perkeep https://en.m.wikipedia.org/wiki/Perkeep
  - breakingcups 2y ago
    Perkeep is very interesting on a fundamental level, as a concept and as a building block, but sadly hasn't received enough love to fully fulfill its vision. I wish there were more people organizing around Perkeep and breathing life into it.
fosco 2y ago
I’m reasonable satisfied with Shiori and its text search functionality
packetlost 2y ago
I too am looking for a tool that can decently mirror sites. The next best thing is something like Instapaper or Obsidian Clipper which export copies of a site (into markdown, in the latter's case).
alkh 2y ago
While I haven't personally tried it myself, ArchiveBox seems to be tailored specifically for archiving[1]. Mb you should give it a try? [1]https://archivebox.io/ https://archivebox.io/
didgetmaster 2y ago
Modern hard drives are now above 20TB in capacity. If your average file size is 100KB, then they will hold 200 million+ files. File systems with their directory or folder hierarchy are poor organizers of files. Searching for files is a huge hassle. File systems, which were designed over 50 years ago when storage was measured in MB, need to be replaced by object stores that are much better at organizing and finding data.
- packetlost 2y ago
  In what way are object stores better than hierarchical filesystems? With links you can create arbitrary indices over the same data with different (or the same) names.
  - 3036e4 2y ago
    This is what I do. Some 5 million files (3 TB) with hard and soft links everywhere to make the files show up in places I hope to find them again. Ripgrep Anything or whatever it is called is pretty good for finding things, at least as long as I can restrict the search to a not too huge subset of my archive. I also have org-mode documents full of links to interesting files, and sometimes to locations within files. A bit messy, but I do not want to have dependencies on any special software. My oldest files have been with me since I first started hoarding on 360 kB DOS floppies.
  - didgetmaster 2y ago
    With file systems, searching for files (e.g. find all documents) requires traversing the entire directory hierarchy. With hard drives this comes at a cost of many seeks. The last time I traversed a folder tree with over 100 million files, it took more than 30 minutes to complete. A properly designed object store can find everything in less than a minute.