3 ms·

I don't think it's a good idea to depend mainly on low-level or device-specific features (in a NAS or your filesystem) to detect bitrot and corruption in files.

• AnonC 6y ago
  fim looks interesting. I skimmed through the linked page but couldn’t find an answer for a question I have — can fim be used for data stored elsewhere, by which I mean that the fim repository should be located outside the directory structure that it’s tracking? If I have a Photos directory with files and subdirectories, I don’t want that structure to have anything else (including fim).
  • vczf 6y ago
    There isn't a command line argument for pointing to a .fim/ located elsewhere, if that's what you mean. It probably wouldn't be hard to add that to the utility. What you could do is put your photos subdirectory inside another directory, something like below. data/ ├── .fim │ ├── settings.json │ └── states │ └── state_1.json.gz ├── photos └── records
• benibela 6y ago
  >The data architecture I use combines, in order of importance: (1) borg, for incremental deduplicated backups onsite and offsite; (2) syncthing, for duplication and synchronization across devices; (3) fim, for managing file integrity; (4) rsync, local copy to another drive for convenient restores; and (5) git, in places where commit history is actually useful, like code or dotfiles. That looks overly complicated Why is there no single tool that can do all of those? I think one tool could be much more efficient. Especially since they could reuse the databases. With one database of hashes of each file, it could find the changed files and then incrementally only backup those. No need to have one program to search changed files to copy them, and then again have one program to search changed files to hash them I mostly use rsync. Occasionally I destroy my backups by calling it wrongly, like missing a trailing slash. That could not happen if the copying and file integrity checking was combined in one tool >fim combined with incremental backup solves this problem, is filesystem agnostic and painless to migrate around, and is for some reason completely unknown and obscure: https://evrignaud.github.io/fim/ https://evrignaud.github.io/fim/ That looks very useful But it stores the database as json? That is a bad format to store file names. And written in Java it is probably going to be slow (I have over a million files in ~, and multiple copies of it in the backups, so I care a lot about performance) So many emojis in the commit log. Is that really necessary nowadays?
  • vczf 6y ago
    > That looks overly complicated In practice, it's simple enough. I have a backup script that does all the heavy lifting for borg and rsync. Syncthing operates independently in the background, and I rarely log changes with git. The most tedious part of the process is logging new/changed/deleted files with fim. > Why is there no single tool that can do all of those? Good question, and I totally agree. I was actually looking for a solution that could do deduplicated incremental backups (versioning, like time machine) plus file integrity management. I couldn't find one. Nobody seems to care much about file integrity, and nowadays I'm actually unsure if this is a real problem for anyone except paranoid geeks. Could also use a good frontend GUI that could notify you about changed files, allow you to easily mark directories for different levels of "tracking", easily browse past versions and restore them, etc. The CLI isn't doing any favors here. > I mostly use rsync. Occasionally I destroy my backups by calling it wrongly, like missing a trailing slash. That could not happen if the copying and file integrity checking was combined in one tool Try borg-backup, it'll be a massive improvement over plain rsync when it comes to backups. > But it stores the database as json? That is a bad format to store file names. And written in Java it is probably going to be slow (I have over a million files in ~, and multiple copies of it in the backups, so I care a lot about performance) Gzipped json on-disk, but I haven't noticed any problems with it over the last few years. I doubt java is bottlenecking the performance at all: it's limited by IO and CPU throughput for hashing the files. For everyday use, you don't need to hash every file in entirety to detect changes. There's a "fast" mode that checks a couple of blocks. You can also operate in a subdirectory of the repo to ignore files outside that subdirectory. I also don't just use one massive fim repository. I have a .fim/ for each directory I care about (types of data) that a secondary scripts loops through for checking the status. That weakens the integrity guarantees since you can't log the movement of files across directories, but makes it easier to reorganize things. > So many emojis in the commit log. Is that really necessary nowadays? Yah, that's the most emojis I've ever seen in a git repo. Maybe 2017 was a different era.
• ahnick 6y ago
  Fim is interesting, but I think most people (including myself here) don't worry about this because we rely on snapshots of backups. If a file was updated incorrectly then you just go back to the last known snapshot that worked. Of course this is only as good as the interval you use for your snapshots. Fim seems like it would be a lot of extra work to do for every file/folder, but I could potentially see using Fim though for important files (e.g. tax records?, password vault?)
  • vczf 6y ago
    Fim works best for the types of files that don't change frequently, or shouldn't change at all. I use it for photos, digital art files, videos, music, books, records/receipts, etc. When I started, I put everything into fim, but there was too much noise to be useful. Files that change frequently (password database) or won't totally bork out if there's a bit flip (text notes and org files) don't really need their integrity managed manually.