15 ms·

Linux NILFS file system: automatic continuous snapshots

242 points by solene 4y ago

cmurf 4y ago
Very nice introduction to NILSFS, which has been in the Linux kernel since 2009.
throwaway787544 4y ago
koolba 4y ago
How does this compare to ZFS + cron to create snapshots every X minutes?
- goodpoint 4y ago
  There is no comparison. NILFS provides *continuous* snaphots, so you can inspect and rollback changes as needed. It does without a performance penalty compared to other logging filesystems. And without using additional space forever. The backlog rotates forward continuously. It's a really unique feature that makes a lot of sense for desktop use, where you might want to recover files that were created and deleted after a short time.
  - harvie 4y ago
    "It does without a performance penalty" yeah. it's already so terribly slow that it's unlikely that taking snapshots can make it any slower :-D
    - Volundr 4y ago
      That was not my experience with NILFS. It outperformed ext4 on my laptop NVME.
      - harvie 4y ago
        i think i was running it on 6TB conventional HDD RAID1. also note that the read and write speeds might be quite asymetrical... in general also depends on workload type.
      - akvadrako 4y ago
        The benchmarks here look pretty bad: https://www.phoronix.com/review/linux-58-filesystems/4 https://www.phoronix.com/review/linux-58-filesystems/4
        Volundr 4y ago
        The last page looks pretty bad. If you look at the others it's more of a mixed bag, but yeah. I don't remember what benchmark I ran before deciding to run it on my laptop. Given my work at the time probably pgbench, but I couldn't say for sure. It was long enough ago I also might've been benchmarking against ext3, not 4.
  - fuckstick 4y ago
    > It does without a performance penalty. What is the basis for comparison? Sounds like a pretty meaningless statement at its face.
    - goodpoint 4y ago
      Compared to other logging filesystems obviously.
      - deleted 4y ago
        [deleted]
      - fuckstick 4y ago
        Nilfs baseline (write throughput especially) is slow as shit compared to other filesystems including f2fs. So just because you have this feature that doesn’t make it even slower isn’t that interesting - you pay for it one way or the other.
        usr1106 4y ago
        For many users filesystem speed of your home directory is completely irrelevant unless you run on a Raspberry Pi using SD cards. You just don't notice it. Of course if you haver server handling let's say video files things will be very different. And there are some users who process huge amounts of data. I run 2 lvm snapshots (daily and weekly) on my home partition for years. Write performance is abysmal if you measure it, but you don't note it in daily development work.
  - 1MachineElf 4y ago
    >It's a really unique feature that makes a lot of sense for desktop us Sounds like it could serve as a basis for a Linux implementation of something like Apple Time Machine.
    - deleted 4y ago
      [deleted]
    - mustache_kimono 4y ago
      With 'httm', a few of us are already living in that bright future: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
    - masklinn 4y ago
      Afaik Time Machine does not do continuous snapshots, just periodic (and triggered). So you can already do that with zfs: take a snapshot and send it to the backup drive.
  - harvie 4y ago
    Perhaps we can leverage "inotify" API to make ZFS snapshot everytime some file had been changed... But i think ZFS is not really good at handling huge amounts of snapshots. The NILFS2 snapshots are probably more lightweight when compared to ZFS ones.
    - goodpoint 4y ago
      The NILFS snapshots are practically free (for a logging filesystem, obviously).
    - mustache_kimono 4y ago
      > Perhaps we can leverage "inotify" API to make ZFS snapshot everytime some file had been changed... ZFS and btrfs users are already living in the future: inotifywait -r -m --format %w%f -e close_write "/srv/downloads/" | while read -r line; do # command below will snapshot the dataset # upon which the closed file is located sudo httm --snap "$line" done See: https://kimono-koans.github.io/inotifywait/ https://kimono-koans.github.io/inotifywait/
      - harvie 4y ago
        What is httm? I like this script as a proof of concept. But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script). All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome. So for real reliability you would probably want this to actualy run in ZFS/kernel context...
        mustache_kimono 4y ago
        > What is httm? I like this script as a proof of concept. See: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm > But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script). I mean, sure, scripts gonna script. You're gonna have to make the POC work for you. But, for instance, I'm not sure half of your issues are problems with a systemd service. I'm not sure one is a problem with a well designed script, which accounts for your particular issues, and a systemd service. > All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome. Yes, you can? Just re this POC, you can inotifywait a single directory, which contains multiple datasets, and httm will correctly determine and snapshot the correct one upon command. Your real bottleneck here is not sudo or bash. It's the zfs command waiting for a transaction group sync, or arranging for the trans group (or even something else, but its definitely zfs?), to snap. You can also use `httm -m` to simply identify the dataset and use a channel program and/or a separate script to sync. sudo and bash may not have the performance for your use case, hell, they are composable with everything else? > So for real reliability you would probably want this to actualy run in ZFS/kernel context... Yeesh, I'm not sure? Maybe for your/a few specific use cases? Note, inotify (a kernel facility) is your other bottleneck. You're never going to want to watch more than a few/10s of thousand files. The overhead is just going to be too great. But for most use cases (your documents folder)? Give httm and inotifywait a shot.
  - deleted 4y ago
    [deleted]
  - pkulak 4y ago
    > There is no comparison. What if I compare it to BTRFS + Snapper? No performance penalty there, plus checksumming.
    - AshamedCaptain 4y ago
      btrfs and snapperd do have a performance penalty as the number of snapshots increases. Having 100+ usually means snapper list will take north of an hour. You can easily reach these numbers if you are taking a snapshot every handful of minutes. Even background snapper cleanups will start to take a toll, since even if they are done with ionice they tend to block simultaneous accesses to the filesystem while they are in progress. If you have your root on the same filesystem, it's not pretty -- lots of periodic system-wide freezes with the HDD LEDs non-stop blinking. I tend to limit snapshots always to < 20 for that reason (and so does the default snapperd config).
    - mike256 4y ago
      About 2 years ago I believed the same. Then I used BTRFS as a store for VM images (with periodoc snapshot) and performance went down to really really bad. After I deleted all snapshots performance was good again. There is a big performance penalty in btrfs with more than about 100 snapshots.
      - jinnko 4y ago
        Did you disable CoW on the VM image files? This makes a significant difference to performance on BTRFS
- harvie 4y ago
  Week ago my client lost data on ZFS by accidentaly deleting folder. Unfortunately the data was created and deleted in the meantime between two snapshots. One would expect that it still might be possible to recover, because ZFS is CoW. There are some solutions like photorec (which now has ZFS support), but it expects you can identify the file by footprint of its contents, which was not the case. Also many of these solutions would require ZFS to go offline for forensic analysis and that was also not possible because lots of other clients were using the same pool at the time. So this had failed me and i really wished at the time that ZFS had continuous snapshots. BTW on ZFS i use ZnapZend. It's second best thing after continuous snapshots: https://www.znapzend.org/ https://www.znapzend.org/ https://github.com/oetiker/znapzend/ https://github.com/oetiker/znapzend/ There are also some ZFS snapshotting daemons in Debian, but this is much more elegant and flexible. But since znapzend is userspace daemon (as are all ZFS snapshoters) you need some kind of monitoring and warning mechanism for cases something goes wrong and it can't longer create snapshots (crashes, gets killed by OOM or something...). In NILFS2 every write/delete is snapshot, so you are basicaly guaranteed by kernel to have everything snapshoted without having to watch it.
- yonrg 4y ago
  I run this setup. zfs + zfsnap (not cron anymore, now systemd.timer). I cannot tell if NILFS is doing this too, with zfsnap I maintain different retention times. 5-minutely for 1hour, hourly for 1day, daily for a week. That are less than 60 snapshots. The older ones are cleaned up. In addition, zfs brings compression and encryption. That's why I have it on the laptops, too.
Rygian 4y ago
How close is this to a large continuous tape loop for video surveillance? I would very much welcome a filesystem that breaks away from the directories/files paradigm. Any time-based data store would greatly benefit from that.
- rcthompson 4y ago
  I think all you would need to add is a daemon that automatically deletes the oldest file(s) whenever free space drops below a certain threshold, so that the filesystem GC can reclaim that space for new files.
  - Rygian 4y ago
    I know and use 'logrotate'. My point was more on the tracks of a filesystem where a single file can be overwritten over and over again, and it's up to the filesystem to transparently ensure the full capacity of the disk is put towards retaining old versions of the file.
    - nix23 4y ago
      Hmm maybe something like Bluestore? https://docs.ceph.com/en/latest/rados/configuration/storage-devices/#bluestore https://docs.ceph.com/en/latest/rados/configuration/storage-...
      - Rygian 4y ago
        I definitely need to dive into Ceph, thanks for the pointer :-)
  - tommiegannert 4y ago
    If NILFS is continuously checkpointing, couldn't you even remove the file right after you add it, for simplicity?
Volundr 4y ago
NILFS is really, really cool. In concept. Unfortunately the tooling and support just isn't there. I ran it for quite some time on my laptop and the continuous snapshoting is everything I hoped it'd be. At one point however there was a change to the kernel that rendered it unbootable. Despite being a known and recorded bug it took forever to get fixed (about a year if I recall correctly) leaving me stuck on an old kernel the whole time. This was made more frustrating by the lack of any tooling such as fsck to help me diagnose the issue. The only reason I figured out it was a bug was that I booted a live CD to try to rescue the system and it booted fine. When I finally replaced that laptop I went back to ZFS and scripted snapshots. As much as I want to, I just can't recommend NILFS for daily use.
- CGamesPlay 4y ago
  How did Linus not go on a rampage after breaking userspace for an entire year? Is NILFS not part of the kernel mainline, I guess?
  - yjftsjthsd-h 4y ago
    > Is NILFS not part of the kernel mainline, I guess? Good guess, but no: https://github.com/torvalds/linux/tree/master/fs/nilfs2 https://github.com/torvalds/linux/tree/master/fs/nilfs2 > How did Linus not go on a rampage after breaking userspace for an entire year? I would very much like to know that as well. Any chance it didn't get reported (at least, not as "this broke booting")?
    - Volundr 4y ago
      I reported it along with a few other users in https://marc.info/?l=linux-nilfs&m=157540765215806&w=2 https://marc.info/?l=linux-nilfs&m=157540765215806&w=2. I think it just isn't widely enough used that Linus noticed we were broken. If I recall correctly it also wasn't directly fixed so much as incidentally. I just kept checking new kernel versions as they were released until one worked. There was never anything in the change-log (that I recall) about fixing the bug, just another change that happened to fix the issue. Edit: Looking through the archives, it looks like my memory was somewhat uncharitable. It was reported in November and directly patched in June (https://marc.info/?l=linux-nilfs&m=159154670627428&w=2 https://marc.info/?l=linux-nilfs&m=159154670627428&w=2) so about 7 months after reporting. Not sure what kernel release that would've landed in, so could've been closer to 8.
newcup 4y ago
I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version of the file system from few hours or weeks ago to access overwritten or deleted data. I use NILFS also on backup disks to provide combined deduplication and snapshots easily (just rsync & NILFS’ mkss, latter to make sure the “checkpoints” aren’t unnoticedly garbage collected in case the backup disk gets full).
- nix23 4y ago
  >I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014 Yes it's really sad, there we have a native and stable check-summing fs, and nearly no one knows about it.
  - yjftsjthsd-h 4y ago
    > check-summing fs Is it? Last I'd heard was > nilfs2 store checksums for all data. However, at least the current implementation does not verify it when reading. https://www.spinics.net/lists/linux-nilfs/msg01063.html https://www.spinics.net/lists/linux-nilfs/msg01063.html
    - nix23 4y ago
      Hmm you could be right, i found nothing about that it is calculated at read-time. Just with fsck.
  - conradev 4y ago
    BTRFS is also a native copy on write filesystem that verifies a configurable checksum and supports snapshots. The snapshots are not automatic, but short of that it is pretty feature complete
    - 4y ago
harvie 4y ago
I had issues with file locking when running some legacy database software on NILFS2. Probably caused data corruption in that database (not the FS itself). SF website of NILFS2 suggests that there are some unimplemented features, one of them being synchronous IO, which might have caused that issue? https://nilfs.sourceforge.io/en/current_status.html https://nilfs.sourceforge.io/en/current_status.html In some cases, the NILFS2 is safer storage for your data than ZFS. So NILFS might work for some simple usecases (eg. localy storing documents that you modify often), but it's certainly not ready to be deployed as generic filesystem. It's relatively slow and sometimes behaves bit weird. If something goes really bad, the recovery might be bit painfull. There is no fsck yet, nor community support. NILFS2 can self-heal itself to some extent. I really like the idea of NILFS2 but at this point i would prefer patch adding continuous snapshotting to ZFS. Unlike NILFS2 the ZFS have lots of active developers and big community. While NILFS2 is almost dead. The fact it's been in kernel for quite some time and most people didn't even noticed it (despite it's very interresting features) speaks for itself. Don't get me wrong. I wish that more developers get interested in NILFS2 and fix these issues and make it on par with EXT4, XFS and ZFS... But still ZFS has more features overall, so we might just add continuous snapshots in memoriam of NILFS2.
- yjftsjthsd-h 4y ago
  > In some cases, the NILFS2 is safer storage for your data than ZFS. What cases? Do you just mean due to continuous snapshots protecting against accidental deletes or such, or are there more "under the covers" things it fixes?
  - ComputerGuru 4y ago
    It’s basically append-only for recent things so you theoretically you can’t lose anything (within a reasonable timeframe). I don’t know if the porcelain exposes everything you need to avail yourself of that design functionality, though.
  - harvie 4y ago
    Yeah. i mean just due to continuous snapshots. Otherwise i don't really trust it that much. It's like this outsider kid which is kinda cool. But still needs to grow up and sell it :-D
jerf 4y ago
What happens if you run "dd if=/dev/zero of=/any/file/here", thus simply loading the disk with all the zeros it can handle? Do you lose all your snapshots as they are deleted to make room, or does it keep some space aside for this situation? (Not a "gotcha" question, a legitimate question.)
- solene 4y ago
  the garbage collector daemon will delete older checkpoints beyond the preserve time to make some room.
- Volundr 4y ago
  It's configurable: https://nilfs.sourceforge.io/en/man5/nilfs_cleanerd.conf.5.html https://nilfs.sourceforge.io/en/man5/nilfs_cleanerd.conf.5.h.... Cleanerd is responsible for maintaining a certain amount of free space on the system, and you can control the rules for doing so (e.x. a checkpoint won't be eligible for being cleaned until it is 1 week old). It's also worth knowing NILFS2 has checkpoints and snapshots. What you actually get are continuous "checkpoints". These can be upgraded to snapshots at any time with a simple command. Checkpoints are garbage collected, snapshots are not (until they are downgraded back into checkpoints).
- regularfry 4y ago
  I know this isn't what you're getting at, but is it smart enough to create a sparse file when you specifically pick zero as your filler byte?
wazoox 4y ago
I've been running NILFS2 on my main work NAS for 8 years. It never failed us :)
- mdaniel 4y ago
  I mean this honestly: how did you evaluate such a new filesystem in order to bet a work NAS upon it?
  - yonrg 4y ago
    I would do it by using it! ... and probably some backup
  - wazoox 4y ago
    I've made some testing, and installed it on a secondary system that in the beginning mostly hosted unimportant files. Then we added more things, and as after a few years it posed absolutely no problem we went further (and added a backup procedure). Then we migrated to new hardware, and it's still going strong (it's quite small, about 15 TB volume).
didgetmaster 4y ago
Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file system must treat your precious thesis paper the same way it treats a huge cat video you downloaded off the Internet. We need some kind of 'object store' where each object can be given a set of attributes that govern how the file system treats it. Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about. I have been working on a kind of object file system that addresses this problem.
- nix23 4y ago
  Well you can do that kind of with zfs filesystems, and the "object" is the recordsize.
  - mustache_kimono 4y ago
    I was going to ask: "Is there any limit on the number of ZFS filesystems in a pool?" Google says 2^64 is the limit. Couldn't one just just generate a filesystem per object if snapshots, etc., on a per object level is what one cared about? Wonder how quickly this would fall over? > Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about. This GP comment is a little goofy though. There was a user I once encountered who wanted ZFS, but a la carte. "I want the snapshots but I don't need COW." You have to explain, "You don't get the snapshots unless you have the COW", etc.
    - Conan_Kudo 4y ago
      On Btrfs, you can mark a folder/file/subvolume to have nocow, which has the effect of only doing a COW operation when you are creating snapshots.
remram 4y ago
How is this pronounced? Nil-F-S? Nilfuss? Nai-L-F-S? N-I-L-F-S?
- heavyset_go 4y ago
  The first one.
sargun 4y ago
I've always wondered why NILFS (or similar) isn't used for cases where ransomware is a risk. I'm honestly surprised that it's not mandated to use a append-only / log-structured filesystem for some critical systems (think patient records), where the cost of losing data is so high, rarely mutated, and trading it off for wasting storage isn't that bad (after all, HDD storage is incredibly cheap, and nobody said you had to keep the working set and the log on the same device).
- compsciphd 4y ago
  you don't need a log structured fs to do this, you could just have regular zfs/btrfs snapshots too. BUT if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, the only reason they might not is due to "security through obscurity". now, what I have argued is that an append only file system which works in a SAN like environment (i.e. you have random reads, but only append writes properties that are enforced remotely) could give you that, but to an extent you'd still get a similar behavior by just exporting ZFS shares (or even as block devices) and snapshotting them regularly on the remote end.
  - ephbit 4y ago
    > if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, .. How so? Let's say you have one machine holding the actual data for working on it. And some backup server. You could use btrfs send over ssh and regularly btrfs receive the data on the backup machine. Even it they got encrypted by ransomware they wouldn't be lost in the backups. As long as they're not deleted there how could a compromised work machine compromise the data on the backup machine?
    - compsciphd 4y ago
      I'm not anywhere near a user of btrfs send/recieve api, but I was under the impression that it essentially streams commands and hence could conceptually also be used to delete snapshots on the remote side?
compsciphd 4y ago
we used NILFS 15 years ago in dejaview - https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search it and enable you to recreate the state as it was at that time with non noticeable interruption to the user (processes downtime was like 0.02s).
- heavyset_go 4y ago
  This is cool, thanks for sharing it.
- bcjordan 4y ago
  Super cool work. Are any tools like this available today? I know some VM tools have snapshotting but full history and high speed scrubbing sounds awesome.
  - compsciphd 4y ago
    sadly (as with much work form phd students like I was), the closest one could get to it today is trying to duplicate it. i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it. In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in research papers, but also not a standard) and something that was built as an X display driver 15 years ago (so not directly usable today even if code is available). Could probably duplicate it with vnc based screencasting. vnc didn't work for us as we needed better performance (i.e. it was built to demonstrate remote display of video and games and there was no real remote audio setup back then so we had to create our own). the "text" search just used gnome's accessible API much like a screenreader would do (with a bit of per application optimizations as can filter out things like menus and the like, primarily was to dump text out of terminals, firefox and perhaps open office and maybe even a pdf reader if memory serves me correctly, but a long time ago).
Nifty3929 4y ago
Do any file systems have good, native support for tagging and complex searched based on those tags?
- DannyBee 4y ago
  BeFS was the last real one i'm aware of at the complexity you are talking about (plenty of FSen have some very basic indexed support for say file sizes , but not the kind of generic tagging you are talking about) At this point, the view seems to be "attributes happen in the file system, indexing happens in user space". Especially on linux. Part of the reason is, as i understand it, the surface/complexity of including query languages in the kernel, which is not horribly unreasonable So all the common FSen have reasonable xattr support, and inotify/etc that support notification of attribute changes. The expectation seems to be that the fact that inotify might drop events now and then is not a dealbreaker. The modern queue length is usually 16384 anyway. I'm not saying there aren't tradeoffs here, but this seems to be the direction taken overall. I actually would love to have an FS with native indexed xattr and a way to get at them. I just don't think we'll get back there again anytime soon.
  - Nifty3929 4y ago
    Okay - how about tagging and non-complex searches then. Beggars can't be choosers :-) Really what I'd like is just to search for some specific tags, or maybe list a directory excluding some tag, or similar. For bonus points, maybe a virtual directory that represents a search like this, and which "contains" the results of that search. (A "Search Folder") I'll check out BeFS. Thanks!
darau1 4y ago
What's the difference between a snapshot, and a checkpoint?
- okasaki 4y ago
  from TA: > A checkpoint is a snapshot of your system at a given point in time, but it can be deleted automatically if some disk space must be reclaimed. A checkpoint can be transformed into a snapshot that will never be removed.
nintendo1889 4y ago
I remember DEC/HP releasing the source to the digital unix AdvFS filesystem on sourceforge with the intent of porting it over to linux, but it never materialized. AdvFS had many advanced features. The source is still available and within it are some PDF slides that explain a lot of it's features.
- nix23 4y ago
  https://en.wikipedia.org/wiki/AdvFS https://en.wikipedia.org/wiki/AdvFS
  - nintendo1889 4y ago
    I noticed 'log based recovery'. Basically ReiserFS.
ggm 4y ago
Didn't VMS have this baked in? My memory is that all 8.3 file names had 8.3[;nnn] version tagging under the hood
- usr1106 4y ago
  That's what it looked like, but I doubt it was deep in the filesystem. It was basically just a naming convention. User had to purge old versions manually. This gets tedious if you have many files that change often. Snapshots are a safety net, not something you want to have in your way all day long.
  - ggm 4y ago
    Er.. my memory is that it did COW inside VMS fs semantics and was not manually achived. You did have to manually delete. So I don't think it was just a hack. It didn't do directories so was certainly not as good as snapshot but we're talking 40 years ago!
    - nix23 4y ago
      https://en.wikipedia.org/wiki/OpenVMS#File_system https://en.wikipedia.org/wiki/OpenVMS#File_system >> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
      - ggm 4y ago
        The link you want is https://en.m.wikipedia.org/wiki/Files-11 https://en.m.wikipedia.org/wiki/Files-11 Every file has a version number, which defaults to 1 if no other versions of the same filename are present (otherwise one higher than the greatest version). Every time a file is saved, rather than overwriting the existing version, a new file with the same name but an incremented version number is created. Old versions can be deleted explicitly, with the DELETE or the PURGE command, or optionally, older versions of a file can be deleted automatically when the file's version limit is reached (set by SET FILE/VERSION_LIMIT). Old versions are thus not overwritten, but are kept on disk and may be retrieved at any time. The architectural limit on version numbers is 32767. The versioning behavior is easily overridden if it is unwanted. In particular, files which are directly updated, such as databases, do not create new versions unless explicitly programmed.
LeoPanthera 4y ago
What happens when you store a virtual machine hard disk image on this? When you boot the VM, is the entire image duplicated via CoW every time something changes in the VM?
warinukraine 4y ago

242 points by solene 4y ago

cmurf 4y ago

Very nice introduction to NILSFS, which has been in the Linux kernel since 2009.

throwaway787544 4y ago

koolba 4y ago

How does this compare to ZFS + cron to create snapshots every X minutes?

goodpoint 4y ago
There is no comparison. NILFS provides *continuous* snaphots, so you can inspect and rollback changes as needed. It does without a performance penalty compared to other logging filesystems. And without using additional space forever. The backlog rotates forward continuously. It's a really unique feature that makes a lot of sense for desktop use, where you might want to recover files that were created and deleted after a short time.
- harvie 4y ago
  "It does without a performance penalty" yeah. it's already so terribly slow that it's unlikely that taking snapshots can make it any slower :-D
  - Volundr 4y ago
    That was not my experience with NILFS. It outperformed ext4 on my laptop NVME.
    - harvie 4y ago
      i think i was running it on 6TB conventional HDD RAID1. also note that the read and write speeds might be quite asymetrical... in general also depends on workload type.
    - akvadrako 4y ago
      The benchmarks here look pretty bad: https://www.phoronix.com/review/linux-58-filesystems/4 https://www.phoronix.com/review/linux-58-filesystems/4
      - Volundr 4y ago
        The last page looks pretty bad. If you look at the others it's more of a mixed bag, but yeah. I don't remember what benchmark I ran before deciding to run it on my laptop. Given my work at the time probably pgbench, but I couldn't say for sure. It was long enough ago I also might've been benchmarking against ext3, not 4.
- fuckstick 4y ago
  > It does without a performance penalty. What is the basis for comparison? Sounds like a pretty meaningless statement at its face.
  - goodpoint 4y ago
    Compared to other logging filesystems obviously.
    - deleted 4y ago
      [deleted]
    - fuckstick 4y ago
      Nilfs baseline (write throughput especially) is slow as shit compared to other filesystems including f2fs. So just because you have this feature that doesn’t make it even slower isn’t that interesting - you pay for it one way or the other.
      - usr1106 4y ago
        For many users filesystem speed of your home directory is completely irrelevant unless you run on a Raspberry Pi using SD cards. You just don't notice it. Of course if you haver server handling let's say video files things will be very different. And there are some users who process huge amounts of data. I run 2 lvm snapshots (daily and weekly) on my home partition for years. Write performance is abysmal if you measure it, but you don't note it in daily development work.
- 1MachineElf 4y ago
  >It's a really unique feature that makes a lot of sense for desktop us Sounds like it could serve as a basis for a Linux implementation of something like Apple Time Machine.
  - deleted 4y ago
    [deleted]
  - mustache_kimono 4y ago
    With 'httm', a few of us are already living in that bright future: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm
  - masklinn 4y ago
    Afaik Time Machine does not do continuous snapshots, just periodic (and triggered). So you can already do that with zfs: take a snapshot and send it to the backup drive.
- harvie 4y ago
  Perhaps we can leverage "inotify" API to make ZFS snapshot everytime some file had been changed... But i think ZFS is not really good at handling huge amounts of snapshots. The NILFS2 snapshots are probably more lightweight when compared to ZFS ones.
  - goodpoint 4y ago
    The NILFS snapshots are practically free (for a logging filesystem, obviously).
  - mustache_kimono 4y ago
    > Perhaps we can leverage "inotify" API to make ZFS snapshot everytime some file had been changed... ZFS and btrfs users are already living in the future: inotifywait -r -m --format %w%f -e close_write "/srv/downloads/" | while read -r line; do # command below will snapshot the dataset # upon which the closed file is located sudo httm --snap "$line" done See: https://kimono-koans.github.io/inotifywait/ https://kimono-koans.github.io/inotifywait/
    - harvie 4y ago
      What is httm? I like this script as a proof of concept. But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script). All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome. So for real reliability you would probably want this to actualy run in ZFS/kernel context...
      - mustache_kimono 4y ago
        > What is httm? I like this script as a proof of concept. See: https://github.com/kimono-koans/httm https://github.com/kimono-koans/httm > But i still can imagine failure modes, eg. inotify might start acting weird when ZFS remounts the watched directory, OOM killer terminates it without anyone noticing, bash loop go haywire when package manager updates that script (bash is running directly from the file and when it changes during execution, it might just continue running from the same byt offset in completely different script). I mean, sure, scripts gonna script. You're gonna have to make the POC work for you. But, for instance, I'm not sure half of your issues are problems with a systemd service. I'm not sure one is a problem with a well designed script, which accounts for your particular issues, and a systemd service. > All these things actualy happened to me in the past. Not to say that if you have multiple datasets in ZFS you cannot inotify wait on all of them at once, so you will have to manage one bash process per dataset. And performance of bash and sudo might not be that awesome. Yes, you can? Just re this POC, you can inotifywait a single directory, which contains multiple datasets, and httm will correctly determine and snapshot the correct one upon command. Your real bottleneck here is not sudo or bash. It's the zfs command waiting for a transaction group sync, or arranging for the trans group (or even something else, but its definitely zfs?), to snap. You can also use `httm -m` to simply identify the dataset and use a channel program and/or a separate script to sync. sudo and bash may not have the performance for your use case, hell, they are composable with everything else? > So for real reliability you would probably want this to actualy run in ZFS/kernel context... Yeesh, I'm not sure? Maybe for your/a few specific use cases? Note, inotify (a kernel facility) is your other bottleneck. You're never going to want to watch more than a few/10s of thousand files. The overhead is just going to be too great. But for most use cases (your documents folder)? Give httm and inotifywait a shot.
- deleted 4y ago
  [deleted]
- pkulak 4y ago
  > There is no comparison. What if I compare it to BTRFS + Snapper? No performance penalty there, plus checksumming.
  - AshamedCaptain 4y ago
    btrfs and snapperd do have a performance penalty as the number of snapshots increases. Having 100+ usually means snapper list will take north of an hour. You can easily reach these numbers if you are taking a snapshot every handful of minutes. Even background snapper cleanups will start to take a toll, since even if they are done with ionice they tend to block simultaneous accesses to the filesystem while they are in progress. If you have your root on the same filesystem, it's not pretty -- lots of periodic system-wide freezes with the HDD LEDs non-stop blinking. I tend to limit snapshots always to < 20 for that reason (and so does the default snapperd config).
  - mike256 4y ago
    About 2 years ago I believed the same. Then I used BTRFS as a store for VM images (with periodoc snapshot) and performance went down to really really bad. After I deleted all snapshots performance was good again. There is a big performance penalty in btrfs with more than about 100 snapshots.
    - jinnko 4y ago
      Did you disable CoW on the VM image files? This makes a significant difference to performance on BTRFS
harvie 4y ago
Week ago my client lost data on ZFS by accidentaly deleting folder. Unfortunately the data was created and deleted in the meantime between two snapshots. One would expect that it still might be possible to recover, because ZFS is CoW. There are some solutions like photorec (which now has ZFS support), but it expects you can identify the file by footprint of its contents, which was not the case. Also many of these solutions would require ZFS to go offline for forensic analysis and that was also not possible because lots of other clients were using the same pool at the time. So this had failed me and i really wished at the time that ZFS had continuous snapshots. BTW on ZFS i use ZnapZend. It's second best thing after continuous snapshots: https://www.znapzend.org/ https://www.znapzend.org/ https://github.com/oetiker/znapzend/ https://github.com/oetiker/znapzend/ There are also some ZFS snapshotting daemons in Debian, but this is much more elegant and flexible. But since znapzend is userspace daemon (as are all ZFS snapshoters) you need some kind of monitoring and warning mechanism for cases something goes wrong and it can't longer create snapshots (crashes, gets killed by OOM or something...). In NILFS2 every write/delete is snapshot, so you are basicaly guaranteed by kernel to have everything snapshoted without having to watch it.
yonrg 4y ago
I run this setup. zfs + zfsnap (not cron anymore, now systemd.timer). I cannot tell if NILFS is doing this too, with zfsnap I maintain different retention times. 5-minutely for 1hour, hourly for 1day, daily for a week. That are less than 60 snapshots. The older ones are cleaned up. In addition, zfs brings compression and encryption. That's why I have it on the laptops, too.

Rygian 4y ago

How close is this to a large continuous tape loop for video surveillance? I would very much welcome a filesystem that breaks away from the directories/files paradigm. Any time-based data store would greatly benefit from that.

rcthompson 4y ago
I think all you would need to add is a daemon that automatically deletes the oldest file(s) whenever free space drops below a certain threshold, so that the filesystem GC can reclaim that space for new files.
- Rygian 4y ago
  I know and use 'logrotate'. My point was more on the tracks of a filesystem where a single file can be overwritten over and over again, and it's up to the filesystem to transparently ensure the full capacity of the disk is put towards retaining old versions of the file.
  - nix23 4y ago
    Hmm maybe something like Bluestore? https://docs.ceph.com/en/latest/rados/configuration/storage-devices/#bluestore https://docs.ceph.com/en/latest/rados/configuration/storage-...
    - Rygian 4y ago
      I definitely need to dive into Ceph, thanks for the pointer :-)
- tommiegannert 4y ago
  If NILFS is continuously checkpointing, couldn't you even remove the file right after you add it, for simplicity?

Volundr 4y ago

NILFS is really, really cool. In concept. Unfortunately the tooling and support just isn't there. I ran it for quite some time on my laptop and the continuous snapshoting is everything I hoped it'd be. At one point however there was a change to the kernel that rendered it unbootable. Despite being a known and recorded bug it took forever to get fixed (about a year if I recall correctly) leaving me stuck on an old kernel the whole time. This was made more frustrating by the lack of any tooling such as fsck to help me diagnose the issue. The only reason I figured out it was a bug was that I booted a live CD to try to rescue the system and it booted fine. When I finally replaced that laptop I went back to ZFS and scripted snapshots. As much as I want to, I just can't recommend NILFS for daily use.

CGamesPlay 4y ago
How did Linus not go on a rampage after breaking userspace for an entire year? Is NILFS not part of the kernel mainline, I guess?
- yjftsjthsd-h 4y ago
  > Is NILFS not part of the kernel mainline, I guess? Good guess, but no: https://github.com/torvalds/linux/tree/master/fs/nilfs2 https://github.com/torvalds/linux/tree/master/fs/nilfs2 > How did Linus not go on a rampage after breaking userspace for an entire year? I would very much like to know that as well. Any chance it didn't get reported (at least, not as "this broke booting")?
  - Volundr 4y ago
    I reported it along with a few other users in https://marc.info/?l=linux-nilfs&m=157540765215806&w=2 https://marc.info/?l=linux-nilfs&m=157540765215806&w=2. I think it just isn't widely enough used that Linus noticed we were broken. If I recall correctly it also wasn't directly fixed so much as incidentally. I just kept checking new kernel versions as they were released until one worked. There was never anything in the change-log (that I recall) about fixing the bug, just another change that happened to fix the issue. Edit: Looking through the archives, it looks like my memory was somewhat uncharitable. It was reported in November and directly patched in June (https://marc.info/?l=linux-nilfs&m=159154670627428&w=2 https://marc.info/?l=linux-nilfs&m=159154670627428&w=2) so about 7 months after reporting. Not sure what kernel release that would've landed in, so could've been closer to 8.

newcup 4y ago

I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014. Apart from one kernel regression bug related to NILFS2 it’s worked flawlessly (no data corruption even with the bug just no access to the file system; effectively it forced running older kernel while the bug was fixed). The continuous snapshotting has saved me a couple of times; I’ve just mounted a version of the file system from few hours or weeks ago to access overwritten or deleted data. I use NILFS also on backup disks to provide combined deduplication and snapshots easily (just rsync & NILFS’ mkss, latter to make sure the “checkpoints” aren’t unnoticedly garbage collected in case the backup disk gets full).

nix23 4y ago
>I think NILFS is a hidden gem. I’ve been using it exclusively in my Linux laptops, desktops etc. since ca. 2014 Yes it's really sad, there we have a native and stable check-summing fs, and nearly no one knows about it.
- yjftsjthsd-h 4y ago
  > check-summing fs Is it? Last I'd heard was > nilfs2 store checksums for all data. However, at least the current implementation does not verify it when reading. https://www.spinics.net/lists/linux-nilfs/msg01063.html https://www.spinics.net/lists/linux-nilfs/msg01063.html
  - nix23 4y ago
    Hmm you could be right, i found nothing about that it is calculated at read-time. Just with fsck.
- conradev 4y ago
  BTRFS is also a native copy on write filesystem that verifies a configurable checksum and supports snapshots. The snapshots are not automatic, but short of that it is pretty feature complete
  - 4y ago

harvie 4y ago

I had issues with file locking when running some legacy database software on NILFS2. Probably caused data corruption in that database (not the FS itself). SF website of NILFS2 suggests that there are some unimplemented features, one of them being synchronous IO, which might have caused that issue? https://nilfs.sourceforge.io/en/current_status.html https://nilfs.sourceforge.io/en/current_status.html In some cases, the NILFS2 is safer storage for your data than ZFS. So NILFS might work for some simple usecases (eg. localy storing documents that you modify often), but it's certainly not ready to be deployed as generic filesystem. It's relatively slow and sometimes behaves bit weird. If something goes really bad, the recovery might be bit painfull. There is no fsck yet, nor community support. NILFS2 can self-heal itself to some extent. I really like the idea of NILFS2 but at this point i would prefer patch adding continuous snapshotting to ZFS. Unlike NILFS2 the ZFS have lots of active developers and big community. While NILFS2 is almost dead. The fact it's been in kernel for quite some time and most people didn't even noticed it (despite it's very interresting features) speaks for itself. Don't get me wrong. I wish that more developers get interested in NILFS2 and fix these issues and make it on par with EXT4, XFS and ZFS... But still ZFS has more features overall, so we might just add continuous snapshots in memoriam of NILFS2.

yjftsjthsd-h 4y ago
> In some cases, the NILFS2 is safer storage for your data than ZFS. What cases? Do you just mean due to continuous snapshots protecting against accidental deletes or such, or are there more "under the covers" things it fixes?
- ComputerGuru 4y ago
  It’s basically append-only for recent things so you theoretically you can’t lose anything (within a reasonable timeframe). I don’t know if the porcelain exposes everything you need to avail yourself of that design functionality, though.
- harvie 4y ago
  Yeah. i mean just due to continuous snapshots. Otherwise i don't really trust it that much. It's like this outsider kid which is kinda cool. But still needs to grow up and sell it :-D

jerf 4y ago

What happens if you run "dd if=/dev/zero of=/any/file/here", thus simply loading the disk with all the zeros it can handle? Do you lose all your snapshots as they are deleted to make room, or does it keep some space aside for this situation? (Not a "gotcha" question, a legitimate question.)

solene 4y ago
the garbage collector daemon will delete older checkpoints beyond the preserve time to make some room.
Volundr 4y ago
It's configurable: https://nilfs.sourceforge.io/en/man5/nilfs_cleanerd.conf.5.html https://nilfs.sourceforge.io/en/man5/nilfs_cleanerd.conf.5.h.... Cleanerd is responsible for maintaining a certain amount of free space on the system, and you can control the rules for doing so (e.x. a checkpoint won't be eligible for being cleaned until it is 1 week old). It's also worth knowing NILFS2 has checkpoints and snapshots. What you actually get are continuous "checkpoints". These can be upgraded to snapshots at any time with a simple command. Checkpoints are garbage collected, snapshots are not (until they are downgraded back into checkpoints).
regularfry 4y ago
I know this isn't what you're getting at, but is it smart enough to create a sparse file when you specifically pick zero as your filler byte?

wazoox 4y ago

I've been running NILFS2 on my main work NAS for 8 years. It never failed us :)

mdaniel 4y ago
I mean this honestly: how did you evaluate such a new filesystem in order to bet a work NAS upon it?
- yonrg 4y ago
  I would do it by using it! ... and probably some backup
- wazoox 4y ago
  I've made some testing, and installed it on a secondary system that in the beginning mostly hosted unimportant files. Then we added more things, and as after a few years it posed absolutely no problem we went further (and added a backup procedure). Then we migrated to new hardware, and it's still going strong (it's quite small, about 15 TB volume).

didgetmaster 4y ago

Does NILFS do checksums and snapshotting for every single file in the system? One of my biggest complaints about file systems in general is that they are all designed to treat every file the exact same way. We now have storage systems (even SSDs) that are big enough to hold hundreds of millions of files. Those files can be a mix of small files, big files, temp files, personal files, and public files. Yet every file system must treat your precious thesis paper the same way it treats a huge cat video you downloaded off the Internet. We need some kind of 'object store' where each object can be given a set of attributes that govern how the file system treats it. Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about. I have been working on a kind of object file system that addresses this problem.

nix23 4y ago
Well you can do that kind of with zfs filesystems, and the "object" is the recordsize.
- mustache_kimono 4y ago
  I was going to ask: "Is there any limit on the number of ZFS filesystems in a pool?" Google says 2^64 is the limit. Couldn't one just just generate a filesystem per object if snapshots, etc., on a per object level is what one cared about? Wonder how quickly this would fall over? > Backup, encryption, COW, checksums, and other operations should not be wasted on a bunch of data that no one really cares about. This GP comment is a little goofy though. There was a user I once encountered who wanted ZFS, but a la carte. "I want the snapshots but I don't need COW." You have to explain, "You don't get the snapshots unless you have the COW", etc.
  - Conan_Kudo 4y ago
    On Btrfs, you can mark a folder/file/subvolume to have nocow, which has the effect of only doing a COW operation when you are creating snapshots.

remram 4y ago

How is this pronounced? Nil-F-S? Nilfuss? Nai-L-F-S? N-I-L-F-S?

heavyset_go 4y ago
The first one.

sargun 4y ago

I've always wondered why NILFS (or similar) isn't used for cases where ransomware is a risk. I'm honestly surprised that it's not mandated to use a append-only / log-structured filesystem for some critical systems (think patient records), where the cost of losing data is so high, rarely mutated, and trading it off for wasting storage isn't that bad (after all, HDD storage is incredibly cheap, and nobody said you had to keep the working set and the log on the same device).

compsciphd 4y ago
you don't need a log structured fs to do this, you could just have regular zfs/btrfs snapshots too. BUT if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, the only reason they might not is due to "security through obscurity". now, what I have argued is that an append only file system which works in a SAN like environment (i.e. you have random reads, but only append writes properties that are enforced remotely) could give you that, but to an extent you'd still get a similar behavior by just exporting ZFS shares (or even as block devices) and snapshotting them regularly on the remote end.
- ephbit 4y ago
  > if an attack has the ability to delete an entire file system / encrypt it, they really have the ability to delete the snapshots as well, .. How so? Let's say you have one machine holding the actual data for working on it. And some backup server. You could use btrfs send over ssh and regularly btrfs receive the data on the backup machine. Even it they got encrypted by ransomware they wouldn't be lost in the backups. As long as they're not deleted there how could a compromised work machine compromise the data on the backup machine?
  - compsciphd 4y ago
    I'm not anywhere near a user of btrfs send/recieve api, but I was under the impression that it essentially streams commands and hence could conceptually also be used to delete snapshots on the remote side?

compsciphd 4y ago

we used NILFS 15 years ago in dejaview - https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf https://www.cs.columbia.edu/~nieh/pubs/sosp2007_dejaview.pdf We combined nilfs + our process snapshotting tech (we tried to mainline it, but it didn't go, but many of the concepts ended up in CRIU though) + our remote display + screen reading tech (i.e. normal APIs) to create an environment that could record everything you ever saw visually and textually. enable you to search it and enable you to recreate the state as it was at that time with non noticeable interruption to the user (processes downtime was like 0.02s).

heavyset_go 4y ago
This is cool, thanks for sharing it.
bcjordan 4y ago
Super cool work. Are any tools like this available today? I know some VM tools have snapshotting but full history and high speed scrubbing sounds awesome.
- compsciphd 4y ago
  sadly (as with much work form phd students like I was), the closest one could get to it today is trying to duplicate it. i.e. combining criu with nilfs (but a lot of the work that we did to get process downtime to minimal numbers requires being in kernel, as described in paper) and unsure criu can do it. In addition our screenrecording mechanism was our own "proprietary" (not really proprietary as fully described in research papers, but also not a standard) and something that was built as an X display driver 15 years ago (so not directly usable today even if code is available). Could probably duplicate it with vnc based screencasting. vnc didn't work for us as we needed better performance (i.e. it was built to demonstrate remote display of video and games and there was no real remote audio setup back then so we had to create our own). the "text" search just used gnome's accessible API much like a screenreader would do (with a bit of per application optimizations as can filter out things like menus and the like, primarily was to dump text out of terminals, firefox and perhaps open office and maybe even a pdf reader if memory serves me correctly, but a long time ago).

Nifty3929 4y ago

Do any file systems have good, native support for tagging and complex searched based on those tags?

DannyBee 4y ago
BeFS was the last real one i'm aware of at the complexity you are talking about (plenty of FSen have some very basic indexed support for say file sizes , but not the kind of generic tagging you are talking about) At this point, the view seems to be "attributes happen in the file system, indexing happens in user space". Especially on linux. Part of the reason is, as i understand it, the surface/complexity of including query languages in the kernel, which is not horribly unreasonable So all the common FSen have reasonable xattr support, and inotify/etc that support notification of attribute changes. The expectation seems to be that the fact that inotify might drop events now and then is not a dealbreaker. The modern queue length is usually 16384 anyway. I'm not saying there aren't tradeoffs here, but this seems to be the direction taken overall. I actually would love to have an FS with native indexed xattr and a way to get at them. I just don't think we'll get back there again anytime soon.
- Nifty3929 4y ago
  Okay - how about tagging and non-complex searches then. Beggars can't be choosers :-) Really what I'd like is just to search for some specific tags, or maybe list a directory excluding some tag, or similar. For bonus points, maybe a virtual directory that represents a search like this, and which "contains" the results of that search. (A "Search Folder") I'll check out BeFS. Thanks!

darau1 4y ago

What's the difference between a snapshot, and a checkpoint?

okasaki 4y ago
from TA: > A checkpoint is a snapshot of your system at a given point in time, but it can be deleted automatically if some disk space must be reclaimed. A checkpoint can be transformed into a snapshot that will never be removed.

nintendo1889 4y ago

I remember DEC/HP releasing the source to the digital unix AdvFS filesystem on sourceforge with the intent of porting it over to linux, but it never materialized. AdvFS had many advanced features. The source is still available and within it are some PDF slides that explain a lot of it's features.

nix23 4y ago
https://en.wikipedia.org/wiki/AdvFS https://en.wikipedia.org/wiki/AdvFS
- nintendo1889 4y ago
  I noticed 'log based recovery'. Basically ReiserFS.

ggm 4y ago

Didn't VMS have this baked in? My memory is that all 8.3 file names had 8.3[;nnn] version tagging under the hood

usr1106 4y ago
That's what it looked like, but I doubt it was deep in the filesystem. It was basically just a naming convention. User had to purge old versions manually. This gets tedious if you have many files that change often. Snapshots are a safety net, not something you want to have in your way all day long.
- ggm 4y ago
  Er.. my memory is that it did COW inside VMS fs semantics and was not manually achived. You did have to manually delete. So I don't think it was just a hack. It didn't do directories so was certainly not as good as snapshot but we're talking 40 years ago!
  - nix23 4y ago
    https://en.wikipedia.org/wiki/OpenVMS#File_system https://en.wikipedia.org/wiki/OpenVMS#File_system >> DEC attempted to replace it with a log-structured file system file system named Spiralog first released in 1995. However, Spiralog was discontinued due to a variety of problems, including issues with handling full volumes.
    - ggm 4y ago
      The link you want is https://en.m.wikipedia.org/wiki/Files-11 https://en.m.wikipedia.org/wiki/Files-11 Every file has a version number, which defaults to 1 if no other versions of the same filename are present (otherwise one higher than the greatest version). Every time a file is saved, rather than overwriting the existing version, a new file with the same name but an incremented version number is created. Old versions can be deleted explicitly, with the DELETE or the PURGE command, or optionally, older versions of a file can be deleted automatically when the file's version limit is reached (set by SET FILE/VERSION_LIMIT). Old versions are thus not overwritten, but are kept on disk and may be retrieved at any time. The architectural limit on version numbers is 32767. The versioning behavior is easily overridden if it is unwanted. In particular, files which are directly updated, such as databases, do not create new versions unless explicitly programmed.

LeoPanthera 4y ago

What happens when you store a virtual machine hard disk image on this? When you boot the VM, is the entire image duplicated via CoW every time something changes in the VM?

warinukraine 4y ago