9 ms·

Interactive, file-level Time Machine-like tool for ZFS

253 points by codetrotter 5y ago

gigatexal 5y ago
I've been waiting for someone to build this. Awesome!
water8 5y ago
Very cool! ZFS is a fantastic filesystem. Very easy to use and effective at preventing things like bit-rot which can plague large infrequently accessed datasets that are stored on cheap HDDs
aphrax 5y ago
Looks good. I still miss the OpenSolaris ZFS / Gnome Integration - 'time slider' iirc..
- joseph 5y ago
  Yes, me too. That was peak desktop experience over ten years ago.
- marcodiego 5y ago
  https://news.ycombinator.com/item?id=31185021 https://news.ycombinator.com/item?id=31185021
  - gclawes 5y ago
    Broken link?
    - ssl232 5y ago
      Enable showdead. It points to https://www.dailymotion.com/video/x7oe8l https://www.dailymotion.com/video/x7oe8l btw.
  - aphrax 5y ago
    thats the one, and it was great!
    - marcodiego 5y ago
      Most interesting is how snappy the thing looked, especially combined with compiz. Looks like we lost our way somewhere between 2008-2012 or these effects are really only impressive for a few hours. Nevertheless, I'm pretty satisfied with my desktop nowadays.
jedberg 5y ago
Fun fact: When Apple first built Time Machine, the goal was that ZFS would be the root filesystem. That's why the TM UX fits so nicely with ZFS. Sadly the licensing issues scared Apple off from adopting ZFS.
- vondur 5y ago
  I think it was due to a lawsuit EMC had going against Sun over ZFS.
  - GekkePrutser 5y ago
    I thought it was Jonathan Schwartz embarassing Steve Jobs by announcing it was coming to macOS himself. Which was a big deal at the time. I always thought that was a petty reason but I didn't really consider it beyond Steve Jobs to do that to be honest. He really took such matters very personally. But your explanation sounds more reasonable. I really thought this until now though.
    - throw0101a 5y ago
      It was a 'support contract' between Sun and Apple not being able to be agreed to: * https://web.archive.org/web/20121221111757/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/20121221111757/http://mail.opens...
  - dburkland 5y ago
    I think you mean with NetApp
- jjoonathan 5y ago
  Yeah, that original interface was wild https://youtu.be/2HixPtvIWhY?t=2174 https://youtu.be/2HixPtvIWhY?t=2174 if only ZFS hadn't fallen through! Speaking of which, are there any public plans to do this with APFS or are they just trying to push everyone into the cloud?
  - 5y ago
GekkePrutser 5y ago
Thanks for the link. This will be cool, I use FreeBSD as daily driver so ZFS is already a given.
Melatonic 5y ago
People kinda love shitting on Windows here but VSS (Shadow Copy) is actually a very underrated and solid feature. Works with NTFS and even better ReFS
- lacrosse_tannin 5y ago
  A long time ago, I set up some zfs-in-a-box-OS thing, and hooked to active directory (i think?) and samba, and windows explorer had a built in context menu to restore files to older versions.
- lacrosse_tannin 5y ago
  A long time ago, I set up some zfs-in-a-box-OS thing (some solaris fork), and hooked to active directory (i think?) and samba, and windows explorer had a built in context menu to restore files to older versions.
  - matja 5y ago
    Yep - this is the VFS shadow copy extension for samba, works great! Actually it doesn't require ZFS, just a method to create snapshots and find them - example smb.conf : vfs objects = shadow_copy2 shadow:snapdir = /home/.zfs/snapshot/ shadow:basedir = /home shadow:sort = desc shadow:format = auto:%Y-%m-%dT%H_%M_%SZ
- paulkon 5y ago
  Only thing with VSS is that ransomware can delete all your snapshots. Samba shares on ZFS via TrueNAS or the like with VSS support mitigates that attack vector. Speaking of, any stats out there on which proportion of ransomware is targeted at which distro for linux, freebsd, windows and mac os?
  - Melatonic 5y ago
    Lot of AV's specifically try to lock that down but yeah its a first pass for a lot of ransomware.
- 5y ago
tjoff 5y ago
This is great, I've often wondered why so little focus is spent on tools like this. Feels like low hanging fruit given the great usability improvements they can bring. All the talks about backups but the reason people don't do it is partly because the tooling is so bad. (no, snapshots is not a proper backup by itself, but a great addition to). Wondering if btrfs will get some love too.
- atoav 5y ago
  There is timeshift for btrfs. Sadly it only works for certain very standard btrfs subvolume layouts and if you deviate only slightly from that it is no longer supported..
  - vondur 5y ago
    Yes. I watched a tutorial video on YouTube where it shows how to label the BTRFS sub volumes correctly for Timeshift to work correctly in Fedora.
  - ravi-delia 5y ago
    It's not super hard to hack it if you need to, but yeah zfs exposes exactly what's needed without any global configuration. In exchange btrfs gets to be a little more flexible, but honestly idk how useful it is to treat snapshots as normal directories anyway. That's my only real complaint about btrfs on the desktop.
- Affric 5y ago
  My theory based on reading comments from ZFS devs is that as resources are somewhat limited and most people who use it are a bit technical these kind of things are low priority so they go slowly.
- mustache_kimono 5y ago
  > This is great, Thanks! (It's my little project.) > I've often wondered why so little focus is spent on tools like this. Feels like low hanging fruit given the great usability improvements they can bring. 100% agree. Definitely a QoL tool for the ZFS lover. The zsh key bindings open up uses I never really even dreamed up, because it's just so quick to get to a usable interface. Definitely also a very UNIX-y tool too. Recently I started fiddling with piping to git repos and tar archives and it's very cool -- see: https://github.com/kimono-koans/httm#example-usage https://github.com/kimono-koans/httm#example-usage > Wondering if btrfs will get some love too. btrfs makes this exact thing harder, but, cross fingers, the btrfs devs are listening, because I think btrfs folks would love a ".btrfs" directory like the ".zfs" directory.
rbanffy 5y ago
Interestingly, Solaris had a UI component for ZFS snapshots built into Gnome's file explorer. Not sure if it's still there though. IIRC, OpenSolaris had it. Sadly, the screenshots seem to be gone... https://web.archive.org/web/20081016104234/http://blogs.sun.com:80/erwann/entry/zfs_on_the_desktop_zfs https://web.archive.org/web/20081016104234/http://blogs.sun.... edit: found a couple places with the UI: https://www.linuxjournal.com/content/zfs-finding-its-way-linux-near-you https://www.linuxjournal.com/content/zfs-finding-its-way-lin... https://arstechnica.com/information-technology/2008/10/opensolaris-gets-zfs-snapshot-visualization-feature/ https://arstechnica.com/information-technology/2008/10/opens...
- yjftsjthsd-h 5y ago
  I couldn't say what Oracle's been doing, but writing this comment from OpenIndiana (Illumos) the feature is still here and works fine. It's integrated into MATE's Caja (forked GNOME/nautilus) and AFAICT it's the same exact feature and it still just works; you click the little clock button in the toolbar, click to the snapshot you want, and browse the filesystem. Actually looking more closely, it looks like it's a pretty thin veneer over the .zfs/snapshot virtual directory so it should be dead simple to build into any file browser that supports extensions (because all you need to do is provide a list of directories under $FS/.zfs/snapshot, and then when the user clicks one change path to $FS/.zfs/snapshot/$SNAPSHOT). I mean, there's a little more integration than that (some context menu items to snapshot and view old versions, an extra column in the detailed view) but that's the main part and I suspect the whole thing could be reimplemented in a day or two. Edit: Nicer screenshot: https://www.openindiana.org/documentation/gallery/oi-b148-gui-timeslider/ https://www.openindiana.org/documentation/gallery/oi-b148-gu...
  - brnt 5y ago
    Wonder why it's not used for browsing btrfs snapshots. It's such a great bit of ui.
    - yjftsjthsd-h 5y ago
      I have no specific knowledge but suspect that it just never cross-pollinated; illumos is a somewhat separate community from the Linux-sphere. For all I know, you can build the same code on Linux with OpenZFS and trivially switch it to BTRFS by replacing the paths.
seletskiy 5y ago
Shameless plug — https://github.com/reconquest/zeus https://github.com/reconquest/zeus Daemon for automatic incremental backup of your ZFS systems to the another (physically connected right now) pool.
seletskiy 5y ago
Also, some obscure yet very helpful ZFS feature is that all your ZFS datasets have hidden (e.g. not visible via `ls -a`) `.zfs` directory in their roots. This directory contains all snapshots of a given ZFS dataset mounted as directories. For example, if your home is a separate dataset, then `/home/operator/.zfs/` will contain all snapshots automatically mounted for your convenience.
- SkyMarshal 5y ago
  How do you show the .zfs directory if it ignores `ls -a`?
  - mappu 5y ago
    The whole VFS is an illusion, it's totally possible for the filesystem driver to say one thing in readdir(3) and another thing in fopen(3).
    - tbrownaw 5y ago
      Related fun fact: if you have +x but not +r on a directory, you can traverse it (access things whose names you know), but not view its contents (`ls` won't work).
  - 4oo4 5y ago
    You have to go into zfs settings to enable it, I think it's something like this: `zfs set snapdir=visible poolname/dataset` After that you can see it with `ls -la`.
- RandomBK 5y ago
  Huh, this is new to me. How does zfs handle regular directories that happen to be named .zfs? Might this be a security issue as well? What permissions are needed to access this path? If I somehow have a webserver serving static files from a zfs dataset, might someone use this to access old or deleted files?
  - 5y ago
mustache_kimono 5y ago
My little project. ;) Surprised and delighted its on the HN front page! If you use ZFS a bunch, like I do, you will use this little tool more than you may imagine. Also, FYI, for the trivia/history buffs -- cool bit of happenstance -- 20 years ago yesterday Jeff Bonwick filed the PSARC case for the ZFS filesystem. See: https://illumos.org/opensolaris/ARChive/PSARC/2002/240/onepager.opensolaris https://illumos.org/opensolaris/ARChive/PSARC/2002/240/onepa...
velcrovan 5y ago
The ideal file-level Time Machine-like tool for ZFS already exists and it is Windows Explorer. Not even kidding. Right click on any file or folder residing on a ZFS share and click on Properties, then click the Previous Versions tab. You can restore the file or folder from any snapshot, or just open it and look at it.
- 4oo4 5y ago
  That is, until you get NTFS bitrot since you don't have checksumming.
  - Infernal 5y ago
    GP specifies a ZFS share.
    - 4oo4 5y ago
      Ahh, interesting, I never thought about that. So that means you have a share mounted probably via NFS/SMB, Windows is able to read the zfs snapshots? Or are the Windows previous versions snapshots separate from the zfs ones?
      - mastax 5y ago
        You can configure samba to look for ZFS snapshots: vfs objects = shadow_copy2 shadow:snapdir = .zfs/snapshot shadow:sort = desc shadow:format = zfs-auto-snap_frequent-%Y-%m-%d-%H%M
        4oo4 5y ago
        That's awesome. I never expected Windows could integrate with zfs like that. Speaking of, I always forget this exists too: https://github.com/openzfsonwindows/openzfs/ https://github.com/openzfsonwindows/openzfs/
        5y ago

253 points by codetrotter 5y ago

I've been waiting for someone to build this. Awesome!

Very cool! ZFS is a fantastic filesystem. Very easy to use and effective at preventing things like bit-rot which can plague large infrequently accessed datasets that are stored on cheap HDDs

aphrax 5y ago

Looks good. I still miss the OpenSolaris ZFS / Gnome Integration - 'time slider' iirc..

joseph 5y ago
Yes, me too. That was peak desktop experience over ten years ago.
marcodiego 5y ago
https://news.ycombinator.com/item?id=31185021 https://news.ycombinator.com/item?id=31185021
- gclawes 5y ago
  Broken link?
  - ssl232 5y ago
    Enable showdead. It points to https://www.dailymotion.com/video/x7oe8l https://www.dailymotion.com/video/x7oe8l btw.
- aphrax 5y ago
  thats the one, and it was great!
  - marcodiego 5y ago
    Most interesting is how snappy the thing looked, especially combined with compiz. Looks like we lost our way somewhere between 2008-2012 or these effects are really only impressive for a few hours. Nevertheless, I'm pretty satisfied with my desktop nowadays.

jedberg 5y ago

Fun fact: When Apple first built Time Machine, the goal was that ZFS would be the root filesystem. That's why the TM UX fits so nicely with ZFS. Sadly the licensing issues scared Apple off from adopting ZFS.

vondur 5y ago
I think it was due to a lawsuit EMC had going against Sun over ZFS.
- GekkePrutser 5y ago
  I thought it was Jonathan Schwartz embarassing Steve Jobs by announcing it was coming to macOS himself. Which was a big deal at the time. I always thought that was a petty reason but I didn't really consider it beyond Steve Jobs to do that to be honest. He really took such matters very personally. But your explanation sounds more reasonable. I really thought this until now though.
  - throw0101a 5y ago
    It was a 'support contract' between Sun and Apple not being able to be agreed to: * https://web.archive.org/web/20121221111757/http://mail.opensolaris.org/pipermail/zfs-discuss/2009-October/033125.html https://web.archive.org/web/20121221111757/http://mail.opens...
- dburkland 5y ago
  I think you mean with NetApp
jjoonathan 5y ago
Yeah, that original interface was wild https://youtu.be/2HixPtvIWhY?t=2174 https://youtu.be/2HixPtvIWhY?t=2174 if only ZFS hadn't fallen through! Speaking of which, are there any public plans to do this with APFS or are they just trying to push everyone into the cloud?
- 5y ago

GekkePrutser 5y ago

Thanks for the link. This will be cool, I use FreeBSD as daily driver so ZFS is already a given.

Melatonic 5y ago

People kinda love shitting on Windows here but VSS (Shadow Copy) is actually a very underrated and solid feature. Works with NTFS and even better ReFS

lacrosse_tannin 5y ago
A long time ago, I set up some zfs-in-a-box-OS thing, and hooked to active directory (i think?) and samba, and windows explorer had a built in context menu to restore files to older versions.
lacrosse_tannin 5y ago
A long time ago, I set up some zfs-in-a-box-OS thing (some solaris fork), and hooked to active directory (i think?) and samba, and windows explorer had a built in context menu to restore files to older versions.
- matja 5y ago
  Yep - this is the VFS shadow copy extension for samba, works great! Actually it doesn't require ZFS, just a method to create snapshots and find them - example smb.conf : vfs objects = shadow_copy2 shadow:snapdir = /home/.zfs/snapshot/ shadow:basedir = /home shadow:sort = desc shadow:format = auto:%Y-%m-%dT%H_%M_%SZ
paulkon 5y ago
Only thing with VSS is that ransomware can delete all your snapshots. Samba shares on ZFS via TrueNAS or the like with VSS support mitigates that attack vector. Speaking of, any stats out there on which proportion of ransomware is targeted at which distro for linux, freebsd, windows and mac os?
- Melatonic 5y ago
  Lot of AV's specifically try to lock that down but yeah its a first pass for a lot of ransomware.
5y ago

tjoff 5y ago

This is great, I've often wondered why so little focus is spent on tools like this. Feels like low hanging fruit given the great usability improvements they can bring. All the talks about backups but the reason people don't do it is partly because the tooling is so bad. (no, snapshots is not a proper backup by itself, but a great addition to). Wondering if btrfs will get some love too.

atoav 5y ago
There is timeshift for btrfs. Sadly it only works for certain very standard btrfs subvolume layouts and if you deviate only slightly from that it is no longer supported..
- vondur 5y ago
  Yes. I watched a tutorial video on YouTube where it shows how to label the BTRFS sub volumes correctly for Timeshift to work correctly in Fedora.
- ravi-delia 5y ago
  It's not super hard to hack it if you need to, but yeah zfs exposes exactly what's needed without any global configuration. In exchange btrfs gets to be a little more flexible, but honestly idk how useful it is to treat snapshots as normal directories anyway. That's my only real complaint about btrfs on the desktop.
Affric 5y ago
My theory based on reading comments from ZFS devs is that as resources are somewhat limited and most people who use it are a bit technical these kind of things are low priority so they go slowly.
mustache_kimono 5y ago
> This is great, Thanks! (It's my little project.) > I've often wondered why so little focus is spent on tools like this. Feels like low hanging fruit given the great usability improvements they can bring. 100% agree. Definitely a QoL tool for the ZFS lover. The zsh key bindings open up uses I never really even dreamed up, because it's just so quick to get to a usable interface. Definitely also a very UNIX-y tool too. Recently I started fiddling with piping to git repos and tar archives and it's very cool -- see: https://github.com/kimono-koans/httm#example-usage https://github.com/kimono-koans/httm#example-usage > Wondering if btrfs will get some love too. btrfs makes this exact thing harder, but, cross fingers, the btrfs devs are listening, because I think btrfs folks would love a ".btrfs" directory like the ".zfs" directory.

rbanffy 5y ago

Interestingly, Solaris had a UI component for ZFS snapshots built into Gnome's file explorer. Not sure if it's still there though. IIRC, OpenSolaris had it. Sadly, the screenshots seem to be gone... https://web.archive.org/web/20081016104234/http://blogs.sun.com:80/erwann/entry/zfs_on_the_desktop_zfs https://web.archive.org/web/20081016104234/http://blogs.sun.... edit: found a couple places with the UI: https://www.linuxjournal.com/content/zfs-finding-its-way-linux-near-you https://www.linuxjournal.com/content/zfs-finding-its-way-lin... https://arstechnica.com/information-technology/2008/10/opensolaris-gets-zfs-snapshot-visualization-feature/ https://arstechnica.com/information-technology/2008/10/opens...

yjftsjthsd-h 5y ago
I couldn't say what Oracle's been doing, but writing this comment from OpenIndiana (Illumos) the feature is still here and works fine. It's integrated into MATE's Caja (forked GNOME/nautilus) and AFAICT it's the same exact feature and it still just works; you click the little clock button in the toolbar, click to the snapshot you want, and browse the filesystem. Actually looking more closely, it looks like it's a pretty thin veneer over the .zfs/snapshot virtual directory so it should be dead simple to build into any file browser that supports extensions (because all you need to do is provide a list of directories under $FS/.zfs/snapshot, and then when the user clicks one change path to $FS/.zfs/snapshot/$SNAPSHOT). I mean, there's a little more integration than that (some context menu items to snapshot and view old versions, an extra column in the detailed view) but that's the main part and I suspect the whole thing could be reimplemented in a day or two. Edit: Nicer screenshot: https://www.openindiana.org/documentation/gallery/oi-b148-gui-timeslider/ https://www.openindiana.org/documentation/gallery/oi-b148-gu...
- brnt 5y ago
  Wonder why it's not used for browsing btrfs snapshots. It's such a great bit of ui.
  - yjftsjthsd-h 5y ago
    I have no specific knowledge but suspect that it just never cross-pollinated; illumos is a somewhat separate community from the Linux-sphere. For all I know, you can build the same code on Linux with OpenZFS and trivially switch it to BTRFS by replacing the paths.

seletskiy 5y ago

Shameless plug — https://github.com/reconquest/zeus https://github.com/reconquest/zeus Daemon for automatic incremental backup of your ZFS systems to the another (physically connected right now) pool.

seletskiy 5y ago

Also, some obscure yet very helpful ZFS feature is that all your ZFS datasets have hidden (e.g. not visible via `ls -a`) `.zfs` directory in their roots. This directory contains all snapshots of a given ZFS dataset mounted as directories. For example, if your home is a separate dataset, then `/home/operator/.zfs/` will contain all snapshots automatically mounted for your convenience.

SkyMarshal 5y ago
How do you show the .zfs directory if it ignores `ls -a`?
- mappu 5y ago
  The whole VFS is an illusion, it's totally possible for the filesystem driver to say one thing in readdir(3) and another thing in fopen(3).
  - tbrownaw 5y ago
    Related fun fact: if you have +x but not +r on a directory, you can traverse it (access things whose names you know), but not view its contents (`ls` won't work).
- 4oo4 5y ago
  You have to go into zfs settings to enable it, I think it's something like this: `zfs set snapdir=visible poolname/dataset` After that you can see it with `ls -la`.
RandomBK 5y ago
Huh, this is new to me. How does zfs handle regular directories that happen to be named .zfs? Might this be a security issue as well? What permissions are needed to access this path? If I somehow have a webserver serving static files from a zfs dataset, might someone use this to access old or deleted files?
- 5y ago

mustache_kimono 5y ago

My little project. ;) Surprised and delighted its on the HN front page! If you use ZFS a bunch, like I do, you will use this little tool more than you may imagine. Also, FYI, for the trivia/history buffs -- cool bit of happenstance -- 20 years ago yesterday Jeff Bonwick filed the PSARC case for the ZFS filesystem. See: https://illumos.org/opensolaris/ARChive/PSARC/2002/240/onepager.opensolaris https://illumos.org/opensolaris/ARChive/PSARC/2002/240/onepa...

velcrovan 5y ago

The ideal file-level Time Machine-like tool for ZFS already exists and it is Windows Explorer. Not even kidding. Right click on any file or folder residing on a ZFS share and click on Properties, then click the Previous Versions tab. You can restore the file or folder from any snapshot, or just open it and look at it.

4oo4 5y ago
That is, until you get NTFS bitrot since you don't have checksumming.
- Infernal 5y ago
  GP specifies a ZFS share.
  - 4oo4 5y ago
    Ahh, interesting, I never thought about that. So that means you have a share mounted probably via NFS/SMB, Windows is able to read the zfs snapshots? Or are the Windows previous versions snapshots separate from the zfs ones?
    - mastax 5y ago
      You can configure samba to look for ZFS snapshots: vfs objects = shadow_copy2 shadow:snapdir = .zfs/snapshot shadow:sort = desc shadow:format = zfs-auto-snap_frequent-%Y-%m-%d-%H%M
      - 4oo4 5y ago
        That's awesome. I never expected Windows could integrate with zfs like that. Speaking of, I always forget this exists too: https://github.com/openzfsonwindows/openzfs/ https://github.com/openzfsonwindows/openzfs/
        5y ago