7 ms·

This is a good reminder for everyone that snapshots are not backups. Also, thank you for all of the hard work on ZoL!

ryao 9y ago
You are welcome. I should clarify the snapshots remark. The problem with how this interacts with snapshots is that the snapshots containing orphaned files cannot be repaired by software without BPR. They can only be listed by software for deletion by the administrator. Also, for the dataset’s tip, the future tool to repair it can only put the orphaned files into a lost+found directory without the original file names.
- gigatexal 9y ago
  Seriously you and the other maintainers are my heroes. Keep it up! And thanks for the awesome write up.
- Annatar 9y ago
  lost+found is an incorrect approach for ZFS. If you feel you have to regress to what was 40 years ago and was a bad solution, then the misfeature causing it needs to be engineered instead of hacked together, or better yet not implemented at all. If you have to break from mainstream ZFS and regress that violently, it’s the wrong approach.
  - barrkel 9y ago
    Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?
    - Annatar 9y ago
      I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.
      - deleted 9y ago
        [deleted]
      - ryao 9y ago
        > I suggest completely backing out the code in question This makes sense and we did that. It does not fix the fact that datasets exist where directories have incorrect sizes and files are orphaned because of the bad patch. > and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. While those are fine choices, they are not immune to bugs either. We had a space map corruption bug that we inherited from Illumos several years ago. We also inherited an issue in send/recv from illumos several years ago when hole_birth was introduced. FreeBSD’s TRIM implementation was far from perfect too and likely messed some things up given the long history of bug fixes that it required. I do not mean to discourage people from using those systems, but it is silly to adopt them because a bug got past joint OpenZFS review and landed in the Linux port first. While they do a good job too, they are not infallible. The major OpenZFS platforms are roughly at parity in terms of their risk for bugs and regressions. > metadata are not files so they have no business going into lost+found. This makes no sense. How are orphaned files “metadata”? Why should they not be placed in a lost+found directory?
        Annatar 9y ago
        I do not mean to discourage people from using those systems, but it is silly to adopt them because a bug got past joint OpenZFS review and landed in the Linux port first. That’s your opinion and while I extremely disagree with it, it is what it is. It might be even true in the case if ZFS on Linux team but on the whole of GNU/Linux, my professional experience is that the amount of bugs and breakage is much higher than on BSD or illumos based operating systems, and to me, quality and stability are the most important things. For me a computer is a tool to make money and so reliability of hardware and especially the software trumps everything else. In fact, apart from reliability and simplicity nothing else is important to me when it comes to software.
        monetus 9y ago
        Your postings reminded me of illumOS's existence, so I went looking for an arm build to no avail. You wouldn't happen to know if that is something that is available would you?
  - iforgotpassword 9y ago
    So what exactly is your suggestion? "lost+found is ugly so don't recover the files at all"? He said meta data is lost so I'm really curious to hear your elegant modern solution. This is a tool that might be needed on a few systems that suffered from this relatively short loved bug and then never again after that, so your solution better not require ten times the effort.
    - Annatar 9y ago
      If it's a one time throwaway tool, then it doesn't need lost+found!
      - ryao 9y ago
        I was talking more about the idea of a lost+found directory. Brian suggested that we make it ‘.zfs/lost+found‘, which might be what we do. I am leaning toward extending scrub to add the function when a flag for an erratum check is passed. That implies that the tool would be a part of the driver so that it can be done online for minimal downtime.
        Annatar 9y ago
        But something is majorly wrong with this approach if you can even have orphaned files! This should be impossible on ZFS!
        luke0016 9y ago
        That's why it's a bug?
walrus01 9y ago
also a good time to remind people that backups are not backups unless they're geographically diverse. back up to an off site location in case your (home|office) burns down or suffers some other total catastrophe.
- 1024core 9y ago
  > also a good time to remind people that backups are not backups unless they're geographically diverse. ... and backups are not backups until you can reliably restore from them.
  - rkagerer 9y ago
    ...and a DR plan isn't worth the paper it's written on unless you test it regularly (people leave, media gets corrupted... just like the military; do drills!)
    - dcuthbertson 9y ago
      Paper? You write on paper? My DR plan is in a file and backed up locally, and ... um ... oh oh.
    - walrus01 9y ago
      ... and a DR plan isn't worth anything unless you test the "people leave" part without any knowledge in their heads. your internal documentation and credential storage needs to be solid in case four people flying together all die abruptly in the same plane crash. the traditional "bus problem".
  - dspillett 9y ago
    And that you regularly test that property.
    - amelius 9y ago
      But how do you test that reliably if the filesystem makes files disappear? I.e., you'll have an original that misses some files, and a backup that misses the same files. A diff will show nothing.
Annatar 9y ago
“This is a good reminder for everyone that snapshots are not backups.” Were this not ZFS, I would agree with you, however, since zfs rollback will revert to previous state and if the snapshot is on a redundant vdev, there is no difference between that and losing your backup due to a damaged tape. In fact, zfs snapshots are exactly how time machine on illumos based operating systems is implemented.
- barrkel 9y ago
  Backups have three characteristics: redundancy, versioning and distribution. ZFS only fulfils two. If ZFS is your backup target, then you're making more sense.
  - Annatar 9y ago
    ZFS has distribution via zfs send subcommand.

ryao 9y ago

You are welcome. I should clarify the snapshots remark. The problem with how this interacts with snapshots is that the snapshots containing orphaned files cannot be repaired by software without BPR. They can only be listed by software for deletion by the administrator. Also, for the dataset’s tip, the future tool to repair it can only put the orphaned files into a lost+found directory without the original file names.

gigatexal 9y ago
Seriously you and the other maintainers are my heroes. Keep it up! And thanks for the awesome write up.
Annatar 9y ago
lost+found is an incorrect approach for ZFS. If you feel you have to regress to what was 40 years ago and was a bad solution, then the misfeature causing it needs to be engineered instead of hacked together, or better yet not implemented at all. If you have to break from mainstream ZFS and regress that violently, it’s the wrong approach.
- barrkel 9y ago
  Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?
  - Annatar 9y ago
    I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.
    - deleted 9y ago
      [deleted]
    - ryao 9y ago
      > I suggest completely backing out the code in question This makes sense and we did that. It does not fix the fact that datasets exist where directories have incorrect sizes and files are orphaned because of the bad patch. > and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. While those are fine choices, they are not immune to bugs either. We had a space map corruption bug that we inherited from Illumos several years ago. We also inherited an issue in send/recv from illumos several years ago when hole_birth was introduced. FreeBSD’s TRIM implementation was far from perfect too and likely messed some things up given the long history of bug fixes that it required. I do not mean to discourage people from using those systems, but it is silly to adopt them because a bug got past joint OpenZFS review and landed in the Linux port first. While they do a good job too, they are not infallible. The major OpenZFS platforms are roughly at parity in terms of their risk for bugs and regressions. > metadata are not files so they have no business going into lost+found. This makes no sense. How are orphaned files “metadata”? Why should they not be placed in a lost+found directory?
      - Annatar 9y ago
        I do not mean to discourage people from using those systems, but it is silly to adopt them because a bug got past joint OpenZFS review and landed in the Linux port first. That’s your opinion and while I extremely disagree with it, it is what it is. It might be even true in the case if ZFS on Linux team but on the whole of GNU/Linux, my professional experience is that the amount of bugs and breakage is much higher than on BSD or illumos based operating systems, and to me, quality and stability are the most important things. For me a computer is a tool to make money and so reliability of hardware and especially the software trumps everything else. In fact, apart from reliability and simplicity nothing else is important to me when it comes to software.
        monetus 9y ago
        Your postings reminded me of illumOS's existence, so I went looking for an arm build to no avail. You wouldn't happen to know if that is something that is available would you?
- iforgotpassword 9y ago
  So what exactly is your suggestion? "lost+found is ugly so don't recover the files at all"? He said meta data is lost so I'm really curious to hear your elegant modern solution. This is a tool that might be needed on a few systems that suffered from this relatively short loved bug and then never again after that, so your solution better not require ten times the effort.
  - Annatar 9y ago
    If it's a one time throwaway tool, then it doesn't need lost+found!
    - ryao 9y ago
      I was talking more about the idea of a lost+found directory. Brian suggested that we make it ‘.zfs/lost+found‘, which might be what we do. I am leaning toward extending scrub to add the function when a flag for an erratum check is passed. That implies that the tool would be a part of the driver so that it can be done online for minimal downtime.
      - Annatar 9y ago
        But something is majorly wrong with this approach if you can even have orphaned files! This should be impossible on ZFS!
        luke0016 9y ago
        That's why it's a bug?

walrus01 9y ago

also a good time to remind people that backups are not backups unless they're geographically diverse. back up to an off site location in case your (home|office) burns down or suffers some other total catastrophe.

1024core 9y ago
> also a good time to remind people that backups are not backups unless they're geographically diverse. ... and backups are not backups until you can reliably restore from them.
- rkagerer 9y ago
  ...and a DR plan isn't worth the paper it's written on unless you test it regularly (people leave, media gets corrupted... just like the military; do drills!)
  - dcuthbertson 9y ago
    Paper? You write on paper? My DR plan is in a file and backed up locally, and ... um ... oh oh.
  - walrus01 9y ago
    ... and a DR plan isn't worth anything unless you test the "people leave" part without any knowledge in their heads. your internal documentation and credential storage needs to be solid in case four people flying together all die abruptly in the same plane crash. the traditional "bus problem".
- dspillett 9y ago
  And that you regularly test that property.
  - amelius 9y ago
    But how do you test that reliably if the filesystem makes files disappear? I.e., you'll have an original that misses some files, and a backup that misses the same files. A diff will show nothing.

Annatar 9y ago

“This is a good reminder for everyone that snapshots are not backups.” Were this not ZFS, I would agree with you, however, since zfs rollback will revert to previous state and if the snapshot is on a redundant vdev, there is no difference between that and losing your backup due to a damaged tape. In fact, zfs snapshots are exactly how time machine on illumos based operating systems is implemented.

barrkel 9y ago
Backups have three characteristics: redundancy, versioning and distribution. ZFS only fulfils two. If ZFS is your backup target, then you're making more sense.
- Annatar 9y ago
  ZFS has distribution via zfs send subcommand.