7 ms·

Why would each file system need a "native encryption"? What gains are there fromt this? Encryption seems it would be more cleanly implemented transparently und

Why would each file system need a "native encryption"? What gains are there fromt this? Encryption seems it would be more cleanly implemented transparently underneath the file system level.

daenney 10y ago
As noted in the PR, in this case it's a licensing issue.
- black_knight 10y ago
  No, I mean why would ZFS have anything to do with encryption at all? It doesn't seem to be a filesystem issue, rather something you put underneath it (encrypting the block device, no matter what file system is on top) or on top of it (encrypting separate files).
  - otterley 10y ago
    It's a feature that trades off layering purity (which ZFS has never adhered to) for increased usability.
    - black_knight 10y ago
      I see. When crypto is concerned, this kind of purity seems attractive though… Easier to reason about when using it. If I were to use ZFS’s native crypto I would have to look into the details of their implementation to understand what guarantees are provided, and where in the stack the crypto is applied etc…
      - tptacek 10y ago
        In fact, it's significantly harder to reason about encryption at the block layer, and not just because it's difficult to reason about the most popular construction (XTS). Among other things, you have to come up with a semantic for handling authentication failures at a sector level. What does it mean for one sector of a file to fail authentication? Have you lost the whole file? Do you fill with zeroes? How do you signal the precise error to the kernel or users?
        black_knight 10y ago
        The semantics seem pretty strait forward. That sector is untrusted. You do not fill with zeroes(!), but fail as hard as you can. I understand that kernel currently may not have any semantics for files or sectors failing authentication, but that issue would doesn't seem to depend on the file/sector dilemma, and would have to be solved anyhow.
        tptacek 10y ago
        To me, this is very simple: File-level encryption has total format flexibility. A filesystem can store arbitrary metadata. It turns out that strong encryption schemes want internal metadata. You can go through contortions --- contortions no block encryption scheme currently goes through --- to get some sort of space for metadata at the block layer, at significant expense. Or you can get it for free with files. File-level encryption is also message-aware. Encryption cares about the boundaries between messages (just for instance: because it leaves no ambiguity about the set of disk sectors that makes up any particular file at any particular moment in time). Block-level crypto can't provide this; it's by definition oblivious to messages. We have block-level crypto because it's a cheap easy win, for the minimal security it provides. If the FBI is interdicting you, block crypto isn't going to help you: they incurred weeks worth of planning before anyone with handcuffs approached you, and the extra planning it takes to make sure they grab you when your key is resident is a rounding error. But if you leave your laptop full of financial secrets in a taxi cab, block crypto will probably keep you from having to roll all your keys, change your account numbers, notifying your clients, etc.
        tedunangst 10y ago
        But won't the FBI also wait to make the grab when using filesystem encryption? We're not talking about some filesystem that uses unique passwords for every file.
        tptacek 10y ago
        Unlike block encryption, filesystem encryption doesn't have to be all-or-none; you can have lots of different keys, unlock on a directory-by-directory basis, whatever. There's room for arbitrary metadata. If you're using your computer at all, block encryption is unlikely to help you. I'm not saying any particular extant system does those things; I'm just saying they're a possibility for filesystems and foreclosed for device-level encryption.
  - vbit 10y ago
    With this you can encrypt one dataset and leave another unencrypted, on the same block device, paying the cost only or the encrypted one.
    - mrsteveman1 10y ago
      Which means you can also potentially use different keys for each encrypted filesystem as well, one for each user or various other scenarios
      - black_knight 10y ago
        That’s interesting, but it again seems like something which can be implemented independently of the particular choice of file system.
        rincebrain 10y ago
        Only sort of? ZFS is both filesystem and logical volume abstraction, so to implement different keys on different "filesystems" on ZFS, you'd need to expose them as block devices, not filesystems, and then use your encryption du jour on them, then your filesystem atop that - which also kills most of your compression or encryption properties, since you're doing it before ZFS sees the data, so to speak.
        black_knight 10y ago
        I guess I was thinking more like a 9P[0] approach where a file system can be given to a module (or any program) and based on it the module can export a related file system. For example the module would encrypt the files before storing it to the given file system. However, I realise this does not quite fit into the current kernel architectures. [0] https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol)
    - black_knight 10y ago
      What is a dataset in this setting?
  - Titanous 10y ago
    In order to do filesystem encryption properly, it needs to be done at the file layer, not the block layer. Block-level encryption is not authenticated because there are no extra bytes to add the authentication tag. If a ciphertext is not authenticated, it can be trivially tampered with. This means that someone with access to the encrypted drive could add backdoors or otherwise tamper with the executables and data even though it is encrypted.
    - black_knight 10y ago
      You can perfectly fine divide the block device into a bit smaller chunks so that you can fit a MAC or similar at the end.
      - Titanous 10y ago
        My understanding is that in practice this is quite hard to do. As far as I know, none of the mainstream block-layer encryption systems (BitLocker, FileVault, dm-crypt) provide authenticated encryption. Here's a short thread about the problems with adding it to dm-crypt: http://comments.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/6689 http://comments.gmane.org/gmane.linux.kernel.device-mapper.d... I'm sure you can find more threads if you look around.
        black_knight 10y ago
        Thanks! That’s good to know. Indeed, encryption without authentication is poor security.
      - deleted 10y ago
        [deleted]
      - mioelnir 10y ago
        This is how FreeBSD's GELI (which has authenticated encryption for blockdevices) did it. For every 4k data block it presented up the stack, it consumed 9 512 sectors on disk. Each of them contained 480 bytes, the rest for MAC. With 4k native drives, this became completely impractical. To keep ratios similar, you would have to present 32k byte devices up the stack, which filesystems have troubles with. Or have 1 MAC sector per data sector or similar, cutting your storage in half.
  - anorwell 10y ago
    Encrypting on top of ZFS negates ZFS' compression and de-duplication. My understanding is that this implementation preserves both of those. For example, ZFS can compress before encrypting.
rodgerd 10y ago
Because in the unified storage model that ZFS (and btrfs and friends) have, being force to set up specifically encrypted zvols and zpools is a bunch of arse.
mrsteveman1 10y ago
Here's one example where it would be helpful, say you have a simple mirrored pool built from 2x LUKS encrypted drives in a low powered NAS device. With the encryption underneath ZFS, the encryption during a write necessarily happens twice, once for each LUKS mapping, which increases CPU load, reduces throughput, or both. With the encryption in the ZFS layer, data only needs to be encrypted once during a write, after that the data can be written to as many drives as necessary without any additional overhead.
- black_knight 10y ago
  I see. Encryption is one possible filter one could apply to data before it goes to the block device. Another is compression. I guess ZFS has that one also covered. Chould ZFS provide some generic way to apply different such filters which would be applied to each "mirror"?
  - mioelnir 10y ago
    No, datasets do not map to vdevs (mirrors). Since it came up in another comment as well: a dataset is a child object of a pool, but it is also the root-object of a subtree that consumes storage in that pool. To the user this dataset is presented as either a filesystem or a blockdevice to name the two most common options. Apart from internal accounting things like the spacemap, every consumed storage space in a zpool belongs to a dataset in some way. It might be a data block for a file in that dataset, or it might be an old storage block still referenced by a snapshot of a dataset. A lot of zfs commands work on datasets (send/receive, snapshot, clone, ...). They are also the point where settings such as compression, deduplication etc can be enabled/disabled as well as traditional filesystem mount options like noatime or noexec. All the datasets consume storage from the pool, which dynamically stripes over all configured vdevs. If you enable an option for a dataset, you enable it for all storage of that dataset which ends up on all vdevs. You can not delegate a dataset to a specific vdev and then enable some option on that vdev. Also sorry to everyone who knows enough about ZFS internals to realize that I just took their design, pulled it behind a shed and hit it with a blunt, heavy object.
tptacek 10y ago
Encryption is easier to implement at the block level, but it's also much less effective. http://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/ http://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/
- throwaway7767 10y ago
  Thomas, have you looked at this code? Can you confirm if it's actually doing seemingly-sensible authenticated filesystem-level encryption? I know some previous efforts at "native ZFS encryption" essentially re-implemented AES-XTS block-level encryption below the FS, which seems unlikely to offer any advantages.
johncolanduoni 10y ago
There's a few things. First off, ZFS takes advantage of doing its own volume management on bare metal. Some aspects of data recovery/resilience work better when it can interact directly with the device, particularly for resistance to bit rot (which is one of ZFS's biggest advantages). Putting something like LUKS in between isn't as bad as LVM or hardware RAID, but is not great. Second, it enables the use of different ways of using a particular block cipher like AES (namely GCM instead of say XTS) that have some advantages such as authentication of data. I'm not sure if it's an option for this particular implementation, but nothing about the way ZFS encryption works would preclude using XTS, while GCM doesn't really map well to block device encryption (there is no good way to store the extra IV and authentication code, while ZFS can put it in the metadata). There are of course disadvantages. Some information about the structure of the data on your drive is accessible that would not be if you used block encryption. Also, unlike LUKS or LUKS w/ LVM you can't easily mix filesystem types on the same drive set.
- therealmarv 10y ago
  I wonder how/what is better with direct hard drive access here, if there is bit rot on LUKS I expect it is also fixed with ZFS, what should be different? "Some aspects of data recovery/resilience work better when it can interact directly with the device, particularly for resistance to bit rot (which is one of ZFS's biggest advantages)."

Why would each file system need a "native encryption"? What gains are there fromt this? Encryption seems it would be more cleanly implemented transparently underneath the file system level.

daenney 10y ago

As noted in the PR, in this case it's a licensing issue.

black_knight 10y ago
No, I mean why would ZFS have anything to do with encryption at all? It doesn't seem to be a filesystem issue, rather something you put underneath it (encrypting the block device, no matter what file system is on top) or on top of it (encrypting separate files).
- otterley 10y ago
  It's a feature that trades off layering purity (which ZFS has never adhered to) for increased usability.
  - black_knight 10y ago
    I see. When crypto is concerned, this kind of purity seems attractive though… Easier to reason about when using it. If I were to use ZFS’s native crypto I would have to look into the details of their implementation to understand what guarantees are provided, and where in the stack the crypto is applied etc…
    - tptacek 10y ago
      In fact, it's significantly harder to reason about encryption at the block layer, and not just because it's difficult to reason about the most popular construction (XTS). Among other things, you have to come up with a semantic for handling authentication failures at a sector level. What does it mean for one sector of a file to fail authentication? Have you lost the whole file? Do you fill with zeroes? How do you signal the precise error to the kernel or users?
      - black_knight 10y ago
        The semantics seem pretty strait forward. That sector is untrusted. You do not fill with zeroes(!), but fail as hard as you can. I understand that kernel currently may not have any semantics for files or sectors failing authentication, but that issue would doesn't seem to depend on the file/sector dilemma, and would have to be solved anyhow.
        tptacek 10y ago
        To me, this is very simple: File-level encryption has total format flexibility. A filesystem can store arbitrary metadata. It turns out that strong encryption schemes want internal metadata. You can go through contortions --- contortions no block encryption scheme currently goes through --- to get some sort of space for metadata at the block layer, at significant expense. Or you can get it for free with files. File-level encryption is also message-aware. Encryption cares about the boundaries between messages (just for instance: because it leaves no ambiguity about the set of disk sectors that makes up any particular file at any particular moment in time). Block-level crypto can't provide this; it's by definition oblivious to messages. We have block-level crypto because it's a cheap easy win, for the minimal security it provides. If the FBI is interdicting you, block crypto isn't going to help you: they incurred weeks worth of planning before anyone with handcuffs approached you, and the extra planning it takes to make sure they grab you when your key is resident is a rounding error. But if you leave your laptop full of financial secrets in a taxi cab, block crypto will probably keep you from having to roll all your keys, change your account numbers, notifying your clients, etc.
        tedunangst 10y ago
        But won't the FBI also wait to make the grab when using filesystem encryption? We're not talking about some filesystem that uses unique passwords for every file.
        tptacek 10y ago
        Unlike block encryption, filesystem encryption doesn't have to be all-or-none; you can have lots of different keys, unlock on a directory-by-directory basis, whatever. There's room for arbitrary metadata. If you're using your computer at all, block encryption is unlikely to help you. I'm not saying any particular extant system does those things; I'm just saying they're a possibility for filesystems and foreclosed for device-level encryption.
- vbit 10y ago
  With this you can encrypt one dataset and leave another unencrypted, on the same block device, paying the cost only or the encrypted one.
  - mrsteveman1 10y ago
    Which means you can also potentially use different keys for each encrypted filesystem as well, one for each user or various other scenarios
    - black_knight 10y ago
      That’s interesting, but it again seems like something which can be implemented independently of the particular choice of file system.
      - rincebrain 10y ago
        Only sort of? ZFS is both filesystem and logical volume abstraction, so to implement different keys on different "filesystems" on ZFS, you'd need to expose them as block devices, not filesystems, and then use your encryption du jour on them, then your filesystem atop that - which also kills most of your compression or encryption properties, since you're doing it before ZFS sees the data, so to speak.
        black_knight 10y ago
        I guess I was thinking more like a 9P[0] approach where a file system can be given to a module (or any program) and based on it the module can export a related file system. For example the module would encrypt the files before storing it to the given file system. However, I realise this does not quite fit into the current kernel architectures. [0] https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol)
  - black_knight 10y ago
    What is a dataset in this setting?
- Titanous 10y ago
  In order to do filesystem encryption properly, it needs to be done at the file layer, not the block layer. Block-level encryption is not authenticated because there are no extra bytes to add the authentication tag. If a ciphertext is not authenticated, it can be trivially tampered with. This means that someone with access to the encrypted drive could add backdoors or otherwise tamper with the executables and data even though it is encrypted.
  - black_knight 10y ago
    You can perfectly fine divide the block device into a bit smaller chunks so that you can fit a MAC or similar at the end.
    - Titanous 10y ago
      My understanding is that in practice this is quite hard to do. As far as I know, none of the mainstream block-layer encryption systems (BitLocker, FileVault, dm-crypt) provide authenticated encryption. Here's a short thread about the problems with adding it to dm-crypt: http://comments.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/6689 http://comments.gmane.org/gmane.linux.kernel.device-mapper.d... I'm sure you can find more threads if you look around.
      - black_knight 10y ago
        Thanks! That’s good to know. Indeed, encryption without authentication is poor security.
    - deleted 10y ago
      [deleted]
    - mioelnir 10y ago
      This is how FreeBSD's GELI (which has authenticated encryption for blockdevices) did it. For every 4k data block it presented up the stack, it consumed 9 512 sectors on disk. Each of them contained 480 bytes, the rest for MAC. With 4k native drives, this became completely impractical. To keep ratios similar, you would have to present 32k byte devices up the stack, which filesystems have troubles with. Or have 1 MAC sector per data sector or similar, cutting your storage in half.
- anorwell 10y ago
  Encrypting on top of ZFS negates ZFS' compression and de-duplication. My understanding is that this implementation preserves both of those. For example, ZFS can compress before encrypting.

rodgerd 10y ago

Because in the unified storage model that ZFS (and btrfs and friends) have, being force to set up specifically encrypted zvols and zpools is a bunch of arse.

mrsteveman1 10y ago

Here's one example where it would be helpful, say you have a simple mirrored pool built from 2x LUKS encrypted drives in a low powered NAS device. With the encryption underneath ZFS, the encryption during a write necessarily happens twice, once for each LUKS mapping, which increases CPU load, reduces throughput, or both. With the encryption in the ZFS layer, data only needs to be encrypted once during a write, after that the data can be written to as many drives as necessary without any additional overhead.

black_knight 10y ago
I see. Encryption is one possible filter one could apply to data before it goes to the block device. Another is compression. I guess ZFS has that one also covered. Chould ZFS provide some generic way to apply different such filters which would be applied to each "mirror"?
- mioelnir 10y ago
  No, datasets do not map to vdevs (mirrors). Since it came up in another comment as well: a dataset is a child object of a pool, but it is also the root-object of a subtree that consumes storage in that pool. To the user this dataset is presented as either a filesystem or a blockdevice to name the two most common options. Apart from internal accounting things like the spacemap, every consumed storage space in a zpool belongs to a dataset in some way. It might be a data block for a file in that dataset, or it might be an old storage block still referenced by a snapshot of a dataset. A lot of zfs commands work on datasets (send/receive, snapshot, clone, ...). They are also the point where settings such as compression, deduplication etc can be enabled/disabled as well as traditional filesystem mount options like noatime or noexec. All the datasets consume storage from the pool, which dynamically stripes over all configured vdevs. If you enable an option for a dataset, you enable it for all storage of that dataset which ends up on all vdevs. You can not delegate a dataset to a specific vdev and then enable some option on that vdev. Also sorry to everyone who knows enough about ZFS internals to realize that I just took their design, pulled it behind a shed and hit it with a blunt, heavy object.

tptacek 10y ago

Encryption is easier to implement at the block level, but it's also much less effective. http://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/ http://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/

throwaway7767 10y ago
Thomas, have you looked at this code? Can you confirm if it's actually doing seemingly-sensible authenticated filesystem-level encryption? I know some previous efforts at "native ZFS encryption" essentially re-implemented AES-XTS block-level encryption below the FS, which seems unlikely to offer any advantages.

johncolanduoni 10y ago

There's a few things. First off, ZFS takes advantage of doing its own volume management on bare metal. Some aspects of data recovery/resilience work better when it can interact directly with the device, particularly for resistance to bit rot (which is one of ZFS's biggest advantages). Putting something like LUKS in between isn't as bad as LVM or hardware RAID, but is not great. Second, it enables the use of different ways of using a particular block cipher like AES (namely GCM instead of say XTS) that have some advantages such as authentication of data. I'm not sure if it's an option for this particular implementation, but nothing about the way ZFS encryption works would preclude using XTS, while GCM doesn't really map well to block device encryption (there is no good way to store the extra IV and authentication code, while ZFS can put it in the metadata). There are of course disadvantages. Some information about the structure of the data on your drive is accessible that would not be if you used block encryption. Also, unlike LUKS or LUKS w/ LVM you can't easily mix filesystem types on the same drive set.

therealmarv 10y ago
I wonder how/what is better with direct hard drive access here, if there is bit rot on LUKS I expect it is also fixed with ZFS, what should be different? "Some aspects of data recovery/resilience work better when it can interact directly with the device, particularly for resistance to bit rot (which is one of ZFS's biggest advantages)."