4 ms·

Very cool and probably a fun exercise, but I would probably put the data on a ZFS volume with dedupe instead, which from reading this implementation is pretty m

Very cool and probably a fun exercise, but I would probably put the data on a ZFS volume with dedupe instead, which from reading this implementation is pretty much the same thing to my layman’s perspective. I could also add compression to the same dataset.

traceroute66 4y ago
> I would probably put the data on a ZFS volume with dedupe instead But doesn't ZFS dedupe eat RAM for breakfast ? Double-digit GB RAM per TB data IIRC ?
- giantrobot 4y ago
  It's usually along the lines of 1GB per TB. Some factors can affect that number but I've found it pretty accurate. Note that's 1GB of RAM that ends up wired to ZFS and never usable by the rest of the system.
- redundantly 4y ago
  The DDT (deduplication table) used to require ~1GB RAM for every 1TB data written over the life of the file system. Deleting data from the file system wouldn't remove the dedupe references, you'd have to recreate the pool entirely and start over with a new DDT. However, there are now special devices that can be used stored to store DDT. Typically this is done with two SSDs configured as a mirrored vdev for the DDT metadata. This reduces the overhead on memory, but does cost some performance and still has the same limitation that the DDT size can only be reduced by re-creating the pool.
  - hultner 4y ago
    Yes this is what I do, and I wouldn't have dedupe for all datasets, for instance in this case he wanted dedupe on his games, so I would just enable it for that dataset. But I do run disk-wide compression with no problems and have done so on all my datasets for many years now, and it's been a tremendous space saver. Especially on machines where I have a lot of VMs/containers, it's not unusual for me to have a compression ratio of 2 on these with good old lz4, it will be interesting to see what damage zstd will do once I start experimenting with that.
    - redundantly 4y ago
      +1 for compression! Unless you have very specific performance needs there's no reason to not to use compression. In fact TrueNAS (previously known as FreeNAS) has it enabled by default. I've yet to compare lz4 to zstd myself, but I've read great things about it.
- seized 4y ago
  No, only for deduplication, which is optional and doesn't make sense for most workloads. And usage depends on record (like block) size and other factors. Usually 1GB per TB of deduplicated storage is the max. ZFS works on datasets for configs like this, so you can dedup a small "filesystem" of data but not all 40TB on a pool. It will use lots of RAM for ARC (adaptive read cache) but that can be limited. ZFS RAM usage is greatly overblown in my opinion and experience.
  - deleted 4y ago
    [deleted]
  - anotherhue 4y ago
    Agreed. People forget how much more RAM we have now versus when ZFS was developed. It's like old emacs swapping jokes.
tripleo1 4y ago
ZFS is fun but it locks up for unexplained reasons sometimes
- justinclift 4y ago
  Any more info? That sounds like a problem specific to the setup you were using. (?) Saying that because if it was something that commonly happened, then either a) it would have been fixed, or b) people would have stopped using it. :)
  - quectophoton 4y ago
    If the specific setup is an external HDD (or maybe a very slow disk, and trying to write too fast to it), then I can make sense of parent's comment. Like I mentioned in my sibling comment, I can 100% reproduce something that sounds like what parent mentioned (most recent attempt was like 1-2 months ago); but for my specific case, I can see how ZFS on an external HDD might not be that common.
    - anotherhue 4y ago
      I suggest playing with the ZFS tunings to add a write throttle. My hunch is your disk's buffer is filling and then blocking.
      - quectophoton 4y ago
        That sounds like a good idea, you just gave me new keywords to search for my next attempt. Thanks!
      - tripleo1 4y ago
        I think that there was an electrical problem (mismatched chargers on a laptop) in the first instance of mine. Then there's encrypted datasets being very slow, especially when copying between two pools. Then there's having two pools on the same disk corrupting each other. Though in this case the data might have been recoverable using some recovery tool (like a "modified zdb"). Also, I read on some website somewhere (maybe zfsonlinux.org) that USB devices probably shouldn't be used with ZFS. Honestly I "wish" some bunch of crazy filesystem people would just clean room ZFS...
totetsu 4y ago
My recent 'copy-on-write' foot gun was using ZFS in ubuntu. There seems to be some automatic config that makes zfs snapshots each time you apt install something. This lead to wondering why on earth no matter how many files I deleted my disk usage was not reducing from 100%. All those deleted files still existed in the sanpshots. coupled with some bug that makes the snaps shots report their size as much smaller than they really were.

traceroute66 4y ago

> I would probably put the data on a ZFS volume with dedupe instead But doesn't ZFS dedupe eat RAM for breakfast ? Double-digit GB RAM per TB data IIRC ?

giantrobot 4y ago
It's usually along the lines of 1GB per TB. Some factors can affect that number but I've found it pretty accurate. Note that's 1GB of RAM that ends up wired to ZFS and never usable by the rest of the system.
redundantly 4y ago
The DDT (deduplication table) used to require ~1GB RAM for every 1TB data written over the life of the file system. Deleting data from the file system wouldn't remove the dedupe references, you'd have to recreate the pool entirely and start over with a new DDT. However, there are now special devices that can be used stored to store DDT. Typically this is done with two SSDs configured as a mirrored vdev for the DDT metadata. This reduces the overhead on memory, but does cost some performance and still has the same limitation that the DDT size can only be reduced by re-creating the pool.
- hultner 4y ago
  Yes this is what I do, and I wouldn't have dedupe for all datasets, for instance in this case he wanted dedupe on his games, so I would just enable it for that dataset. But I do run disk-wide compression with no problems and have done so on all my datasets for many years now, and it's been a tremendous space saver. Especially on machines where I have a lot of VMs/containers, it's not unusual for me to have a compression ratio of 2 on these with good old lz4, it will be interesting to see what damage zstd will do once I start experimenting with that.
  - redundantly 4y ago
    +1 for compression! Unless you have very specific performance needs there's no reason to not to use compression. In fact TrueNAS (previously known as FreeNAS) has it enabled by default. I've yet to compare lz4 to zstd myself, but I've read great things about it.
seized 4y ago
No, only for deduplication, which is optional and doesn't make sense for most workloads. And usage depends on record (like block) size and other factors. Usually 1GB per TB of deduplicated storage is the max. ZFS works on datasets for configs like this, so you can dedup a small "filesystem" of data but not all 40TB on a pool. It will use lots of RAM for ARC (adaptive read cache) but that can be limited. ZFS RAM usage is greatly overblown in my opinion and experience.
- deleted 4y ago
  [deleted]
- anotherhue 4y ago
  Agreed. People forget how much more RAM we have now versus when ZFS was developed. It's like old emacs swapping jokes.

tripleo1 4y ago

ZFS is fun but it locks up for unexplained reasons sometimes

justinclift 4y ago
Any more info? That sounds like a problem specific to the setup you were using. (?) Saying that because if it was something that commonly happened, then either a) it would have been fixed, or b) people would have stopped using it. :)
- quectophoton 4y ago
  If the specific setup is an external HDD (or maybe a very slow disk, and trying to write too fast to it), then I can make sense of parent's comment. Like I mentioned in my sibling comment, I can 100% reproduce something that sounds like what parent mentioned (most recent attempt was like 1-2 months ago); but for my specific case, I can see how ZFS on an external HDD might not be that common.
  - anotherhue 4y ago
    I suggest playing with the ZFS tunings to add a write throttle. My hunch is your disk's buffer is filling and then blocking.
    - quectophoton 4y ago
      That sounds like a good idea, you just gave me new keywords to search for my next attempt. Thanks!
    - tripleo1 4y ago
      I think that there was an electrical problem (mismatched chargers on a laptop) in the first instance of mine. Then there's encrypted datasets being very slow, especially when copying between two pools. Then there's having two pools on the same disk corrupting each other. Though in this case the data might have been recoverable using some recovery tool (like a "modified zdb"). Also, I read on some website somewhere (maybe zfsonlinux.org) that USB devices probably shouldn't be used with ZFS. Honestly I "wish" some bunch of crazy filesystem people would just clean room ZFS...

totetsu 4y ago

My recent 'copy-on-write' foot gun was using ZFS in ubuntu. There seems to be some automatic config that makes zfs snapshots each time you apt install something. This lead to wondering why on earth no matter how many files I deleted my disk usage was not reducing from 100%. All those deleted files still existed in the sanpshots. coupled with some bug that makes the snaps shots report their size as much smaller than they really were.