3 ms·

ZFS uses a large amount of ram, i think the old rule of thumb was 1GB ram per 1TB of storage

by mewse-hn 11mo ago

01HNNWZ0MV43FF 11mo ago
Huh. More than just the normal page cache on other filesystems?
- tekla 11mo ago
  ZFS will eat up as much RAM as you give it as it caches files in memory as accessed.
  - ac29 11mo ago
    All filesystems do this (at least all modern ones, on linux)
- WarOnPrivacy 11mo ago
  Yes. Parent's comment matches everything I've heard. 32GB is a common recommendation for home lab setups. I run 32 in my TrueNAS builds (36TB and 60TB).
  - magicalhippo 11mo ago
    You can run it with much less. I don't recall the bare minimum but with a bit of tweaking 2GB should be plenty[1]. I recall reading some running it on a 512MB system, but that was a while ago so not sure if you can still go that low. Performance can suffer though, for example low memory will limit the size of the transaction groups. So for decent performance you will want 8GB or more depending on workloads. [1]: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#hardware-requirements https://openzfs.github.io/openzfs-docs/Project%20and%20Commu...
yjftsjthsd-h 11mo ago
That's only for deduplication. https://superuser.com/a/993019 https://superuser.com/a/993019
- Lammy 11mo ago
  I do like to deduplicate my BitTorrent downloads/seeding directory with my media directories so I can edit metadata to my heart's content while still seeding forever without having to incur 2x storage usage. I tune the `recordsize` to 1MiB so it has vastly fewer blocks to keep track of compared to the default 128K, at the cost of any modification wasting very slightly more space. Really not a big deal though when talking about multi-gibibyte media containers, multi-megapixel art embeds, etc.
  - yegle 11mo ago
    cross-seed | cross-seed https://www.cross-seed.org/ https://www.cross-seed.org/
    - bscphil 11mo ago
      I believe they are saying they literally edit the media files to add / change metadata. Cross-seeding is only possible if the files are kept the same.
  - zenoprax 11mo ago
    Have you considered "reflinks"? Supported as of [OpenZFS 2.2](https://github.com/openzfs/zfs/pull/13392 https://github.com/openzfs/zfs/pull/13392). Haven't used them yet myself but seems like a nice use case for things like minor metadata changes to media files. The bulk of the file is shared and only the delta between the two are saved.
    - Lammy 11mo ago
      Neat; will look into this. My setup is several years older than this, predating even FreeBSD's move to OpenZFS, and I just haven't touched the config of it since then since it works flawlessly (and since I already bought the RAM lol)

ZFS uses a large amount of ram, i think the old rule of thumb was 1GB ram per 1TB of storage

01HNNWZ0MV43FF 11mo ago

Huh. More than just the normal page cache on other filesystems?

tekla 11mo ago
ZFS will eat up as much RAM as you give it as it caches files in memory as accessed.
- ac29 11mo ago
  All filesystems do this (at least all modern ones, on linux)
WarOnPrivacy 11mo ago
Yes. Parent's comment matches everything I've heard. 32GB is a common recommendation for home lab setups. I run 32 in my TrueNAS builds (36TB and 60TB).
- magicalhippo 11mo ago
  You can run it with much less. I don't recall the bare minimum but with a bit of tweaking 2GB should be plenty[1]. I recall reading some running it on a 512MB system, but that was a while ago so not sure if you can still go that low. Performance can suffer though, for example low memory will limit the size of the transaction groups. So for decent performance you will want 8GB or more depending on workloads. [1]: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/FAQ.html#hardware-requirements https://openzfs.github.io/openzfs-docs/Project%20and%20Commu...

yjftsjthsd-h 11mo ago

That's only for deduplication. https://superuser.com/a/993019 https://superuser.com/a/993019

Lammy 11mo ago
I do like to deduplicate my BitTorrent downloads/seeding directory with my media directories so I can edit metadata to my heart's content while still seeding forever without having to incur 2x storage usage. I tune the `recordsize` to 1MiB so it has vastly fewer blocks to keep track of compared to the default 128K, at the cost of any modification wasting very slightly more space. Really not a big deal though when talking about multi-gibibyte media containers, multi-megapixel art embeds, etc.
- yegle 11mo ago
  cross-seed | cross-seed https://www.cross-seed.org/ https://www.cross-seed.org/
  - bscphil 11mo ago
    I believe they are saying they literally edit the media files to add / change metadata. Cross-seeding is only possible if the files are kept the same.
- zenoprax 11mo ago
  Have you considered "reflinks"? Supported as of [OpenZFS 2.2](https://github.com/openzfs/zfs/pull/13392 https://github.com/openzfs/zfs/pull/13392). Haven't used them yet myself but seems like a nice use case for things like minor metadata changes to media files. The bulk of the file is shared and only the delta between the two are saved.
  - Lammy 11mo ago
    Neat; will look into this. My setup is several years older than this, predating even FreeBSD's move to OpenZFS, and I just haven't touched the config of it since then since it works flawlessly (and since I already bought the RAM lol)