6 ms·

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy. With the exception of a couple people like Jim Salters, I don't take any advice I see about ZFS unless I know the person is a dev or has commits on the project. And most of the time I take the advice I've seen 'ryao' post which is, paraphrased, "use the defaults unless you know what you're tuning". That should go without saying, but there are a lot of armchair "geniuses" posting online about ZFS and they are just guessing. And, how to you get any new developers if the expectation is an immediate, complete understanding of the system? That just isn't realistic and, in most cases, the dev that knows a little bit probably knows exponentially more than an average user. Interacting on the bug tracker and debugging things they don't completely understand is going to improve their understanding. There's value there. Things have gotten more complicated too. We aren't using spinning disks without caches any more, right? I can understand a disk from 20 years ago. They're relatively simple. I couldn't tell you the first thing about SSDs or NVMe stuff. There are going to be bugs. Just be glad the devs acknowledge them and fix them. Personally I still rank ZFS as the least likely to lose my data and I'll keep using it. I put roughly zero value in a forum full of people complaining.

solarkraft 3y ago
A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me. Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.
- favorited 3y ago
  I think it comes down to the fact that the TrueNAS developers treat it as an appliance (which is a good thing, IMO). However, because so many people's only experience with ZFS is TrueNAS, when iXsystems says "Doing X on TrueNAS is a bad idea, that's not a configuration we support," it gets cargo culted as "doing X on ZFS is bad."
- bpbp-mango 3y ago
  Using it through something like TrueNAS with sensible defaults is probably a better idea for the average person than trying to roll their own deployment, no?
  - angus-prune 3y ago
    Im not sure running TrueNAS counts as learning ZFS, which I think is GP's point. If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. If you want to learn ZFS enough to roll your own deployment, change the configuration correctly and understand the changes you make then don't start with TrueNAS, and definitely don't listen to the TrueNAS community. Or at least, that's how I read the GP comment.
    - donmcronald 3y ago
      > If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. I evaluated TrueNAS Scale about 6 months ago and bailed on it very quickly. My install with their "sensible defaults" ended up having swap partitions on all my spinning disks. That alone was enough for me to decide against using it. One bad choice likely means other bad choices in my experience. They also trampled / reset my 'zfs_zrc_max' tunable every time a VM got started (I think) and it wasn't obvious what was doing it. I didn't think there was any value in learning the quirks of TrueNAS over doing a setup from scratch, so I ended up going with Ubuntu LTS and doing all my own config / testing.
    - pixl97 3y ago
      I mean, pretty much if you want to learn anything you're going to have to put a lot of time into it. With ZFS it will be building different servers with different disk layouts and figuring out how to test and benchmark those options. Which just takes an absolutely massive amount of time. I know, I was doing just that 5+ years ago. Commercial storage vendors aren't really any different. Mostly the biggest difference here is they offer a very limited subset of hardware and disk configurations and tend to iterate over that very small configuration space. Step outside of that config space and they won't have much insight. This remind me of the earlier days of Mac vs PC. Mac tended to have a very small hardware set of supported devices, and in many ways helped stability. PC supported pretty much anything you threw at it, which commonly had issues with crappy drivers and untested interactions.
- withinboredom 3y ago
  Reminds me of a game (MMO) I play on the weekends. Any time someone new joins, they always ask how come frame rates are so low. Then we begin the ”set your graphic settings to high” conversation. They immediately say we are full of shit. It’s already set to low and getting 10 fps. Eventually, someone explains: “just try it, or stop complaining.” 30s later, they exclaim: “holy ** I’m getting 60 fps now!” Sometimes, what looks like misinformation isn’t misinformation. TL;DR: this game switches to a software rendering pipeline on “medium” or lower. You have to set it on “high” to switch to the hardware rendering pipeline.
  - jeegsy 3y ago
    Wow! Thats a great story
  - Am4TIfIsER0ppos 3y ago
    Is that a visible option somewhere or do you only have one setting, the preset?
    - withinboredom 3y ago
      The preset is the only way to influence it.
hotpotamus 3y ago
One of my favorite comments chains here was was something to the affect of: Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression. Another commenter asks how they know that. First commenter replies that they wrote the code for it. Nothing more than a ZFS user (to some extent my company relies on it even) here, so I just try and color inside the lines, but I'm ever impressed by it.
- rincebrain 3y ago
  You probably mean [1]. [1] - https://news.ycombinator.com/item?id=36608024 https://news.ycombinator.com/item?id=36608024
  - hotpotamus 3y ago
    That is it, thank you.
RantyDave 3y ago
On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more akin to a directory, and many of them can be changed on a live filesystem. Here, have at it: https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. It's a shame a bug crept in to an otherwise pretty much faultless filesystem. I'm sure there are some truths in there about commercial support or not running the bleeding edge or whatever. And making backups, even for ZFS backed filesystems. Sacrilege, I know.
- 3np 3y ago
  > whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for ZFS backed filesystems. Sacrilege, I know. Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.
  - colechristensen 3y ago
    Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it. In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
nailer 3y ago
> So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy. No, that is not plausible. It's a filesystem. Behaviour in all cases for a filesystem should be predictable. There are zero things a user should be able to do that would make a filesystem non deterministic.
hdjdkdbdbe 3y ago
Tbh guessing is part of the way to solve a bug. You need to make a hypothesis and the test if it is true. If not try another. Saying the zfs/btrfs teams have no idea means the person has no idea about software development process ... It is a system none fully knows a system that complex. If they did, the bug wouldn't have happened.

solarkraft 3y ago

A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me. Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.

favorited 3y ago
I think it comes down to the fact that the TrueNAS developers treat it as an appliance (which is a good thing, IMO). However, because so many people's only experience with ZFS is TrueNAS, when iXsystems says "Doing X on TrueNAS is a bad idea, that's not a configuration we support," it gets cargo culted as "doing X on ZFS is bad."
bpbp-mango 3y ago
Using it through something like TrueNAS with sensible defaults is probably a better idea for the average person than trying to roll their own deployment, no?
- angus-prune 3y ago
  Im not sure running TrueNAS counts as learning ZFS, which I think is GP's point. If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. If you want to learn ZFS enough to roll your own deployment, change the configuration correctly and understand the changes you make then don't start with TrueNAS, and definitely don't listen to the TrueNAS community. Or at least, that's how I read the GP comment.
  - donmcronald 3y ago
    > If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. I evaluated TrueNAS Scale about 6 months ago and bailed on it very quickly. My install with their "sensible defaults" ended up having swap partitions on all my spinning disks. That alone was enough for me to decide against using it. One bad choice likely means other bad choices in my experience. They also trampled / reset my 'zfs_zrc_max' tunable every time a VM got started (I think) and it wasn't obvious what was doing it. I didn't think there was any value in learning the quirks of TrueNAS over doing a setup from scratch, so I ended up going with Ubuntu LTS and doing all my own config / testing.
  - pixl97 3y ago
    I mean, pretty much if you want to learn anything you're going to have to put a lot of time into it. With ZFS it will be building different servers with different disk layouts and figuring out how to test and benchmark those options. Which just takes an absolutely massive amount of time. I know, I was doing just that 5+ years ago. Commercial storage vendors aren't really any different. Mostly the biggest difference here is they offer a very limited subset of hardware and disk configurations and tend to iterate over that very small configuration space. Step outside of that config space and they won't have much insight. This remind me of the earlier days of Mac vs PC. Mac tended to have a very small hardware set of supported devices, and in many ways helped stability. PC supported pretty much anything you threw at it, which commonly had issues with crappy drivers and untested interactions.
withinboredom 3y ago
Reminds me of a game (MMO) I play on the weekends. Any time someone new joins, they always ask how come frame rates are so low. Then we begin the ”set your graphic settings to high” conversation. They immediately say we are full of shit. It’s already set to low and getting 10 fps. Eventually, someone explains: “just try it, or stop complaining.” 30s later, they exclaim: “holy ** I’m getting 60 fps now!” Sometimes, what looks like misinformation isn’t misinformation. TL;DR: this game switches to a software rendering pipeline on “medium” or lower. You have to set it on “high” to switch to the hardware rendering pipeline.
- jeegsy 3y ago
  Wow! Thats a great story
- Am4TIfIsER0ppos 3y ago
  Is that a visible option somewhere or do you only have one setting, the preset?
  - withinboredom 3y ago
    The preset is the only way to influence it.

hotpotamus 3y ago

One of my favorite comments chains here was was something to the affect of: Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression. Another commenter asks how they know that. First commenter replies that they wrote the code for it. Nothing more than a ZFS user (to some extent my company relies on it even) here, so I just try and color inside the lines, but I'm ever impressed by it.

rincebrain 3y ago
You probably mean [1]. [1] - https://news.ycombinator.com/item?id=36608024 https://news.ycombinator.com/item?id=36608024
- hotpotamus 3y ago
  That is it, thank you.

RantyDave 3y ago

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more akin to a directory, and many of them can be changed on a live filesystem. Here, have at it: https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. It's a shame a bug crept in to an otherwise pretty much faultless filesystem. I'm sure there are some truths in there about commercial support or not running the bleeding edge or whatever. And making backups, even for ZFS backed filesystems. Sacrilege, I know.

3np 3y ago
> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for ZFS backed filesystems. Sacrilege, I know. Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.
- colechristensen 3y ago
  Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it. In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.

nailer 3y ago

> So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy. No, that is not plausible. It's a filesystem. Behaviour in all cases for a filesystem should be predictable. There are zero things a user should be able to do that would make a filesystem non deterministic.

hdjdkdbdbe 3y ago

Tbh guessing is part of the way to solve a bug. You need to make a hypothesis and the test if it is true. If not try another. Saying the zfs/btrfs teams have no idea means the person has no idea about software development process ... It is a system none fully knows a system that complex. If they did, the bug wouldn't have happened.