3 ms·

> Machines dies. Hardware has bugs, or is broken. Things just bork. It's a fact of life. I know, I'm a sysadmin. I care for hardware, mend it, heal it, and som

> Machines dies. Hardware has bugs, or is broken. Things just bork. It's a fact of life. I know, I'm a sysadmin. I care for hardware, mend it, heal it, and sometimes donate, cann-bird or bury it. I'm used to it. > worst case scenario is an host reimage... While hosting PBs of data on it? No, thanks. > Would I build a file storage system around btrfs? No - without proper redundancy at least. Everything is easy for small n. When you store 20TB on 4x5TB drives, everything can be done. When you have a >5PB of storage on racks, you need at least a copy of that system running hot-standby. That's not cheap in any sense. Instead, I'd use ZFS, Lustre, anything, but not BTRFS. > I can live with that - applications will generally break production ten billion times before btrfs does. In our case, no. Our systems doesn't stop because a daemon decided to stop because a server among many fried itself.

chronid 3y ago
I have worked on and around systems with an order of magnitude more data and a single node failing did not matter. We weren't using btrfs anyway (for data drives) and it definitely was not cheap. But storage never is. But again, most systems are not like that. Kubernetes cluster nodes? Reimage at will. Compute nodes for vms backed by SAN? Reimage at will. Btrfs can actually make that reimage faster and it's pretty reliable on a single flash drive so why not?
- bayindirh 3y ago
  Well, that was my primary point. BTRFS is not ready for these kind of big installations handled by ZFS or Lustre at this point. On the other hand, BTRFS’ single disk performance, esp, for small files is visibly lower than EXT4 and XFS, so why bother? There are many solutions for EXT4 which allows versioning, and if I can reimage a node (or 200) in 5 minute flat, why should I bother with the overhead of BTRFS? It’s not that I haven’t tried BTRFS. Its features are nice, but from my perspective, it’s not ready for prime time, yet. What bothers me is the mental gymnastics pretending that it’s mature at this point. It’ll be good file system. An excellent one in fact, but it still needs to cook.

chronid 3y ago

I have worked on and around systems with an order of magnitude more data and a single node failing did not matter. We weren't using btrfs anyway (for data drives) and it definitely was not cheap. But storage never is. But again, most systems are not like that. Kubernetes cluster nodes? Reimage at will. Compute nodes for vms backed by SAN? Reimage at will. Btrfs can actually make that reimage faster and it's pretty reliable on a single flash drive so why not?

bayindirh 3y ago
Well, that was my primary point. BTRFS is not ready for these kind of big installations handled by ZFS or Lustre at this point. On the other hand, BTRFS’ single disk performance, esp, for small files is visibly lower than EXT4 and XFS, so why bother? There are many solutions for EXT4 which allows versioning, and if I can reimage a node (or 200) in 5 minute flat, why should I bother with the overhead of BTRFS? It’s not that I haven’t tried BTRFS. Its features are nice, but from my perspective, it’s not ready for prime time, yet. What bothers me is the mental gymnastics pretending that it’s mature at this point. It’ll be good file system. An excellent one in fact, but it still needs to cook.