6 ms·

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real pro

by jjirsa 9y ago

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtime would be the least painful part of the eventual failure.

codys 9y ago
> There's at least one unpatched and there are bound to be more. References, please.
- jjirsa 9y ago
  https://news.ycombinator.com/item?id=14938092 https://news.ycombinator.com/item?id=14938092
codys 9y ago
It might be worthwhile mentioning that at some point in the past, HN ran on 1 server (and might still do so). https://news.ycombinator.com/item?id=5229522 https://news.ycombinator.com/item?id=5229522
- jjirsa 9y ago
  Cute but irrelevant?
patrickg_zill 9y ago
Actually the real issue is, "when the system is 65% full and you reboot, how long will it take for ZFS to mount it"? Perhaps he has split up ZFS into a number of different pools and they can be mounted in parallel (depends on the init script and whether ZFS can do this). But I do recall that larger ZFS pools can take a bit of time to mount; maybe the updated ZFS for Linux is faster....
- exikyut 9y ago
  > how long will it take for ZFS to mount it? What? How long would it take, roughly? Genuinely curious.
  - XorNot 9y ago
    Depends on the number of snapshots in my experience. 84,000 was slightly too many.
- deleted 9y ago
  [deleted]
exikyut 9y ago
I'm planning on doing a tiny ZFS pool (once I've finally saved up for it - financials can be fun sometimes!), and was thinking of doing FreeBSD for ZFS and Linux for everything else on top of Xen. I'm currently unsure how to make Linux see the ZFS pool though. I.... don't really like NFS. It's too glitchy in my experience. I use it to listen to music stored on a different machine from my laptop, which uses a long-range USB Wi-Fi adapter. If I unplug the adapter without cleanly unmounting /nfs, I get a kworker in an infinite loop. I googled around one afternoon and discovered that RHEL apparently found and fixed this in kernel 3.x. Interesting - I'm on Slackware, with kernel 4.1.x. >.< I wish you could do cross-VM virtio. That would be awesome. Then I could export the device node corresponding to the whole pool from FreeBSD and just mount it as a gigantic ext4 filesystem on Linux. (Can you do that?!)
- deleted 9y ago
  [deleted]
- drvdevd 9y ago
  You could in theory have a zvol exported from a ZFS pool on FreeBSD to Linux via some remote block dev protocol. iSCSI comes to mind, as it would have to be cross platform. Then slap ext4 on top of that in Linux. The idea feels "janky" though. Lots of overhead compared to just running ZFS on Linux.
  - inopinatus 9y ago
    Not just in theory. I've done this for a large dev/test Xen cluster (pool was actually on Solaris). Worked well. We made snapshots to match major release tags.
sathackr 9y ago
That's something I was thinking about. This thing has to go down occasionally for patches, service, maintenance. 500TB is alot to trust to a single box or two. I'm building a Ceph cluster right now, about the same size as the one in the article, except with 9x36 drive chassis.
- KaiserPro 9y ago
  Clustered filesystems just means that one machine can wreak havock across many domains of fun. (I used early lustre, I know from fun experience) ceph seems nice, but appears to be very CPU hungry and not very fast. Seems easier to have sharded vanilla linux file servers. It requires a decent asset manager, but once you have that, backups and rebalancing become trivial.
ams6110 9y ago
My (admittedly limted) experience with ZFS has been bad. I know it's supposed to be great but that is not what I have experienced. Multi-hour outages. Having to have Oracle help resolve problems. This was a few year ago now, maybe it's better these days. I was not the administrator, just a user, but we had ZFS issues at least quarterly on a ~20TB shared filesystem on a Linux cluster.
- walrus 9y ago
  Was ZFS determined to be the problem in the root cause analysis, or was it another part of the storage system that failed? It's easy to pass the blame down to the lowest level of the system because no one wants to take the blame. If it was actually determined to be the fault of ZFS, what was the problem? I'd like to avoid it in my own deployment.
- deelowe 9y ago
  Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?
  - FullyFunctional 9y ago
    With utmost respect: this isn't a super valuable data point. A "production deployment" with lots of users and/or workloads will see issues you will never encounter in a moderate setting. FWIW, I use FreeNAS in a similar small but diverse setting with nary a problem, but when I sat it up for a 30+ organization, issue arose that I hadn't expected (not data loss, but usability and performance issues).
    - cyphar 9y ago
      ZFS has definitely been used in very large production deployments in a variety of scenarios for well over a decade. The question seems reasonable given that history. ZFS isn't some toy filesystem hacked together over a weekend.
    - deelowe 9y ago
fnj 9y ago
> Zfs on Linux has data loss bugs. Please don't spread untruths. Somebody who doesn't know better might actually believe you.
- jjirsa 9y ago
  https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1600060 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/160... Unpatched on 16.04, referenced as supported in the article
  - cyphar 9y ago
    I read the patch and linked issues. It looks like an OpenZFS bug to me, which wasn't ported to ZoL until recently but it definitely doesn't look like a ZoL-exclusive bug. Not to mention that the issue description doesn't say it's a data loss bug. It's a bug that means that ZFS send would not include holes in very specific scenarios. On-disk data would still be safe as far as I can see (though I'm not an FS expert by any stretch).
  - raattgift 9y ago
    That was strictly speaking an openzfs bug that hit all the ports. There's another one, somewhat related: https://github.com/zfsonlinux/zfs/issues/6224 https://github.com/zfsonlinux/zfs/issues/6224 which (so far) seems tied to using recordsize > 128k without either of the -L or -c flags on the zfs send side, with the result that the sendstream is corrupted in such a way that the receiver cannot detect the corruption. As with the filled-holes problem, the problem is real but rare. Unlike the filled-holes problem, it is unlikely to affect many people since it is (probably) very rare that anyone uses large records and does not use -L (or -c, or both), although there are certainly automatic snapshot-send systems (e.g. znapzend) that use a common minimal set of options to zfs send. This is especially unfortunate because of the rarity of the corruptions, the apparent rarity of people using POSIX-layer checksumming (e.g. rsync -c, or sha256deep or the like) on large datasets (with large files that had holes made and refilled, for example) to validate that a received dataset really is the same as the original, and the apparent rarity of people doing this sort of validation specifically targetting backwards compatibility mechanisms (e.g. zfs recv into a version 28 or earlier pool from a source dataset that uses all the most recent bells and whistles). Finally, it is extra-especially unfortunate because recovering from this sort of corruption is awkward and time-consuming; at the minimum the source and destination have to be entirely read at least once or alternatively the destination needs to be destroyed and sent again from scratch once the fix or workaround for send|recv corruptions is known.
deelowe 9y ago
You care to back up those data loss claims with any proof? I agree that this smells a bit funny, but the author seems to dance around the important topics like vpools, raid levels, and use case. Instead they just focus on the hardware, which is arguably the least interesting bit (until you know the former).
petre 9y ago
> It's like a blog written by a 22 year old straight out of college Also no mention of FreeBSD. Instead he picked the less battle tested ZoL and only mentioned Solaris and licensing, so the author is not even aware of Illumos.

by jjirsa 9y ago

codys 9y ago

> There's at least one unpatched and there are bound to be more. References, please.

jjirsa 9y ago
https://news.ycombinator.com/item?id=14938092 https://news.ycombinator.com/item?id=14938092

codys 9y ago

It might be worthwhile mentioning that at some point in the past, HN ran on 1 server (and might still do so). https://news.ycombinator.com/item?id=5229522 https://news.ycombinator.com/item?id=5229522

jjirsa 9y ago
Cute but irrelevant?

patrickg_zill 9y ago

Actually the real issue is, "when the system is 65% full and you reboot, how long will it take for ZFS to mount it"? Perhaps he has split up ZFS into a number of different pools and they can be mounted in parallel (depends on the init script and whether ZFS can do this). But I do recall that larger ZFS pools can take a bit of time to mount; maybe the updated ZFS for Linux is faster....

exikyut 9y ago
> how long will it take for ZFS to mount it? What? How long would it take, roughly? Genuinely curious.
- XorNot 9y ago
  Depends on the number of snapshots in my experience. 84,000 was slightly too many.
deleted 9y ago
[deleted]

exikyut 9y ago

I'm planning on doing a tiny ZFS pool (once I've finally saved up for it - financials can be fun sometimes!), and was thinking of doing FreeBSD for ZFS and Linux for everything else on top of Xen. I'm currently unsure how to make Linux see the ZFS pool though. I.... don't really like NFS. It's too glitchy in my experience. I use it to listen to music stored on a different machine from my laptop, which uses a long-range USB Wi-Fi adapter. If I unplug the adapter without cleanly unmounting /nfs, I get a kworker in an infinite loop. I googled around one afternoon and discovered that RHEL apparently found and fixed this in kernel 3.x. Interesting - I'm on Slackware, with kernel 4.1.x. >.< I wish you could do cross-VM virtio. That would be awesome. Then I could export the device node corresponding to the whole pool from FreeBSD and just mount it as a gigantic ext4 filesystem on Linux. (Can you do that?!)

deleted 9y ago
[deleted]
drvdevd 9y ago
You could in theory have a zvol exported from a ZFS pool on FreeBSD to Linux via some remote block dev protocol. iSCSI comes to mind, as it would have to be cross platform. Then slap ext4 on top of that in Linux. The idea feels "janky" though. Lots of overhead compared to just running ZFS on Linux.
- inopinatus 9y ago
  Not just in theory. I've done this for a large dev/test Xen cluster (pool was actually on Solaris). Worked well. We made snapshots to match major release tags.

sathackr 9y ago

That's something I was thinking about. This thing has to go down occasionally for patches, service, maintenance. 500TB is alot to trust to a single box or two. I'm building a Ceph cluster right now, about the same size as the one in the article, except with 9x36 drive chassis.

KaiserPro 9y ago
Clustered filesystems just means that one machine can wreak havock across many domains of fun. (I used early lustre, I know from fun experience) ceph seems nice, but appears to be very CPU hungry and not very fast. Seems easier to have sharded vanilla linux file servers. It requires a decent asset manager, but once you have that, backups and rebalancing become trivial.

ams6110 9y ago

My (admittedly limted) experience with ZFS has been bad. I know it's supposed to be great but that is not what I have experienced. Multi-hour outages. Having to have Oracle help resolve problems. This was a few year ago now, maybe it's better these days. I was not the administrator, just a user, but we had ZFS issues at least quarterly on a ~20TB shared filesystem on a Linux cluster.

walrus 9y ago
Was ZFS determined to be the problem in the root cause analysis, or was it another part of the storage system that failed? It's easy to pass the blame down to the lowest level of the system because no one wants to take the blame. If it was actually determined to be the fault of ZFS, what was the problem? I'd like to avoid it in my own deployment.
deelowe 9y ago
Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?
- FullyFunctional 9y ago
  With utmost respect: this isn't a super valuable data point. A "production deployment" with lots of users and/or workloads will see issues you will never encounter in a moderate setting. FWIW, I use FreeNAS in a similar small but diverse setting with nary a problem, but when I sat it up for a 30+ organization, issue arose that I hadn't expected (not data loss, but usability and performance issues).
  - cyphar 9y ago
    ZFS has definitely been used in very large production deployments in a variety of scenarios for well over a decade. The question seems reasonable given that history. ZFS isn't some toy filesystem hacked together over a weekend.
  - deelowe 9y ago

fnj 9y ago

> Zfs on Linux has data loss bugs. Please don't spread untruths. Somebody who doesn't know better might actually believe you.

jjirsa 9y ago
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1600060 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/160... Unpatched on 16.04, referenced as supported in the article
- cyphar 9y ago
  I read the patch and linked issues. It looks like an OpenZFS bug to me, which wasn't ported to ZoL until recently but it definitely doesn't look like a ZoL-exclusive bug. Not to mention that the issue description doesn't say it's a data loss bug. It's a bug that means that ZFS send would not include holes in very specific scenarios. On-disk data would still be safe as far as I can see (though I'm not an FS expert by any stretch).
- raattgift 9y ago
  That was strictly speaking an openzfs bug that hit all the ports. There's another one, somewhat related: https://github.com/zfsonlinux/zfs/issues/6224 https://github.com/zfsonlinux/zfs/issues/6224 which (so far) seems tied to using recordsize > 128k without either of the -L or -c flags on the zfs send side, with the result that the sendstream is corrupted in such a way that the receiver cannot detect the corruption. As with the filled-holes problem, the problem is real but rare. Unlike the filled-holes problem, it is unlikely to affect many people since it is (probably) very rare that anyone uses large records and does not use -L (or -c, or both), although there are certainly automatic snapshot-send systems (e.g. znapzend) that use a common minimal set of options to zfs send. This is especially unfortunate because of the rarity of the corruptions, the apparent rarity of people using POSIX-layer checksumming (e.g. rsync -c, or sha256deep or the like) on large datasets (with large files that had holes made and refilled, for example) to validate that a received dataset really is the same as the original, and the apparent rarity of people doing this sort of validation specifically targetting backwards compatibility mechanisms (e.g. zfs recv into a version 28 or earlier pool from a source dataset that uses all the most recent bells and whistles). Finally, it is extra-especially unfortunate because recovering from this sort of corruption is awkward and time-consuming; at the minimum the source and destination have to be entirely read at least once or alternatively the destination needs to be destroyed and sent again from scratch once the fix or workaround for send|recv corruptions is known.

deelowe 9y ago

You care to back up those data loss claims with any proof? I agree that this smells a bit funny, but the author seems to dance around the important topics like vpools, raid levels, and use case. Instead they just focus on the hardware, which is arguably the least interesting bit (until you know the former).

petre 9y ago

> It's like a blog written by a 22 year old straight out of college Also no mention of FreeBSD. Instead he picked the less battle tested ZoL and only mentioned Solaris and licensing, so the author is not even aware of Illumos.