5 ms·

What are the limits of ZFS, can it scale up to the petabyte range like Ceph?

gbrown_ 6y ago
Ceph is a distributed storage system whereas ZFS is not so you can't really compare the two by the capacity people run them at.
- tutfbhuf 6y ago
  Okay, if I'm not allowed to compare them directly, what are their different use cases?
hosteur 6y ago
I believe the z in zfs is for zettabyte
Tuna-Fish 6y ago
ZFS can scale a lot further than Ceph. The limits were designed to be large enough to be never encountered in practice. Not just large enough to be never encountered by the people working on it, but large enough that it is not possible to fit a filesystem that needs more on earth, no matter how good your technology is. The only limits that can be reached is that the maximum size of a single file is 2^64 bytes, or 16 exabytes, and the maximum amount of files in a single directory is 2^48, or 281 trillion. The other limits are large enough that to reach them would generally require more energy than it would take to literally boil the oceans.
- tutfbhuf 6y ago
  I mean practically speaking. Lets say I actually were to buy 100 x 10TB drives. I know that I can span a ceph cluster across those drives (e.g. CERN has created some 30PB cluster in the past). Can I use 100 drives in a RAID-Z? I'm asking because the main advantage I can see for a zettabyte file system (ZFS) is at larger scale, thus I'm interested in how to actually use it e.g. for a server setup of a company that needs to save data on many disks.
  - gh02t 6y ago
    ZFS can use that many drives, but it's not a distributed filesystem so it's intended to use within a single server. For 100 disks (or some arbitrarily large number) I'd assume you want something distributed, in which case ZFS could serve as the underlying FS on each node with e.g. Gluster or Ceph. ZFS as the backing store for Gluster is fairly common, for example.
    - deleted 6y ago
      [deleted]
    - secabeen 6y ago
      I have two systems with 140 drives in a single ZFS pool. (14 VDEVS of 10 drives each). They work great, and we haven't had any issues with them.
      - tutfbhuf 6y ago
        What type of connection do these system have (e.g. 10GbE) and how much latency? Would it be possible to span a ZFS Pool across multiple regions (us, europe, ...)?
        secabeen 6y ago
        The two systems are in different buildings on campus. Each system (with 140) drives is a single unit, with 60-drive JBODs running on SAS-external connections. If you want to span a pool across multiple regions, you'll want a distributed file system on top of your ZFS to manage that. Something like Lustre or Ceph. It would still be very challenging, though.
  - magicalhippo 6y ago
    > Can I use 100 drives in a RAID-Z AFAIK yes you could, but you wouldn't want to put them in a single RAID-Z(1,2 or 3) VDEV, but rather multiple VDEVs. This is because I/O operations scale with the number of VDEVs rather than number of disks[1]. So there's a tradeoff between space efficiency and IOPS. But AFAIK you absolutely could put 100 disks in a single RAID-Z VDEV. On the mailing lists there's frequently people posting with that or more disks in a single pool (split across multiple VDEVs). [1]: https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz https://www.delphix.com/blog/delphix-engineering/zfs-raidz-s...
rsync 6y ago
"What are the limits of ZFS, can it scale up to the petabyte range like Ceph?" We (rsync.net) have scaled zpools to petabyte range. A current, example configuration would be: - 60-drive JBODs - 15 drive raidz3 vdevs, four per JBOD - 16TB SAS drives That ends up being ~192TB per vdev, 768 TB per JBOD ... and if you span a pool across two JBODs, you have ~1.5 PB. I should note that what makes it possible to sleep at night with such a configuration is the fact that raidz3 exists. If not for that, I would not configure 15 drive vdevs with jut raidz2 ("raid6") protection.
- tutfbhuf 6y ago
  Has rsync.net ever considered Ceph and if yes, why decided against it in favour of ZFS?
  - rsync 6y ago
    No, we did not ever consider ceph. rsync.net architecture is purely FreeBSD and there was just a very good fit - and roadmap - from UFS2 which we used from 2001-2012 and ZFS which we have used since. Also, for better or worse, rsync.net is about UNIX and ZFS is very unixy. We think in filesystems and files and directories and ZFS let's us keep that set of abstractions.

Ceph is a distributed storage system whereas ZFS is not so you can't really compare the two by the capacity people run them at.

tutfbhuf 6y ago
Okay, if I'm not allowed to compare them directly, what are their different use cases?

hosteur 6y ago

I believe the z in zfs is for zettabyte

Tuna-Fish 6y ago

ZFS can scale a lot further than Ceph. The limits were designed to be large enough to be never encountered in practice. Not just large enough to be never encountered by the people working on it, but large enough that it is not possible to fit a filesystem that needs more on earth, no matter how good your technology is. The only limits that can be reached is that the maximum size of a single file is 2^64 bytes, or 16 exabytes, and the maximum amount of files in a single directory is 2^48, or 281 trillion. The other limits are large enough that to reach them would generally require more energy than it would take to literally boil the oceans.

tutfbhuf 6y ago
I mean practically speaking. Lets say I actually were to buy 100 x 10TB drives. I know that I can span a ceph cluster across those drives (e.g. CERN has created some 30PB cluster in the past). Can I use 100 drives in a RAID-Z? I'm asking because the main advantage I can see for a zettabyte file system (ZFS) is at larger scale, thus I'm interested in how to actually use it e.g. for a server setup of a company that needs to save data on many disks.
- gh02t 6y ago
  ZFS can use that many drives, but it's not a distributed filesystem so it's intended to use within a single server. For 100 disks (or some arbitrarily large number) I'd assume you want something distributed, in which case ZFS could serve as the underlying FS on each node with e.g. Gluster or Ceph. ZFS as the backing store for Gluster is fairly common, for example.
  - deleted 6y ago
    [deleted]
  - secabeen 6y ago
    I have two systems with 140 drives in a single ZFS pool. (14 VDEVS of 10 drives each). They work great, and we haven't had any issues with them.
    - tutfbhuf 6y ago
      What type of connection do these system have (e.g. 10GbE) and how much latency? Would it be possible to span a ZFS Pool across multiple regions (us, europe, ...)?
      - secabeen 6y ago
        The two systems are in different buildings on campus. Each system (with 140) drives is a single unit, with 60-drive JBODs running on SAS-external connections. If you want to span a pool across multiple regions, you'll want a distributed file system on top of your ZFS to manage that. Something like Lustre or Ceph. It would still be very challenging, though.
- magicalhippo 6y ago
  > Can I use 100 drives in a RAID-Z AFAIK yes you could, but you wouldn't want to put them in a single RAID-Z(1,2 or 3) VDEV, but rather multiple VDEVs. This is because I/O operations scale with the number of VDEVs rather than number of disks[1]. So there's a tradeoff between space efficiency and IOPS. But AFAIK you absolutely could put 100 disks in a single RAID-Z VDEV. On the mailing lists there's frequently people posting with that or more disks in a single pool (split across multiple VDEVs). [1]: https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz https://www.delphix.com/blog/delphix-engineering/zfs-raidz-s...

rsync 6y ago

"What are the limits of ZFS, can it scale up to the petabyte range like Ceph?" We (rsync.net) have scaled zpools to petabyte range. A current, example configuration would be: - 60-drive JBODs - 15 drive raidz3 vdevs, four per JBOD - 16TB SAS drives That ends up being ~192TB per vdev, 768 TB per JBOD ... and if you span a pool across two JBODs, you have ~1.5 PB. I should note that what makes it possible to sleep at night with such a configuration is the fact that raidz3 exists. If not for that, I would not configure 15 drive vdevs with jut raidz2 ("raid6") protection.

tutfbhuf 6y ago
Has rsync.net ever considered Ceph and if yes, why decided against it in favour of ZFS?
- rsync 6y ago
  No, we did not ever consider ceph. rsync.net architecture is purely FreeBSD and there was just a very good fit - and roadmap - from UFS2 which we used from 2001-2012 and ZFS which we have used since. Also, for better or worse, rsync.net is about UNIX and ZFS is very unixy. We think in filesystems and files and directories and ZFS let's us keep that set of abstractions.