3 ms·

Yeah, I've got a Ceph cluster in my homelab that tops out about 80mb/s and it's pretty darn painful compared to a 2/3 disk mirrored ZFS on spinning rust.

polskibus 8y ago
Are there any good ceph alternatives? Why can't you use zfs directly?
- maccam94 8y ago
  Ceph is distributed storage, ZFS is direct-attached to one host at a time.
- vvanders 8y ago
  Yeah, there's a few but I haven't had a chance to give them a spin yet. I was mostly using Ceph to explore live migrations in Proxmox, although they just released support for ZFS based migrations(you have to migrate the whole local disk though so they aren't nearly as snappy). There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication. It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.
scurvy 8y ago
You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.
- vvanders 8y ago
  Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.
  - scurvy 8y ago
    10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything. 125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).
    - vvanders 8y ago
      Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch. I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
      - scurvy 8y ago
        You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap. Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
        vvanders 8y ago
        Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link. However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.

polskibus 8y ago

Are there any good ceph alternatives? Why can't you use zfs directly?

maccam94 8y ago
Ceph is distributed storage, ZFS is direct-attached to one host at a time.
vvanders 8y ago
Yeah, there's a few but I haven't had a chance to give them a spin yet. I was mostly using Ceph to explore live migrations in Proxmox, although they just released support for ZFS based migrations(you have to migrate the whole local disk though so they aren't nearly as snappy). There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication. It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.

scurvy 8y ago

You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.

vvanders 8y ago
Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.
- scurvy 8y ago
  10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything. 125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).
  - vvanders 8y ago
    Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch. I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
    - scurvy 8y ago
      You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap. Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
      - vvanders 8y ago
        Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link. However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.