3 ms·

Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more ju

Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.

scurvy 8y ago
10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything. 125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).
- vvanders 8y ago
  Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch. I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
  - scurvy 8y ago
    You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap. Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
    - vvanders 8y ago
      Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link. However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.

scurvy 8y ago

10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything. 125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).

vvanders 8y ago
Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch. I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).
- scurvy 8y ago
  You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap. Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.
  - vvanders 8y ago
    Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link. However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.