4 ms·

Their ceph usage is quite odd IMO, that few disks per node and such a (for storage) slow network interconnect cannot give you good results. We have a two-year-o

Their ceph usage is quite odd IMO, that few disks per node and such a (for storage) slow network interconnect cannot give you good results. We have a two-year-old (time really does fly) ceph benchmark with 100G Ethernet interconnect for cluster network[0] if anybody is interested. IMO using NVMe's flash and then connecting that with anything below 25G makes not much sense and is a waste of money, as network isn't that expensive anymore; especially with three to five nodes you can set up a full mesh to avoid an often more expensive switch. IME Ceph really starts to shine on a) slightly bigger setups and b) maintenance like upgrading nodes and swapping faulty disks, that's just a bliss if one doesn't violate a few basic rules (like maybe don't use 2/1 replica (two copies but return already after only one got written)). But also smaller cluster can work fine and can be quite performant, well at least if the interconnect bandwidth isn't one that was specified at the end of the 90s.. Also, ZFS 0.8.6 is from 2020, lots of stuff happened since then, a bit odd to post a new benchmark using almost two-year-old software. [0]: https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09 https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-be...

yjftsjthsd-h 4y ago
> Also, ZFS 0.8.6 is from 2020, lots of stuff happened since then, a bit odd to post a new benchmark using almost two-year-old software. That's probably a side effect of using Ubuntu 20.04, just like having a 5.4 kernel (which is likewise getting old). Although yes, that probably isn't helping, and it would be interesting to do the same test and change nothing but moving to Ubuntu 22.04 and the corresponding kernel and ZFS versions, both of which should help.
- hardwaresofton 4y ago
  It's absolutely a side effect of using Ubuntu -- I had to build ZFS (was quite easy) to get newer versions on Ubuntu and do a bit to prevent the system from picking up the default version (pakages & *-dkms packages)
IcePic 4y ago
"maybe don't use 2/1 replica (two copies but return already after only one got written)" I don't think ceph ever does that.
- tlamponi 4y ago
  Sure does, you can create a pool with that target/minimum replica just fine, there are some hosters that thought they were improving performance and data usage (which, well, is true), but ignored the impossibility of having any real redundancy at all; note that 2/2 is just fine though, but the pool will go read only if one failure domain goes down (compared to 3/2 where you can continue operating just fine in such a case). IOW. 2/1 will never ever make sense, it's just crying for data loss, if you want to save on data usage, and if you also don't mind the (slightly) higher CPU load, you can use erasure coding instead: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_ec_pools https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ce... https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod...
  - IcePic 4y ago
    It still doesn't mean it accepts the write as finished when only 1 PG has it. I agree on the risks on why 2/1 is stupid, just not that this also means writes finish faster with min_size < replication_factor.
    - tlamponi 4y ago
      It does though, it defines when to complete the write command to a client and when general any IO should be allowed on an object (both need #copies >= min-size), from the Quincy PG settings docs: > Sets the minimum number of written replicas for objects in the pool in order > to acknowledge an I/O operation to the client. If minimum is not met, Ceph > will not acknowledge the I/O to the client, which may result in data loss. > This setting ensures a minimum number of replicas when operating in degraded > mode.
somat 4y ago
I have always wanted to make a ceph cluster with one disk per node, just to see how it would behave, from a performance view point it would suck, but from a fault tolerant viewpoint it would be amazing.

yjftsjthsd-h 4y ago

> Also, ZFS 0.8.6 is from 2020, lots of stuff happened since then, a bit odd to post a new benchmark using almost two-year-old software. That's probably a side effect of using Ubuntu 20.04, just like having a 5.4 kernel (which is likewise getting old). Although yes, that probably isn't helping, and it would be interesting to do the same test and change nothing but moving to Ubuntu 22.04 and the corresponding kernel and ZFS versions, both of which should help.

hardwaresofton 4y ago
It's absolutely a side effect of using Ubuntu -- I had to build ZFS (was quite easy) to get newer versions on Ubuntu and do a bit to prevent the system from picking up the default version (pakages & *-dkms packages)

IcePic 4y ago

"maybe don't use 2/1 replica (two copies but return already after only one got written)" I don't think ceph ever does that.

tlamponi 4y ago
Sure does, you can create a pool with that target/minimum replica just fine, there are some hosters that thought they were improving performance and data usage (which, well, is true), but ignored the impossibility of having any real redundancy at all; note that 2/2 is just fine though, but the pool will go read only if one failure domain goes down (compared to 3/2 where you can continue operating just fine in such a case). IOW. 2/1 will never ever make sense, it's just crying for data loss, if you want to save on data usage, and if you also don't mind the (slightly) higher CPU load, you can use erasure coding instead: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_ec_pools https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ce... https://docs.ceph.com/en/latest/rados/operations/erasure-code/ https://docs.ceph.com/en/latest/rados/operations/erasure-cod...
- IcePic 4y ago
  It still doesn't mean it accepts the write as finished when only 1 PG has it. I agree on the risks on why 2/1 is stupid, just not that this also means writes finish faster with min_size < replication_factor.
  - tlamponi 4y ago
    It does though, it defines when to complete the write command to a client and when general any IO should be allowed on an object (both need #copies >= min-size), from the Quincy PG settings docs: > Sets the minimum number of written replicas for objects in the pool in order > to acknowledge an I/O operation to the client. If minimum is not met, Ceph > will not acknowledge the I/O to the client, which may result in data loss. > This setting ensures a minimum number of replicas when operating in degraded > mode.

somat 4y ago

I have always wanted to make a ceph cluster with one disk per node, just to see how it would behave, from a performance view point it would suck, but from a fault tolerant viewpoint it would be amazing.