3 ms·
Not sure what you mean about Ceph wanting to be in a single rack. I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has ove
by antongribok 8mo ago
Not sure what you mean about Ceph wanting to be in a single rack.
I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has over 100 racks.
In a typical Leaf-Spine network architecture, you can easily have sub 100 microsecond network latency which would translate to sub millisecond Ceph latencies.
We have one site that is Leaf-Spine-SuperSpine, and the difference in network latency is barely measurable between machines in the same network pod and between different network pods.
- PunchyHamster 8mo agoI meant that you can't span single cluster over high latency link, unlike for example garagefs
- __turbobrew__ 8mo agoCeph has synchronous replication, writes have to be acked by all replicas before the client gets an ack. Fundamentally, the latency of ceph is at least the latency between the OSDs. This is a tradeoff ceph makes for strong consistency.
- PunchyHamster 8mo agoI know. We run it for near a decade now. I mentioned it because a lot of uses for minio are pretty small. I had 2 servers at home running their builtin site replication and it was super easy setup that would take far more of both hardware and work to replicate to ceph so while ceph might be theoretically fitting feature list, realisticially it isn't an option.
- yencabulator 8mo agoThe Ceph way of doing asynchronous replication would be to run separate clusters and ship incremental snapshots between them. I don't know if anyone's programmed the automation for that, but it's definitely doable. For S3 only, radosgw has it's own async replication thing. https://ceph.io/en/news/blog/2025/stretch-cluuuuuuuuusters-part1/#replication-options-in-ceph https://ceph.io/en/news/blog/2025/stretch-cluuuuuuuuusters-p... Disclaimer: ex-Ceph-developer.