3 ms·

Seeing a ton of adoption of this after the Minio debacle https://www.repoflow.io/blog/benchmarking-self-hosted-s3-compatible-storage-a-practical-performance-co

Seeing a ton of adoption of this after the Minio debacle https://www.repoflow.io/blog/benchmarking-self-hosted-s3-compatible-storage-a-practical-performance-comparison https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful. RustFS also looks interesting but for entirely non-technical reasons we had to exclude it. Anyone have any advice for swapping this in for Minio?

dpedu 10mo ago
I have not tried either myself, but I wanted to mention that Versity S3 Gateway looks good too. https://github.com/versity/versitygw https://github.com/versity/versitygw I am also curious how Ceph S3 gateway compares to all of these.
- zipzad 10mo ago
  I'd be curious to know how versitygw compares to rclone serve S3.
- skrtskrt 10mo ago
  When I was there, DigitalOcean was writing a complete replacement for the Ceph S3 gateway because its performance under high concurrency was awful. They just completely swapped out the whole service from the stack and wrote one in Go because of how much better the concurrency management was, and Ceph's team and codebase C++ was too resistant to change.
  - jiqiren 10mo ago
    Unrelated, but one of the more annoying aspects of whatever software they use now is lack of IPv6 for the CDN layer of DigitalOcean Spaces. It means I need to proxy requests myself. :(
Implicated 10mo ago
> but for entirely non-technical reasons we had to exclude it Able/willing to expand on this at all? Just curious.
- NitpickLawyer 10mo ago
  Not the same person you asked, but my guess would be that it is seen as a chinese product.
  - lima 10mo ago
    RustFS appears to be very early-stage with no real distributed systems architecture: https://github.com/rustfs/rustfs/pull/884 https://github.com/rustfs/rustfs/pull/884 I'm not sure if it even has any sort of cluster consensus algorithm? I can't imagine it not eating committed writes in a multi-node deployment. Garage and Ceph (well, radosgw) are the only open source S3-compatible object storage which have undergone serious durability/correctness testing. Anything else will most likely eat your data.
    - KevinatRustFS 10mo ago
      Hi there, RustFS team member here! Thanks for taking a look. To clarify our architecture: RustFS is purpose-built for high-performance object storage. We intentionally avoid relying on general-purpose consensus algorithms like Raft in the data path, as they introduce unnecessary latency for large blobs. Instead, we rely on Erasure Coding for durability and Quorum-based Strict Consistency for correctness. A write is strictly acknowledged only after the data has been safely persisted to the majority of drives. This means the concern about "eating committed writes" is addressed through strict read-after-write guarantees rather than a background consensus log. While we avoid heavy consensus for data transfer, we utilize dsync—a custom, lightweight distributed locking mechanism—for coordination. This specific architectural strategy has been proven reliable in production environments at the EiB scale.
      - lima 9mo ago
        Is there a paper or some other architecture document for dsync? It's really hard to solve this problem without a consensus algorithm in a way that doesn't sacrifice something (usually correctness in edge cases/network partitions). Data availability is easy(ish), but keeping the metadata consistent requires some sort of consensus, either using Raft/Paxos/..., using strictly commutative operations, or similar. I'm curious how RustFS solves this, and I couldn't find any documentation. EiB scale doesn't mean much - some workloads don't require strict metadata consistency guarantees, but others do.
scottydelta 10mo ago
From what I have seen in the previous discussions here (since and before Minio debacle) and at work, Garage is a solid replacement.
klooney 10mo ago
Seaweed looks good in those benchmarks, I haven't heard much about it for a while.
chrislusf 10mo ago
Disclaim: I work on SeaweedFS. Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.
- dionian 10mo ago
  can you link benchmarks
  - chrislusf 10mo ago
    It is in the parent comment.
- meotimdihia 10mo ago
  I confirm this, I used SeaweedFS to serve 1M users daily with 56 million images / ~100TB with 2 servers + HDD only, while Minio can't do this. Seaweedfs performance is much better than Minio's. The only problem is that SeaweedFS documentation is hard to understand.
  - magicalhippo 10mo ago
    SeaweedFS is also so optimized for small objects, it can't store larger objects (max 32GiB[1]). Not a concern for many use-cases, just something to be aware of as it's not a universal solution. [1]: https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#storage-size https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#st...
    - chrislusf 10mo ago
      Not correct. The files are chunked into smaller pieces and spread to all volume servers.
      - magicalhippo 10mo ago
        Well, then I suggest updating the incorrect readme. It's why I've ignored SeaweedFS.
deleted 10mo ago
[deleted]
elvinagy 10mo ago
I’m Elvin from the RustFS team in the U.S. Thanks for sharing the benchmark; it’s helpful to see how RustFS performs in real-world setups. We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us. If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.

dpedu 10mo ago

I have not tried either myself, but I wanted to mention that Versity S3 Gateway looks good too. https://github.com/versity/versitygw https://github.com/versity/versitygw I am also curious how Ceph S3 gateway compares to all of these.

zipzad 10mo ago
I'd be curious to know how versitygw compares to rclone serve S3.
skrtskrt 10mo ago
When I was there, DigitalOcean was writing a complete replacement for the Ceph S3 gateway because its performance under high concurrency was awful. They just completely swapped out the whole service from the stack and wrote one in Go because of how much better the concurrency management was, and Ceph's team and codebase C++ was too resistant to change.
- jiqiren 10mo ago
  Unrelated, but one of the more annoying aspects of whatever software they use now is lack of IPv6 for the CDN layer of DigitalOcean Spaces. It means I need to proxy requests myself. :(

Implicated 10mo ago

> but for entirely non-technical reasons we had to exclude it Able/willing to expand on this at all? Just curious.

NitpickLawyer 10mo ago
Not the same person you asked, but my guess would be that it is seen as a chinese product.
- lima 10mo ago
  RustFS appears to be very early-stage with no real distributed systems architecture: https://github.com/rustfs/rustfs/pull/884 https://github.com/rustfs/rustfs/pull/884 I'm not sure if it even has any sort of cluster consensus algorithm? I can't imagine it not eating committed writes in a multi-node deployment. Garage and Ceph (well, radosgw) are the only open source S3-compatible object storage which have undergone serious durability/correctness testing. Anything else will most likely eat your data.
  - KevinatRustFS 10mo ago
    Hi there, RustFS team member here! Thanks for taking a look. To clarify our architecture: RustFS is purpose-built for high-performance object storage. We intentionally avoid relying on general-purpose consensus algorithms like Raft in the data path, as they introduce unnecessary latency for large blobs. Instead, we rely on Erasure Coding for durability and Quorum-based Strict Consistency for correctness. A write is strictly acknowledged only after the data has been safely persisted to the majority of drives. This means the concern about "eating committed writes" is addressed through strict read-after-write guarantees rather than a background consensus log. While we avoid heavy consensus for data transfer, we utilize dsync—a custom, lightweight distributed locking mechanism—for coordination. This specific architectural strategy has been proven reliable in production environments at the EiB scale.
    - lima 9mo ago
      Is there a paper or some other architecture document for dsync? It's really hard to solve this problem without a consensus algorithm in a way that doesn't sacrifice something (usually correctness in edge cases/network partitions). Data availability is easy(ish), but keeping the metadata consistent requires some sort of consensus, either using Raft/Paxos/..., using strictly commutative operations, or similar. I'm curious how RustFS solves this, and I couldn't find any documentation. EiB scale doesn't mean much - some workloads don't require strict metadata consistency guarantees, but others do.

scottydelta 10mo ago

From what I have seen in the previous discussions here (since and before Minio debacle) and at work, Garage is a solid replacement.

klooney 10mo ago

Seaweed looks good in those benchmarks, I haven't heard much about it for a while.

chrislusf 10mo ago

Disclaim: I work on SeaweedFS. Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.

dionian 10mo ago
can you link benchmarks
- chrislusf 10mo ago
  It is in the parent comment.
meotimdihia 10mo ago
I confirm this, I used SeaweedFS to serve 1M users daily with 56 million images / ~100TB with 2 servers + HDD only, while Minio can't do this. Seaweedfs performance is much better than Minio's. The only problem is that SeaweedFS documentation is hard to understand.
- magicalhippo 10mo ago
  SeaweedFS is also so optimized for small objects, it can't store larger objects (max 32GiB[1]). Not a concern for many use-cases, just something to be aware of as it's not a universal solution. [1]: https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#storage-size https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#st...
  - chrislusf 10mo ago
    Not correct. The files are chunked into smaller pieces and spread to all volume servers.
    - magicalhippo 10mo ago
      Well, then I suggest updating the incorrect readme. It's why I've ignored SeaweedFS.

deleted 10mo ago

[deleted]

elvinagy 10mo ago

I’m Elvin from the RustFS team in the U.S. Thanks for sharing the benchmark; it’s helpful to see how RustFS performs in real-world setups. We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us. If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.