5 ms·

They point to AIStor as alternative. Other alternatives: https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage https://github.com

They point to AIStor as alternative. Other alternatives: https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage https://github.com/rustfs/rustfs https://github.com/rustfs/rustfs https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs https://github.com/supabase/storage https://github.com/supabase/storage https://github.com/scality/cloudserver https://github.com/scality/cloudserver https://github.com/ceph/ceph https://github.com/ceph/ceph Among others

courtcircuits 8mo ago
From my experience, Garage is the best replacement to replace MinIO *in a dev environment*. It provides a pretty good CLI that makes automatic setup easier than MinIO. However in a production environment, I guess Ceph is still the best because of how prominent it is.
- egorfine 8mo ago
  Garage doesn't support CORS which makes it impossible to use for development for scenarios where web site visitors PUT files to pre-signed URLs.
  - courtcircuits 8mo ago
    Yep I know, I had to build a proxy for s3 which supports custom pre-signed URLs. In my case it was worth it because my team needs to verify uploaded content for security reasons. But for most cases I guess that you can't really bother deploying a proxy just for CORS. https:/github.com/beep-industries/content
  - lxpz 8mo ago
    Garage does actually support CORS (via PutBucketCORS). If there is anything specific missing, would you be wiling to open an issue on our issue tracker so that we can track it as a feature request?
    - egorfine 8mo ago
      Oh, indeed! Although the doc only mentions CORS [1] on an "exposing website page" which is not exactly related; also the mention strongly suggests using reverse proxy for CORS [1] which is an overkill and perhaps not needed if supported natively? Also googling the question only points out to the same reverse proxy page. Now that I know about PutBucketCORS it's perfectly clear but perhaps it's not easily discoverable. I am willing to write a cookbook article on signed browser uploads once I figure out all the details. [1] https://imgur.com/a/r3KoNkM https://imgur.com/a/r3KoNkM
dijit 8mo ago
Would be cool to understand the tradeoffs of the various block storage implementations. I'm using seaweedfs for a single-machine S3 compatible storage, and it works great. Though I'm missing out on a lot of administrative nice-to-haves (like, easy access controls and a good understanding of capacity vs usage, error rates and so on... this could be a pebcak issue though). Ceph I have also used and seems to care a lot more about being distributed. If you have less than 4 hosts for storage it feels like it scoffs at you when setting up. I was also unable to get it to perform amazingly, though to be fair I was doing it via K8S/Rook atop the Flannel CNI, which is an easy to use CNI for toy deployments, not performance critical systems - so that could be my bad. I would trust a ceph deployment with data integrity though, it just gives me that feel of "whomever worked on this, really understood distributed systems".. but, I can't put that feeling into any concrete data.
0xUndefined 8mo ago
Had great experience with garage for an easy to setup distributed s3 cluster for home lab use (connecting a bunch of labs run by friends in a shared cluster via tailscale/headscale). They offer a "eventual consistency" mode (consistency_mode = dangerous is the setting, so perhaps don't use it for your 7-nines SaaS offering) where your local s3 node will happily accept (and quickly process) requests and it will then duplicate it to other servers later. Overall great philosophy (target at self-hosting / independence) and clear and easy maintenance, not doing anything fancy, easy to understand architecture and design / operation instructions.
justincormack 8mo ago
Wrote a bit about differences between rustfs and garage here https://buttondown.com/justincormack/archive/ignore-previous-directions-10-new-rusty-object/ https://buttondown.com/justincormack/archive/ignore-previous... - since then rustfs fixed the issue I found. They are for very different use cases. Rustfs really is close to a minio rewrite.
- PunchyHamster 8mo ago
  there is one thing that worries me about rustfs: https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... I expect rugpull in the future
  - _bobm 8mo ago
    hah, good on them. nice catch.
  - rbtrsv 8mo ago
    Chineeez
- victormy 8mo ago
  I don't think this is a problem. The CLA is there to avoid future legal disputes. It prevents contributors from initiating IP lawsuits later on, which could cause significantly more trouble for the project.
  - everfrustrated 8mo ago
    The only possible "trouble" would be because the project does a rug-pull to closed source. Be honest about what you're saying.
GeertJohan 8mo ago
That's a great list. I've just opened a pull request on the minio repository to add these to the list of alternatives. https://github.com/minio/minio/pull/21746 https://github.com/minio/minio/pull/21746
- bluepuma77 8mo ago
  The mentioned AIStor "alternative" is on the min.io website. It seems like a re-brand. I doubt they will link to competing products.
- hinata08 8mo ago
  While I do approve of that MR, doing it is ironic considering the topic was "MinIO repository is no longer maintained" Let's hope the editor has second thoughts on some parts
  - GeertJohan 8mo ago
    I'm well aware of the irony surrounding minio, adding a little bit more doesn't hurt :P
- augusto-moura 8mo ago
  I believe the Minio developers are aware of the alternatives, having only their own commercial solution listed as alternatives might be a deliberate decision. But you can try merging the PR, there's nothing wrong with it
wvh 8mo ago
Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system underneath, for local, testing and small scale deployments. Not sure that exists? Of course a lot of stuff is being bolted onto S3 and it's not as simple as it initially claimed to be.
- memset 8mo ago
  For testing, consider https://github.com/localstack/localstack https://github.com/localstack/localstack
  - 9dev 8mo ago
    WAY too much. I just need a tiny service that translates common S3 ops into filesystem ops and back.
- seddonm1 8mo ago
  What about s3 stored in SQLite? https://github.com/seddonm1/s3ite https://github.com/seddonm1/s3ite This was written to store many thousands of images for machine learning
- magicalhippo 8mo ago
  From what I can gather, S3Proxy[1] can do this, but relies on a Java library that's no longer maintained[2], so not really much better. I too think it would be great with a simple project that can serve S3 from filesystem, for local deployments that doesn't need balls to the walls performance. [1]: https://github.com/gaul/s3proxy https://github.com/gaul/s3proxy [2]: https://jclouds.apache.org/ https://jclouds.apache.org/
- hobofan 8mo ago
  SeaweedFS's new `weed mini` command[0] does a great job at that. Previously our most flakey tests in CI were due to MinIO sometimes not starting up properly, but with `weed mini` that was completely resolved. [0]: https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with-weed-mini https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with...
mickael-kerjean 8mo ago
I'm the author of another option (https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash) which has a S3 gateway that expose itself as a S3 server but is just a proxy that forward your S3 call onto anything else like SFTP, local FS, FTP, NFS, SMB, IPFS, Sharepoint, Azure, git repo, Dropbox, Google Drive, another S3, ... it's entirely stateless and act as a proxy translating S3 call onto whatever you have connected in the other end
- cookiengineer 8mo ago
  Didn't know about filestash yet. Kudos, this framework seems to be really well implemented, I really like the plugin and interface based architecture.
- havnagiggle 8mo ago
  I was looking at running [versitygw](https://github.com/versity/versitygw https://github.com/versity/versitygw) but filestash looks pretty sweet! Any chance you're familiar with Versity and how the S3 proxy may differ?
  - mickael-kerjean 8mo ago
    I did a project with Monash university who were using Versity on their storage to handle multi tiers storage on their 12PB cluster, with glacier like capabilities on tape storage with a robot picking up data on their tape backup and a hot storage tier for better access performance, lifecycle rules to move data from hot storage to cold, etc.... The underlying storage was all Versity and they had Filestash working on top, effectively we did some custom plugins so you could recall the data on their own selfhosted glacier while using it through the frontend so their user had a Dropbox like experience. Depending on what you want to do they can be very much complimentary
    - antongribok 8mo ago
      Monash University is also a Ceph Foundation member. They've been active in the Ceph community for a long time. I don't know any specifics, but I'm pretty sure their Ceph installation is pretty big and used to support critical data.
lhaussknecht 8mo ago
We are using ruftfs for our simple usecases as a replacement for minio. Very slim footprint and very fast.
philipwhiuk 8mo ago
To be clear AIStor is built by the MinIO team so this is just an upsell.
PunchyHamster 8mo ago
from my experiences rustfs have promise, supports a lot of features, even allows to bring your own secret/access keys (if you want to migrate without changing creds on clients) but it's very much still in-development; and they have already prepared for bait-and-switch in code ( https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... ) Ceph is closest feature wise to actual S3 feature-set wise but it's a lot to setup. It pretty much wants few local servers, you can replicate to another site but each site on its own is pretty latency sensitive between storage servers. It also offers many other features aside, as S3 is just built on top of their object store that can be also used for VM storage or even FUSE-compatible FS Garage is great but it is very much "just to store stuff", it lacks features on both S3 side (S3 have a bunch of advanced ACLs many of the alternatives don't support, and stuff for HTTP headers too) and management side (stuff like "allow access key to access only certain path on the bucket is impossible for example). Also the clustering feature is very WAN-aware, unlike ceph where you pretty much have to have all your storage servers in same rack if you want a single site to have replication.
- antongribok 8mo ago
  Not sure what you mean about Ceph wanting to be in a single rack. I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has over 100 racks. In a typical Leaf-Spine network architecture, you can easily have sub 100 microsecond network latency which would translate to sub millisecond Ceph latencies. We have one site that is Leaf-Spine-SuperSpine, and the difference in network latency is barely measurable between machines in the same network pod and between different network pods.
  - PunchyHamster 8mo ago
    I meant that you can't span single cluster over high latency link, unlike for example garagefs
    - __turbobrew__ 8mo ago
      Ceph has synchronous replication, writes have to be acked by all replicas before the client gets an ack. Fundamentally, the latency of ceph is at least the latency between the OSDs. This is a tradeoff ceph makes for strong consistency.
dizhn 8mo ago
Both rustfs and seaweedfs are pretty pretty good based on my light testing.

courtcircuits 8mo ago

From my experience, Garage is the best replacement to replace MinIO *in a dev environment*. It provides a pretty good CLI that makes automatic setup easier than MinIO. However in a production environment, I guess Ceph is still the best because of how prominent it is.

egorfine 8mo ago
Garage doesn't support CORS which makes it impossible to use for development for scenarios where web site visitors PUT files to pre-signed URLs.
- courtcircuits 8mo ago
  Yep I know, I had to build a proxy for s3 which supports custom pre-signed URLs. In my case it was worth it because my team needs to verify uploaded content for security reasons. But for most cases I guess that you can't really bother deploying a proxy just for CORS. https:/github.com/beep-industries/content
- lxpz 8mo ago
  Garage does actually support CORS (via PutBucketCORS). If there is anything specific missing, would you be wiling to open an issue on our issue tracker so that we can track it as a feature request?
  - egorfine 8mo ago
    Oh, indeed! Although the doc only mentions CORS [1] on an "exposing website page" which is not exactly related; also the mention strongly suggests using reverse proxy for CORS [1] which is an overkill and perhaps not needed if supported natively? Also googling the question only points out to the same reverse proxy page. Now that I know about PutBucketCORS it's perfectly clear but perhaps it's not easily discoverable. I am willing to write a cookbook article on signed browser uploads once I figure out all the details. [1] https://imgur.com/a/r3KoNkM https://imgur.com/a/r3KoNkM

dijit 8mo ago

Would be cool to understand the tradeoffs of the various block storage implementations. I'm using seaweedfs for a single-machine S3 compatible storage, and it works great. Though I'm missing out on a lot of administrative nice-to-haves (like, easy access controls and a good understanding of capacity vs usage, error rates and so on... this could be a pebcak issue though). Ceph I have also used and seems to care a lot more about being distributed. If you have less than 4 hosts for storage it feels like it scoffs at you when setting up. I was also unable to get it to perform amazingly, though to be fair I was doing it via K8S/Rook atop the Flannel CNI, which is an easy to use CNI for toy deployments, not performance critical systems - so that could be my bad. I would trust a ceph deployment with data integrity though, it just gives me that feel of "whomever worked on this, really understood distributed systems".. but, I can't put that feeling into any concrete data.

0xUndefined 8mo ago

Had great experience with garage for an easy to setup distributed s3 cluster for home lab use (connecting a bunch of labs run by friends in a shared cluster via tailscale/headscale). They offer a "eventual consistency" mode (consistency_mode = dangerous is the setting, so perhaps don't use it for your 7-nines SaaS offering) where your local s3 node will happily accept (and quickly process) requests and it will then duplicate it to other servers later. Overall great philosophy (target at self-hosting / independence) and clear and easy maintenance, not doing anything fancy, easy to understand architecture and design / operation instructions.

justincormack 8mo ago

Wrote a bit about differences between rustfs and garage here https://buttondown.com/justincormack/archive/ignore-previous-directions-10-new-rusty-object/ https://buttondown.com/justincormack/archive/ignore-previous... - since then rustfs fixed the issue I found. They are for very different use cases. Rustfs really is close to a minio rewrite.

PunchyHamster 8mo ago
there is one thing that worries me about rustfs: https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... I expect rugpull in the future
- _bobm 8mo ago
  hah, good on them. nice catch.
- rbtrsv 8mo ago
  Chineeez
victormy 8mo ago
I don't think this is a problem. The CLA is there to avoid future legal disputes. It prevents contributors from initiating IP lawsuits later on, which could cause significantly more trouble for the project.
- everfrustrated 8mo ago
  The only possible "trouble" would be because the project does a rug-pull to closed source. Be honest about what you're saying.

GeertJohan 8mo ago

That's a great list. I've just opened a pull request on the minio repository to add these to the list of alternatives. https://github.com/minio/minio/pull/21746 https://github.com/minio/minio/pull/21746

bluepuma77 8mo ago
The mentioned AIStor "alternative" is on the min.io website. It seems like a re-brand. I doubt they will link to competing products.
hinata08 8mo ago
While I do approve of that MR, doing it is ironic considering the topic was "MinIO repository is no longer maintained" Let's hope the editor has second thoughts on some parts
- GeertJohan 8mo ago
  I'm well aware of the irony surrounding minio, adding a little bit more doesn't hurt :P
augusto-moura 8mo ago
I believe the Minio developers are aware of the alternatives, having only their own commercial solution listed as alternatives might be a deliberate decision. But you can try merging the PR, there's nothing wrong with it

wvh 8mo ago

Apart from Minio, we tried Garage and Ceph. I think there's definitely a need for something that interfaces using S3 API but is just a simple file system underneath, for local, testing and small scale deployments. Not sure that exists? Of course a lot of stuff is being bolted onto S3 and it's not as simple as it initially claimed to be.

memset 8mo ago
For testing, consider https://github.com/localstack/localstack https://github.com/localstack/localstack
- 9dev 8mo ago
  WAY too much. I just need a tiny service that translates common S3 ops into filesystem ops and back.
seddonm1 8mo ago
What about s3 stored in SQLite? https://github.com/seddonm1/s3ite https://github.com/seddonm1/s3ite This was written to store many thousands of images for machine learning
magicalhippo 8mo ago
From what I can gather, S3Proxy[1] can do this, but relies on a Java library that's no longer maintained[2], so not really much better. I too think it would be great with a simple project that can serve S3 from filesystem, for local deployments that doesn't need balls to the walls performance. [1]: https://github.com/gaul/s3proxy https://github.com/gaul/s3proxy [2]: https://jclouds.apache.org/ https://jclouds.apache.org/
hobofan 8mo ago
SeaweedFS's new `weed mini` command[0] does a great job at that. Previously our most flakey tests in CI were due to MinIO sometimes not starting up properly, but with `weed mini` that was completely resolved. [0]: https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with-weed-mini https://github.com/seaweedfs/seaweedfs/wiki/Quick-Start-with...

mickael-kerjean 8mo ago

I'm the author of another option (https://github.com/mickael-kerjean/filestash https://github.com/mickael-kerjean/filestash) which has a S3 gateway that expose itself as a S3 server but is just a proxy that forward your S3 call onto anything else like SFTP, local FS, FTP, NFS, SMB, IPFS, Sharepoint, Azure, git repo, Dropbox, Google Drive, another S3, ... it's entirely stateless and act as a proxy translating S3 call onto whatever you have connected in the other end

cookiengineer 8mo ago
Didn't know about filestash yet. Kudos, this framework seems to be really well implemented, I really like the plugin and interface based architecture.
havnagiggle 8mo ago
I was looking at running [versitygw](https://github.com/versity/versitygw https://github.com/versity/versitygw) but filestash looks pretty sweet! Any chance you're familiar with Versity and how the S3 proxy may differ?
- mickael-kerjean 8mo ago
  I did a project with Monash university who were using Versity on their storage to handle multi tiers storage on their 12PB cluster, with glacier like capabilities on tape storage with a robot picking up data on their tape backup and a hot storage tier for better access performance, lifecycle rules to move data from hot storage to cold, etc.... The underlying storage was all Versity and they had Filestash working on top, effectively we did some custom plugins so you could recall the data on their own selfhosted glacier while using it through the frontend so their user had a Dropbox like experience. Depending on what you want to do they can be very much complimentary
  - antongribok 8mo ago
    Monash University is also a Ceph Foundation member. They've been active in the Ceph community for a long time. I don't know any specifics, but I'm pretty sure their Ceph installation is pretty big and used to support critical data.

lhaussknecht 8mo ago

We are using ruftfs for our simple usecases as a replacement for minio. Very slim footprint and very fast.

philipwhiuk 8mo ago

To be clear AIStor is built by the MinIO team so this is just an upsell.

PunchyHamster 8mo ago

from my experiences rustfs have promise, supports a lot of features, even allows to bring your own secret/access keys (if you want to migrate without changing creds on clients) but it's very much still in-development; and they have already prepared for bait-and-switch in code ( https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... ) Ceph is closest feature wise to actual S3 feature-set wise but it's a lot to setup. It pretty much wants few local servers, you can replicate to another site but each site on its own is pretty latency sensitive between storage servers. It also offers many other features aside, as S3 is just built on top of their object store that can be also used for VM storage or even FUSE-compatible FS Garage is great but it is very much "just to store stuff", it lacks features on both S3 side (S3 have a bunch of advanced ACLs many of the alternatives don't support, and stuff for HTTP headers too) and management side (stuff like "allow access key to access only certain path on the bucket is impossible for example). Also the clustering feature is very WAN-aware, unlike ceph where you pretty much have to have all your storage servers in same rack if you want a single site to have replication.

antongribok 8mo ago
Not sure what you mean about Ceph wanting to be in a single rack. I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has over 100 racks. In a typical Leaf-Spine network architecture, you can easily have sub 100 microsecond network latency which would translate to sub millisecond Ceph latencies. We have one site that is Leaf-Spine-SuperSpine, and the difference in network latency is barely measurable between machines in the same network pod and between different network pods.
- PunchyHamster 8mo ago
  I meant that you can't span single cluster over high latency link, unlike for example garagefs
  - __turbobrew__ 8mo ago
    Ceph has synchronous replication, writes have to be acked by all replicas before the client gets an ack. Fundamentally, the latency of ceph is at least the latency between the OSDs. This is a tradeoff ceph makes for strong consistency.

dizhn 8mo ago

Both rustfs and seaweedfs are pretty pretty good based on my light testing.