12 ms·
Garage – An S3 object store so reliable you can run it outside datacenters
- SomaticPirate 11mo agoSeeing a ton of adoption of this after the Minio debacle https://www.repoflow.io/blog/benchmarking-self-hosted-s3-compatible-storage-a-practical-performance-comparison https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful. RustFS also looks interesting but for entirely non-technical reasons we had to exclude it. Anyone have any advice for swapping this in for Minio?
- dpedu 11mo agoI have not tried either myself, but I wanted to mention that Versity S3 Gateway looks good too. https://github.com/versity/versitygw https://github.com/versity/versitygw I am also curious how Ceph S3 gateway compares to all of these.
- zipzad 11mo agoI'd be curious to know how versitygw compares to rclone serve S3.
- skrtskrt 11mo agoWhen I was there, DigitalOcean was writing a complete replacement for the Ceph S3 gateway because its performance under high concurrency was awful. They just completely swapped out the whole service from the stack and wrote one in Go because of how much better the concurrency management was, and Ceph's team and codebase C++ was too resistant to change.
- jiqiren 11mo agoUnrelated, but one of the more annoying aspects of whatever software they use now is lack of IPv6 for the CDN layer of DigitalOcean Spaces. It means I need to proxy requests myself. :(
- Implicated 11mo ago> but for entirely non-technical reasons we had to exclude it Able/willing to expand on this at all? Just curious.
- NitpickLawyer 11mo agoNot the same person you asked, but my guess would be that it is seen as a chinese product.
- lima 11mo agoRustFS appears to be very early-stage with no real distributed systems architecture: https://github.com/rustfs/rustfs/pull/884 https://github.com/rustfs/rustfs/pull/884 I'm not sure if it even has any sort of cluster consensus algorithm? I can't imagine it not eating committed writes in a multi-node deployment. Garage and Ceph (well, radosgw) are the only open source S3-compatible object storage which have undergone serious durability/correctness testing. Anything else will most likely eat your data.
- KevinatRustFS 11mo agoHi there, RustFS team member here! Thanks for taking a look. To clarify our architecture: RustFS is purpose-built for high-performance object storage. We intentionally avoid relying on general-purpose consensus algorithms like Raft in the data path, as they introduce unnecessary latency for large blobs. Instead, we rely on Erasure Coding for durability and Quorum-based Strict Consistency for correctness. A write is strictly acknowledged only after the data has been safely persisted to the majority of drives. This means the concern about "eating committed writes" is addressed through strict read-after-write guarantees rather than a background consensus log. While we avoid heavy consensus for data transfer, we utilize dsync—a custom, lightweight distributed locking mechanism—for coordination. This specific architectural strategy has been proven reliable in production environments at the EiB scale.
- lima 10mo ago
- scottydelta 11mo agoFrom what I have seen in the previous discussions here (since and before Minio debacle) and at work, Garage is a solid replacement.
- klooney 11mo agoSeaweed looks good in those benchmarks, I haven't heard much about it for a while.
- chrislusf 11mo agoDisclaim: I work on SeaweedFS. Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a lot of features.
- dionian 11mo agocan you link benchmarks
- chrislusf 11mo agoIt is in the parent comment.
- meotimdihia 11mo agoI confirm this, I used SeaweedFS to serve 1M users daily with 56 million images / ~100TB with 2 servers + HDD only, while Minio can't do this. Seaweedfs performance is much better than Minio's. The only problem is that SeaweedFS documentation is hard to understand.
- magicalhippo 11mo agoSeaweedFS is also so optimized for small objects, it can't store larger objects (max 32GiB[1]). Not a concern for many use-cases, just something to be aware of as it's not a universal solution. [1]: https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#storage-size https://github.com/seaweedfs/seaweedfs?tab=readme-ov-file#st...
- chrislusf 11mo agoNot correct. The files are chunked into smaller pieces and spread to all volume servers.
- magicalhippo 11mo agoWell, then I suggest updating the incorrect readme. It's why I've ignored SeaweedFS.
- deleted 11mo ago[deleted]
- elvinagy 11mo agoI’m Elvin from the RustFS team in the U.S. Thanks for sharing the benchmark; it’s helpful to see how RustFS performs in real-world setups. We know trust matters, especially for a newer project, and we try to earn it through transparency and external validation. we were excited to see RustFS recently added as an optional service in Laravel Sail’s official Docker environment (PR #822). Having our implementation reviewed and accepted by a major ecosystem like Laravel was an encouraging milestone for us. If the “non-technical reasons” you mentioned are around licensing or governance, I’m happy to discuss our long-term Apache 2.0 commitment and path to a stable GA.
- ai-christianson 11mo agoI love garage. I think it has applications beyond the standard self host s3 alternative. It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.
- singpolyma3 11mo agoI'd love to hear what configuration you are using for this
- Powdering7082 11mo agoNo erasure coding seems like a pretty big loss in terms of how much resources do you need to get good resiliency & efficiency
- munro 11mo agoI was looking at using this on an LTO tape library, it seems the only resiliency is through replication, but this was my main concern with this project, what happens with HW goes bad
- lxpz 11mo agoIf you have replication, you can lose one of the replica, that's the point. This is what Garage was designed for, and it works. Erasure coding is another debate, for now we have chosen not to implement it, but I would personally be open to have it supported by Garage if someone codes it up.
- hathawsh 11mo agoErasure coding is an interesting topic for me. I've run some calculations on the theoretical longevity of digital storage. If you assume that today's technology is close to what we'll be using for a long time, then cross-device erasure coding wins, statistically. However, if you factor in the current exponential rate of technological development, simply making lots of copies and hoping for price reductions over the next few years turns out to be a winning strategy, as long as you don't have vendor lock-in. In other words, I think you're making great choices.
- Dylan16807 11mo agoI question that math. Erasure coding needs less than half as much space as replication, and imposes pretty small costs itself. Maybe we can say the difference is irrelevant if storage prices will drop 4x over the next five years? But looking at pricing trends right now... that's not likely. Hard drives and SSDs are about the same price they were 5 years ago. The 5 years before that SSDs were seeing good advancements, but hard drive prices only advanced 2x.
- deleted 11mo ago[deleted]
- fabian2k 11mo agoLooks interesting for something like local development. I don't intend to run production object storage myself, but some of the stuff in the guide to the production setup (https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/ https://garagehq.deuxfleurs.fr/documentation/cookbook/real-w...) would scare me a bit: > For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation. It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.
- igor47 11mo agoI've been using minio for local dev but that version is unmaintained now. However, I was put off by the minimum requirements for garage listed on the page -- does it really need a gig of RAM?
- archon810 11mo agoThe current latest Minio release that is working for us for local development is now almost a year old and soon enough we will have to upgrade. Curious what others have replaced it with that is as easy to set up and has a management UI.
- mbreese 11mo agoI think that's part of the pitch here... swapping out Minio for Garage. Both scale a lot more than for just local development, but local dev certainly seems like a good use-case here.
- lxpz 11mo agoIt does not, at least not for a small local dev server. I believe RAM usage should be around 50-100MB, increasing if you have many requests with large objects.
- doctorpangloss 11mo agohttps://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v1/script/jepsen.garage https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-... this is the reliability question no?
- lxpz 11mo agoI talked about the meaning of the Jepsen test and the results we obtained in the FOSDEM'24 talk: https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3009-advances-in-garage-the-low-tech-storage-platform-for-geo-distributed-clusters/ https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3... Slides are available here: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8bac07df7fd96ea5f3df7052e70f5577c753/doc/talks/2024-02-03-fosdem/talk.pdf https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...
- agwa 11mo agoDoes this support conditional PUT (If-Match / If-None-Match)?
- codethief 11mo agohttps://news.ycombinator.com/item?id=46328218 https://news.ycombinator.com/item?id=46328218
- faizshah 11mo agoOne really useful usecase for Garage for me has been data engineering scripts. I can just use the S3 integration that every tool has to dump to garage and then I can more easily scale up to cloud later.
- Eikon 11mo agoUnfortunately, this doesn’t support conditional writes through if-match and if-none-match [0] and thus is not compatible with ZeroFS [1]. [0] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052 [1] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS
- chrislusf 11mo agoI work on SeaweedFS. It has support for these if conditions, and a lot more.
- wyattjoh 11mo agoWasn't expecting to see it hosted on forgejo. Kind of a breath of fresh air to be honest.
- thhck 11mo agoBTW https://deuxfleurs.fr/ https://deuxfleurs.fr/ is one of the most beautiful website I have ever seen
- codethief 11mo agoIt's beautiful from an artistic point of view but also rather hard to read and probably not very accessible (haven't checked it, though, since I'm on my phone).
- isoprophlex 11mo agoWorks perfectly on an iphone. I can't attest to the accessibility features, but the aesthetic is absolutely wonderful. Something I love, and went for on my own portfolio/company website... this is executed 100x better tho, clearly a labor of love and not 30 minutes of shitting around in vi.
- self_awareness 11mo agoWell it's ASCII-themed but it's completely unreadable in terminal links/lynx.
- apawloski 11mo agoIs it the same consistency model as S3? I couldn't see anything about it in their docs.
- lxpz 11mo agoRead-after-write consistency : yes (after PutObject has finished, the object will be immediately visible in all subsequent requests, including GetObject and ListObjects) Conditionnal writes : no, we can't do it with CRDTs, which are the core of Garage's design.
- skrtskrt 11mo agoDoes RAMP or CURE offer any possibility of conditional writes with CRDTs? I have had these papers on my list to read for months, specifically wondering if it could be applied to Garage https://dd.thekkedam.org/assets/documents/publications/Report/RR-8858.pdf https://dd.thekkedam.org/assets/documents/publications/Repor... http://www.bailis.org/papers/ramp-sigmod2014.pdf http://www.bailis.org/papers/ramp-sigmod2014.pdf
- lxpz 11mo agoI had a very rapid look at these two papers, it looks like none of them allow the implementation of compare-and-swap, which is required for if-match / if-none-match support. They have a weaker definition of a "transaction". Which is to be expected as they only implement causal consistency at best and not consensus, whereas consensus is required for compare-and-swap.
- skrtskrt 11mo agoack - makes sense, thank you for looking!
- topspin 11mo agoNo tags on objects. Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs. But no tags on objects is a pretty big gap, and I had to shelve it. If Garage folk see this: please think on this. You obviously have the talent to make a killer application, but tags are table stakes in the "cloud" API world.
- lxpz 11mo agoThank you for your feedback, we will take it into account.
- topspin 11mo agoGreat, and thank you. I really, really appreciate that Garage accommodates running as a single node without work-arounds and special configuration to yield some kind of degraded state. Despite the single minded focus on distributed operation you no doubt hear endlessly (as seen among some comments here,) there are, in fact, traditional use cases where someone will be attracted to Garage only for the API compatibility, and where they will achieve availability in production sufficient to their needs by means other than clustering.
- VerifiedReports 11mo agoWhat are "tags on objects?"
- topspin 11mo agohttps://docs.aws.amazon.com/AmazonS3/latest/userguide/object-tagging.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object... Arbitrary name+value pairs attached to S3 objects and buckets, and readily available via the S3 API. Metadata, basically. AWS has some tie-ins with permissions and other features, but tags can be used for any purpose. You might encode video multiple times at different bitrates, and store the rate in a tag on each object, for example. Tags are an affordance used by many applications for countless purposes.
- JonChesterfield 11mo agoCorrupts data on power loss according to their own docs. Like what you get outside of data centers. Not reliable then.
- lxpz 11mo agoLosing a node is a regular occurrence, and a scenario for which Garage has been designed. The assumption Garage makes, which is well-documented, is that of 3 replica nodes, only 1 will be in a crash-like situation at any time. With 1 crashed node, the cluster is still fully functional. With 2 crashed nodes, the cluster is unavailable until at least one additional node is recovered, but no data is lost. In other words, Garage makes a very precise promise to its users, which is fully respected. Database corruption upon power loss enters in the definition of a "crash state", similarly to a node just being offline due to an internet connection loss. We recommend making metadata snapshots so that recovery of a crashed node is faster and simpler, but it's not required per se: Garage can always start over from an empty database and recover data from the remaining copies in the cluster. To talk more about concrete scenarios: if you have 3 replicas in 3 different physical locations, the assumption of at-most one crashed node is pretty reasonable, it's quite unlikely that 2 of the 3 locations will be offline at the same time. Concerning data corruption on a power loss, the probability to lose power at 3 distant sites at the exact same time with the same data in the write buffers is extremely low, so I'd say in practice it's not a problem. Of course, this all implies a Garage cluster running with 3-way replication, which everyone should do.
- jiggawatts 11mo agoSo if you put a 3-way cluster in the same building and they lose power together, then what? Is your data toast?
- lxpz 11mo agoIf I make certain assumptions and you respect them, I will give you certain guarantees. If you don't respect them, I won't guarantee anything. I won't guarantee that your data will be toast either.
- deleted 11mo ago[deleted]
- deleted 11mo ago[deleted]
- ekjhgkejhgk 11mo agoAnybody understand how this compares with Vast?
- allanrbo 11mo agoI use Syncthing a lot. Is Garage only really useful if you specifically want to expose an S3 drop in compatible API, or does it also provide other benefits over syncthing?
- lxpz 11mo agoThey are not solving the same problem. Syncthing will synchronize a full folder between an arbitrary number of machines, but you still have to access this folder one way or another. Garage provides an HTTP API for your data, and handles internally the placement of this data among a set of possible replica nodes. But the data is not in the form of files on disk like the ones you upload to the API. Syncthing is good for, e.g., synchronizing your documents or music collection between computers. Garage is good as a storage service for back-ups with e.g. Restic, for media files stored by a web application, for serving personal (static) web sites to the Internet. Of course, you can always run something like Nextcloud in front of Garage and get folder synchronization between computers somewhat like what you would get with Syncthing. But to answer your question, yes, Garage only provides a S3-compatible API specifically.
- sippeangelo 11mo agoYou use Syncthing for object storage?
- deleted 11mo ago[deleted]
- supernes 11mo agoI tried it recently. Uploaded around 300 documents (1GB) and then went to delete them. Maybe my client was buggy, because the S3 service inside the container crashed and couldn't recover - I had to restart it. It's a really cool project, but I wouldn't really call it "reliable" from my experience.
- awoimbee 11mo agoHow is garage for a simple local dev env ? I recently used seaweedfs since they have a super simple minimal setup compared to garage which seemed to require a config file just to get started.
- adamcharnock 11mo agoCopy/paste from a previous thread [0]: We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes) I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case. --- In addition: Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2]. We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking. [0]: https://news.ycombinator.com/item?id=46140342 https://news.ycombinator.com/item?id=46140342 [1]: https://rustfs.com/ https://rustfs.com/ [2]: https://rook.io/ https://rook.io/
- hardwaresofton 11mo agoPlease also consider including SeaweedFS in the testing.
- __turbobrew__ 11mo agoI wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.
- breakingcups 11mo agoIS there a better solution for self-healing S3 storage that you could recommend? I'm also curious what will make a rook cluster croak after some time and what kind of maintenance is required in your experience.
- adastra22 11mo agoceph?
- eduardogarza 11mo agoI use this for booting up S3-compatible buckets for local development and testing -- paired up with s5cmd, I can seed 15GB and over 60,000 items (seed/mock data) in < 60s... have a perfect replica of a staging environment with Docker containers (api, db, cache, objects) all up in less than 2mins. Super simple to set up for my case and been working great. Previously I used LocalStack S3 but ultimately didn't like the lack of persistance thats not available on the OSS verison. MinIO OSS is apparently no longer maintained? Also looked at SeaweedFS and RustFS but from a quick reading into them this once was the easiest to set up.
- chrislusf 11mo agoI work on SeaweedFS. So very biased. :) Just run "weed sever -s3 -dir=..." to have an object store.
- eduardogarza 11mo agoI'll try it!
- k__ 11mo agoHalf-OT: Does anyone know a good open source S3 alternarive that's easily extendable with custom storage backends? For example, AWS offers IA and Glacier in addition to the defaults.
- onionjake 11mo agoStorj supports arbitrary configured backends each with different erasure coding, node placement, etc.
- yupyupyups 11mo agoGarage is amazing! But it would be even more amazing if it had immutable object support. :) This is used for ransomware resistant backups.
- tenacious_tuna 11mo agoAnyone know if it's possible to bandwidth-limit the sync operations? I'd love to set up garage instances across my families' houses to act as a distributed backup, but I don't want to hose their (or my) down/uplink during awake hours. Having redundant selfhosted S3like storage would solve many problems for me, but I really need that capability.
- BOOSTERHIDROGEN 11mo agoI use juicefs
- ianopolous 11mo ago@lxpz It would be great to do a follow up to this blog post with the latest Peergos. All the issues with baseline bandwidth and requests have gone away, even with federation on. The baseline is now 0, and even many locally initiated requests will be served directly from a Peergos cache without touching S3. https://garagehq.deuxfleurs.fr/blog/2022-ipfs/ https://garagehq.deuxfleurs.fr/blog/2022-ipfs/ Let's talk!
- PunchyHamster 11mo agoFor someone recently migrating from minio, caveats * no lifecycle management of any kind - if you're using it for backups you can't set "don't delete versions for 3 months", so if anyone takes hold of your key, you backups are gone. I relied on minio's lifecycle management for that but it's feature missing in garage (and to be fair, most other) S3 * no automatic mirroring (if you want to have second copy in something other than garage or just don't want to have a cluster but rather have more independent nodes) * ACLs for access are VERY limited - can't make a key access only sub-path, can't make a "master key" (AFAIK, couldn't find an option) that can access all the buckets so the previous point is also harder - I can't easily use rclone to mirror entire instance somewhere else unless I write scrip iterating over buckets and adding them bucket by bucket to key ACK * Web hosting features are extremely limited so you won't be able to say set CORS headers for the bucket * No ability to set keys - you can only generate on inside garage or import garage-formatted one - which means you can't just migrate storage itself, you have to re-generate every key. It also makes automating it harder, in case of minio you can pre-generate key and pass then fed it to clients and to the minio key command, here you have to do the dance of "generate with tool" -> "scrape and put into DB" -> put onto clients. Overall I like the software a lot but if you have setup that uses those features, beware.
- coldtea 11mo ago>no lifecycle management of any kind - if you're using it for backups you can't set "don't delete versions for 3 months", so if anyone takes hold of your key, you backups are gone If someone gets a hold of your key, can't they also just change your backup deletion policy, even if it supported one?
- PunchyHamster 11mo ago> If someone gets a hold of your key, can't they also just change your backup deletion policy, even if it supported one? Minio have full on ACLs so you can just create a key that can only write/read but not change any settings like that. So you just need to keep the "master key" that you use for setup away from potentially vulnerable devices, the "backup key" doesn't need those permissions.
- JonChesterfield 11mo agoI think this works. A subset of S3's API does look like a CRDT. Metadata can go in sqlite. Compiles to a static binary easily. I've spent a mostly pleasant day seeing whether I can reasonably use garage + rclone as a replacement for NFS and the answer appears to be yes. Not really a recommended thing to do. Garage setup was trivial, somewhat reminiscent of wireguard. Rclone setup was a nuisance, accumulated a lot of arguments to get latency down and I think the 1.6 in trixie is buggy. Each node has rclone's fuse mount layer on it with garage as the backing store. Writes are slow and a bit async, debugging shows that to be wholly my fault for putting rclone in front of it. Reads are fast, whether pretending to be a filesystem or not. Yep, I think I'm sold. There will be better use cases for this than replacing NFS. Thanks for sharing :)