10 ms·
Garage: An open-source distributed object storage service
- ddorian43 4y agoA very good alternative is seaweedfs https://github.com/chrislusf/seaweedfs/ https://github.com/chrislusf/seaweedfs/ based on facebook haystack paper (efficient small files) & more.
- mdaniel 4y agoDid you pull that URL out of some blog post or something? The accurate URL is https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- ddorian43 4y agoHe (the developer) had the repo on his own profile. Then he moved the repo to seaweedfs organization. My browser auto-completed the old url.
- ollybee 4y agoIt looks similar to minio, which as also an AGPL single binary that implements the S3 API. However Minio is written in Go and Garage is in Rust. I'd love to see a detailed comparison.
- Terretta 4y agoGarage design goals and non-goals: https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/ Seaweed design goals / features: https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs MinIO: https://min.io/docs/minio/linux/operations/concepts.html https://min.io/docs/minio/linux/operations/concepts.html Note Garage list of non goals specify priority features of Seaweed or MinIO, for example erasure coding.
- ilyt 4y agoDoes it have API parity with S3 ? On top of normal get/put S3 offers a retention policies and abilities to apply say a given header to a file or whole bucket
- remram 4y agoReading that only makes Minio and Garage seem more similar. They are both single-binary, clustered, S3-compatible file storage solutions. They are both meant for self-hosting on consumer-grade hard drives, provide redundancy to deal with drive or node failures, and don't aim to implement the entire AWS S3 API surface.
- wvh 4y agoWe're running Ceph with Rook and Minio in Kubernetes. I'm evaluating Garage as an alternative for some purposes such as application instance static files for Python or Javascript applications to avoid block volumes in deployments. Garage has a simpler high availability distribution story than Minio's erasure coding striping, but it works well for applications that don't require sub-second writes. It can do high availability with only three nodes versus four for Minio, which works well for smaller setups. It can also serve static websites straight from a bucket, which would work well for static sites like those made with static site generators. Ceph has an S3 API gateway, but it's a more wholesome solution better suited for larger setups compared to Minio and Garage.
- inportb 4y agoIt seems that Garage is designed to be fast when inter-node latency is high. A geodistributed "cluster," for example, would profit from using Garage instead of Minio.
- throwawayk8s 4y agoIs there any OSS implementation that is not AGPL?
- ilyt 4y agoWhy would that matter? Apps using its api don't need to be AGPL
- tyingq 4y agoFor some places, the in-house legal department makes it very difficult to use anything AGPL even if you assure them you're only using the api.
- throwawayk8s 4y agoWe would like to self host and our legal team forbids us from using AGPL stuff.
- remram 4y agoSounds like a problem created entirely by your legal team, not software or licenses.
- tomrod 4y agoWorking with legal is advisable instead of ignoring it.
- remram 4y agoThe solution to "colleague got it wrong" is not "silently ignore them" and I never suggested that. In fact my suggestion is precisely that you "work with them" and have them review this specific case. "Don't use AGPL" is a good baseline rule if you don't have a legal team but does not apply in this case as I'm sure they'd advise if they reviewed it.
- speedgoose 4y ago
- ilyt 4y ago
- CommanderData 4y agoReally clean docs page, nice design!
- bayesian_horse 4y agoComparison to Minio?
- seized 4y agoJust finished installing it on my OpenIndiana NAS to replace Minio. Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. Minios listing operations were horribly slow, still have to see if Garage resolves that.
- KronisLV 4y ago> Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over to storing the blobs in some attached storage that had XFS: https://blog.kronis.dev/tutorials/3-4-pidgeot-a-system-for-millions-of-documents-deploying-and-scaling https://blog.kronis.dev/tutorials/3-4-pidgeot-a-system-for-m... With abstracted storage (say, files bunches up into X MB large containers or chunked into such when too large, with something else to keep track of what is where) that wouldn't be such an issue, though you might end up with other issues along the way. It's curious that we don't advocate for storing blobs in relational databases anymore, even though I can also understand the reasoning (or at least why having a separate DB for your system data and your blob data would be a good idea, for backups/test data/deciding where to host what and so on).
- alex_sf 4y ago> I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. Honestly, I'd expect the exact opposite. Filesystems are really good at storing files. Why not leverage all that work? > I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over to storing the blobs in some attached storage that had XFS That's a misconfiguration issue though, not a reason to not store blobs as files on disk. Ext4 can handle 2^32 files. ZFS can handle 2^128(?). > With abstracted storage (say, files bunches up into X MB large containers or chunked into such when too large, with something else to keep track of what is where) that wouldn't be such an issue, though you might end up with other issues along the way. A few issues that come to mind for me: * This requires tuning to actually reduce the number of inodes of used for certain datasets. E.g., if I'm storing large media files, that chunking would _increase_ the number of files on disk, not reduce it. At which point, if inode limits are the issue, we're just making it worse. * It adds additional complexity. Now you need to account for these chunks, and, if you care about the data, check it periodically. * You need specific tooling to work with it. Files on a filesystem are.. files on a filesystem. Easy to backup, easy to view. Arbitrary chunking and such requires tooling to perform operations on it. Tooling that may break, or have the wrong versions, or.. etc. > It's curious that we don't advocate for storing blobs in relational databases anymore, even though I can also understand the reasoning In my experience, the popular RDBMS out there just aren't good at it. With the way locking semantics and their transaction queueing works, storing and retrieving lots of blobs just isn't performant. You can get away with it for a long time though, and it can be pretty nice when you can.
- deleted 4y ago[deleted]
- rodolphoarruda 4y agoHow does this compare to Wasabi for hot backups?
- anaganisk 4y agoWasabi is a service this is a filesystem
- g9yuayon 4y agoDo we know the largest size of data that these object stores have handled? People seem have been moving away from HDFS, yet companies could host exabytes of data on HDFS and serve TBs of scans per second with a single team of fewer than 15 people. I was wondering how production-ready the other OSS alternatives are for such scale of data.
- barefeg 4y agoKind of topic. But I’m wondering if it’s possible to get comparable pricing when self-hosting a solution similar to this in the cloud vs say AWS S3.
- rglullis 4y agoYes. Specially S3 which is way overpriced. I have setup already a multi-region minio cluster at the cost of $0.0039/GB. The only "cloud" solution that could get closer to that is Storj. [0] [0]: https://www.storj.io/ https://www.storj.io/
- toomim 4y agoThanks! Where are you hosting the machines? Are they on a vps somewhere, or do you own the hardware?
- rglullis 4y agoHetzner dedicated servers in their different data centers.
- deleted 4y ago[deleted]
- brunoqc 4y agoIsn't storj using a cryptocurrency?
- rglullis 4y agoYou pay for the service in cash if you want. The token is to pay out the nodes storing it.
- didip 4y agoFor this use-case, I like JuiceFS better. * https://juicefs.com/en/ https://juicefs.com/en/ * https://github.com/juicedata/juicefs https://github.com/juicedata/juicefs I am not affiliated with them, just a regular user.
- gengstrand 4y agoI remember working at a company that got started before cloud took off. They used mogile fs which I recently found at https://mogilefs.github.io/mogilefs-docs/HighLevelOverview.html https://mogilefs.github.io/mogilefs-docs/HighLevelOverview.h... but I never hear about anyone else using it. It wasn't as stable as S3 but it was okay, I guess. Does anyone else here remember that distributed open source file system?
- acidburnNSA 4y agoSo is this supposed to be a simpler, less "omg you need a full time staff of 10 to manage" version of Ceph? https://docs.ceph.com/en/quincy/ https://docs.ceph.com/en/quincy/
- pauldardeau 4y agoDoubtful, as Garage's features are a small subset of Ceph's. Ceph supports block storage, object storage (via RadosGW), and file storage (via cephfs). Ceph is highly sophisticated and very complex. It's anything but simple.
- dang 4y agoRelated: Garage, our self-hosted distributed object storage solution - https://news.ycombinator.com/item?id=30256753 https://news.ycombinator.com/item?id=30256753 - Feb 2022 (130 comments)