26 ms·
Garage: Open-Source Distributed Object Storage
- bluepuma77 2y agoCan it be easily deployed with old-school Docker Swarm?
- TristanBall 2y agoI don't think I have ever personally felt older than having someone describe anything docker related as "old-school"
- bluepuma77 2y agoThat’s was not my intention! Docker is young and fashionable, every windows script kiddy uses it nowadays! And then comes to the Docker forum complaining about strange issues, not realizing Docker Desktop is a different product, it uses a Linux VM to run the Docker engine, which was build for Linux ;-) I explicitly wrote "old-school Docker Swarm", as that is missing love for years and everyone with 2 IT FTEs seems to be moving to k8s.
- CoolCold 2y agoI was insulted as well - luckily just mental insult, assuming my age and the context of what's old and what's not :)
- TechDebtDevin 2y agoSeaweedFS is great as well. https://github.com/seaweedfs/seaweedfs https://github.com/seaweedfs/seaweedfs
- evanjrowley 2y agoLooks awesome. Been looking for some flexible self-hosted WebDAV solutions and SeaweedFS would be an interesting choice.
- genewitch 2y agodepending on what you need it for nextcloud has WebDAV (clients can interact with it, and windows can mount your home folder directly, i just tried it out a couple days ago.) I've never used webdav before so i'm unsure of what other use cases there are, but the nextcloud implementation (whatever it may be) was friction-free - everything just worked.
- n_ary 2y agoTried this for my own homelab, either I misconfigured it or it consumes x2(linearly) memory(working) of the stored data. So, for example, if I put 1GB of data, seaweed would immediately consume 2GB of memory constantly! Edit: memory = RAM
- crest 2y agoAre you claiming that SeaweedFS requires twice as much RAM as the sum of the sizes of the stored objects?
- TechDebtDevin 2y agoThat is odd. It likely has something to do with the index caching and how many replication volumes you configured. By default it indexes all file metadata in RAM (I think) but that wouldn't justify that type of memory usage. I've always used mostly default configurations in Docker Swarm, similar to this: https://github.com/cycneuramus/seaweedfs-docker-swarm/blob/master/docker-compose.yml https://github.com/cycneuramus/seaweedfs-docker-swarm/blob/m...
- fijiaarone 2y agoI don’t understand why everyone wants to replicate AWS APIs for things that are not AWS. S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter. Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files, when you used to have a toolbox with drawers, folders, and labels. Replicating a real file system is not that hard, and when you lose the original reason for using a bucket —- because your were stuck in the swamp with nothing else to carry your files in — why keep using it when you’re out of the mud?
- TheColorYellow 2y agoBecause at this point it's a well known API. I bet people want to recreate AWS without the Amazon part, and so this is for them. Which, to your point, makes no sense because as you rightly point out, people use S3 because of the Amazon services and ecosystem it is integrated with - not at all because it is "good tech"
- acdha 2y agoS3 was the second AWS service, behind SQS, and saw rapid adoption which cannot be explained by integration with services introduced later.
- vlovich123 2y agoStorage is generally sticky but I wouldn’t be so quick to dismiss that reason because it might explain why anything would fail to displace it; a bunch of software is written against S3 and the entire ecosystem around it is quite rich. It doesn’t explain the initial popularity but does explain stickiness. Initial popularity was because it was the first good REST API to do cloud storage AND the price was super reasonable.
- acdha 2y agoOh, I’m definitely not saying integration or compatibility have nothing to do with it - only that “horrible interface with a terrible lack of features” seems impossible to reconcile with its immense popularity.
- comvidyarthi 2y agoIs this open source ?
- kevlened 2y agoAGPL https://git.deuxfleurs.fr/Deuxfleurs/garage https://git.deuxfleurs.fr/Deuxfleurs/garage
- computerfan494 2y agoI have used Garage for a long time. It's great, but the AWS sigv4 protocol for accessing it is just frustrating. Why can't I just send my API key as a header? I don't need the full AWS SDK to get and put files, and the AWS sigv4 is a ton of extra complexity to add to my projects. I don't care about the "security benefits" of AWS sigv4. I hope the authors consider a different authentication scheme so I can recommend Garage more readily.
- 6LLvveMx2koXfwn 2y agoImplementing v4 on the server side also requires the service to keep the token as plain text. If it's a persistent password, rather than an ephemeral key, that opens up another whole host of security issues around password storage. And on the flip side requiring the client to hit an endpoint to receive a session based token is even more crippling from a performance perspective.
- klysm 2y agoSending your api key in the header is equivalent to basic auth.
- computerfan494 2y agoYep, and that's fine with me. I don't have a problem with basic auth.
- vineyardmike 2y agoThis is not intended for commercial services. Realistically, this software was made for people who keep servers in their basement. The security profile of LAN users is very different than public AWS.
- anonzzzies 2y agoThe site says it was made (initially) and used for a commercial French hoster.
- CyberDildonics 2y agoWhat is the difference between a "distributed object storage" and a file system?
- vineyardmike 2y agoIt’s an S3 api compatible object store that supports distributed storage across different servers. Object store = store blobs of bytes. Usually by bucket + key accessible over HTTP. No POSIX expectation. Distributed = works spread across multiple servers in different locations.
- CyberDildonics 2y agostore blobs of bytes Files by bucket Directories key accessible File names over HTTP Web server
- crest 2y agoFiles are normally stored hierarchically (e.g. atomically move directories), and updated in place. Objects are normally considered to exist in a flat namespace and are written/replaced atomically. Object storage requires less expensive (in a distributed system) metadata operations. This means it's both easier and faster to scale out object storage.
- crabbone 2y agoThere are few. From the perspective of consistency guarantees, object storage gives fewer of such guarantees (this is seen as allowing implementations to be faster than typical file-systems). For example, since there isn't a concept of directories in object store, the implementation doesn't need to deal with the problems that arise while copying or moving directories with files open in those directories. There are some non-storage functions that are performed only by filesystems, but not object storage. For example, suid bits. It's also much more common to use object stores for larger chunks of data s.a. whole disk snapshots, VM images etc. While filesystems aim for the middle-size (small being RDBMs) s.a. text files you'd open in a text editor. Subsequently, they are optimized for these objectives. Filesystems care a lot about what happens when random small incremental and possibly overlapping updates happen to the same file, while object stores care about performance of sequential reads and writes the most. This excludes the notion of "distributed" as both can be distributed (and in different ways). I suppose you meant to ask about the difference between "distributed object storage" and "distributed filesystem".
- deleted 2y ago[deleted]
- surfingdino 2y agoThere's also OpenStack Swift.
- giulivo 2y agoI believe OpenStack Swift in particular is known to work well in some large organizations [1], NVIDIA is one of those and also invested in its maintenance [2]. 1. https://www.youtube.com/watch?v=H1DunJM1zoc https://www.youtube.com/watch?v=H1DunJM1zoc 2. https://platform.swiftstack.com/docs/ https://platform.swiftstack.com/docs/
- Daviey 2y agoLast time I looked at Garage it only supported paired storage replication, such that if I had a 10GB disk in location A and a 1TB disk is location 2 and 3, it would only support "RAID1-esq" mirroring, so my storage would be limited to 10GB
- leansensei 2y agoThat's a deliberate design decision.
- icy 2y agoI've been running this on K3s at home (for my website and file server) and it's been very well behaved: https://git.icyphox.sh/infra/tree/master/apps/garage https://git.icyphox.sh/infra/tree/master/apps/garage I find it interesting that they chose CRDTs over Raft for distributed consensus.
- iscoelho 2y agofrom an operations point of view, I am surprised anyone likes Raft. I have yet to see any application implement Raft in a way that does not spectacularly fail in production and require manual intervention to resolve. CRDTs do not have the same failure scenarios and favor uptime over consistency.
- j-pb 2y agoWhat I'm really missing in this space is something like this for content addressed blob storage. I feel like a lot of complexity and performance overhead could be reduced if you only store immutable blobs under their hash (e.g Blake3). Combined with a soft delete this would make all operations idempotent, blobs trivially cacheable, and all state a CRDT/monotonically mergeable/coordination free. There is stuff like IPFS in the large, but I want this for local deployments as a S3 replacement, when the metadata is stored elsewhere like git or a database.
- the_duke 2y agoGarage splis the data into chunks for deduplication, so it basically already does content addressed storage under the hood.. They probably don't expose it publicly though.
- j-pb 2y agoYeah, and as far as I understood they use the key hash to address the overall object descriptor. So in theory using the hash of the file instead of the hash of the key should be a simple-ish change. Tbh I'm not sure if content aware chunking isn't a sirens call: - It sounds great on paper, but once you start storing encrypted (which you have to do if you want e2e encryption) or compressed blobs (e.g. images) it won't work anymore. - Ideally you would store things with enough fine grained blobs that blob-level deduplication would suffice. - Storing a blob across your cluster has additional compute, lookup, bookkeeping, and communication overhead, resulting in worse latency. Storing an object as a contiguous unit makes the cache/storage hierarchies happy and allows for optimisations like using `sendfile`. - Storing the blobs as a unit makes computational storage easier to implement, where instead of reading the blob and processing it, you would send a small WASM program to the storage server (or drive? https://semiconductor.samsung.com/us/ssd/smart-ssd/) and only receive the computation result back.
- singinwhale 2y agoSounds a little like Kademlia, the DHT implementation that BitTorrent uses. It's a distributed hash table where the value mapped to a hash is immutable after it is STOREd (at least in the implementations that I know)
- neon_me 2y agoWhats the motivation behind project like this one? We got ceph, minio, seaweedfs ... and a dozen of others. I am genuinly curious what is the goal here?
- koito17 2y agoMinio assumes each node has identical hardware. Garage is designed for use-cases like self-hosting, where nodes are not expected to have identical hardware.
- otabdeveloper4 2y agoMinio doesn't, it has bucket replication and it works okay.
- WhereIsTheTruth 2y agoperformance, therefore cheaper
- iscoelho 2y agonot just about cost! improved performance/latency can make workloads that previously required a local SSD/NVME to be actually able run to run on distributed storage or an object store. it can not be understated how slow Ceph/Minio/etc can be compared to local NVME. there is plenty of room for improvement.
- rakoo 2y agoI can only answer for Garage and not others. Garage is the result of the desired organization of the collective behind it: deuxfleurs. The model is that of people willing to establish a horizontal governance, with none being forced to do anything because it all works by consensus. The idea is to have an infrastructure serving the collective, not a self hosted thing that everyone has to maintain, not something in a data center because it has clear ecological impacts, but something in-between. Something that can be hosted on secon-hand machines, at home, but taking the low reliability of machines/electricity/residential internet into account. Some kind of cluster, but not the kind you find in the cloud where machines are supposed to be kind of always on, linked with high-bandwidth, low-latency network: quite the opposite actually. deuxfleurs thought long and hard about the kind of infra this would translate to. The base came fast enough: some kind of storage, based on a standard (even de-facto only is good because it means it is proven), that would tolerate some nodes go down. The decision of doing a Dynamo-like thing to be accessed through S3 with eventual consistency made sense So Garage is not "simply" a S3 storage system: it is a system to store blobs in an unreliable but still trusted coonsumer-grade network of passable machines.
- makkesk8 2y agoWe moved over to garage after running minio in production with about ~2PB after about 2 years of headache. Minio does not deal with small files very well, rightfully so, since they don't keep a separate index of the files other than straight on disk. While ssd's can mask this issue to some extent, spinning rust, not so much. And speaking of replication, this just works... Minio's approach even with synchronous mode turned on, tends to fall behind, and again small files will pretty much break it all together. We saw about 20-30x performance gain overall after moving to garage for our specific use case.
- sandGorgon 2y agoquick question for advice - we have been evaluating minio for a in-house deployed storage for ML data. this is financial data which we have to comply on a crap ton of regulations. so we wanted lots of compliance features - like access logs, access approvals, short lived (time bound) accesses, etc etc. how would you compare garage vs minio on that front ?
- withinboredom 2y agoYou will probably put a proxy in front of it, so do your audit logging there (nginx ingress mirror mode works pretty good for that)
- mdaniel 2y agoAs a competing theory, since both Minio and Garage are open source, if it were my stack I'd patch them to log with the granularity one wished since in my mental model the system of record will always have more information than a simple HTTP proxy in front of them Plus, in the spirit of open source, it's very likely that if one person has this need then others have this need, too, and thus the whole ecosystem grows versus everyone having one more point of failure in the HTTP traversal
- withinboredom 2y agoHmm... maybe??? If you have a central audit log, what is the probability that whatever gets implemented in all the open (and closed) source projects will be compatible?
- sunshine-o 2y agoI really appreciate the low memory usage of Garage compared to Minio. The only thing I am missing is the ability to automatically replicate some buckets on AWS S3 for backup.
- anonzzzies 2y agoNLNet sponsored a lot of nice things.
- lifty 2y agoThe EU, but yeah. NLNet are the ones that judged the applications and disbursed the funds.
- tgjk 2y ago[flagged]
- MoodyMoon 2y agoApache Ozone is an alternative for an object store running on top of Hadoop. Maybe someone who has experience running this in a production environment can comment on it. https://ozone.apache.org/ https://ozone.apache.org/
- seaghost 2y agoI want something very simple to run locally that has s3 compatibility just for the dev work and testing. Any recommendations?
- zmj 2y agohttps://hub.docker.com/r/localstack/localstack https://hub.docker.com/r/localstack/localstack
- rlonstein 2y agohttps://min.io/ https://min.io/
- zX41ZdbW 2y agoMinio is fairly easy to setup locally or in CI. We use it for CI in ClickHouse, for example: https://github.com/ClickHouse/ClickHouse/blob/master/docker/test/stateless/setup_minio.sh https://github.com/ClickHouse/ClickHouse/blob/master/docker/...
- storagenerd 2y agoCheck out this one - https://github.com/NVIDIA/aistore https://github.com/NVIDIA/aistore https://aiatscale.org/ https://aiatscale.org/ It is an object storage system and more..
- arcanemachiner 2y agoGitHub mirror: https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage
- moffkalast 2y agoFinally one can launch startups from their own Garage again.
- thecleaner 2y agoIst this formally verified by any chance ? I feel like there's space where formal designs could be expressed in TLA+ such that its easier for the community to keep track of the design.
- halfa 2y agoThere is formal proof for some parts of garage layout system, see https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/doc/optimal_layout_report/optimal_layout.pdf https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/...
- dtag00 2y agoA bit of an off-topic question: I would like to programmatically generate S3 credentials that allow only read access or r/w access to only a certain set of prefixes. Imagine something like "Dropbox": You have a set of users, each user has his own prefix, but also users want to be able to share certain prefixes with other users. (Users are managed externally in a Postgres DB - MinIO does currently not know about them). I found this really difficult to achieve with MinIO, since this appears to require an AssumeRole request, which is almost not documented in any way and I did not find a Typescript example. Additionally, there's a weird set of restrictions in place for MinIO (and also AWS) that makes this really difficult to do, e.g. the size of policies is limited, which effectively limits the number of prefixes a user can share. I found this really difficult to work around. Can anyone suggest a way to do this? Can garage do this? Am I just approaching this from the wrong side? Thanks
- ajbfbasvbv 2y ago[flagged]
- ajbfbasvbv 2y ago[flagged]