17 ms·

Garage, our self-hosted distributed object storage solution

409 points by lxpz 5y ago

skratlo 5y ago
What's with the obsession with S3 API? Why not FUSE and just use any programming language without libraries to access the files? (Obviously without some S3/FUSE layer, without HTTP, just direct access to garage with the fastest protocol possible)
- skybrian 5y ago
  S3 is simpler because you don't have to deal with hierarchical directories.
leathersoft 5y ago
I swear to god, I thought this was a parody on yet another product with fancy names for just a physical garage
- adrn10 5y ago
  This is an actual object store - we just have a talent for naming artifacts
adrn10 5y ago
So basically, we are a hosting association that wanted to put their servers at home _and_ sleep at night. This demanded inter-home redundancy of our data. But none of the existing solutions (MinIO, Ceph...) are designed for high inter-node latency. Hence Garage! Your cheap and proficient object store: designed to run on a toaster, through carrier-pigeon-grade networks, while still supporting a tons of workloads (static websites, backups, Nextcloud... you name it)
- mro_name 5y ago
  I love that attitude towards self-hosting and it not being the sole hobby from that day on. Casual self-hosting if you will. House-keeping UX is key for self-hosting by laypersons.
- ddorian43 5y ago
  Did you try seaweedfs? Or contributing to any of them?
  - adrn10 5y ago
    Basically, our association is trying to move away from file systems, for scalability and availability purposes. There's a similarity between RDBMS (say SQL) and filesystems, in the sense that they distribute like crap. Main reason being that they have too strong consistency properties. Although, one can use Garage as a file system, notably using WebDAV. It's a beta feature, called Bagage: https://git.deuxfleurs.fr/Deuxfleurs/bagage https://git.deuxfleurs.fr/Deuxfleurs/bagage I'll let my knowledgeable fellows answer, if you'd like to hear about differences between seaweedfs and Garage in more detail :)
    - ddorian43 5y ago
      seaweedfs is an object storage at the core. In my (personal) case, I disqualify cause of missing erasure coding.
      - Already__Taken 5y ago
        Maybe they added it since you looked or did something not match your needs? https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-for-warm-storage https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-f...
        ddorian43 5y ago
        I disqualify Garage because of missing erasure-coding. I know seaweedfs has it.
      - StillBored 5y ago
        Garage says isn't a goal either. https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/ "Storage optimizations: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication."
- RobLach 5y ago
  I've been researching a solution for this problem for some time and you all have landed on exactly how I thought this should be architected. Excellent work and I foresee much of the future infrastructure of the internet headed in this direction.
lxpz 5y ago
For people interested in digging in, we have an interesting benchmark here showing how it significantly outperforms Minio in the geo-distributed setting: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/ https://garagehq.deuxfleurs.fr/documentation/design/benchmar... We also made a presentation at FOSDEM'22, the video should be out soon.
deleted 5y ago
[deleted]
Loic 5y ago
Interesting point to note is that they have some external funding from the EU. > The Deuxfleurs association has received a grant from NGI POINTER[0], to fund 3 people working on Garage full-time for a year : from October 2021 to September 2022. [0]: https://pointer.ngi.eu/ https://pointer.ngi.eu/
- sofixa 5y ago
  That's really great, i didn't know this programme existed.
  - adrn10 5y ago
    (Deuxfleurs administrative council member here:) Starting from a little association with 200€ in a cash box last year, we were indeed amazed to actually get this funding. I strongly recommend NGI grants to any European citizen. Just look at the variety of profiles that NGI POINTER (one of the grant type) funded last year: https://pointer.ngi.eu/wp-content/uploads/2021/10/NGI-POINTER-Architects-Booklet-2021.pdf https://pointer.ngi.eu/wp-content/uploads/2021/10/NGI-POINTE... They even finance individuals wanting to contribute to FOSS up to 50k€ for a year.
superboum 5y ago
I tried to answer to a comment that was deleted, probably due to the form. Instead of deleting my answer, I want to share the reworded critics and my answers. > Why not using Riak and adding an S3 API around it. Riak was developed by a company named Basho that went bankrupt some years ago, the software is not developed anymore. In fact, we do no need to add an S3 API around Riak KV, Basho even released "Riak Cloud Storage"[0] that exactly does this: provide an S3 API on top of Riak KV architecture. We plan to release a comparison between Garage and Riak CS, Garage has some interesting features that Riak CS does not have! In practice, implementing an object store on top of a DynamoDB-like KV store is not that straightforward. For example, Exoscale, a cloud provider went this way for their first implementation of their KV store, Pithos[1], but rewrote it later as you need special logic to handle your chunks (they did not publish Pithos v2). > Most apps don't have S3 support We are maintaining in our documentation an "integration" section listing all the compatible applications. Garage already works with Matrix, Mastodon, Peertube, Nextcloud, Restic (an alternative to Borg), Hugo and Publii (a static site generator with a GUI). These applications are only a fraction of all existing applications, but our software is targeted at its users/hosters. > A distributed system is not necessarily highly available I will not fight on the wording: we come from an academic background where the term "distributed computing" has a specific meaning that may differ outside. In our field, we define models where we study systems made of processes that can crash. Depending on your algorithms and the properties you want, you can prove that your system will work despite some crashes. We want to build software on these academic foundations. This is also the reason we put "Standing on the shoulders of giants" on our front page and linking to research papers. To put it in a nutshell, one critic we address to other software is that sometimes they lack theoretical/academic foundations that lead to unexpected failures/more work to sysadmins. But on the theoretical point, Basho and Riak were exemplary and a model for us! [0]: https://docs.riak.com/riak/cs/2.1.1/index.html https://docs.riak.com/riak/cs/2.1.1/index.html [1]: https://github.com/exoscale/pithos https://github.com/exoscale/pithos
- lxpz 5y ago
  I also had some things to answer to that comment which I'll put back here for the record. > I have been building distributed systems for 20 years, and they are not more reliable. They are probabilistically more likely to fail. It depends on what kind of faults you want to protect against. In our case, we are hosting servers at home, meaning that any one of them could be disconnected at any time due to a power outage, a fiber being cut off, or any number of reasons. We are also running old hardware where individual machines are more likely to fail. We also do not run clusters composed of very large numbers of machines, meaning that the number of simultaneous failures that can be expected actually remains quite low. This means that the choices made by Garage's architecture make sense for us. But maybe your point was about the distinction between distributed systems and high availability, in which case I agree. Several of us have studied distributed systems in the academic setting, and in our vocabulary, distributed systems almost by definition includes crash-tolerance and thus making systems HA. I understand that in the engineering community the vocabulary might be different and we might orient our communication more towards presenting Garage as HA thanks to your insight, as it is one of our core, defining features. > However, this isn't it. This is distributed S3 with CRDTs. Still too application-specific, because every app that wants to use it has to be integrated with S3. They could have just downloaded something like Riak and added an S3 API around it. Garage is almost that, except that we didn't download Riak but made our own CRDT-based distributed storage system. It's actually not the most complex part at all, and most of the developpement time was spent on S3 compatibility. Rewriting the storage layer means that we have better integration between components, as everything is built in Rust and heavily depends on the type system to ensure things work well together. In the future, we plan to reuse the storage layer we built for Garage for other projects, in particular to build an e-mail storage server.
morelish 5y ago
Think the idea is a good one. As S3 becomes standard, it would be nice if someone could rip amazon's face-off (like amazon have done for other open-source companies, e.g. elastic).
- social_quotient 5y ago
  Elastic was a bit with cause?
  - morelish 5y ago
    How do you mean?
- malteg 5y ago
  what about https://min.io/ https://min.io/
  - morelish 5y ago
    Cheers, I'll have a look at that.
Rygu 5y ago
Github repo (mirror): https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage
detritus 5y ago
Y'all might want to get to the point a bit quicker in your pitch. A lot of fluff to wade through before the lede.. "Garage is a distributed storage solution, that automatically replicates your data on several servers. Garage takes into account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems. Garage implements the Amazon S3 protocol, a de-facto standard that makes it compatible with a large variety of existing software. For instance it can be used as a storage back-end for many self-hosted web applications such as NextCloud, Matrix, Mastodon, Peertube, and many others, replacing the local file system of a server by a distributed storage layer. Garage can also be used to synchronize your files or store your backups with utilities such as Rclone or Restic. Last but not least, Garage can be used to host static websites, such as the one you are currently reading, which is served directly by the Garage cluster we host at Deuxfleurs."
- lxpz 5y ago
  Thanks for the feedback! We will definitely be adjusting our pitch in further iterations.
  - JKCalhoun 5y ago
    I'm the layest of laymen, is Garage "a distributed Dropbox"?
    - adrn10 5y ago
      It's the layer below: a sound distributed storage backend. You can use it to distribute your Nextcloud data, store your backups, host websites... You can even mount it in your OS' file manager and use it as an external drive!
  - erwincoumans 5y ago
    Who owns the servers?
    - 5y ago
iam-TJ 5y ago
Your git repository is not reachable over IPv6 despite publishing a DNS AAAA record: $ curl -v https://git.deuxfleurs.fr/Deuxfleurs/garage/issues * Trying 2001:41d0:8:ba0b::1:443... * Trying 5.135.179.11:443... * Immediate connect fail for 5.135.179.11: Network is unreachable $ dig +noquestion +nocmd aaaa git.deuxfleurs.fr ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46366 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; ANSWER SECTION: git.deuxfleurs.fr. 6822 IN CNAME hammerhead.machine.deuxfleurs.fr. hammerhead.machine.deuxfleurs.fr. 6822 IN AAAA 2001:41d0:8:ba0b::1 This from an IPv6-only network. Since you publish an IPv6 record our DNS64/NAT64 gateway doesn't get involved hence the IPv4 immediate connect fail.
- superboum 5y ago
  Thanks for reporting the problem. I just deleted the AAAA entry for this machine. In the meantime, if the result is cached for you, you can pass the `-4` argument to force IPv4: git clone -4 git@git.deuxfleurs.fr:Deuxfleurs/garage.git And, in a second time, we will work on a better/working IPv6 configuration for our Git repository and all of our services (we use Nomad+Docker and did not find a way to expose IPv6 in a satisfying way yet).
  - iam-TJ 5y ago
    Looks like the TTL on the records is still affecting local resolution. I'm aware of the git protocol work-around to force IPv4 but that doesn't help when using a regular web-browser to visit the published URL for issues (first thing I evaluate for a new-to-me project): https://git.deuxfleurs.fr/Deuxfleurs/garage/issues https://git.deuxfleurs.fr/Deuxfleurs/garage/issues
- adrn10 5y ago
  My bad T_T My colleagues keep picking on me for this. Now that you, complete stranger, you see it too, I'm feeling obliged to improve the situation!
k8sToGo 5y ago
Do you guys expose Prometheus metrics or how is monitoring supposed to be done (e.g. on replication status)?
- superboum 5y ago
  Currently, we have no elegant way to achieve what you want. When failures occur, repair is done through workers that says when they launch, when they repair chunks, and when they exit in the logs. We also have `garage status` and `garage stats`. The first command displays healthy and non healthy nodes, the second one displays the queue length of our tables and chunks, if their values are greater than zero, we are repairing the cluster. We are documenting failure recovery in our documentation: https://garagehq.deuxfleurs.fr/documentation/cookbook/recovering/ https://garagehq.deuxfleurs.fr/documentation/cookbook/recove... For the near future, we plan to integrate opentelemetry. But we are still discussing the design and information we want to track and report. We are currently discussing these questions in our issue tracker: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/111 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/111 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/207 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/207 If you have some knowledge/experience on this subject, feel free to share it in these issues.
- lxpz 5y ago
  Exposing Prometheus metrics is an ongoing work which we haven't had much time to advance yet. For now we can check on replication progress or overall cluster health by inspecting Garage's state from the command line.
malteg 5y ago
what is the benefit compared to https://min.io/ https://min.io/
- rhim 5y ago
  This is written in Rust!
- adrn10 5y ago
  We made a comparison here: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/ https://garagehq.deuxfleurs.fr/documentation/design/benchmar... Other comments from my colleagues also shed light on Garage's specific features (do check the discussion on Raft). In a nutshell, Garage is designed for higher inter-node latency, thanks to less round-trips for reads & writes. Garage does not intend to compete with MinIO, though - rather, to expand the application domain of object stores.
- superboum 5y ago
  Another benefit compared to MinIO is we have "flexible topologies". Due to our design choice, you can add and remove nodes without any constraint on number of nodes and size of the storage. So you do not have to overprovision your cluster as recommended by MinIO[0]. Additionally, and we planned a full blog post on this subject, adding or removing a node in the cluster does not lead to a full rebalance of the cluster. To understand why, I must explain how it works traditionally and how we improved on existing work. When you initialize the cluster, we split the cluster in partitions, then assign partitions to nodes (see Maglev[1]). Later, based on their hash, we will store data in its corresponding partition. When a node is added or removed, traditional approaches rerun the whole algorithm and comes with a totally different partition assignation. Instead, we try to compute a new partition distribution that minimize partitions assignment change, which in the end minimize the number of partitions moved. On the drawback side, Garage does not implement erasure coding (as it also the reason of many MinIO's limitations) and duplicate data 3 times which is less efficient. Garage also implements less S3 endpoints than Minio (for example we do not support versioning), the full list is available in our documentation[2]. [0]: https://docs.min.io/minio/baremetal/installation/deploy-minio-distributed.html#capacity-based-planning https://docs.min.io/minio/baremetal/installation/deploy-mini... [1]: https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud https://www.usenix.org/conference/nsdi16/technical-sessions/... [2]: https://garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility/ https://garagehq.deuxfleurs.fr/documentation/reference-manua...
prmoustache 5y ago
I understand the founders are Terry Pratchett / Discworld fans from the name of the associations. For non french speakers, Deuxfleurs is the french traduction of the Twoflower character appearing in The Colour of Magic, The Light Fantastic and Interesting Times from Terry Pratchett's Discworld books series.
- adrn10 5y ago
  Busted! We might even have a Bagage[0] side-project... [0] https://git.deuxfleurs.fr/Deuxfleurs/bagage https://git.deuxfleurs.fr/Deuxfleurs/bagage
merb 5y ago
this might be a stupid comment. but is this safe to use inside a commercial project as a backend s3 solution? i.e. just using it instead of lcoal storage and not changing it?
- superboum 5y ago
  Our software is published under the AGPLv3 license and comes with no guarantee, like any other FOSS project (if you do not pay for support). We are considering our software as "public beta" quality, so we think it works well, at least for us. On the plus side, it survived Hacker News Hug of Death. Indeed, the website we linked is hosted on our own Garage cluster made of old Lenovo ThinkCentre M83 (with Intel Pentium G3420 and 8GB of RAM) and the cluster seems fine. We also host more than 100k objects in our Matrix (a chat service) bucket. On the minus side, this is the first time we have so much coverage, so our software has not yet been tested by thousands of people. It is possible that in the near future, some edge cases we never triggered are reported. This is the reason why most people wait that an application reaches a certain level of adoption before using it, in other words they don't want to pay "the early adopter cost". In the end, it's up to you :-)
  - cvwright 5y ago
    Awesome. Are you using the matrix-media-repo to connect the Matrix homeserver to Garage?
    - lxpz 5y ago
      No, we are simply using synapse-s3-storage-provider: https://github.com/matrix-org/synapse-s3-storage-provider https://github.com/matrix-org/synapse-s3-storage-provider
- deleted 5y ago
  [deleted]
- lxpz 5y ago
  License-wise, Garage is AGPLv3 like many similar software projects such as MinIO and MongoDB. We are not lawyers, but in the spirit, this should not preclude from using it in commercial software as long as it is not modified. In the case where you are modifying it, like e.g. adding a nice administration GUI, we kindly ask that you publish your modifications as open source so that we can incorporate them back in the main product if we want.
southerntofu 5y ago
Hello, quick question: from what i understand since it's based on CRDTs it means any server in the cluster may save something to the storage, that gets replicated elsewhere. That's fine when there is absolute trust between the server operators, but am i correct to assume it's not the same threat model as encrypted/signed backups pushed to some friends lending storage space for you (who could corrupt your backup but can't corrupt your working data)? If my understanding is correct, maybe your homepage should outline the threat model more clearly (i.e. a single trusted operator for the whole cluster) and point to other solutions like TAHOE-LAFS for other use-cases. Congratulations on doing selfhosting with friends, that's pretty cool! Do you have any idea if some hosting cooperatives (CHATONS) or ISPs (FFDN) have practical use-cases in mind? They already have many physical locations and rather good bandwidth, but i personally can't think of an interesting idea.
- ClumsyPilot 5y ago
  I was thiniing StorJ and fulecoin are the ones attempting to provide storage with a 'tryst noone' model
  - southerntofu 5y ago
    "Trust noone" cannot work. Humanity is built on trust. The question is what level of trust you place on different operators. From a quick look StorJ and fuelcoin both appear to be classic crypto-scams where the entire tech stack is controlled by a single company... talk about "zero trust" :-) If you're really interested about low trust tech for file storage, i recommend you check out TAHOE-LAFS. It's pretty cool tech and there's no money to be made out of it, nor is there a company dictating who can join what pool and how much you have to pay for what kind of storage. It's an actually decentralized system, unlike most blockchains.
- lxpz 5y ago
  You are absolutely correct concerning the trust model of Garage: any administrator of one node is an administrator of the entire cluster, and can manipulate data arbitrarily in the system. From an ideological perspective, we are strongly attached to the building of tight-knit communities in which strong trust bonds can emerge, as it gives us more meaning than living in an individualized society where all exchanges between individuals are mediated by a market, or worse, by blockchain technology. This means that trusting several system's administrator makes sense to us. Note that in the case of cooperatives such as CHATONS, most users are non-technical and have to trust their sysadmin anyways; here, they just have to trust several sysadmins instead of just one. We know that several hosting cooperatives of the CHATONS network are thinking like us and are interested in setting up systems such as Garage that work under this assumption. In the meantime, we also do perfectly recognize the possibility of a variety of attack scenarios against which we want to consider practical defenses, such as the following: 1/ An honest-but-curious system administrator or an intruder in the network that wants to read user's private data, or equivalently, a police raid where server software is embarked for inspection by state services; 2/ A malicious administrator or an intruder that wants to manipulate the users by introducing fake data; 3/ A malicious administrator or an intruder that simply wants to wreak havoc by deleting everything. Point 1 is the biggest risk in my eyes. Several solutions can be built to add an encryption layer over S3 for different usage scenarios. For instance for storing personnal files, Rclone can be used to add simple file encryption to an S3 bucket and can also be mounted directly via FUSE, allowing us to access Garage as an end-to-end encrypted network drive. For backups, programs like Restic and Borg allow us to upload our backups encrypted into Garage. Point 2 can be at least partially solved by adding signatures and verification in the encryption layer which is handled on the client (I don't know for sure if Rclone, Restic and Borg are doing this). Point 3 is harder and probably requires adaptation on the side of Garage to be solved, for instance by adding a restriction on which nodes are allowed to propagate updates in the network and thus establishing a hierarchy between two categories of nodes: those that implement the S3 gateway and thus have full power in the network, and those that are only responsible for storing data given by the gateway but cannot originate modifications themselves (under the assumption that modifications must be accompanied by a digital signature and that only gateway nodes have the private keys to generate such signatures). Then we could separate gateway nodes according to the different buckets in which they can write for better separation of concernts. However, as long as we are trying to implement the S3 protocol I think we are stuck with some imperfect solution like this, because S3 itself does not implement using public-key cryptography to attest operations sent by the users. It is perfectly true that solutions that are explicitely designed to handle these threats (such as TAHOE-LAFS) would provide better guarantees in a system that is maybe more consistent as a whole. However we are also trying to juggle these security constraints with deployment contraints such as keeping compatibility with standard protocols such as S3 (and soon also IMAP for mailbox storage), which restricts us in the design choices we make. Just to clarify, the fact that any node can tamper with any of the data in the cluster is not strictly linked to the fact that we use CRDTs internally. It is true that CRDTs make it more difficult, but we do believe that there are solutions to this and we intend to implement at least some of them in our next project that involves mailbox storage.
Fiahil 5y ago
NICE ! Very Nice ! Congrats on the project and the naming ! If I may add a few thoughts : You may be very well placed for being a more suitable replacement to Minio in Data science / AI projects. Why ? Because a hard requirement for any serious MLOps construction needs 2 things : An append-only storage and a way to stream large content to a single place. First one is very hard to get right, second is relatively easy. Being CRDT-based it should be _very easy_ for you to provide an append-only storage that can store partitioned, ordered logs of immutable objects (think dataframes, and think kafka). Once you have that, it's really easy to build the remaining missing pieces (UI, API, ...) for creating a _much better_ (and distributed) version of MLFlow. Finally, S3 protocol is an "okay" version for file storage, but as you are probably aware, it's clearly a huge limiter. So, trash it. Provide a read-only S3 gateway for compatibility, but writes should use a different API. PS: Galette-Saucisse <3
- adrn10 5y ago
  Thanks for your enthusiastic feedback, fellow breton! It's an interesting take you make about ML workloads. We haven't investigated that yet (next in line is e-mail: you see we target ubiquitous, low-tech needs that are horrendous to host). But we will definitely consider it for future works: small breton food trucks do need their ML solutions for better galettes.
  - Fiahil 5y ago
    If you ever need further insights into the ML/AI use case, don't hesitate to ping me at {my-username}-at-gmail. I'll forward it to my work address :)
    - adrn10 5y ago
      And you are cordially invited to our chatroom on Matrix, if you want to hack around Garage: https://matrix.to/#/#garage:deuxfleurs.fr https://matrix.to/#/#garage:deuxfleurs.fr We used to mostly talk French there - until yesterday!
bamazizi 5y ago
Github mirror? It's extremely easy to forget and stop following up on a project that's not on Github! You might have personal reasons by your current choice, but maybe the project can have shot at success if it's in a more social and high traffic ecosystem.
- lxpz 5y ago
  Exists, already linked in another comment : https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage
pnocera 5y ago
I'd see a nice fit with Tailscale. Or how to create a secured private network of redundant and HA storage...
- adrn10 5y ago
  We haven't looked too much into VPNs, although we do need some plumbing to distribute our whole infrastructure at Deuxfleurs.
mediocregopher 5y ago
I've been looking through the available options for this use-case, with the same motivation given in this post, and was surprised to see there wasn't something which could fit this niche (until now!). Really happy to see this being worked on by a dedicated team. Combined with a tool like https://github.com/slackhq/nebula https://github.com/slackhq/nebula this could help form the foundation of a fully autonomous, durable, online community.
belfalas 5y ago
Looks very worth checking out! Will this be made available for installation via Cloudron some time in the future?
razzio 5y ago
This is a fantastic project and I can't wait to try it out! One question on the network requirements. The web page says for networking: "200 ms or less, 50 Mbps or more". How hard are these requirements? For folks like me that can't afford a guaranteed 50Mbps internet connection, is this still usable? There are plenty of places in the world where 50Mbps internet connectivity would be a dream. Even here in Canada there are plenty of places with a max of 10Mbps. The African continent for example will have many more.
- adrn10 5y ago
  This depends on your workload. It would work, but we first target fiber optics domestic connections.
- lxpz 5y ago
  It's not a hard requirement, you might just have a harder time handling voluminous files as Garage nodes will in all case have to transfer data internally in the cluster (remember that files have to be sent to three nodes when they are stored in Garage, meaning 3x more bandwidth needs to be used). 10Mbps is already pretty good if it is stable and your ping isn't off the charts, and it might be totally workable depending on your use case.
latchkey 5y ago
I see zero unit/integration tests in the source tree, not even a `test` target in the Makefile. Terrifying.
- lxpz 5y ago
  Then you didn't look close enough. Have a look at the .drone.yml to see the entry points, we have both unit tests and integration tests that run at each build.
lloydatkinson 5y ago
> account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems. Azure already does this, so claiming it's unique seems untrue.
- lxpz 5y ago
  I mean, sure, but we're operating under the premise that we don't want to give out our data to a cloud provider and instead want to store data ourselves. Half of the post is dedicated to explaining that, so pretending that you can't infer that from context seems a bit unfair.
discardedrefuse 5y ago
Hi this is a very cool project, and I love that it's geared towards self-hosting. I'm about inherit some hardware so I'm actually looking at clustered network storage options right now. In my setup, I don't care about redundancy. I much prefer to maximize storage capacity. Can Garage be configured without replication? If not, maybe someone else can point me in the right direction. Something like a glusterfs distributed volume. At first glance, SeaweedFS has a "no replication" option, but their docs make it seem like they're geared towards handling billions of tiny files. I have about 24TB of 1Gb to 20GB files.
- lxpz 5y ago
  You probably want at least some redundancy otherwise you're at risk of losing data as soon as a single one of your hard drive fails. However if you're interested in maximizing capacity, you probably need erasure coding, which Garage does not provide. Gluster, Ceph and Minio might all be good candidates for your use case. Or if you have the possibility of putting all your drives in a single box, just do that and make a ZFS pool.
  - discardedrefuse 5y ago
    > You probably want at least some redundancy otherwise you're at risk of losing data as soon as a single one of your hard drive fails. If by "fails" you mean the network connection drops out. Then yes, that would be a huge problem. I was hoping some project had a built-in solution to this. Currently, I'm using MergerFS to effectively create 1 disk out of 3 external USB drives and it handles accidental drive disconnects with no problems (I can't gush enough over how great mergerfs is). But, if by "fails" you mean actual hardware failure. Then, I don't really care. I keep 1 to 1 backups. A few days of downtime to restore the data isn't a big deal; this is just my home network. > Or if you have the possibility of putting all your drives in a single box... Unfortunately, I've maxed out the drive bays on my TS140 server. Buying new, larger drives to replace existing drives seems wasteful. Also, I've just been gifted another TS140, which is a good platform to start building another file server. You've given me something to think about, thanks. I appreciate you taking the time to respond!
- 5y ago
toolslive 5y ago
Some remarks: - replication is way too expensive for what it's worth. As soon as you can afford it you really should go for some kind of erasure coding. The exact moment when erasure coding is more efficient (storage wise, transport wise) and more robust (in terms of devices you can lose) depends on the size of the objects (fixed cost of metadata). - you don't need distributed consistency for the data storage actions just for the meta data (the map that tells you where the data is). You can start pushing data, and if the metadata for the object metadata never is created, you just pushed garbage which can be collected.
kalstone 5y ago
Does Garage integrate with Kubernetes?
rad_gruchalski 5y ago
Hey, looks pretty interesting but I have a couple of questions: 1) the license of Garage is AGPL, MinIO is also AGPL so I'm not sure what's the problem with it as a valid choice for self-hosting? You seem to not have a problem using Apache 2 licensed software (HashiCorp stack). 2) if you're a non-profit, how come MinIO is your "closest competitor"?

409 points by lxpz 5y ago

What's with the obsession with S3 API? Why not FUSE and just use any programming language without libraries to access the files? (Obviously without some S3/FUSE layer, without HTTP, just direct access to garage with the fastest protocol possible)

skybrian 5y ago
S3 is simpler because you don't have to deal with hierarchical directories.

leathersoft 5y ago

I swear to god, I thought this was a parody on yet another product with fancy names for just a physical garage

adrn10 5y ago
This is an actual object store - we just have a talent for naming artifacts

adrn10 5y ago

So basically, we are a hosting association that wanted to put their servers at home _and_ sleep at night. This demanded inter-home redundancy of our data. But none of the existing solutions (MinIO, Ceph...) are designed for high inter-node latency. Hence Garage! Your cheap and proficient object store: designed to run on a toaster, through carrier-pigeon-grade networks, while still supporting a tons of workloads (static websites, backups, Nextcloud... you name it)

mro_name 5y ago
I love that attitude towards self-hosting and it not being the sole hobby from that day on. Casual self-hosting if you will. House-keeping UX is key for self-hosting by laypersons.
ddorian43 5y ago
Did you try seaweedfs? Or contributing to any of them?
- adrn10 5y ago
  Basically, our association is trying to move away from file systems, for scalability and availability purposes. There's a similarity between RDBMS (say SQL) and filesystems, in the sense that they distribute like crap. Main reason being that they have too strong consistency properties. Although, one can use Garage as a file system, notably using WebDAV. It's a beta feature, called Bagage: https://git.deuxfleurs.fr/Deuxfleurs/bagage https://git.deuxfleurs.fr/Deuxfleurs/bagage I'll let my knowledgeable fellows answer, if you'd like to hear about differences between seaweedfs and Garage in more detail :)
  - ddorian43 5y ago
    seaweedfs is an object storage at the core. In my (personal) case, I disqualify cause of missing erasure coding.
    - Already__Taken 5y ago
      Maybe they added it since you looked or did something not match your needs? https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-for-warm-storage https://github.com/chrislusf/seaweedfs/wiki/Erasure-coding-f...
      - ddorian43 5y ago
        I disqualify Garage because of missing erasure-coding. I know seaweedfs has it.
    - StillBored 5y ago
      Garage says isn't a goal either. https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/ "Storage optimizations: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication."
RobLach 5y ago
I've been researching a solution for this problem for some time and you all have landed on exactly how I thought this should be architected. Excellent work and I foresee much of the future infrastructure of the internet headed in this direction.

lxpz 5y ago

For people interested in digging in, we have an interesting benchmark here showing how it significantly outperforms Minio in the geo-distributed setting: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/ https://garagehq.deuxfleurs.fr/documentation/design/benchmar... We also made a presentation at FOSDEM'22, the video should be out soon.

deleted 5y ago

[deleted]

Loic 5y ago

Interesting point to note is that they have some external funding from the EU. > The Deuxfleurs association has received a grant from NGI POINTER[0], to fund 3 people working on Garage full-time for a year : from October 2021 to September 2022. [0]: https://pointer.ngi.eu/ https://pointer.ngi.eu/

sofixa 5y ago
That's really great, i didn't know this programme existed.
- adrn10 5y ago
  (Deuxfleurs administrative council member here:) Starting from a little association with 200€ in a cash box last year, we were indeed amazed to actually get this funding. I strongly recommend NGI grants to any European citizen. Just look at the variety of profiles that NGI POINTER (one of the grant type) funded last year: https://pointer.ngi.eu/wp-content/uploads/2021/10/NGI-POINTER-Architects-Booklet-2021.pdf https://pointer.ngi.eu/wp-content/uploads/2021/10/NGI-POINTE... They even finance individuals wanting to contribute to FOSS up to 50k€ for a year.

superboum 5y ago

I tried to answer to a comment that was deleted, probably due to the form. Instead of deleting my answer, I want to share the reworded critics and my answers. > Why not using Riak and adding an S3 API around it. Riak was developed by a company named Basho that went bankrupt some years ago, the software is not developed anymore. In fact, we do no need to add an S3 API around Riak KV, Basho even released "Riak Cloud Storage"[0] that exactly does this: provide an S3 API on top of Riak KV architecture. We plan to release a comparison between Garage and Riak CS, Garage has some interesting features that Riak CS does not have! In practice, implementing an object store on top of a DynamoDB-like KV store is not that straightforward. For example, Exoscale, a cloud provider went this way for their first implementation of their KV store, Pithos[1], but rewrote it later as you need special logic to handle your chunks (they did not publish Pithos v2). > Most apps don't have S3 support We are maintaining in our documentation an "integration" section listing all the compatible applications. Garage already works with Matrix, Mastodon, Peertube, Nextcloud, Restic (an alternative to Borg), Hugo and Publii (a static site generator with a GUI). These applications are only a fraction of all existing applications, but our software is targeted at its users/hosters. > A distributed system is not necessarily highly available I will not fight on the wording: we come from an academic background where the term "distributed computing" has a specific meaning that may differ outside. In our field, we define models where we study systems made of processes that can crash. Depending on your algorithms and the properties you want, you can prove that your system will work despite some crashes. We want to build software on these academic foundations. This is also the reason we put "Standing on the shoulders of giants" on our front page and linking to research papers. To put it in a nutshell, one critic we address to other software is that sometimes they lack theoretical/academic foundations that lead to unexpected failures/more work to sysadmins. But on the theoretical point, Basho and Riak were exemplary and a model for us! [0]: https://docs.riak.com/riak/cs/2.1.1/index.html https://docs.riak.com/riak/cs/2.1.1/index.html [1]: https://github.com/exoscale/pithos https://github.com/exoscale/pithos

lxpz 5y ago
I also had some things to answer to that comment which I'll put back here for the record. > I have been building distributed systems for 20 years, and they are not more reliable. They are probabilistically more likely to fail. It depends on what kind of faults you want to protect against. In our case, we are hosting servers at home, meaning that any one of them could be disconnected at any time due to a power outage, a fiber being cut off, or any number of reasons. We are also running old hardware where individual machines are more likely to fail. We also do not run clusters composed of very large numbers of machines, meaning that the number of simultaneous failures that can be expected actually remains quite low. This means that the choices made by Garage's architecture make sense for us. But maybe your point was about the distinction between distributed systems and high availability, in which case I agree. Several of us have studied distributed systems in the academic setting, and in our vocabulary, distributed systems almost by definition includes crash-tolerance and thus making systems HA. I understand that in the engineering community the vocabulary might be different and we might orient our communication more towards presenting Garage as HA thanks to your insight, as it is one of our core, defining features. > However, this isn't it. This is distributed S3 with CRDTs. Still too application-specific, because every app that wants to use it has to be integrated with S3. They could have just downloaded something like Riak and added an S3 API around it. Garage is almost that, except that we didn't download Riak but made our own CRDT-based distributed storage system. It's actually not the most complex part at all, and most of the developpement time was spent on S3 compatibility. Rewriting the storage layer means that we have better integration between components, as everything is built in Rust and heavily depends on the type system to ensure things work well together. In the future, we plan to reuse the storage layer we built for Garage for other projects, in particular to build an e-mail storage server.

morelish 5y ago

Think the idea is a good one. As S3 becomes standard, it would be nice if someone could rip amazon's face-off (like amazon have done for other open-source companies, e.g. elastic).

social_quotient 5y ago
Elastic was a bit with cause?
- morelish 5y ago
  How do you mean?
malteg 5y ago
what about https://min.io/ https://min.io/
- morelish 5y ago
  Cheers, I'll have a look at that.

Rygu 5y ago

Github repo (mirror): https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage

detritus 5y ago

Y'all might want to get to the point a bit quicker in your pitch. A lot of fluff to wade through before the lede.. "Garage is a distributed storage solution, that automatically replicates your data on several servers. Garage takes into account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems. Garage implements the Amazon S3 protocol, a de-facto standard that makes it compatible with a large variety of existing software. For instance it can be used as a storage back-end for many self-hosted web applications such as NextCloud, Matrix, Mastodon, Peertube, and many others, replacing the local file system of a server by a distributed storage layer. Garage can also be used to synchronize your files or store your backups with utilities such as Rclone or Restic. Last but not least, Garage can be used to host static websites, such as the one you are currently reading, which is served directly by the Garage cluster we host at Deuxfleurs."

lxpz 5y ago
Thanks for the feedback! We will definitely be adjusting our pitch in further iterations.
- JKCalhoun 5y ago
  I'm the layest of laymen, is Garage "a distributed Dropbox"?
  - adrn10 5y ago
    It's the layer below: a sound distributed storage backend. You can use it to distribute your Nextcloud data, store your backups, host websites... You can even mount it in your OS' file manager and use it as an external drive!
- erwincoumans 5y ago
  Who owns the servers?
  - 5y ago

iam-TJ 5y ago

Your git repository is not reachable over IPv6 despite publishing a DNS AAAA record: $ curl -v https://git.deuxfleurs.fr/Deuxfleurs/garage/issues * Trying 2001:41d0:8:ba0b::1:443... * Trying 5.135.179.11:443... * Immediate connect fail for 5.135.179.11: Network is unreachable $ dig +noquestion +nocmd aaaa git.deuxfleurs.fr ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46366 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; ANSWER SECTION: git.deuxfleurs.fr. 6822 IN CNAME hammerhead.machine.deuxfleurs.fr. hammerhead.machine.deuxfleurs.fr. 6822 IN AAAA 2001:41d0:8:ba0b::1 This from an IPv6-only network. Since you publish an IPv6 record our DNS64/NAT64 gateway doesn't get involved hence the IPv4 immediate connect fail.

superboum 5y ago
Thanks for reporting the problem. I just deleted the AAAA entry for this machine. In the meantime, if the result is cached for you, you can pass the `-4` argument to force IPv4: git clone -4 git@git.deuxfleurs.fr:Deuxfleurs/garage.git And, in a second time, we will work on a better/working IPv6 configuration for our Git repository and all of our services (we use Nomad+Docker and did not find a way to expose IPv6 in a satisfying way yet).
- iam-TJ 5y ago
  Looks like the TTL on the records is still affecting local resolution. I'm aware of the git protocol work-around to force IPv4 but that doesn't help when using a regular web-browser to visit the published URL for issues (first thing I evaluate for a new-to-me project): https://git.deuxfleurs.fr/Deuxfleurs/garage/issues https://git.deuxfleurs.fr/Deuxfleurs/garage/issues
adrn10 5y ago
My bad T_T My colleagues keep picking on me for this. Now that you, complete stranger, you see it too, I'm feeling obliged to improve the situation!

k8sToGo 5y ago

Do you guys expose Prometheus metrics or how is monitoring supposed to be done (e.g. on replication status)?

superboum 5y ago
Currently, we have no elegant way to achieve what you want. When failures occur, repair is done through workers that says when they launch, when they repair chunks, and when they exit in the logs. We also have `garage status` and `garage stats`. The first command displays healthy and non healthy nodes, the second one displays the queue length of our tables and chunks, if their values are greater than zero, we are repairing the cluster. We are documenting failure recovery in our documentation: https://garagehq.deuxfleurs.fr/documentation/cookbook/recovering/ https://garagehq.deuxfleurs.fr/documentation/cookbook/recove... For the near future, we plan to integrate opentelemetry. But we are still discussing the design and information we want to track and report. We are currently discussing these questions in our issue tracker: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/111 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/111 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/207 https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/207 If you have some knowledge/experience on this subject, feel free to share it in these issues.
lxpz 5y ago
Exposing Prometheus metrics is an ongoing work which we haven't had much time to advance yet. For now we can check on replication progress or overall cluster health by inspecting Garage's state from the command line.

malteg 5y ago

what is the benefit compared to https://min.io/ https://min.io/

rhim 5y ago
This is written in Rust!
adrn10 5y ago
We made a comparison here: https://garagehq.deuxfleurs.fr/documentation/design/benchmarks/ https://garagehq.deuxfleurs.fr/documentation/design/benchmar... Other comments from my colleagues also shed light on Garage's specific features (do check the discussion on Raft). In a nutshell, Garage is designed for higher inter-node latency, thanks to less round-trips for reads & writes. Garage does not intend to compete with MinIO, though - rather, to expand the application domain of object stores.
superboum 5y ago
Another benefit compared to MinIO is we have "flexible topologies". Due to our design choice, you can add and remove nodes without any constraint on number of nodes and size of the storage. So you do not have to overprovision your cluster as recommended by MinIO[0]. Additionally, and we planned a full blog post on this subject, adding or removing a node in the cluster does not lead to a full rebalance of the cluster. To understand why, I must explain how it works traditionally and how we improved on existing work. When you initialize the cluster, we split the cluster in partitions, then assign partitions to nodes (see Maglev[1]). Later, based on their hash, we will store data in its corresponding partition. When a node is added or removed, traditional approaches rerun the whole algorithm and comes with a totally different partition assignation. Instead, we try to compute a new partition distribution that minimize partitions assignment change, which in the end minimize the number of partitions moved. On the drawback side, Garage does not implement erasure coding (as it also the reason of many MinIO's limitations) and duplicate data 3 times which is less efficient. Garage also implements less S3 endpoints than Minio (for example we do not support versioning), the full list is available in our documentation[2]. [0]: https://docs.min.io/minio/baremetal/installation/deploy-minio-distributed.html#capacity-based-planning https://docs.min.io/minio/baremetal/installation/deploy-mini... [1]: https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud https://www.usenix.org/conference/nsdi16/technical-sessions/... [2]: https://garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility/ https://garagehq.deuxfleurs.fr/documentation/reference-manua...

prmoustache 5y ago

I understand the founders are Terry Pratchett / Discworld fans from the name of the associations. For non french speakers, Deuxfleurs is the french traduction of the Twoflower character appearing in The Colour of Magic, The Light Fantastic and Interesting Times from Terry Pratchett's Discworld books series.

adrn10 5y ago
Busted! We might even have a Bagage[0] side-project... [0] https://git.deuxfleurs.fr/Deuxfleurs/bagage https://git.deuxfleurs.fr/Deuxfleurs/bagage

merb 5y ago

this might be a stupid comment. but is this safe to use inside a commercial project as a backend s3 solution? i.e. just using it instead of lcoal storage and not changing it?

superboum 5y ago
Our software is published under the AGPLv3 license and comes with no guarantee, like any other FOSS project (if you do not pay for support). We are considering our software as "public beta" quality, so we think it works well, at least for us. On the plus side, it survived Hacker News Hug of Death. Indeed, the website we linked is hosted on our own Garage cluster made of old Lenovo ThinkCentre M83 (with Intel Pentium G3420 and 8GB of RAM) and the cluster seems fine. We also host more than 100k objects in our Matrix (a chat service) bucket. On the minus side, this is the first time we have so much coverage, so our software has not yet been tested by thousands of people. It is possible that in the near future, some edge cases we never triggered are reported. This is the reason why most people wait that an application reaches a certain level of adoption before using it, in other words they don't want to pay "the early adopter cost". In the end, it's up to you :-)
- cvwright 5y ago
  Awesome. Are you using the matrix-media-repo to connect the Matrix homeserver to Garage?
  - lxpz 5y ago
    No, we are simply using synapse-s3-storage-provider: https://github.com/matrix-org/synapse-s3-storage-provider https://github.com/matrix-org/synapse-s3-storage-provider
deleted 5y ago
[deleted]
lxpz 5y ago
License-wise, Garage is AGPLv3 like many similar software projects such as MinIO and MongoDB. We are not lawyers, but in the spirit, this should not preclude from using it in commercial software as long as it is not modified. In the case where you are modifying it, like e.g. adding a nice administration GUI, we kindly ask that you publish your modifications as open source so that we can incorporate them back in the main product if we want.

southerntofu 5y ago

Hello, quick question: from what i understand since it's based on CRDTs it means any server in the cluster may save something to the storage, that gets replicated elsewhere. That's fine when there is absolute trust between the server operators, but am i correct to assume it's not the same threat model as encrypted/signed backups pushed to some friends lending storage space for you (who could corrupt your backup but can't corrupt your working data)? If my understanding is correct, maybe your homepage should outline the threat model more clearly (i.e. a single trusted operator for the whole cluster) and point to other solutions like TAHOE-LAFS for other use-cases. Congratulations on doing selfhosting with friends, that's pretty cool! Do you have any idea if some hosting cooperatives (CHATONS) or ISPs (FFDN) have practical use-cases in mind? They already have many physical locations and rather good bandwidth, but i personally can't think of an interesting idea.

ClumsyPilot 5y ago
I was thiniing StorJ and fulecoin are the ones attempting to provide storage with a 'tryst noone' model
- southerntofu 5y ago
  "Trust noone" cannot work. Humanity is built on trust. The question is what level of trust you place on different operators. From a quick look StorJ and fuelcoin both appear to be classic crypto-scams where the entire tech stack is controlled by a single company... talk about "zero trust" :-) If you're really interested about low trust tech for file storage, i recommend you check out TAHOE-LAFS. It's pretty cool tech and there's no money to be made out of it, nor is there a company dictating who can join what pool and how much you have to pay for what kind of storage. It's an actually decentralized system, unlike most blockchains.
lxpz 5y ago
You are absolutely correct concerning the trust model of Garage: any administrator of one node is an administrator of the entire cluster, and can manipulate data arbitrarily in the system. From an ideological perspective, we are strongly attached to the building of tight-knit communities in which strong trust bonds can emerge, as it gives us more meaning than living in an individualized society where all exchanges between individuals are mediated by a market, or worse, by blockchain technology. This means that trusting several system's administrator makes sense to us. Note that in the case of cooperatives such as CHATONS, most users are non-technical and have to trust their sysadmin anyways; here, they just have to trust several sysadmins instead of just one. We know that several hosting cooperatives of the CHATONS network are thinking like us and are interested in setting up systems such as Garage that work under this assumption. In the meantime, we also do perfectly recognize the possibility of a variety of attack scenarios against which we want to consider practical defenses, such as the following: 1/ An honest-but-curious system administrator or an intruder in the network that wants to read user's private data, or equivalently, a police raid where server software is embarked for inspection by state services; 2/ A malicious administrator or an intruder that wants to manipulate the users by introducing fake data; 3/ A malicious administrator or an intruder that simply wants to wreak havoc by deleting everything. Point 1 is the biggest risk in my eyes. Several solutions can be built to add an encryption layer over S3 for different usage scenarios. For instance for storing personnal files, Rclone can be used to add simple file encryption to an S3 bucket and can also be mounted directly via FUSE, allowing us to access Garage as an end-to-end encrypted network drive. For backups, programs like Restic and Borg allow us to upload our backups encrypted into Garage. Point 2 can be at least partially solved by adding signatures and verification in the encryption layer which is handled on the client (I don't know for sure if Rclone, Restic and Borg are doing this). Point 3 is harder and probably requires adaptation on the side of Garage to be solved, for instance by adding a restriction on which nodes are allowed to propagate updates in the network and thus establishing a hierarchy between two categories of nodes: those that implement the S3 gateway and thus have full power in the network, and those that are only responsible for storing data given by the gateway but cannot originate modifications themselves (under the assumption that modifications must be accompanied by a digital signature and that only gateway nodes have the private keys to generate such signatures). Then we could separate gateway nodes according to the different buckets in which they can write for better separation of concernts. However, as long as we are trying to implement the S3 protocol I think we are stuck with some imperfect solution like this, because S3 itself does not implement using public-key cryptography to attest operations sent by the users. It is perfectly true that solutions that are explicitely designed to handle these threats (such as TAHOE-LAFS) would provide better guarantees in a system that is maybe more consistent as a whole. However we are also trying to juggle these security constraints with deployment contraints such as keeping compatibility with standard protocols such as S3 (and soon also IMAP for mailbox storage), which restricts us in the design choices we make. Just to clarify, the fact that any node can tamper with any of the data in the cluster is not strictly linked to the fact that we use CRDTs internally. It is true that CRDTs make it more difficult, but we do believe that there are solutions to this and we intend to implement at least some of them in our next project that involves mailbox storage.

Fiahil 5y ago

NICE ! Very Nice ! Congrats on the project and the naming ! If I may add a few thoughts : You may be very well placed for being a more suitable replacement to Minio in Data science / AI projects. Why ? Because a hard requirement for any serious MLOps construction needs 2 things : An append-only storage and a way to stream large content to a single place. First one is very hard to get right, second is relatively easy. Being CRDT-based it should be _very easy_ for you to provide an append-only storage that can store partitioned, ordered logs of immutable objects (think dataframes, and think kafka). Once you have that, it's really easy to build the remaining missing pieces (UI, API, ...) for creating a _much better_ (and distributed) version of MLFlow. Finally, S3 protocol is an "okay" version for file storage, but as you are probably aware, it's clearly a huge limiter. So, trash it. Provide a read-only S3 gateway for compatibility, but writes should use a different API. PS: Galette-Saucisse <3

adrn10 5y ago
Thanks for your enthusiastic feedback, fellow breton! It's an interesting take you make about ML workloads. We haven't investigated that yet (next in line is e-mail: you see we target ubiquitous, low-tech needs that are horrendous to host). But we will definitely consider it for future works: small breton food trucks do need their ML solutions for better galettes.
- Fiahil 5y ago
  If you ever need further insights into the ML/AI use case, don't hesitate to ping me at {my-username}-at-gmail. I'll forward it to my work address :)
  - adrn10 5y ago
    And you are cordially invited to our chatroom on Matrix, if you want to hack around Garage: https://matrix.to/#/#garage:deuxfleurs.fr https://matrix.to/#/#garage:deuxfleurs.fr We used to mostly talk French there - until yesterday!

bamazizi 5y ago

Github mirror? It's extremely easy to forget and stop following up on a project that's not on Github! You might have personal reasons by your current choice, but maybe the project can have shot at success if it's in a more social and high traffic ecosystem.

lxpz 5y ago
Exists, already linked in another comment : https://github.com/deuxfleurs-org/garage https://github.com/deuxfleurs-org/garage

pnocera 5y ago

I'd see a nice fit with Tailscale. Or how to create a secured private network of redundant and HA storage...

adrn10 5y ago
We haven't looked too much into VPNs, although we do need some plumbing to distribute our whole infrastructure at Deuxfleurs.

mediocregopher 5y ago

I've been looking through the available options for this use-case, with the same motivation given in this post, and was surprised to see there wasn't something which could fit this niche (until now!). Really happy to see this being worked on by a dedicated team. Combined with a tool like https://github.com/slackhq/nebula https://github.com/slackhq/nebula this could help form the foundation of a fully autonomous, durable, online community.

belfalas 5y ago

Looks very worth checking out! Will this be made available for installation via Cloudron some time in the future?

razzio 5y ago

This is a fantastic project and I can't wait to try it out! One question on the network requirements. The web page says for networking: "200 ms or less, 50 Mbps or more". How hard are these requirements? For folks like me that can't afford a guaranteed 50Mbps internet connection, is this still usable? There are plenty of places in the world where 50Mbps internet connectivity would be a dream. Even here in Canada there are plenty of places with a max of 10Mbps. The African continent for example will have many more.

adrn10 5y ago
This depends on your workload. It would work, but we first target fiber optics domestic connections.
lxpz 5y ago
It's not a hard requirement, you might just have a harder time handling voluminous files as Garage nodes will in all case have to transfer data internally in the cluster (remember that files have to be sent to three nodes when they are stored in Garage, meaning 3x more bandwidth needs to be used). 10Mbps is already pretty good if it is stable and your ping isn't off the charts, and it might be totally workable depending on your use case.

latchkey 5y ago

I see zero unit/integration tests in the source tree, not even a `test` target in the Makefile. Terrifying.

lxpz 5y ago
Then you didn't look close enough. Have a look at the .drone.yml to see the entry points, we have both unit tests and integration tests that run at each build.

lloydatkinson 5y ago

> account the geographical location of servers, and ensures that copies of your data are located at different locations when possible for maximal redundancy, a unique feature in the landscape of distributed storage systems. Azure already does this, so claiming it's unique seems untrue.

lxpz 5y ago
I mean, sure, but we're operating under the premise that we don't want to give out our data to a cloud provider and instead want to store data ourselves. Half of the post is dedicated to explaining that, so pretending that you can't infer that from context seems a bit unfair.

discardedrefuse 5y ago

Hi this is a very cool project, and I love that it's geared towards self-hosting. I'm about inherit some hardware so I'm actually looking at clustered network storage options right now. In my setup, I don't care about redundancy. I much prefer to maximize storage capacity. Can Garage be configured without replication? If not, maybe someone else can point me in the right direction. Something like a glusterfs distributed volume. At first glance, SeaweedFS has a "no replication" option, but their docs make it seem like they're geared towards handling billions of tiny files. I have about 24TB of 1Gb to 20GB files.

lxpz 5y ago
You probably want at least some redundancy otherwise you're at risk of losing data as soon as a single one of your hard drive fails. However if you're interested in maximizing capacity, you probably need erasure coding, which Garage does not provide. Gluster, Ceph and Minio might all be good candidates for your use case. Or if you have the possibility of putting all your drives in a single box, just do that and make a ZFS pool.
- discardedrefuse 5y ago
  > You probably want at least some redundancy otherwise you're at risk of losing data as soon as a single one of your hard drive fails. If by "fails" you mean the network connection drops out. Then yes, that would be a huge problem. I was hoping some project had a built-in solution to this. Currently, I'm using MergerFS to effectively create 1 disk out of 3 external USB drives and it handles accidental drive disconnects with no problems (I can't gush enough over how great mergerfs is). But, if by "fails" you mean actual hardware failure. Then, I don't really care. I keep 1 to 1 backups. A few days of downtime to restore the data isn't a big deal; this is just my home network. > Or if you have the possibility of putting all your drives in a single box... Unfortunately, I've maxed out the drive bays on my TS140 server. Buying new, larger drives to replace existing drives seems wasteful. Also, I've just been gifted another TS140, which is a good platform to start building another file server. You've given me something to think about, thanks. I appreciate you taking the time to respond!
5y ago

toolslive 5y ago

Some remarks: - replication is way too expensive for what it's worth. As soon as you can afford it you really should go for some kind of erasure coding. The exact moment when erasure coding is more efficient (storage wise, transport wise) and more robust (in terms of devices you can lose) depends on the size of the objects (fixed cost of metadata). - you don't need distributed consistency for the data storage actions just for the meta data (the map that tells you where the data is). You can start pushing data, and if the metadata for the object metadata never is created, you just pushed garbage which can be collected.

kalstone 5y ago

Does Garage integrate with Kubernetes?

rad_gruchalski 5y ago

Hey, looks pretty interesting but I have a couple of questions: 1) the license of Garage is AGPL, MinIO is also AGPL so I'm not sure what's the problem with it as a valid choice for self-hosting? You seem to not have a problem using Apache 2 licensed software (HashiCorp stack). 2) if you're a non-profit, how come MinIO is your "closest competitor"?