5 ms·

Copy/paste from a previous thread [0]: We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in compa

Copy/paste from a previous thread [0]: We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes) I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case. --- In addition: Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2]. We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking. [0]: https://news.ycombinator.com/item?id=46140342 https://news.ycombinator.com/item?id=46140342 [1]: https://rustfs.com/ https://rustfs.com/ [2]: https://rook.io/ https://rook.io/

hardwaresofton 10mo ago
Please also consider including SeaweedFS in the testing.
__turbobrew__ 10mo ago
I wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.
- breakingcups 10mo ago
  IS there a better solution for self-healing S3 storage that you could recommend? I'm also curious what will make a rook cluster croak after some time and what kind of maintenance is required in your experience.
  - adastra22 10mo ago
    ceph?
    - yupyupyups 10mo ago
      Rook is ceph.
  - adamcharnock 10mo ago
    Not used it yet, but RustFS sounds like it has self healing https://docs.rustfs.com/troubleshooting/healing.html https://docs.rustfs.com/troubleshooting/healing.html
  - __turbobrew__ 10mo ago
    I have unfortunately got a ceph cluster in a bad enough state that I just had to delete the pools and start from scratch. It was due to improper sequencing when removing OSDs, but that is kindof the point is you have to know what you are doing to know how to do things safely. For the most part I have so far learned by blundering things and learning hard lessons. Ceph clusters when mistreated can get into death spirals that only an experienced practitioner can advert through very carefully modifying cluster state through things like upmaps. You also need to make sure you understand your failure domains and how to spread mons and osds across the domains to properly handle failure. Lots of people don’t think about this and then one day a rack goes poof and you didn’t replicate your data across racks and you have data loss. Same thing with mons, you should be deploying mons across at least 3 failure domains (ideally 3 different datacenters) to maintain quorum during an outage.
nine_k 10mo ago
They explicitly say that top performance is not a goal: «high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only» (https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/) But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.
NL807 10mo ago
>I get the impression that Garage isn’t necessarily focussed on this kind of use case. I wouldn't be surprised if this will be fixed sometime in the future.
throwaway894345 10mo ago
> We can see we're going to have to move away from MinIO in the foreseeable future. My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it. I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.
- BadWolfStartup 10mo ago
  You can also just pay for MinIO and treat it like any other commercial dependency, with support and clearer expectations around licensing and roadmap, but forks are a different story: unless there’s a well-funded company or solid consortium behind them, you’re mostly just trading one risk for another.
  - riku_iki 10mo ago
    > with support and clearer expectations around licensing and roadmap nothing prevents them from hiking pricing, so expectations are not clear.
- deleted 10mo ago
  [deleted]
johncolanduoni 10mo ago
Somewhat unrelated, but I just looked at the RustFS docs intro[1] after seeing it here. It has this statement: > RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language. I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies). [1]: https://docs.rustfs.com/installation/ https://docs.rustfs.com/installation/
- b112 10mo ago
  [flagged]
  - sporkland 10mo ago
    Not dismissing your point, but Looking at the article, it looks like it's in rust unsafe code. Which seems to me to be a point that the rest of the rust code is fine but the place where they turned off the static safety the language provides they got bit.
    - b112 10mo ago
      Hey! Can't I just enjoy my schadenfreude in peace? I guess the takeaway is that, doubly so, trusting rust code to be memory safe, simply because it is rust isn't sensible. All its protections can simple be invalidated, and an end user would never know.
  - teiferer 10mo ago
    Um I doubt Debian maintainers look at every single line of code in the packages they maintain.
  - necovek 10mo ago
Emjayen 10mo ago
Those rates are peanuts considering that a decade ago saturating 40G, per core, was more than reasonable via standard userspace networking, with atleast a few copies in the datapath.
- PunchyHamster 10mo ago
  passing blocks of memory around vs referencing filesystem/database, ACLs, authentication and SSL
Roark66 10mo ago
Having just finished a "hobby size" setup of Rook-Ceph on 3 n100 mini pcs, with every service to fit in a couple hundred MB of ram (one service needs up to 3Gb when starting, but then runs around 250MB) I'd ask why not ceph? At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb". But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing). My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that. There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).
- PunchyHamster 10mo ago
  Ceph is far higher on RAM usage and complexity. Yeah if you need block storage in addition it's a good choice, but for anything smaller than half a rack of devices it's kinda overkill Also from our experience the docs outright lie about ceph's OSD memory usage and we've seen double or more than what docs claim (8-10GB instead of 4)
PunchyHamster 10mo ago
My small adventure with rustfs is that it is somewhat underbaked at the moment. And also it is already rigged for a rug-pull https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs#L47 https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...
- evil-olive 10mo ago
  yeah, their docs look pretty comprehensive, but there's a disturbing number of 404s that scream "not ready for prime-time" to me. from https://rustfs.com/ https://rustfs.com/ if you click Documentation, it takes you to their main docs site. there's a nav header at the top, if you click Docs there...it 404s. "Single Node Multiple Disk Installation" is a 404. ditto "Terminology Explanation". and "Troubleshooting > Node Failure". and "RustFS Performance Comparison". on the 404 page, there's a "take me home" button...which also leads to a 404.
- elvinagy 10mo ago
  Thanks for flagging this and for taking the time to point out the broken links. We open-sourced RustFS only a few months ago, and while we’ve been heavily focused on getting the core system to GA, that has admittedly created some documentation debt. We’re actively reviewing the docs and cleaning up any 404s or navigation issues we can find. For the specific 404 you mentioned, we haven’t been able to reproduce it on our end so far, but we’re continuing to investigate in case it’s environment- or cache-related. On the licensing side, we want to be clear that we’re fully committed to Apache 2.0 for the long term.

hardwaresofton 10mo ago

Please also consider including SeaweedFS in the testing.

__turbobrew__ 10mo ago

I wouldn’t use rook if you solely want S3. It is a massively complex system which you really need to invest in understanding or else your cluster will croak at some point and you will have no idea on how to fix it.

breakingcups 10mo ago
IS there a better solution for self-healing S3 storage that you could recommend? I'm also curious what will make a rook cluster croak after some time and what kind of maintenance is required in your experience.
- adastra22 10mo ago
  ceph?
  - yupyupyups 10mo ago
    Rook is ceph.
- adamcharnock 10mo ago
  Not used it yet, but RustFS sounds like it has self healing https://docs.rustfs.com/troubleshooting/healing.html https://docs.rustfs.com/troubleshooting/healing.html
- __turbobrew__ 10mo ago
  I have unfortunately got a ceph cluster in a bad enough state that I just had to delete the pools and start from scratch. It was due to improper sequencing when removing OSDs, but that is kindof the point is you have to know what you are doing to know how to do things safely. For the most part I have so far learned by blundering things and learning hard lessons. Ceph clusters when mistreated can get into death spirals that only an experienced practitioner can advert through very carefully modifying cluster state through things like upmaps. You also need to make sure you understand your failure domains and how to spread mons and osds across the domains to properly handle failure. Lots of people don’t think about this and then one day a rack goes poof and you didn’t replicate your data across racks and you have data loss. Same thing with mons, you should be deploying mons across at least 3 failure domains (ideally 3 different datacenters) to maintain quorum during an outage.

nine_k 10mo ago

They explicitly say that top performance is not a goal: «high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only» (https://garagehq.deuxfleurs.fr/documentation/design/goals/ https://garagehq.deuxfleurs.fr/documentation/design/goals/) But it might be interesting to see where the time is spent. I suspect they may be doing fewer things in parallel than MinIO, but maybe it's something entirely different.

NL807 10mo ago

>I get the impression that Garage isn’t necessarily focussed on this kind of use case. I wouldn't be surprised if this will be fixed sometime in the future.

throwaway894345 10mo ago

> We can see we're going to have to move away from MinIO in the foreseeable future. My favorite thing about all of this is that I had just invested a ton of time in understanding MinIO and its Kubernetes operator and got everything into a state that I felt good about. I was nearly ready to deploy it to production when the announcement was released that they would not be supporting it. I’m somewhat surprised that no one is forking it (or I haven’t heard about any organizations of consequence stepping up anyway) instead of all of these projects to rebuild it from scratch.

BadWolfStartup 10mo ago
You can also just pay for MinIO and treat it like any other commercial dependency, with support and clearer expectations around licensing and roadmap, but forks are a different story: unless there’s a well-funded company or solid consortium behind them, you’re mostly just trading one risk for another.
- riku_iki 10mo ago
  > with support and clearer expectations around licensing and roadmap nothing prevents them from hiking pricing, so expectations are not clear.
deleted 10mo ago
[deleted]

johncolanduoni 10mo ago

Somewhat unrelated, but I just looked at the RustFS docs intro[1] after seeing it here. It has this statement: > RustFS is a high-performance, distributed object storage software developed using Rust, the world's most popular memory-safe language. I’m actually something of a Rust booster, and have used it professionally more than once (including working on a primarily Rust codebase for a while). But it’s hard to take a project’s docs seriously when it describes Rust as “the world’s most popular memory-safe language”. Java, JavaScript, Python, even C# - these all blow it out of the water in popularity and are unambiguously memory safe. I’ve had a lot more segfaults in Rust dependencies than I have in Java dependencies (though both are minuscule in comparison to e.g. C++ dependencies). [1]: https://docs.rustfs.com/installation/ https://docs.rustfs.com/installation/

b112 10mo ago
[flagged]
- sporkland 10mo ago
  Not dismissing your point, but Looking at the article, it looks like it's in rust unsafe code. Which seems to me to be a point that the rest of the rust code is fine but the place where they turned off the static safety the language provides they got bit.
  - b112 10mo ago
    Hey! Can't I just enjoy my schadenfreude in peace? I guess the takeaway is that, doubly so, trusting rust code to be memory safe, simply because it is rust isn't sensible. All its protections can simple be invalidated, and an end user would never know.
- teiferer 10mo ago
  Um I doubt Debian maintainers look at every single line of code in the packages they maintain.
- necovek 10mo ago

Emjayen 10mo ago

Those rates are peanuts considering that a decade ago saturating 40G, per core, was more than reasonable via standard userspace networking, with atleast a few copies in the datapath.

PunchyHamster 10mo ago
passing blocks of memory around vs referencing filesystem/database, ACLs, authentication and SSL

Roark66 10mo ago

Having just finished a "hobby size" setup of Rook-Ceph on 3 n100 mini pcs, with every service to fit in a couple hundred MB of ram (one service needs up to 3Gb when starting, but then runs around 250MB) I'd ask why not ceph? At work I'm typically a consumer of such services from large cloud providers. I read in few places how "difficult" it is, how you need "4GB minimum RAM for most services" and how "friends do not let friends run Ceph below 10Gb". But this setup runs on a non dedicated 2.5Gb interface (there is VLAN segmentation and careful QoSing). My benchmarks show I'm primarily network latency and bandwidth limited. By the very definition you can't get better than that. There were many factors why I chose Ceph and not Garage, Seaweed or MinIo. (One of the biggest is that ceph does 2 birds with one stone for me - block and object).

PunchyHamster 10mo ago
Ceph is far higher on RAM usage and complexity. Yeah if you need block storage in addition it's a good choice, but for anything smaller than half a rack of devices it's kinda overkill Also from our experience the docs outright lie about ceph's OSD memory usage and we've seen double or more than what docs claim (8-10GB instead of 4)

PunchyHamster 10mo ago

My small adventure with rustfs is that it is somewhat underbaked at the moment. And also it is already rigged for a rug-pull https://github.com/rustfs/rustfs/blob/main/rustfs/src/license.rs#L47 https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...

evil-olive 10mo ago
yeah, their docs look pretty comprehensive, but there's a disturbing number of 404s that scream "not ready for prime-time" to me. from https://rustfs.com/ https://rustfs.com/ if you click Documentation, it takes you to their main docs site. there's a nav header at the top, if you click Docs there...it 404s. "Single Node Multiple Disk Installation" is a 404. ditto "Terminology Explanation". and "Troubleshooting > Node Failure". and "RustFS Performance Comparison". on the 404 page, there's a "take me home" button...which also leads to a 404.
elvinagy 10mo ago
Thanks for flagging this and for taking the time to point out the broken links. We open-sourced RustFS only a few months ago, and while we’ve been heavily focused on getting the core system to GA, that has admittedly created some documentation debt. We’re actively reviewing the docs and cleaning up any 404s or navigation issues we can find. For the specific 404 you mentioned, we haven’t been able to reproduce it on our end so far, but we’re continuing to investigate in case it’s environment- or cache-related. On the licensing side, we want to be clear that we’re fully committed to Apache 2.0 for the long term.