4 ms·

Looks interesting for something like local development. I don't intend to run production object storage myself, but some of the stuff in the guide to the produc

by fabian2k 10mo ago

Looks interesting for something like local development. I don't intend to run production object storage myself, but some of the stuff in the guide to the production setup (https://garagehq.deuxfleurs.fr/documentation/cookbook/real-world/ https://garagehq.deuxfleurs.fr/documentation/cookbook/real-w...) would scare me a bit: > For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation. It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.

igor47 10mo ago
I've been using minio for local dev but that version is unmaintained now. However, I was put off by the minimum requirements for garage listed on the page -- does it really need a gig of RAM?
- archon810 10mo ago
  The current latest Minio release that is working for us for local development is now almost a year old and soon enough we will have to upgrade. Curious what others have replaced it with that is as easy to set up and has a management UI.
  - mbreese 10mo ago
    I think that's part of the pitch here... swapping out Minio for Garage. Both scale a lot more than for just local development, but local dev certainly seems like a good use-case here.
- lxpz 10mo ago
  It does not, at least not for a small local dev server. I believe RAM usage should be around 50-100MB, increasing if you have many requests with large objects.
- dsvf 10mo ago
  I always understood this requirement as "garage will run fine on hardware with 1GB RAM total" - meaning the 1GB includes the RAM used by the OS and other processes. I think that most current consumer hardware that is a, potential garage host, even on the low end, has at least 1GB total RAM.
moffkalast 10mo ago
That's not something you can do reliably in software, datacenter grade NVMe drives come with power loss protection and additional capacitors to handle that gracefully. If power is cut at the wrong moment the partition may not be mountable afterwards otherwise. If you really live somewhere with frequent outages, buy an industrial drive that has a PLP rating. Or get a UPS, they tend to be cheaper.
- crote 10mo ago
  Isn't that the entire point of write-ahead logs, journaling file systems, and fsync in general? A roll-back or roll-forward due to a power loss causing a partial write is completely expected, but surely consumer SSDs wouldn't just completely ignore fsync and blatantly lie that the data has been persisted? As I understood it, the capacitors on datacenter-grade drives are to give it more flexibility, as it allows the drive to issue a successful write response for cached data: the capacitor guarantees that even with a power loss the write will still finish, so for all intents and purposes it has been persisted, so an fsync can return without having to wait on the actual flash itself, which greatly increases performance. Have I just completely misunderstood this?
  - Nextgrid 10mo ago
    > ignore fsync and blatantly lie that the data has been persisted Unfortunately they do: https://news.ycombinator.com/item?id=38371307 https://news.ycombinator.com/item?id=38371307
    - btown 10mo ago
      If the drives continue to have power, but the OS has crashed, will the drives persist the data once a certain amount of time has passed? Are datacenters set up to take advantage of this?
      - Nextgrid 10mo ago
        > will the drives persist the data once a certain amount of time has passed Yes, otherwise those drives wouldn't work at all and would have a 100% warranty return rate. The reason they get away with it is that the misbehavior is only a problem in a specific edge-case (forgetting data written shortly before a power loss).
lxpz 10mo ago
If you know of an embedded key-value store that supports transactions, is fast, has good Rust bindings, and does checksumming/integrity verification by default such that it almost never corrupts upon power loss (or at least, is always able to recover to a valid state), please tell me, and we will integrate it into Garage immediately.
- BeefySwain 10mo ago
  (genuinely asking) why not SQLite by default?
  - lxpz 10mo ago
    We were not able to get good enough performance compared to LMDB. We will work on this more though, there are probably many ways performance can be increased by reducing load on the KV store.
    - skrtskrt 10mo ago
      Could you use something like Fly's Corrosion to shard and distribute the SQLite data? It uses a CRDT reconciliation, which is familiar for Garage.
      - lxpz 10mo ago
        Garage already shards data by itself if you add more nodes, and it is indeed a viable path to increasing throughput.
    - tensor 10mo ago
      Keep in mind that write safety comes with performance penalties. You can turn off write protections and many databases will be super fast, but easily corrupt.
    - srcreigh 10mo ago
      Did you try WITHOUT ROWID? Your sqlite implementation[1] uses a BLOB primary key. In SQLite, this means each operation requires 2 b-tree traversals: The BLOB->rowid tree and the rowid->data tree. If you use WITHOUT ROWID, you traverse only the BLOB->data tree. Looking up lexicographically similar keys gets a huge performance boost since sqlite can scan a B-Tree node and the data is contiguous. Your current implementation is chasing pointers to random locations in a different b-tree. I'm not sure exactly whether on disk size would get smaller or larger. It probably depends on the key size and value size compared to the 64 bit rowids. This is probably a well studied question you could find the answer to. [1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8bac07df7fd96ea5f3df7052e70f5577c753/src/db/sqlite_adapter.rs https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...
yupyupyups 10mo ago
Depending on the underlying storage being reliable is far from unique to garage. This is what most other services do too, unless we're talking about something like Ceph which manages the physical storage itself. Standard filesystems such as ext4 and xfs don't have data checksumming, so you'll have to rely on another layer to provide integrity. Regardless, that's not garage's job imo. It's good that they're keeping their design simple and focus their resources on implementing the S3 spec.
nijave 10mo ago
The assumption is nodes are in different fault domains so it'd be highly unlikely to ruin the whole cluster. LMDB mode also runs with flush/syncing disabled

igor47 10mo ago

I've been using minio for local dev but that version is unmaintained now. However, I was put off by the minimum requirements for garage listed on the page -- does it really need a gig of RAM?

archon810 10mo ago
The current latest Minio release that is working for us for local development is now almost a year old and soon enough we will have to upgrade. Curious what others have replaced it with that is as easy to set up and has a management UI.
- mbreese 10mo ago
  I think that's part of the pitch here... swapping out Minio for Garage. Both scale a lot more than for just local development, but local dev certainly seems like a good use-case here.
lxpz 10mo ago
It does not, at least not for a small local dev server. I believe RAM usage should be around 50-100MB, increasing if you have many requests with large objects.
dsvf 10mo ago
I always understood this requirement as "garage will run fine on hardware with 1GB RAM total" - meaning the 1GB includes the RAM used by the OS and other processes. I think that most current consumer hardware that is a, potential garage host, even on the low end, has at least 1GB total RAM.

moffkalast 10mo ago

That's not something you can do reliably in software, datacenter grade NVMe drives come with power loss protection and additional capacitors to handle that gracefully. If power is cut at the wrong moment the partition may not be mountable afterwards otherwise. If you really live somewhere with frequent outages, buy an industrial drive that has a PLP rating. Or get a UPS, they tend to be cheaper.

crote 10mo ago
Isn't that the entire point of write-ahead logs, journaling file systems, and fsync in general? A roll-back or roll-forward due to a power loss causing a partial write is completely expected, but surely consumer SSDs wouldn't just completely ignore fsync and blatantly lie that the data has been persisted? As I understood it, the capacitors on datacenter-grade drives are to give it more flexibility, as it allows the drive to issue a successful write response for cached data: the capacitor guarantees that even with a power loss the write will still finish, so for all intents and purposes it has been persisted, so an fsync can return without having to wait on the actual flash itself, which greatly increases performance. Have I just completely misunderstood this?
- Nextgrid 10mo ago
  > ignore fsync and blatantly lie that the data has been persisted Unfortunately they do: https://news.ycombinator.com/item?id=38371307 https://news.ycombinator.com/item?id=38371307
  - btown 10mo ago
    If the drives continue to have power, but the OS has crashed, will the drives persist the data once a certain amount of time has passed? Are datacenters set up to take advantage of this?
    - Nextgrid 10mo ago
      > will the drives persist the data once a certain amount of time has passed Yes, otherwise those drives wouldn't work at all and would have a 100% warranty return rate. The reason they get away with it is that the misbehavior is only a problem in a specific edge-case (forgetting data written shortly before a power loss).

lxpz 10mo ago

If you know of an embedded key-value store that supports transactions, is fast, has good Rust bindings, and does checksumming/integrity verification by default such that it almost never corrupts upon power loss (or at least, is always able to recover to a valid state), please tell me, and we will integrate it into Garage immediately.

BeefySwain 10mo ago
(genuinely asking) why not SQLite by default?
- lxpz 10mo ago
  We were not able to get good enough performance compared to LMDB. We will work on this more though, there are probably many ways performance can be increased by reducing load on the KV store.
  - skrtskrt 10mo ago
    Could you use something like Fly's Corrosion to shard and distribute the SQLite data? It uses a CRDT reconciliation, which is familiar for Garage.
    - lxpz 10mo ago
      Garage already shards data by itself if you add more nodes, and it is indeed a viable path to increasing throughput.
  - tensor 10mo ago
    Keep in mind that write safety comes with performance penalties. You can turn off write protections and many databases will be super fast, but easily corrupt.
  - srcreigh 10mo ago
    Did you try WITHOUT ROWID? Your sqlite implementation[1] uses a BLOB primary key. In SQLite, this means each operation requires 2 b-tree traversals: The BLOB->rowid tree and the rowid->data tree. If you use WITHOUT ROWID, you traverse only the BLOB->data tree. Looking up lexicographically similar keys gets a huge performance boost since sqlite can scan a B-Tree node and the data is contiguous. Your current implementation is chasing pointers to random locations in a different b-tree. I'm not sure exactly whether on disk size would get smaller or larger. It probably depends on the key size and value size compared to the 64 bit rowids. This is probably a well studied question you could find the answer to. [1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8bac07df7fd96ea5f3df7052e70f5577c753/src/db/sqlite_adapter.rs https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...

yupyupyups 10mo ago

Depending on the underlying storage being reliable is far from unique to garage. This is what most other services do too, unless we're talking about something like Ceph which manages the physical storage itself. Standard filesystems such as ext4 and xfs don't have data checksumming, so you'll have to rely on another layer to provide integrity. Regardless, that's not garage's job imo. It's good that they're keeping their design simple and focus their resources on implementing the S3 spec.

nijave 10mo ago

The assumption is nodes are in different fault domains so it'd be highly unlikely to ruin the whole cluster. LMDB mode also runs with flush/syncing disabled