3 ms·

Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while tha

Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while that fixed some cases. Looking at the current code this setting is called MINIO_FS_OSYNC now (for some reason) https://github.com/minio/minio/pull/9581/commits/ce63c75575a7ffa7c237b24e11af6c51c6b70399 https://github.com/minio/minio/pull/9581/commits/ce63c75575a... (but I wouldn't trust that... are they fsyncing directories correctly? Making sure object metadata gets deleted with the data in one transaction etc.). Totally undocumented, too. I guess this makes minio "fast". But it might eat your data. Please use something like Ceph+RadosGW instead. It might be okay for running tests where durability isn't a requirement.

tyingq 5y ago
That had me curious, so I searched a bit in their issues. Their attitude about it isn't great: https://github.com/minio/minio/issues/3536 https://github.com/minio/minio/issues/3536 That's too bad, as it seems well thought out in other areas, like clustering.
- dragonsh 5y ago
  MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority. You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system. [0] https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
  - tobias3 5y ago
    I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... . Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.

tyingq 5y ago

That had me curious, so I searched a bit in their issues. Their attitude about it isn't great: https://github.com/minio/minio/issues/3536 https://github.com/minio/minio/issues/3536 That's too bad, as it seems well thought out in other areas, like clustering.

dragonsh 5y ago
MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority. You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system. [0] https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
- tobias3 5y ago
  I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... . Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.