5 ms·

Be warned that their code quality is pretty bad. There was a bug I was dealing with last year where it did not delete objects but returned the correct HTTP resp

Be warned that their code quality is pretty bad. There was a bug I was dealing with last year where it did not delete objects but returned the correct HTTP response code indicating it did. This was widespread, not just some edge case I encountered. Their broken test suite doesn't actually verify the object on disk changed. I tried to engage them but they blew me off.

kapilvt 5y ago
Github issue link? They seem to have a solid ci setup, and I know several large enterprises using. But I found a bug for my usage != bad code quality.
- 0x000000001 5y ago
  My usage was "setup a basic single node for testing, upload a file with mc client, delete a file with mc client". They failed that test. It was responding with 200s but the file was never deleted. There are loads of issues like this on their github: https://github.com/minio/minio/issues/8873 https://github.com/minio/minio/issues/8873
  - tyingq 5y ago
    That's an interesting issue. Boiled down to the object name having '//' in it, which drove a certain direction for the shard location that wasn't the same shard location that the delete function looked in. Sounds like the shard hashing happens before or after object name normalization depending on the operation. Ouch.
  - _hyn3 5y ago
    Based on sib comment, it looks like it was related to a poor name sanitation / matching function (which says, to me, that's risky if you have untrusted names), but this could also be caused by a lazy or delayed deletion strategy.
Areading314 5y ago
Is Ceph with Rados Gateway a better alternative to this?
- 0x000000001 5y ago
  I have a 500PB Ceph setup @ work, but I don't maintain it. It's been solid.
- nateoda 5y ago
  I would say no in production. I was recently testing a ceph + rgw as an on prem s3 solution, but high throughput puts + ls caused an index corruption that “lost” files according to future LS’s, the file was still there if you directly get it. When this was reported it was already found multiple years ago, and never fixed
  - marbu 5y ago
    Could you reference a bug url? I tried to find it via tracker.ceph.com but failed to do so (I don't claim that the problem doesn't exist). That said referencing a bug url would be nice if you want to increase credibility of your claim.
    - AdamProut 5y ago
      Could be: https://tracker.ceph.com/issues/24744 https://tracker.ceph.com/issues/24744 I know this bug has hampered our use of ceph at singlestore. Note that this is not an eventual consistency issue. When it happens the list command will permanently miss files.
- hardwaresofton 5y ago
  CERN runs at least in part on Ceph and it's well documented: https://www.youtube.com/watch?v=OopRMUYiY5E https://www.youtube.com/watch?v=OopRMUYiY5E
mtalantikite 5y ago
I also hit a very frustrating issue in minio where CORS headers weren't being set properly and there were many similar cases in their issues history. Their response was basically "works for me, sorry". I'm pretty sure there was something weird going on with how minio was reading the config state, as I definitely was not the only one hitting it. Luckily I only had to use it for local testing in the project, but the whole thing didn't leave me feeling good. [1] https://github.com/minio/minio/issues/11111 https://github.com/minio/minio/issues/11111
pbadenski 5y ago
We tried to use it a year ago or so, because of the performance promise. We were getting random glitches every few thousand files during operations. There was no obvious pattern, so difficult to reproduce, and as far as I remember there was mentions of it in github. Hopefully they acknowledge and get over this hump, as it seems like a promising project all together.
- tailspin2019 5y ago
  Yep, same experience unfortunately
tobias3 5y ago
Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while that fixed some cases. Looking at the current code this setting is called MINIO_FS_OSYNC now (for some reason) https://github.com/minio/minio/pull/9581/commits/ce63c75575a7ffa7c237b24e11af6c51c6b70399 https://github.com/minio/minio/pull/9581/commits/ce63c75575a... (but I wouldn't trust that... are they fsyncing directories correctly? Making sure object metadata gets deleted with the data in one transaction etc.). Totally undocumented, too. I guess this makes minio "fast". But it might eat your data. Please use something like Ceph+RadosGW instead. It might be okay for running tests where durability isn't a requirement.
- tyingq 5y ago
  That had me curious, so I searched a bit in their issues. Their attitude about it isn't great: https://github.com/minio/minio/issues/3536 https://github.com/minio/minio/issues/3536 That's too bad, as it seems well thought out in other areas, like clustering.
  - dragonsh 5y ago
    MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority. You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system. [0] https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
    - tobias3 5y ago
      I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... . Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.
etaioinshrdlu 5y ago
I had issues with frequent crashes due to various panics a while a go. It eventually went away after a version upgrade. But now reading this I don’t feel terribly confident in using minio long term.
merb 5y ago
what would be a better way to export a nfs storage to s3 than? swift, like it does for glusterfs?
jquaint 5y ago
100% agree. I pin the version we use because you never know if it will come with even more bugs.

kapilvt 5y ago

Github issue link? They seem to have a solid ci setup, and I know several large enterprises using. But I found a bug for my usage != bad code quality.

0x000000001 5y ago
My usage was "setup a basic single node for testing, upload a file with mc client, delete a file with mc client". They failed that test. It was responding with 200s but the file was never deleted. There are loads of issues like this on their github: https://github.com/minio/minio/issues/8873 https://github.com/minio/minio/issues/8873
- tyingq 5y ago
  That's an interesting issue. Boiled down to the object name having '//' in it, which drove a certain direction for the shard location that wasn't the same shard location that the delete function looked in. Sounds like the shard hashing happens before or after object name normalization depending on the operation. Ouch.
- _hyn3 5y ago
  Based on sib comment, it looks like it was related to a poor name sanitation / matching function (which says, to me, that's risky if you have untrusted names), but this could also be caused by a lazy or delayed deletion strategy.

Areading314 5y ago

Is Ceph with Rados Gateway a better alternative to this?

0x000000001 5y ago
I have a 500PB Ceph setup @ work, but I don't maintain it. It's been solid.
nateoda 5y ago
I would say no in production. I was recently testing a ceph + rgw as an on prem s3 solution, but high throughput puts + ls caused an index corruption that “lost” files according to future LS’s, the file was still there if you directly get it. When this was reported it was already found multiple years ago, and never fixed
- marbu 5y ago
  Could you reference a bug url? I tried to find it via tracker.ceph.com but failed to do so (I don't claim that the problem doesn't exist). That said referencing a bug url would be nice if you want to increase credibility of your claim.
  - AdamProut 5y ago
    Could be: https://tracker.ceph.com/issues/24744 https://tracker.ceph.com/issues/24744 I know this bug has hampered our use of ceph at singlestore. Note that this is not an eventual consistency issue. When it happens the list command will permanently miss files.
hardwaresofton 5y ago
CERN runs at least in part on Ceph and it's well documented: https://www.youtube.com/watch?v=OopRMUYiY5E https://www.youtube.com/watch?v=OopRMUYiY5E

mtalantikite 5y ago

I also hit a very frustrating issue in minio where CORS headers weren't being set properly and there were many similar cases in their issues history. Their response was basically "works for me, sorry". I'm pretty sure there was something weird going on with how minio was reading the config state, as I definitely was not the only one hitting it. Luckily I only had to use it for local testing in the project, but the whole thing didn't leave me feeling good. [1] https://github.com/minio/minio/issues/11111 https://github.com/minio/minio/issues/11111

pbadenski 5y ago

We tried to use it a year ago or so, because of the performance promise. We were getting random glitches every few thousand files during operations. There was no obvious pattern, so difficult to reproduce, and as far as I remember there was mentions of it in github. Hopefully they acknowledge and get over this hump, as it seems like a promising project all together.

tailspin2019 5y ago
Yep, same experience unfortunately

tobias3 5y ago

Minio isn't durable. Any S3 operation might not be on disk after it is completed successfully. They had an environment variable MINIO_DRIVE_SYNC for a while that fixed some cases. Looking at the current code this setting is called MINIO_FS_OSYNC now (for some reason) https://github.com/minio/minio/pull/9581/commits/ce63c75575a7ffa7c237b24e11af6c51c6b70399 https://github.com/minio/minio/pull/9581/commits/ce63c75575a... (but I wouldn't trust that... are they fsyncing directories correctly? Making sure object metadata gets deleted with the data in one transaction etc.). Totally undocumented, too. I guess this makes minio "fast". But it might eat your data. Please use something like Ceph+RadosGW instead. It might be okay for running tests where durability isn't a requirement.

tyingq 5y ago
That had me curious, so I searched a bit in their issues. Their attitude about it isn't great: https://github.com/minio/minio/issues/3536 https://github.com/minio/minio/issues/3536 That's too bad, as it seems well thought out in other areas, like clustering.
- dragonsh 5y ago
  MinIO team care about an issue if you are paid customer, not for people who use the open source. Indeed MinIO is not even fully S3 compatible with many edge cases and close the issues related to it by saying it’s not a priority. You might want to look at other options as well like SeaweedFS [0] a POSIX compliant S3 compatible distributed file system. [0] https://github.com/chrislusf/seaweedfs https://github.com/chrislusf/seaweedfs
  - tobias3 5y ago
    I haven't used seaweedfs yet, but it looks better (and small file/object performance should be miles better). W.r.t. to fsync/durability with the seaweedfs API to a volume server you have to turn fsync on via paramter and it is disabled by default. With S3 it is probably also off by default and you can turn it on per bucket: https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Configuration https://github.com/chrislusf/seaweedfs/wiki/Path-Specific-Co... . Both should default to fsync on, with the option to turn it off. So not a great choice of defaults. Again, it probably looks good in benchmarks when people naively compare S3 stores. But it just shouldn't eat your data per default.

etaioinshrdlu 5y ago

I had issues with frequent crashes due to various panics a while a go. It eventually went away after a version upgrade. But now reading this I don’t feel terribly confident in using minio long term.

merb 5y ago

what would be a better way to export a nfs storage to s3 than? swift, like it does for glusterfs?

jquaint 5y ago

100% agree. I pin the version we use because you never know if it will come with even more bugs.