3 ms·
> 1. Does it have the S3/Swift/etc. semantics that you will always get the bits first stored or an error but never a corrupt or truncated file? Yes you are rig
by y4m4b4 11y ago
> 1. Does it have the S3/Swift/etc. semantics that you will always get the bits first stored or an error but never a corrupt or truncated file?
Yes you are right the behavior is exactly similar to how an S3 API would behave for Get/Put().
> Does it have the S3 and (IIRC) Swift feature that allows you to provide an expected hash so data corrupted in transit will be detected and an error will be returned instead of storing the object?
Yes, just like how Content-MD5 is set through a client request for per uploads in S3, Minio honors that as well.
Minio provides three modes of operation
-- memory (in memory object storage)
-- filesystem (point to any directory or disk, all files are contigous and not erasure coded)
-- donut (experimental, distributed erasure coded on disk format)
In Donut individual parts or chunks are separately sha256 summed along with erasure coding on top of these individual parts.
https://github.com/minio/minio/tree/master/pkg/storage/donut https://github.com/minio/minio/tree/master/pkg/storage/donut
Donut also implements rubberband erasure which is a dynamic erasure coding style where in cumulative parities are created for all objects depending on the available disks, total usage etc.
> How does it verify data integrity? Only on access, background scrubs, via API request, etc.
Donut also does data integrity checks on a regular basis as an operational process, where even silent data corruption is fixed automatically without the need for any user intervention.
We are in early stage of our project, as of today Donut is experimental - currently runs on a single node, does sha256 sums for erasured chunks and erasure codes your data on disk.
Much of the operational process and distributed donut is a work in progress, we are planning to get them ready by end of this year.
> Which strong hashes are used for all of the above? Are they configurable / selectable so e.g. if I wanted something strong than S3's Content-MD5 header for uploads can I use SHA-256 instead?
Yes while Content-MD5 is what we do now, along with erasure coding and sha256 summing individual chunks. We can provide the similar semantics of ``x-amz-content-sha256`` from client side. We would be happy to implement it feel free to open a bug - https://github.com/minio/minio/issues https://github.com/minio/minio/issues
> The classic easy-to-ask, hard-to-answer one: what level of testing backs up all of the previous answers?
I agree with hard-to-answer :-). Minio is written in Golang, which largely drives us to "write tests first" style and we have rightfully followed it on a day to day basis. Not being limited to that we are also working towards building a continous integration testing framework which would be baked into the product, which would help any user perform self diagnostics of his/her setup. In-turn generating reports for quality of their setup, any unknown parameters which might show up etc.
Also testing is an ongoing process, so we are striving to be better everyday :-)
Thanks for the questions, feel free to hop on to - gitter.im/minio/minio and find us there.
- acdha 11y agoThanks for the reply – it sounds like you're definitely hitting the areas I care about. One quick thought: I was at USENIX the year that ZFS was announced and one thing which was very popular was when the Sun team talked about how much random testing they'd done with billions of runs exercising the major features with simulated failures, memory corruption, etc. so they could stress-test the detection and recovery mechanisms: https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing https://blogs.oracle.com/bill/entry/zfs_and_the_all_singing You could see a lot of sysadmins nodding their heads as they described each class of failure mode and how they were trying to simulate it. These days I'd probably toss in something like running afl-fuzz against your storage code, too.
- y4m4b4 11y ago> Thanks for the reply – it sounds like you're definitely hitting the areas I care about. Thank you, appreciate the inputs :-)