5 ms·
> JuiceFS, written in Go, can manage tens of billions of files in a single namespace. At that scale, I care about integrity. Can someone working in this space
by amluto 3y ago
> JuiceFS, written in Go, can manage tens of billions of files in a single namespace.
At that scale, I care about integrity. Can someone working in this space please have a real integrity story as part of the offering? Give each object (object, version pair, perhaps) a cryptographic hash of the contents, and make that hash be part of the inventory. Allow the entire bucket to opt in to mandatory hashing. Let me ask the system to do a scrub in which it verifies those cryptographic hashes.
If this blows up metadata to 164 bytes per object, so be it. But the hash can probably get away with being stored with data, not metadata, as long as there is a mechanism to inventory those hashes. Keeping them in memory doesn’t seem necessary.
Even S3 has only desultory support for this. A lot of competitors have nothing of the sort.
- jorticka 3y agoRocksDB supports hashing at multiple levels (key, value, files) because Meta also realized the importance of integrity. It also supports verifying them in bulk. Presumably filesystems built over rocksdb also support this.
- snissn 3y agodo you use the hash of the items you're storing as the key/filename?
- daviesliu 3y agoJuiceFS relies on the object store to provide integrity for data. Besides that, JuiceFS stores the checksum of each object as tags in S3, and verifies that when downloading the objects. Inside the metadata service, it uses merkle tree (hash of hash) to verify the integrity of whole namespace (including id of data blocks) between RAFT replicas. Once we store the hash (4 bytes) of each objects into metadata, it should provide the integrity of the whole namespace.
- amluto 3y agoDoes JuiceFS allow the user to specify the hash of a file when uploaded? And then to read that hash back later? Otherwise there’s no end-to-end integrity check.
- markhahn 3y agosurely you mean that the FS should calculate the hash on file creation/update, not take some random value from the user. but I agree that a FS that maintains file-content hash should allow clients to query it.
- amluto 3y agoNo, the FS should verify the hash on creation/update. Otherwise corruption during creation/update would just cause the hash to match the corrupted data.
- deleted 3y ago[deleted]
- daviesliu 3y agoThe S3 API allow user to specify the hash of content as HTTP header, it will be verified by the JuiceFS gateway and persisted into JuiceFS as ETag. With POSIX API or HDFS, there is no such API to do that, unfortunately.
- fh973 3y agoThe Quobyte DCFS does end-to-end CRC32 for each 4k block of data. All metadata and communication is also CRC protected, although one other frame boundaries.
- EGreg 3y agoIPFS, MaidSAFE and even BitTorrent have had this from day 1, and they are distributed and not under the control of one company. FileCoin is used to pay for storage, but for now it seems storage is mostly free because of block rewards.
- kfrzcode 3y agoAnd they are slow, expensive and subject to node-based 33% attacks Hedera Hashgraph is none of these
- Geisterde 3y agoIs bittorrent subject to sybil too?
- wizzwizz4 3y agoAll distributed networks are vulnerable to Sybil attacks (unless you can ensure provenance somehow, out-of-band), but unless you can break the hash function, all that gets you with BitTorrent is denial of service (and traffic interception, I suppose, but that should already be part of your threat model).
- Geisterde 3y agoI see, so there isnt a risk of downloading malicious content, just that you might not get the file. Thanks.
- KaiserPro 3y agoThe next bit: > Its metadata engine uses an all-in-memory approach and achieves remarkable memory optimization Sounds like a fun thing to recover after a crash.
- ted_dunning 3y agoThey have a blog on that: https://juicefs.com/en/blog/engineering/increase-performance-metadata-backup-recovery https://juicefs.com/en/blog/engineering/increase-performance...
- daviesliu 3y agoThis post said another topic: how to backup the metadata of JuiceFS in readable format (JSON) and restore it into an empty database.
- daviesliu 3y agoThe recover process is similar to what a database does after a crash, it loads the recent snapshot in disk and apply any newer transaction logs.
- kfrzcode 3y agoI think you could achieve this on Hedera's hashgraph - ABFT DLT
- yjftsjthsd-h 3y agoI was under the impression that minio with erasure coding did that? ( https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html https://min.io/docs/minio/linux/operations/concepts/erasure-... )
- amluto 3y agoLooks like minio added this in 2022: https://github.com/minio/minio/pull/15433 https://github.com/minio/minio/pull/15433