3 ms·
What triggered me was just throwing out "reed solomon" when talking about random writes. How does that work? We'll read from 5 places to complete your write?
by jbooth 10y ago
What triggered me was just throwing out "reed solomon" when talking about random writes. How does that work? We'll read from 5 places to complete your write?
- nickpsecurity 10y agoMy impression was that they heard reed solomon was used in robust systems like they are describing. They intend to use it in theirs. It will therefore be just as robust. Similar to how some firms describe their security after adding "256-bit, military-grade AES." ;)
- zzzcpan 10y agoThey operate on blocks and can implement Reed-Solomon with no issues. Random writes do not matter with the architecture like this. The tricky part would be latency and performance during periods of growth.
- jbooth 10y agoYeah, but they have to read several data/parity blocks, and then rewrite all parity blocks plus one data block, for any write to a given block. This creates big difficulties for both consistency and performance, and fixes for consistency make the performance worse (and vice versa). Google's filesystem could use reed-solomon because they're append-only, making consistency a non-issue and performance can be fixed by buffering on the client side.
- betawaffle 10y agoTorus is append-only too. We also plan to support something more like what Facebook's paper describes, where they have extra parity (xor) to support more efficient local repair.
- jbooth 10y agoHow? I thought you're exporting a block device, not a filesystem? You can't append to a block device, and certainly every filesystem out there expects block devices to be random-writable, right?
- betawaffle 10y agoThe "interface" we're exporting is very different from the underlying storage. The block device interface we currently provide supports random writes just fine, but the underlying storage we use (which involves memory-mapped files) is append-only. Once written, blocks are only ever GC'd, not modified.
- jbooth 10y agoOk, so that plus a little MVCC can make you consistent, but you've still got the read-many-to-write-one thing from the perspective of your block device interface, right? And block devices, if I'm remembering right, don't leave you any room to buffer pending writes.
- betawaffle 10y agoTorus implements a kind of MVCC, yes. As for read-many-to-write-one, I assume you're talking about Reed-Solomon or similar erasure coding? There have been some papers written about ways to reduce that, a good one is from Facebook: https://code.facebook.com/posts/536638663113101/saving-capacity-with-hdfs-raid/ https://code.facebook.com/posts/536638663113101/saving-capac.... And that's just one option. Also, this is all speculative since we have yet to implement erasure coding.
- jbooth 10y agoWell, good luck with it. Don't get me wrong, I'd love 1.5x redundancy overhead instead of 3x. But even if you have to downgrade to offering either replication or XOR, it's still a huge missing piece of the typical container deployment, so good luck.