3 ms·
Yep looks like we may have had slightly different goals here. Another one of our design requirements was that we wanted to be able to move data around within th
by james_cowling 10y ago
Yep looks like we may have had slightly different goals here. Another one of our design requirements was that we wanted to be able to move data around within the system (repair operations, rebalancing, encoding, compaction etc) without dealing with the metadata overhead of addressing hundreds of billions of blocks. Statically mapping a block to a 1GB bucket allows us to manage placement decisions on the granularity of buckets rather than orders of magnitude more individual files/blocks.
- jorangreef 10y agoThanks for sharing a great post. Do you think a 512MB bucket might be more optimal compared to a 1GB bucket? 2x less data required to be read when restoring an erasure coded bucket, and 2x quicker restore times, for only a 2x increase in bucket metadata costs? Erasure coded bucket slices (around 85MB per slice assuming 6/9 Reed Solomon) should be just right to amortize disk seeks.
- james_cowling 10y agoThanks! It's always a bit of a tradeoff. A 512MB bucket size would double the size of the Replication Table and we want make sure we keep the entire working set in the buffer cache. 1GB also has the advantage of filling up slower, which gives us more time to close full buckets and create new ones as needed. That said, 1GB is a bit of an arbitrary number which just happens to work pretty well. 512MB might work just fine. The exact number isn't as significant since typically we lose a whole disk at a time and thus needs to recover a certain number of TB regardless of the bucket size.
- jorangreef 10y agoThanks, that makes sense!