3 ms·
Thanks for sharing a great post. Do you think a 512MB bucket might be more optimal compared to a 1GB bucket? 2x less data required to be read when restoring an
by jorangreef 10y ago
Thanks for sharing a great post.
Do you think a 512MB bucket might be more optimal compared to a 1GB bucket? 2x less data required to be read when restoring an erasure coded bucket, and 2x quicker restore times, for only a 2x increase in bucket metadata costs? Erasure coded bucket slices (around 85MB per slice assuming 6/9 Reed Solomon) should be just right to amortize disk seeks.
- james_cowling 10y agoThanks! It's always a bit of a tradeoff. A 512MB bucket size would double the size of the Replication Table and we want make sure we keep the entire working set in the buffer cache. 1GB also has the advantage of filling up slower, which gives us more time to close full buckets and create new ones as needed. That said, 1GB is a bit of an arbitrary number which just happens to work pretty well. 512MB might work just fine. The exact number isn't as significant since typically we lose a whole disk at a time and thus needs to recover a certain number of TB regardless of the bucket size.
- jorangreef 10y agoThanks, that makes sense!