2 ms·
That I know of (and is public so I'm not breaching any NDA) AWS S3[1], Azure[2], GCP[3], Backblaze[4], Facebook's storage layer uses it[5][6], and Oracle Cloud'
by Twirrim 1y ago
That I know of (and is public so I'm not breaching any NDA) AWS S3[1], Azure[2], GCP[3], Backblaze[4], Facebook's storage layer uses it[5][6], and Oracle Cloud's Object Storage Platform[7].
The economies of scale mean that you really have to have something like erasure encoding in place to operate at large scale. The biggest single cost for cloud providers is the per-rack operational costs, so keeping the number of racks down is critical.
[1]https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_Deep-dive-on-Amazon-S3.pdf https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...
[2]https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf https://www.usenix.org/system/files/conference/atc12/atc12-f...
[3]https://cloud.google.com/storage/docs/availability-durability https://cloud.google.com/storage/docs/availability-durabilit...
[4]https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/
[5]https://www.usenix.org/conference/hotstorage13/workshop-program/presentation/rashmi https://www.usenix.org/conference/hotstorage13/workshop-prog...
[6]https://research.facebook.com/publications/a-hitchhikers-guide-to-fast-and-efficient-data-reconstruction-in-erasure-coded-data-centers/ https://research.facebook.com/publications/a-hitchhikers-gui... they even do some interesting things with erasure encoding and HDFS
[7] https://blogs.oracle.com/cloud-infrastructure/post/first-principles-redundancy-recovery-durability https://blogs.oracle.com/cloud-infrastructure/post/first-pri...