4 ms·

"For Backblaze Vaults, we threw out the Linux RAID software we had been using and wrote a Reed-Solomon implementation from scratch. It was exciting to be able t

"For Backblaze Vaults, we threw out the Linux RAID software we had been using and wrote a Reed-Solomon implementation from scratch. It was exciting to be able to use our group theory and matrix algebra from college. We’ll be talking more about this in an upcoming blog post." I hope I'm not the only one uncomfortable about this. I mean, I understand the need for greater flexibility and features that MDRaid doesn't provide, but this wording stinks of NIH and rebuilding the wheel from scratch "because it was fun", discarding the maturity and reliabilty of established software. And data storage is all about reliability. I hope I'm just reading too much from this, and it isn't actually representative of Backblaze's engineering practices.

jbellis 12y ago
Agreed. R-S encoding is hard. You should use something like zfec (extensively battle-tested by Tahoe-LAFS) or JErasure rather than rolling your own. (I say this as someone who worked at a company where we rolled our own, but that was ten years ago and we didn't have those options.)
kenko 12y ago
They don't say they did it because it was fun, only that it was fun. (To be fair, they also don't seem to say why they did do it.)
brianwski 12y ago
Brian Wilson from Backblaze here (not the "Brian Beach" who is author of the blog post) - this was not a case of NIH. We use lots and lots and lots of existing software like Debian, Java, Ext4. We use tools like ansible and Zabbix. But this one thing just didn't exist for us in the form we needed it. We looked, we really did. We did write the Reed-Solomon ourselves in a "clean room" so we did not have to pay any licensing fees and we clearly didn't steal anybody else's source code, but that is a very small amount of code. Like 80 lines of Java. Seriously. We referenced the technical papers we read to implement it in that blog post, but here it is again: http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solomon_codes.html http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solom... And we unit tested the living heck out of that code, plus we mathematically verified various parts. But I'm open to an alternative solution if you can suggest one? Remember our three highest priorities are: 1) reliable, 2) low cost, 3) simple. The "low cost" includes things like we do not want to pay ongoing licensing fees to other companies.
- AceJohnny2 12y ago
  Thanks, that's basically what I wanted to hear. :) My first thought was that you could've reused the R-S code from mdraid or dm-raid or ZFS, but on second thought 1) it may be too specialized to be reusable, and 2) it's GPL (or CDDL), so you can't just plonk it into your own code. And yeah, if it's just 80 lines of Java, I'm worrying about the wrong things.
  - fragmede 12y ago
    > 2) it's GPL Backblaze is a web service, so for better or worse, the GPL doesn't apply here, since we never have access to their binaries. The AGPL would apply, but that's not the license used.
- laymil 12y ago
  I'm actually rather surprised you didn't just go with JErasure (http://jerasure.org/ http://jerasure.org/).
skuhn 12y ago
I doubt that Backblaze decided to go this route on a whim, particularly because it's really their only option for cost efficient large scale storage. MD is terrible for this use case. There's no mature, open source object store with R-S coding. So everyone uses MD, but why does it suck for Backblaze-like storage? It only provides RAID levels, so you have partition-level redundancy but not necessarily object-level redundancy. You still need another layer to put objects in the right places to achieve the level of redundancy you want, and to re-balance as necessary. And a RAID array is local to one system, so achieving multi-host redundancy means duplicating the data as many times as necessary. MD can also be finicky. Since it's part of the kernel, kernel upgrades can (rarely) produce weirdness. You can get stuck between a rock and a hard place, where you need to upgrade to fix a vulnerability but upgrading too quickly puts your data at risk. According to Backblaze, they used to do 3 RAID6 arrays of 15 drives each per pod. This gives an overhead of 1.15:1, and two devices can fail before you lose data. Write performance is not going to be great, nor is rebuild performance. That is probably part of the reason why they could only push 950MB/s per host. This only provides disk-level redundancy, and not host-level. So now you have to at minimum duplicate your array onto a second host, and your overhead is 2.30:1. A third host brings it to 3.46:1. I'm surprised that they were even using RAID for non-boot devices at all, given the overhead. Erasure coding allows you to safely store data with far less than a 2:1 overhead. Their current design claims that they can lose 3 storage devices before they risk losing data permanently, with an overhead of 1.17:1. That is pretty compelling from a cost perspective.

jbellis 12y ago

Agreed. R-S encoding is hard. You should use something like zfec (extensively battle-tested by Tahoe-LAFS) or JErasure rather than rolling your own. (I say this as someone who worked at a company where we rolled our own, but that was ten years ago and we didn't have those options.)

kenko 12y ago

They don't say they did it because it was fun, only that it was fun. (To be fair, they also don't seem to say why they did do it.)

brianwski 12y ago

Brian Wilson from Backblaze here (not the "Brian Beach" who is author of the blog post) - this was not a case of NIH. We use lots and lots and lots of existing software like Debian, Java, Ext4. We use tools like ansible and Zabbix. But this one thing just didn't exist for us in the form we needed it. We looked, we really did. We did write the Reed-Solomon ourselves in a "clean room" so we did not have to pay any licensing fees and we clearly didn't steal anybody else's source code, but that is a very small amount of code. Like 80 lines of Java. Seriously. We referenced the technical papers we read to implement it in that blog post, but here it is again: http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solomon_codes.html http://www.cs.cmu.edu/~guyb/realworld/reedsolomon/reed_solom... And we unit tested the living heck out of that code, plus we mathematically verified various parts. But I'm open to an alternative solution if you can suggest one? Remember our three highest priorities are: 1) reliable, 2) low cost, 3) simple. The "low cost" includes things like we do not want to pay ongoing licensing fees to other companies.

AceJohnny2 12y ago
Thanks, that's basically what I wanted to hear. :) My first thought was that you could've reused the R-S code from mdraid or dm-raid or ZFS, but on second thought 1) it may be too specialized to be reusable, and 2) it's GPL (or CDDL), so you can't just plonk it into your own code. And yeah, if it's just 80 lines of Java, I'm worrying about the wrong things.
- fragmede 12y ago
  > 2) it's GPL Backblaze is a web service, so for better or worse, the GPL doesn't apply here, since we never have access to their binaries. The AGPL would apply, but that's not the license used.
laymil 12y ago
I'm actually rather surprised you didn't just go with JErasure (http://jerasure.org/ http://jerasure.org/).

skuhn 12y ago

I doubt that Backblaze decided to go this route on a whim, particularly because it's really their only option for cost efficient large scale storage. MD is terrible for this use case. There's no mature, open source object store with R-S coding. So everyone uses MD, but why does it suck for Backblaze-like storage? It only provides RAID levels, so you have partition-level redundancy but not necessarily object-level redundancy. You still need another layer to put objects in the right places to achieve the level of redundancy you want, and to re-balance as necessary. And a RAID array is local to one system, so achieving multi-host redundancy means duplicating the data as many times as necessary. MD can also be finicky. Since it's part of the kernel, kernel upgrades can (rarely) produce weirdness. You can get stuck between a rock and a hard place, where you need to upgrade to fix a vulnerability but upgrading too quickly puts your data at risk. According to Backblaze, they used to do 3 RAID6 arrays of 15 drives each per pod. This gives an overhead of 1.15:1, and two devices can fail before you lose data. Write performance is not going to be great, nor is rebuild performance. That is probably part of the reason why they could only push 950MB/s per host. This only provides disk-level redundancy, and not host-level. So now you have to at minimum duplicate your array onto a second host, and your overhead is 2.30:1. A third host brings it to 3.46:1. I'm surprised that they were even using RAID for non-boot devices at all, given the overhead. Erasure coding allows you to safely store data with far less than a 2:1 overhead. Their current design claims that they can lose 3 storage devices before they risk losing data permanently, with an overhead of 1.17:1. That is pretty compelling from a cost perspective.