5 ms·

https://www.backblaze.com/blog/backblaze-storage-pod-4/ https://www.backblaze.com/blog/backblaze-storage-pod-4/ here is the latest version. I have loved being

https://www.backblaze.com/blog/backblaze-storage-pod-4/ https://www.backblaze.com/blog/backblaze-storage-pod-4/ here is the latest version. I have loved being able to follow the iterations of the storage pod. They are very thoughtful about HW choices but leave it open which I think is the best part!

Oculus 12y ago
I've seen their hard drive reliability blog posts, but I never knew they were so open about their storage server designs. I think it's really cool how open they are about something that is definitely a competitive advantage.
- aftbit 12y ago
  The real competitive advantage is the software and IT principles that make managing a dumb array of disks possible.
  - falsestprophet 12y ago
    Is it though? Running OpenStack Swift on these pods would get you pretty far. http://docs.openstack.org/developer/swift/ http://docs.openstack.org/developer/swift/
    - caraboga 12y ago
      Having inherited an openstack install with the swift implementation implemented on earlier generation of pods, these are the issues I and my cohorts ran into: 1) The object replication between the pods will be hard on the array, as the object replication is simply an rsync wrapped inside nested for loops. If you have a bunch of small files with a lot of tenants, it'll hurt. 2) While swift allows you to simply unmount a bad disk, change around the ring files and let replication do its thing, there are actual issues. First of all, out of band smart monitoring of bad sectors actually cases the disk to pre-empt some sata commands and do the smart checks first. On a heavily loaded cluster, under a smart check for a bad block count could kill the drive, and take out the sata controller along with it. We've downed storage pods before by that way. The only way we got around it was to take a pod offline once a week, run all our smart checks, then put it back. 3) To replace a drive, you have to open the machine up and power it off. As any old operator will tell you, drives that have been running for awhile do not like to be turned off. If you are to power off a machine to replace a bad drive, do realize that you might actually break more drives just from the power cycle. 4) Once you change a drive out and use the ring replication of files to rebuild your storage pods, your entire storage cluster will take a non-trivial hit. 5) Last but not least, it is almost of tantamount importance to move the proxy, account and container services away from the hardware that also host the object servers. It's probably good to note that the account and container meta data is stored in sqlite files, with fsync writes. If you add and remove a bunch of files to multiple accounts, the container service is going to get hit first, then the object service. Further more, every single transaction to every meta data/data service, including replication, is federated through the proxy servers. If you look at a swift cluster, the proxy services take up a large chunk of the processing space. Source: Was Openstack admin for a research group in Middle Tennessee. Ran an openstack cluster with 5 gen1.5 pods for the entire swift service, then moved account/container/proxy to three 12 core 2630s with 64 gigs of ram. Cluster was for a DARPA vehicular design project, first part fielded 3000 clients, second part fielded about 1/10 of that, but with more files and bigger files (these were cad and test results respectively).
      - rdkls 12y ago
        Thanks caraboga. I was contemplating this setup a year or so ago, and great to learn your insights from actual implementation. It looked to me like Supermicro JBOD enclosures would be superior (mainly hot swappability) if a bit more expensive, would you agree?
        caraboga 12y ago
        My predecessor went with Backblaze clones for the drive and mb enclosure. You'll run into issues with the backblaze way as they have two power supplies, but one is for the motherboard and the boot drives, and the other power supply is for the actual drives. Further more, as this was a backblaze pod clone, there's no ipmi on the pod motherboards. It makes certain things a bit more annoying than they have to be. If I were to do it again, I would stay away from using enclosures that were inspired by the BackBlaze models. Supermicro enclosures are fine.
      - rincebrain 12y ago
        I'd be curious to know what drives and controllers you had that pre-empted requests due to SMART queries...
        caraboga 12y ago
        I don't recall the model, but they were 3 terabyte Seagates that were bought 3 years ago. I think my predecessor employed the same controller cards as version 1 of the pods. You could tell that the smart queries pre-empted normal io as the smart query would be executed, and disk activity for certain swift object servers would just stall. The object-servers would not return requests for at least some of the drives until the smart query was finished. Curiously enough, I've also ran smartctl against Samsung 840Pros using Highpoint RocketRaid controller cards during the same project. Sometimes this crashes the controller card. (This was for a gluster cluster, when gluster had broken quorum support. A copy of a piece of data, when out of sync with the other copies in a replica set, would be left unchecked. This was produced after one smartctl command took out the controller card, and then the machine along with it.)
- brianwski 12y ago
  Brian from Backblaze here. > open they are about something that is definitely a competitive advantage. From our perspective, our biggest competitor is apathy. Something like 90 percent of people simply don't backup their computers. So pretty much anything we can do to have people hear about Backblaze and possibly overcome their apathy and start backing up their laptops is good for us. These blog posts have worked very well for us - people enjoy the information, and by disclosing the info we have potential customers hear our name and we get a bump in subscriptions. It's a win-win, the BEST kind of win. :-)
atYevP 12y ago
Yev from Backblaze here -> Glad you're enjoying the posts! We enjoy spreading that information, it's a rare thing, so we're pleased as punch that folks are reading them :)
- antr 12y ago
  Hi Yev, do you know if Backblaze is thinking of supporting NAS backups in the near future? I really want to use Backblaze, but what is really stopping me from using you guys is not being able to backup my NAS. Thanks
  - brianwski 12y ago
    Brian from Backblaze here. The NAS problem isn't a technical one, it is a pricing problem. We charge a fixed amount of money "per computer" ($5) and we don't throttle or limit customers in any way. We survive on "averages" - most laptop users only have less than 1 TByte and we make money, some have 50 TBytes and we lose money, but on average we are profitable. If we allowed a single user for $5 to mount network drives with Petabytes of data on them and back those up, this would tip the averages and put us out of business. So we need to solve the pricing problem. With all that said, stay tuned, we are actively trying to figure out a pricing system that will make everybody happy (NAS owners with Petabytes of data included).
    - antr 12y ago
      I totally understand, and respect that. I'm sure there are ways (price plans) to get NAS owners to signup. I might have a NAS, but defiantly don't have petabytes of data, but on average 1TB to 1.5TB on my NAS. I'd be happy to pay $2-4 more dollars more if I could include my NAS... this is my thinking out loud, I haven't run any numbers.
    - e12e 12y ago
      What I'd like to see (been waiting for, for a long time, actually) is "enterprise" or "pro" backblaze, more along the lines of rsync.net or similar. Something that can (willingly) handle backup from servers (at 1Gbs speed). In other words an Amazon S3/glacier-like product -- but with more transparent pricing. I'd be happy to pay a premium (way over the $5 per computer) for something like that -- I'm sure I'm not alone. Extra if the software was released under the AGPL or something -- so that I could migrate to self-host if I wanted/needed to. I don't think too many would be able to start up and compete directly with you wrt your experience with pods etc... so I think it should be viable. Less sure if anyone would "want" to move off Amazon.

Oculus 12y ago

I've seen their hard drive reliability blog posts, but I never knew they were so open about their storage server designs. I think it's really cool how open they are about something that is definitely a competitive advantage.

aftbit 12y ago
The real competitive advantage is the software and IT principles that make managing a dumb array of disks possible.
- falsestprophet 12y ago
  Is it though? Running OpenStack Swift on these pods would get you pretty far. http://docs.openstack.org/developer/swift/ http://docs.openstack.org/developer/swift/
  - caraboga 12y ago
    Having inherited an openstack install with the swift implementation implemented on earlier generation of pods, these are the issues I and my cohorts ran into: 1) The object replication between the pods will be hard on the array, as the object replication is simply an rsync wrapped inside nested for loops. If you have a bunch of small files with a lot of tenants, it'll hurt. 2) While swift allows you to simply unmount a bad disk, change around the ring files and let replication do its thing, there are actual issues. First of all, out of band smart monitoring of bad sectors actually cases the disk to pre-empt some sata commands and do the smart checks first. On a heavily loaded cluster, under a smart check for a bad block count could kill the drive, and take out the sata controller along with it. We've downed storage pods before by that way. The only way we got around it was to take a pod offline once a week, run all our smart checks, then put it back. 3) To replace a drive, you have to open the machine up and power it off. As any old operator will tell you, drives that have been running for awhile do not like to be turned off. If you are to power off a machine to replace a bad drive, do realize that you might actually break more drives just from the power cycle. 4) Once you change a drive out and use the ring replication of files to rebuild your storage pods, your entire storage cluster will take a non-trivial hit. 5) Last but not least, it is almost of tantamount importance to move the proxy, account and container services away from the hardware that also host the object servers. It's probably good to note that the account and container meta data is stored in sqlite files, with fsync writes. If you add and remove a bunch of files to multiple accounts, the container service is going to get hit first, then the object service. Further more, every single transaction to every meta data/data service, including replication, is federated through the proxy servers. If you look at a swift cluster, the proxy services take up a large chunk of the processing space. Source: Was Openstack admin for a research group in Middle Tennessee. Ran an openstack cluster with 5 gen1.5 pods for the entire swift service, then moved account/container/proxy to three 12 core 2630s with 64 gigs of ram. Cluster was for a DARPA vehicular design project, first part fielded 3000 clients, second part fielded about 1/10 of that, but with more files and bigger files (these were cad and test results respectively).
    - rdkls 12y ago
      Thanks caraboga. I was contemplating this setup a year or so ago, and great to learn your insights from actual implementation. It looked to me like Supermicro JBOD enclosures would be superior (mainly hot swappability) if a bit more expensive, would you agree?
      - caraboga 12y ago
        My predecessor went with Backblaze clones for the drive and mb enclosure. You'll run into issues with the backblaze way as they have two power supplies, but one is for the motherboard and the boot drives, and the other power supply is for the actual drives. Further more, as this was a backblaze pod clone, there's no ipmi on the pod motherboards. It makes certain things a bit more annoying than they have to be. If I were to do it again, I would stay away from using enclosures that were inspired by the BackBlaze models. Supermicro enclosures are fine.
    - rincebrain 12y ago
      I'd be curious to know what drives and controllers you had that pre-empted requests due to SMART queries...
      - caraboga 12y ago
        I don't recall the model, but they were 3 terabyte Seagates that were bought 3 years ago. I think my predecessor employed the same controller cards as version 1 of the pods. You could tell that the smart queries pre-empted normal io as the smart query would be executed, and disk activity for certain swift object servers would just stall. The object-servers would not return requests for at least some of the drives until the smart query was finished. Curiously enough, I've also ran smartctl against Samsung 840Pros using Highpoint RocketRaid controller cards during the same project. Sometimes this crashes the controller card. (This was for a gluster cluster, when gluster had broken quorum support. A copy of a piece of data, when out of sync with the other copies in a replica set, would be left unchecked. This was produced after one smartctl command took out the controller card, and then the machine along with it.)
brianwski 12y ago
Brian from Backblaze here. > open they are about something that is definitely a competitive advantage. From our perspective, our biggest competitor is apathy. Something like 90 percent of people simply don't backup their computers. So pretty much anything we can do to have people hear about Backblaze and possibly overcome their apathy and start backing up their laptops is good for us. These blog posts have worked very well for us - people enjoy the information, and by disclosing the info we have potential customers hear our name and we get a bump in subscriptions. It's a win-win, the BEST kind of win. :-)

atYevP 12y ago

Yev from Backblaze here -> Glad you're enjoying the posts! We enjoy spreading that information, it's a rare thing, so we're pleased as punch that folks are reading them :)

antr 12y ago
Hi Yev, do you know if Backblaze is thinking of supporting NAS backups in the near future? I really want to use Backblaze, but what is really stopping me from using you guys is not being able to backup my NAS. Thanks
- brianwski 12y ago
  Brian from Backblaze here. The NAS problem isn't a technical one, it is a pricing problem. We charge a fixed amount of money "per computer" ($5) and we don't throttle or limit customers in any way. We survive on "averages" - most laptop users only have less than 1 TByte and we make money, some have 50 TBytes and we lose money, but on average we are profitable. If we allowed a single user for $5 to mount network drives with Petabytes of data on them and back those up, this would tip the averages and put us out of business. So we need to solve the pricing problem. With all that said, stay tuned, we are actively trying to figure out a pricing system that will make everybody happy (NAS owners with Petabytes of data included).
  - antr 12y ago
    I totally understand, and respect that. I'm sure there are ways (price plans) to get NAS owners to signup. I might have a NAS, but defiantly don't have petabytes of data, but on average 1TB to 1.5TB on my NAS. I'd be happy to pay $2-4 more dollars more if I could include my NAS... this is my thinking out loud, I haven't run any numbers.
  - e12e 12y ago
    What I'd like to see (been waiting for, for a long time, actually) is "enterprise" or "pro" backblaze, more along the lines of rsync.net or similar. Something that can (willingly) handle backup from servers (at 1Gbs speed). In other words an Amazon S3/glacier-like product -- but with more transparent pricing. I'd be happy to pay a premium (way over the $5 per computer) for something like that -- I'm sure I'm not alone. Extra if the software was released under the AGPL or something -- so that I could migrate to self-host if I wanted/needed to. I don't think too many would be able to start up and compete directly with you wrt your experience with pods etc... so I think it should be viable. Less sure if anyone would "want" to move off Amazon.