6 ms·

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81

by edw519 17y ago

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this: Raw Drives $81,000 Backblaze $117,000 Dell $826,000 Sun $1,000,000 NetApp $1,714,000 Amazon $2,806,000 EMC $2,860,000 I had no idea. Kinda makes one rethink what business they want to be in.

spydez 17y ago
Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.
- blasdel 17y ago
  No, they aren't going Google's route! Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters. Backblaze's system is going to have several orders of magnitude less bandwidth: way too many drives in one server (some of them on the PCI bus!) use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s) RAID6 with too many drives per array (15-way XOR is no fun) why use JFS? only access is via HTTPS, not clear if SSL is done on the pod
  - cheriot 17y ago
    I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.
cperciva 17y ago
Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption. Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.
- skolor 17y ago
  Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people. 99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swappable, and will not cause downtime when they are replaced and B) the rest of the components fail once a year (or less) and take ~10 minutes to replace and reboot the system. With 4 main places of failure (PSU, Boot Drive, Motherboard/Ram/CPU, Drive controllers), as long as you have staff constantly on call and they can respond within 5 minutes of a failure, 99.99% uptime seems reasonable. I don't know where you got the idea that your data was on 4 different servers when using S3. I can't find even the slightest amount of information on that. Yes, that would be nice, and it is cool to think that, but its rather doubtful that they're actually doing that (or they could probably add another 9 to their uptime).
  - ShabbyDoo 17y ago
    >but that isn't counting a complete datacenter failure I believe that S3 replicates to multiple physical locations. So, while you might experience downtime, you probably won't experience data loss.
johnrob 17y ago
This is not apples vs apples. Some of those items are living cows, and some are cooked steaks served with a side of mashed potatoes. I'm sure the cow looks cheap (per pound) until you are tasked with turning it into a steak.
mmt 17y ago
As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time. The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart. I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough demand. Obviously, this is starting to change. I am, however, startled that a driveless pod is so expensive ($2467 or $54.82 per drive), considering that external performance is, effectively, limited by gigabit Ethernet. Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. Add another $1k for mobo and cables, and that's $11300 for 240 drives or $47.08 per drive, without having to do a custom case. Granted, it takes up almost twice the rack space, but it's still a decent power density at 50W/U AC (assuming 6.5W per drive DC). It takes 52 groups of 13 1.5TB disks (Hmm.. 13 pops up an awful lot in their setup) to make a PB using their pods. 51 groups means 17 pods is .9945TB for $134k ($134.5/PB). Using my pods, it would be .936TB for $120.3k ($128.5/PB), but what's $6k/PB between friends?
- lsc 17y ago
  you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider. Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system isn't done yet, but I've got 6 disks out there waiting for me to figure it out.
  - mmt 17y ago
    Hmm.. looks suspiciously like exactly the SuperMicro case, for about twice the price of rolling ones own.
    - lsc 17y ago
      from what I understand the coraid systems are plan9-based, and only have a gigabyte of cache ram. If I only needed one I'd seriously consider it, as my time, for setting up one unit and working out the kinks, would unquestionably be more than that, and they have quite a bit of experience working out the problems by now, even if I made my system redundant, their system would unquestionably be more reliable the first iteration. However, I plan on needing a bunch more, so it probably makes sense for me to roll my own, work out the kinks, and hopefully end up with something cheaper that has a lot more cache.

by edw519 17y ago

spydez 17y ago

Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc. They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.

blasdel 17y ago
No, they aren't going Google's route! Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters. Backblaze's system is going to have several orders of magnitude less bandwidth: way too many drives in one server (some of them on the PCI bus!) use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s) RAID6 with too many drives per array (15-way XOR is no fun) why use JFS? only access is via HTTPS, not clear if SSL is done on the pod
- cheriot 17y ago
  I'm guessing that backups are accessed rather infrequently and that someone recovering their data won't mind a few seconds in lag time. This isn't my area of expertise, but their trade off seems sensible. It's just a different scenario than what google is dealing with.

cperciva 17y ago

Backblaze $117,000 [...] Amazon $2,806,000 I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption. Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.

skolor 17y ago
Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people. 99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swappable, and will not cause downtime when they are replaced and B) the rest of the components fail once a year (or less) and take ~10 minutes to replace and reboot the system. With 4 main places of failure (PSU, Boot Drive, Motherboard/Ram/CPU, Drive controllers), as long as you have staff constantly on call and they can respond within 5 minutes of a failure, 99.99% uptime seems reasonable. I don't know where you got the idea that your data was on 4 different servers when using S3. I can't find even the slightest amount of information on that. Yes, that would be nice, and it is cool to think that, but its rather doubtful that they're actually doing that (or they could probably add another 9 to their uptime).
- ShabbyDoo 17y ago
  >but that isn't counting a complete datacenter failure I believe that S3 replicates to multiple physical locations. So, while you might experience downtime, you probably won't experience data loss.

johnrob 17y ago

This is not apples vs apples. Some of those items are living cows, and some are cooked steaks served with a side of mashed potatoes. I'm sure the cow looks cheap (per pound) until you are tasked with turning it into a steak.

mmt 17y ago

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time. The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart. I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough demand. Obviously, this is starting to change. I am, however, startled that a driveless pod is so expensive ($2467 or $54.82 per drive), considering that external performance is, effectively, limited by gigabit Ethernet. Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. Add another $1k for mobo and cables, and that's $11300 for 240 drives or $47.08 per drive, without having to do a custom case. Granted, it takes up almost twice the rack space, but it's still a decent power density at 50W/U AC (assuming 6.5W per drive DC). It takes 52 groups of 13 1.5TB disks (Hmm.. 13 pops up an awful lot in their setup) to make a PB using their pods. 51 groups means 17 pods is .9945TB for $134k ($134.5/PB). Using my pods, it would be .936TB for $120.3k ($128.5/PB), but what's $6k/PB between friends?

lsc 17y ago
you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider. Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system isn't done yet, but I've got 6 disks out there waiting for me to figure it out.
- mmt 17y ago
  Hmm.. looks suspiciously like exactly the SuperMicro case, for about twice the price of rolling ones own.
  - lsc 17y ago
    from what I understand the coraid systems are plan9-based, and only have a gigabyte of cache ram. If I only needed one I'd seriously consider it, as my time, for setting up one unit and working out the kinks, would unquestionably be more than that, and they have quite a bit of experience working out the problems by now, even if I made my system redundant, their system would unquestionably be more reliable the first iteration. However, I plan on needing a bunch more, so it probably makes sense for me to roll my own, work out the kinks, and hopefully end up with something cheaper that has a lot more cache.