9 ms·

This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?

by dmillar 17y ago

PStamatiou 17y ago
Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on the 1.5TB drives that would cause random corruption? (heard about it maybe 6 months ago)
- barrkel 17y ago
  I should think so - I have 4 of them in my ZFS pool, and scrubs are turning up no errors.
- sophacles 17y ago
  I have the 500GB in the same series (with the corrupted firmware) and the fix did very nice things for it. Also, I have never had problems with the 7200.12 (the newer version of the same drive) in a 1TB model.
- jbellis 17y ago
  > While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.
  - PStamatiou 17y ago
    I would have some smart-ass comment here but I just read your HN profile. Good day, Sir.
    - tzury 17y ago
      smart move ;-)
  - Periodic 17y ago
    This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of design over many systems, but for smaller companies it just isn't feasible. We hacker types love to think that we could do the same thing in no time with little budget, and I'm sure we could get a first approximation. But the devil is in the details. Debugging the complex interaction of 20 different hardware components is not my idea of fun. Hats off to them, particularly for sharing.
    - rantfoil 17y ago
      Secondary revenue streams anyone? I know we'd be interested in becoming a customer if Backblaze at some point decided to monetize this part of their knowledge too. This seems like an opportunity to disrupt from the low end of the market for some scrappy enterprising engineers. If they don't do it, someone should.
    - uhgygghhj 17y ago
      I did this as an inhouse backup for a data warehousing app. Just slapping 4 ide cards into a case and putting 16x250gb IDE drives on them resulted in a system that would copy about 1 disk worth before hanging with some fault or suddenly dropping to 1% speed. Just because you can in theory hook 40 drives to n cards doesn't mean it will work - well done to them
    - mmt 17y ago
      I don't think it takes quite as much effort as you imply, since a fairly basic storage array from one of the major vendors is $100k. (And then there's "maintenance" and software upgrades, but that's a separate matter). Even if you got a $100/hr consultant, $100k would be half a year. I suggest it could be done in half that time, by someone who has suitable background. Someone with the specific expertise should be able to do it in 6 weeks elapsed and 2 weeks of full time work, tops. (bt; dt)
- bestes 17y ago
  Yes, they did a firmware update. Here is one article that talks about it: http://www.tomshardware.com/news/seagate-barracuda-1.5TB-freezing,6582.html http://www.tomshardware.com/news/seagate-barracuda-1.5TB-fre...
  - herf 17y ago
    Seagate took about 4 months to fully resolve the issue (their first fix had a bug!) I had a drive that bricked when applying the fix several months after the initial firmware updates. Apparently a buffer overflow in the error log that locked the drive on reboot. Seems dangerous to source all your hardware from one vendor over a small timeframe. Would be nice to have some redundancy over manufacturers too.
- lsc 17y ago
  as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows you to change the behavior, search for WDTLER, but in general. I'm sure the backblaze folks have built something that handles that sort of thing into the system, but just saying; it's something you have to think about before you slap consumer drives into a raid that needs to keep working when one drive fails.
  - Devilboy 17y ago
    The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.
    - lsc 17y ago
      But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.
      - Devilboy 17y ago
        In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?

This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?

by dmillar 17y ago

This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?

Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh That being said - did seagate ever fix their firmware issue on the 1.5TB drives that would cause random corruption? (heard about it maybe 6 months ago)

barrkel 17y ago
I should think so - I have 4 of them in my ZFS pool, and scrubs are turning up no errors.
sophacles 17y ago
I have the 500GB in the same series (with the corrupted firmware) and the fix did very nice things for it. Also, I have never had problems with the 7200.12 (the newer version of the same drive) in a 1TB model.
jbellis 17y ago
> While technology isn't more than any typical hacker can slap together in their apartment I think you missed the part about testing a dozen SATA cards, etc. The attention to detail here is a lot more than something you'd slap together in your apartment.
- PStamatiou 17y ago
  I would have some smart-ass comment here but I just read your HN profile. Good day, Sir.
  - tzury 17y ago
    smart move ;-)
- Periodic 17y ago
  This is what always gets in my way. It takes a lot of work and a lot of expertise to put together a home-grown system that works as well as one from the major vendors. If you're only going to be using one or two, you're much better off going to one of those major vendors because a large part of the price is their expertise and testing that went into it. For a large setup like Backblaze they can spread the cost of design over many systems, but for smaller companies it just isn't feasible. We hacker types love to think that we could do the same thing in no time with little budget, and I'm sure we could get a first approximation. But the devil is in the details. Debugging the complex interaction of 20 different hardware components is not my idea of fun. Hats off to them, particularly for sharing.
  - rantfoil 17y ago
    Secondary revenue streams anyone? I know we'd be interested in becoming a customer if Backblaze at some point decided to monetize this part of their knowledge too. This seems like an opportunity to disrupt from the low end of the market for some scrappy enterprising engineers. If they don't do it, someone should.
  - uhgygghhj 17y ago
    I did this as an inhouse backup for a data warehousing app. Just slapping 4 ide cards into a case and putting 16x250gb IDE drives on them resulted in a system that would copy about 1 disk worth before hanging with some fault or suddenly dropping to 1% speed. Just because you can in theory hook 40 drives to n cards doesn't mean it will work - well done to them
  - mmt 17y ago
    I don't think it takes quite as much effort as you imply, since a fairly basic storage array from one of the major vendors is $100k. (And then there's "maintenance" and software upgrades, but that's a separate matter). Even if you got a $100/hr consultant, $100k would be half a year. I suggest it could be done in half that time, by someone who has suitable background. Someone with the specific expertise should be able to do it in 6 weeks elapsed and 2 weeks of full time work, tops. (bt; dt)
bestes 17y ago
Yes, they did a firmware update. Here is one article that talks about it: http://www.tomshardware.com/news/seagate-barracuda-1.5TB-freezing,6582.html http://www.tomshardware.com/news/seagate-barracuda-1.5TB-fre...
- herf 17y ago
  Seagate took about 4 months to fully resolve the issue (their first fix had a bug!) I had a drive that bricked when applying the fix several months after the initial firmware updates. Apparently a buffer overflow in the error log that locked the drive on reboot. Seems dangerous to source all your hardware from one vendor over a small timeframe. Would be nice to have some redundancy over manufacturers too.
lsc 17y ago
as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows you to change the behavior, search for WDTLER, but in general. I'm sure the backblaze folks have built something that handles that sort of thing into the system, but just saying; it's something you have to think about before you slap consumer drives into a raid that needs to keep working when one drive fails.
- Devilboy 17y ago
  The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.
  - lsc 17y ago
    But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.
    - Devilboy 17y ago
      In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?