10 ms·

Our 6 TB Hard Drive Face-Off

204 points by ehPReth 12y ago

EwanG 12y ago
For those not willing to read a full page, in the end run it is more an energy decision than a reliability decision. And they mention at the end that they are already looking into the 8TB Helium and 8TB SMR drives.
- pbreit 12y ago
  My sense was that the 20% speed advantage was as important or more than the small energy savings. Is that common for slower drives to handle data faster?
  - pfg 12y ago
    Maybe it's a side-effect of increased vibration for 7.2k drives? Hard drive performance drops in environments with high vibration. I could imagine that being a factor when you put 45 drives in a single server.
  - kijin 12y ago
    Anecdotal evidence, but I often find WD drives to be faster than their Seagate cousins, even at lower rotation speeds. I wonder if this is because of better firmware optimizations or because of higher platter density. Maybe it's both. Or maybe WD lies about their rpms ;) Hitachi/HGST, now owned by WD, also makes really fast drives. Their 7200rpm 2.5" drives are the fastest laptop drives I've ever used, except SSDs.
  - miahi 12y ago
    The WD they use is a 5-platter HDD (1.2TBx5). The Seagate has 6 platters (1TBx6). The higher density means that the linear reads/writes are faster at the same spindle speed (more data passes under the heads in the same time interval). The interesting thing is that they are faster even at different speeds.
fubarred 12y ago
Backblaze is pretty awesome for open-sourcing, effectively, their costs in a high-competition segment. It would be difficult (read: pointless) to compete in that market without massive vertical integration, greater scale, deep pockets and compelling, differentiated value (eg something different enough besides all the other entrants). For anything critical in business (customer list, billing, finance, etc.), I would recommend also using Tarsnap in addition to whatever other (non-AWS) vendor/s are used for "offsite"/"offcloud" backup.
java-man 12y ago
Firefox: Unable to Connect Securely Firefox cannot guarantee the safety of your data on www.backblaze.com because it uses SSLv3, a broken security protocol. Advanced info: ssl_error_no_cypher_overlap can we trust them with our data?
- warp 12y ago
  Peculiar. According to https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze... they only support TLS 1.0, not SSLv3. (Though I'm still surprised by how many issues SSLLabs identifies on that site... )
  - Someone1234 12y ago
    They get a "B," Microsoft gets a "B," Google gets a "B," Bank Of America get an "F," and so on. Nothing unusual about their results.
    - brianwski 12y ago
      Backblaze employee here: we try to stay on top of these annoying POODLE and other issues. One thing to realize is we encrypt all the data on the client and THEN push it through HTTPS. So many of the security issues aren't honestly affecting our customer privacy. But we stay on top of the security audits because it's a good idea and to even avoid this conversation or the questions that arise.
- theandrewbailey 12y ago
  I thought it was just me (I've disabled some older cipher suites in about:config). Turns out this site only supports broken ciphers (RC4 and 3DES). Ewwww. https://www.ssllabs.com/ssltest/analyze.html?d=backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=backblaze.com
  - yuhong 12y ago
    3DES is not broken but it is legacy.
Bluestrike2 12y ago
The article mentions having to buy 10k+ new drives in the next few months? Is that number just your average increase for new drives and/or replacements for failed drives? Or does it include planned replacement of older pods with smaller drives? I'm curious if there's a point where you've found it less expensive (or less problematic with drive failure rates per pod) to replace and upgrade older pods compared to just letting them continue to run.
- bluedino 12y ago
  They used to say they don't spend any money on drive replacements (except for labor, obviously) because of the drives having 3-year warranties. Those were using 1.5TB drives. However, the original 67TB pods came out in 2009 and wouldn't be warrantied anymore. According to their blog they took until 2011 to hit 10PB, which would be roughly 150 of the original 67TB pods. The 6TB drives are 4 times better, so they'd be able to replace all of those with just 40 of the new pods, I'm sure the savings in electricity and rack space would be too huge to pass up. Then again, they have their own datacenter space now, so space probably isn't an issue but I'm sure electricity is. They said something around $2,100 a rack (which holds 10 pods) for electricity, space, and data. They were incredibly stoked to cut that in half by moving to 3TB drives with the 2.0 version of the pod so I'm sure cutting that in half excited them as well :)
  - brianwski 12y ago
    Backblaze employee here. Some of the drives are "voided warranty" now, like the ones we "shuck" out of USB enclosures. We just include that into our cost analysis - for example if 5 percent of the drives fail, the warranty saves you 5 percent of cost - so if purchasing the warranty costs 3 percent we do it, but if purchasing the warranty costs 7 percent we do without.
  - brianwski 12y ago
    Also, we still pay "per rack" but prices are now a fraction of the $2,100 (I'm not sure if I'm allowed to say exact numbers).
- 12y ago
justcommenting 12y ago
the critical assumption in this post seems to be that drive speed was the only rate-limiting factor when they were collecting data... but as i understood the post, it seemed like perimeter load-balancing, sata controller/nic load, and/or customer usage patterns could be responsible for some of the differences they observed.
- lsc 12y ago
  You have to screw up a storage system pretty good to have something other than the hard drives be the limiting factor.
  - asdfaoeu 12y ago
    True but that doesn't imply you run at capacity all the time.
  - justcommenting 12y ago
    this is generally true, but even small variations in other factors could propagate through a complex architecture to account for some of the differences presented here, especially over time. these data are still useful, but some of the caveats could be made more explicit. similarly, most drive benchmarks are 'artificial' in any number of ways and come with another set of caveats. since drive testing is already part of backblaze's process for putting drives into production, it could be especially interesting to compare and contrast performance data from testing with data from production use. i pick these nits because real-world storage performance info seems like a valuable public good (particularly for enterprise and enterprise-like usage of consumer drives), and i'm grateful to backblaze for offering it up.
  - SamReidHughes 12y ago
    It's pretty easy to have software to be the limiting factor, such that you aren't taking full advantage of your throughput because the data's not there yet.
- Tepix 12y ago
  Exactly. I found this particular blog post weaker than their previous, highly interesting ones.
lsc 12y ago
hm. I wonder if their data access patterns are largely sequential; I imagine the 7200rpm drives would beat the lower-RPM drives in an environment where random access matters.
- wmf 12y ago
  It's a backup company. Write one, read never.
- justcommenting 12y ago
  i have similarly wondered how important drive cache size might be for these types of use cases. multivariate regressions including brand, cache size, rpm etc. as predictors of various performance metrics would be super interesting in a place like blackblaze.
  - lsc 12y ago
    Cache gets you very little if you can feed in sequential data at a constant rate and at the blocksize of the device. Cache becomes more important as the input data becomes more messy; cache is one tool for turning bursty and random data into constant, sequential data.
rwmj 12y ago
I wonder if anyone has real world experience of Shingled Magnetic Recording (SMR) drives? I saw a talk by Red Hat's Ric Wheeler about a year ago, and they seem both interesting and pretty strange - as in, we may need new filesystems to cope with them.
- brianwski 12y ago
  Backblaze employee here: we will run our first experiment with SMR soon! We're SUPER curious about whether they will work in our application.
benguild 12y ago
It’s really great that BackBlaze does these real world, mass comparisons. Otherwise it’s just limited to what a single test says, often at a single time, with a single batch of drives from various manufacturers. These guys are probably one of the only qualified places to do this at scale and report real world results. Glad they do!
- emsy 12y ago
  I often think of it as open sourcing a company. They give up a competitional advantage but probably gain as much or more in return (reactions on their article as "pull requests" for instance). I don't want to sound hyperbolic but I think it also advances humanity (The information and the spirit behind it). Other companies do talks and papers too, but the Backblaze info seems really unfiltered and close to the real world.
  - Ironchefpython 12y ago
    > the Backblaze info seems really unfiltered and close to the real world. Cloudflare seems to be a similar company. I recommend them both to anyone somewhat technical who needs a CDN. My only complaint about Backblaze is no Unix client. I thought about setting up a Windows VM to manage backups, but I was terrified that Gninjas would sneak into my house at night and shave my beard.
    - akgerber 12y ago
      They have a Unix client— probably just not the Unix you use (OS X).
tzz 12y ago
Backblaze should expand their service to a cheaper cloud storage service similar to Amazon S3. They already have the infrastructure and the know-how.
- venomsnake 12y ago
  Backblaze tech is not that impressive to give them competitive edge over the economy of scale of the big boys have. Amazon & co are not selling at cost, but at whatever the market with bear.
- dsl 12y ago
  The business model is based on only rarely having to retrieve old data. I suspect they spin down the drives (or possibly hibernate the whole box) when not in active "load" mode. You would need to compare the pricing here to Amazon Glacier, if anything.
  - brianwski 12y ago
    Backblaze employee here. Our drives never spin down, although we have considered this because electricity is a large monthly bill to us. Because it is "backup" we are pretty relaxed about shutting down our pods to do maintenance, like replacing a drive. But that only takes 10 minutes, we try not to have machines offline more than that if possible, or it causes support issues from people trying to restore files.
    - rbanffy 12y ago
      Don't the writing patterns make old data tend to remain in older drives and newer data ending up in the newer devices? If that's the case, would reads to those devices become increasingly infrequent to the point it would be worth to spin them down? Are you worried about the stresses induced by spin up/down and thermal variations?
      - brianwski 12y ago
        In general that's my theory also (that after the initial fill, and maybe after an initial "data churn" we might power them down). However, a full pod often has at least one file from several hundred thousand individual customers. If any one of those customers prepares a restore, those drives have to be spinning. Also, we now have an iOS and Android app that you have access to all your files on your laptop from the convenience of your Smartphone - which again means the drives have to be spinning. > Are you worried about the stresses induced by spin up/down and thermal variations? Absolutely. Our datacenter techs are convinced that if they swap a drive QUICKLY the pod has a higher chance of coming back up without any problems, if they let the pod get entirely cool it statistically seems to have more problems when it comes back up - a full cool down and heat up seems to cause issues.
ck2 12y ago
Helium filled drives are literally a dead-end as their lifespan can be shorter than SSD
- ghshephard 12y ago
  Re: "Helium filled drives are a dead end" - Citation? Also, SSDs come with 10 year warranties [1], so, which exceeds anything you can get with a normal spinning HD. If anything, I'd have more confidence with my SSD than I would with a spinning HD - particularly as you don't have to worry about the drives failing to spin up over time. [1] http://www.amazon.com/SanDisk-Extreme-2-5-Inch-Warranty--SDSSDXPS-240G-G25/dp/B00KHRYRNM/ http://www.amazon.com/SanDisk-Extreme-2-5-Inch-Warranty--SDS...
  - dsl 12y ago
    One legitimate argument against helium drives is they contribute to the depletion of the relatively scarce helium supply. http://www.boston.com/news/science/articles/2010/10/17/scientists_warn_worlds_supply_of_helium_close_to_depletion/ http://www.boston.com/news/science/articles/2010/10/17/scien...
    - Smegger 12y ago
      Helium wasted in hard drives though pale in comparison to that used in party balloons.
    - ghshephard 12y ago
      That article reads a lot like "Peak Oil" concerns. There is a lot of Helium available - [1] The global reserves of helium are known to be approximately 41 billion cubic meters. Most of them lie in Qatar, Algeria, the USA and Russia. Annual global production of helium is about 175 million cubic meters, and the USA remains the largest producer. " Regardless, this is something the market can correct for very, very easily. As helium supply becomes more scarce, the price will go up, resulting in greater supply. Most of it comes from natural gas, and is so cheap [2] it's not worth capturing. Oil trades for around $100/barrel. Helium trades at $100 for a thousand cubic feet (albeit in gas form) Some of the problems with Helium is that Physics experiments use a LOT, and previously was so cheap that it wasn't worth trying to conserve. That's changing - [3] A recycling system can recapture about $12,000/year of lost helium for a single scientist. From reading articles - apparently the problems isn't so much that the cost of helium is increasing - but that it's been so cheap because of the US Natural Reserves making it completely non-competitive to capture - they are basically giving the stuff away for next to nothing. And, putting it in a $100+ Hard Drive that will last a half decade is a FAR better use than putting 10x that much in a $0.50/balloon that will last 30 minutes. (Or Macy's parade which uses 400,000 cubic feet of helium) In fact, it may turn out to be the best conceivable use of Helium in terms of a value per m^3 equation. [1] http://www.gazprominfo.com/articles/helium/ http://www.gazprominfo.com/articles/helium/ [2] http://finance.yahoo.com/news/airgas-increase-prices-helium-20-164550580.html http://finance.yahoo.com/news/airgas-increase-prices-helium-... And on Friday, the bureau announced that it was raising the price for a thousand cubic feet for crude helium from $84 to $95 [3] http://www.nature.com/news/united-states-extends-life-of-helium-reserve-1.13819 http://www.nature.com/news/united-states-extends-life-of-hel...
unicornporn 12y ago
I'm so looking forward to the Seagate 8TB Archive HDD. Finally a drive that will hold all my photos and quite a bit more. It will make my backup procedure a whole lot easier.
- medecau 12y ago
  It's also a single basket for all your eggs.
  - ghshephard 12y ago
    As the original poster noted, sometimes it's far better to have a single basket for your eggs, that you keep well backed up, than it is to have multiple baskets, none of which are backed up.
  - deleted 12y ago
    [deleted]
- pjc50 12y ago
  Related ask HN to you and others: what software are you using to backup/sync to that drive? (What would you use on Windows?)
  - flyinghamster 12y ago
    I can't speak for Windows, but the setup where I work involves four drives (currently 2 TB each), with three running as a triple-mirror ZFS pool. For backup, I split a drive off the pool, pull it, slide in the previous backup, and resilver the newly-inserted drive. Each time, I use a different drive bay. The ZFS pool itself provides backup for several other systems, and snapshots are made periodically by a cron job to provide a further line of defense.
  - arm 12y ago
    I use OS X on my main computer, so I haven’t used this personally, but Bvckup 2 was mentioned on HN in the past: https://news.ycombinator.com/item?id=8027405 https://news.ycombinator.com/item?id=8027405
SideburnsOfDoom 12y ago
No failures recorded at all. This is pretty impressive for the drive manufacturers: "Let’s review the Seagate and Western Digital drives so far: Initial reliability (how many drives failed) – No failures. Running reliability (3 months) – No failures SMART Stats (3 months) – No error conditions recorded for the 5 stats that we utilize."
- Cthulhu_ 12y ago
  Of note is that since Backblaze is a backup company, their hard drives will probably not see a lot of churn / activity - write once, wait until something happens. Maybe refresh the data every once in a while. I'd like to see similar tests from companies like Amazon and Dropbox (although the latter probably uses the former), in various use cases.
  - chrisan 12y ago
    Wouldnt they see the same amount of churn a typical user would see? I do not use them but they say "That’s why Backblaze backs up your data automatically and constantly looks for new and changed files to backup. Install Backblaze and never worry about losing a file again." Unless you are running a database it seems like the most common use case for files is write once, update every once in a while. Document creation (code for us or office docs non-coders) would be a bunch of writes up front then follow the same pattern of update every once in a while I suppose their reads are much less than a typical user, but I didnt think reads contributed much to eventual drive failure
    - prawks 12y ago
      Hypothesizing about how Backblaze stores data: because they likely keep a series of backups, it's more likely they write each delta out separately. I'd imagine it's likely they no not have whole copies of your latest backup on their drives, but they could reconstruct one at will if needed. If that were the case, it would indeed be close to write-once.
      - 12y ago
yc1010 12y ago
Does anyone know if there is a set of command line tools (linux) or an api for uploading, downloading and listing files on your account or do they specifically forbid this type of use and obfuscate?
- e12e 12y ago
  I've been hoping for Backblaze to launch a "pro"/"enterprise" tier that allowed for this kind of use. Unfortunately (for me) it would appear they prefer to stay within their chosen niche: home user backups. The service I'm aware of that is closest to what you describe is spideroak[1]. Last I used them (years ago) they were a little too slow, and had some other minor kinks -- but might be worth checking out again now. It looks like they've also finally made good on delivering Nimbus.io -- both as source and as a service (as I said, it's been years since I looked at Spideroak...): https://nimbus.io/ https://nimbus.io/ https://github.com/SpiderOak/nimbus.io https://github.com/SpiderOak/nimbus.io [1] https://spideroak.com/ https://spideroak.com/
  - yc1010 12y ago
    So its not really "unlimited" storage they offer, there is a ceiling that matches the size of a users primary hard drive (max 6-8 TB at present?) with most users probably using laptops with small drives. For example I would love to backup 3-4TB on my NAS, and be able to occasionally access+download and check integrity
    - Xylakant 12y ago
      Incremental backups can easily exceed the size of your drive if you keep them around. I have 300-and-odd gigs of data on my drive, but my backup exceeds 1TB.
    - e12e 12y ago
      Are you reffering to backblaze? They do indeed offer "unlimited" backups AFAIK (to the degree that the phrase makes any sense, obviously). They just don't allow for clients that follow "enterprise usage patterns" so to speak. Another way to put it would be that they specialize in a simple services, with a simple UI -- with the inherit limitations that come with it. I'd personally love it if they (backblaze) started up a sister-service that catered more to what I want -- but I think the cold reality is that it's hard to compete with Amazon S3 on price -- and if you do, in a big way, they can just dump prices to sink your business. At any rate -- you usually don't want to compete on price unless you're the biggest service around -- you wan't the high-margin part of the market...
Tepix 12y ago
It's unfortunate that Backblaze feels the need to transfer the private key to their servers. Their knowledge base entry at https://help.backblaze.com/entries/20203731-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/entries/20203731-Can-you-tell-me-... doesn't clarify wether or not the secret private password is transmitted to their servers to decrypt the private keys there which would make the scheme rather weak. Ideally, the private keys would remain on the user's PCs.
- deleted 12y ago
  [deleted]
- foldor 12y ago
  While I do agree that private keys and transparency is important, I feel they almost need to store the private keys as well. But they should be password protected by something only the user knows, and the password shouldn't need to be transmitted over to their servers. The reason they would need the private keys should be obvious. They are a data disaster recovery service. In the event of a disaster where all of your drives fail, or are inaccessible you would still need access to your data. So if the only copy of the private key isn't on their servers you would be shit out of luck, not exactly a good situation for a company like theirs. But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear.
  - twoodfin 12y ago
    But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear. I think even this is "too secure". I forget my 'secure passwords' regularly, and I'm sure I wouldn't be the only one. It's not a huge improvement over telling a user to print out a private key and keep it somewhere safe. Given the likely threats to Backblaze, I'd encrypt user backup private keys with a threshold or other multi-key scheme. No individual Backblaze employee's key would be sufficient to decrypt a backup; you'd need the active participation (and private knowledge) of at least a few. That prevents individuals from snooping, and requires compromising several employees to decrypt backup data.
plus 12y ago
Unfortunately not the most direct apples-to-apples comparison. The WD Red drives are designed for use in NAS systems, whereas the Seagates used are desktop drives. Seagate does have a brand of NAS drives (called simply enough Seagate NAS), though not apparently at 6TB.
- sp332 12y ago
  It's not saying that the WD drives are better in all cases. Backblaze has a specific use-case and the WD drives are the best fit.

204 points by ehPReth 12y ago

EwanG 12y ago

For those not willing to read a full page, in the end run it is more an energy decision than a reliability decision. And they mention at the end that they are already looking into the 8TB Helium and 8TB SMR drives.

pbreit 12y ago
My sense was that the 20% speed advantage was as important or more than the small energy savings. Is that common for slower drives to handle data faster?
- pfg 12y ago
  Maybe it's a side-effect of increased vibration for 7.2k drives? Hard drive performance drops in environments with high vibration. I could imagine that being a factor when you put 45 drives in a single server.
- kijin 12y ago
  Anecdotal evidence, but I often find WD drives to be faster than their Seagate cousins, even at lower rotation speeds. I wonder if this is because of better firmware optimizations or because of higher platter density. Maybe it's both. Or maybe WD lies about their rpms ;) Hitachi/HGST, now owned by WD, also makes really fast drives. Their 7200rpm 2.5" drives are the fastest laptop drives I've ever used, except SSDs.
- miahi 12y ago
  The WD they use is a 5-platter HDD (1.2TBx5). The Seagate has 6 platters (1TBx6). The higher density means that the linear reads/writes are faster at the same spindle speed (more data passes under the heads in the same time interval). The interesting thing is that they are faster even at different speeds.

fubarred 12y ago

Backblaze is pretty awesome for open-sourcing, effectively, their costs in a high-competition segment. It would be difficult (read: pointless) to compete in that market without massive vertical integration, greater scale, deep pockets and compelling, differentiated value (eg something different enough besides all the other entrants). For anything critical in business (customer list, billing, finance, etc.), I would recommend also using Tarsnap in addition to whatever other (non-AWS) vendor/s are used for "offsite"/"offcloud" backup.

java-man 12y ago

Firefox: Unable to Connect Securely Firefox cannot guarantee the safety of your data on www.backblaze.com because it uses SSLv3, a broken security protocol. Advanced info: ssl_error_no_cypher_overlap can we trust them with our data?

warp 12y ago
Peculiar. According to https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze... they only support TLS 1.0, not SSLv3. (Though I'm still surprised by how many issues SSLLabs identifies on that site... )
- Someone1234 12y ago
  They get a "B," Microsoft gets a "B," Google gets a "B," Bank Of America get an "F," and so on. Nothing unusual about their results.
  - brianwski 12y ago
    Backblaze employee here: we try to stay on top of these annoying POODLE and other issues. One thing to realize is we encrypt all the data on the client and THEN push it through HTTPS. So many of the security issues aren't honestly affecting our customer privacy. But we stay on top of the security audits because it's a good idea and to even avoid this conversation or the questions that arise.
theandrewbailey 12y ago
I thought it was just me (I've disabled some older cipher suites in about:config). Turns out this site only supports broken ciphers (RC4 and 3DES). Ewwww. https://www.ssllabs.com/ssltest/analyze.html?d=backblaze.com https://www.ssllabs.com/ssltest/analyze.html?d=backblaze.com
- yuhong 12y ago
  3DES is not broken but it is legacy.

Bluestrike2 12y ago

The article mentions having to buy 10k+ new drives in the next few months? Is that number just your average increase for new drives and/or replacements for failed drives? Or does it include planned replacement of older pods with smaller drives? I'm curious if there's a point where you've found it less expensive (or less problematic with drive failure rates per pod) to replace and upgrade older pods compared to just letting them continue to run.

bluedino 12y ago
They used to say they don't spend any money on drive replacements (except for labor, obviously) because of the drives having 3-year warranties. Those were using 1.5TB drives. However, the original 67TB pods came out in 2009 and wouldn't be warrantied anymore. According to their blog they took until 2011 to hit 10PB, which would be roughly 150 of the original 67TB pods. The 6TB drives are 4 times better, so they'd be able to replace all of those with just 40 of the new pods, I'm sure the savings in electricity and rack space would be too huge to pass up. Then again, they have their own datacenter space now, so space probably isn't an issue but I'm sure electricity is. They said something around $2,100 a rack (which holds 10 pods) for electricity, space, and data. They were incredibly stoked to cut that in half by moving to 3TB drives with the 2.0 version of the pod so I'm sure cutting that in half excited them as well :)
- brianwski 12y ago
  Backblaze employee here. Some of the drives are "voided warranty" now, like the ones we "shuck" out of USB enclosures. We just include that into our cost analysis - for example if 5 percent of the drives fail, the warranty saves you 5 percent of cost - so if purchasing the warranty costs 3 percent we do it, but if purchasing the warranty costs 7 percent we do without.
- brianwski 12y ago
  Also, we still pay "per rack" but prices are now a fraction of the $2,100 (I'm not sure if I'm allowed to say exact numbers).
12y ago

justcommenting 12y ago

the critical assumption in this post seems to be that drive speed was the only rate-limiting factor when they were collecting data... but as i understood the post, it seemed like perimeter load-balancing, sata controller/nic load, and/or customer usage patterns could be responsible for some of the differences they observed.

lsc 12y ago
You have to screw up a storage system pretty good to have something other than the hard drives be the limiting factor.
- asdfaoeu 12y ago
  True but that doesn't imply you run at capacity all the time.
- justcommenting 12y ago
  this is generally true, but even small variations in other factors could propagate through a complex architecture to account for some of the differences presented here, especially over time. these data are still useful, but some of the caveats could be made more explicit. similarly, most drive benchmarks are 'artificial' in any number of ways and come with another set of caveats. since drive testing is already part of backblaze's process for putting drives into production, it could be especially interesting to compare and contrast performance data from testing with data from production use. i pick these nits because real-world storage performance info seems like a valuable public good (particularly for enterprise and enterprise-like usage of consumer drives), and i'm grateful to backblaze for offering it up.
- SamReidHughes 12y ago
  It's pretty easy to have software to be the limiting factor, such that you aren't taking full advantage of your throughput because the data's not there yet.
Tepix 12y ago
Exactly. I found this particular blog post weaker than their previous, highly interesting ones.

lsc 12y ago

hm. I wonder if their data access patterns are largely sequential; I imagine the 7200rpm drives would beat the lower-RPM drives in an environment where random access matters.

wmf 12y ago
It's a backup company. Write one, read never.
justcommenting 12y ago
i have similarly wondered how important drive cache size might be for these types of use cases. multivariate regressions including brand, cache size, rpm etc. as predictors of various performance metrics would be super interesting in a place like blackblaze.
- lsc 12y ago
  Cache gets you very little if you can feed in sequential data at a constant rate and at the blocksize of the device. Cache becomes more important as the input data becomes more messy; cache is one tool for turning bursty and random data into constant, sequential data.

rwmj 12y ago

I wonder if anyone has real world experience of Shingled Magnetic Recording (SMR) drives? I saw a talk by Red Hat's Ric Wheeler about a year ago, and they seem both interesting and pretty strange - as in, we may need new filesystems to cope with them.

brianwski 12y ago
Backblaze employee here: we will run our first experiment with SMR soon! We're SUPER curious about whether they will work in our application.

benguild 12y ago

It’s really great that BackBlaze does these real world, mass comparisons. Otherwise it’s just limited to what a single test says, often at a single time, with a single batch of drives from various manufacturers. These guys are probably one of the only qualified places to do this at scale and report real world results. Glad they do!

emsy 12y ago
I often think of it as open sourcing a company. They give up a competitional advantage but probably gain as much or more in return (reactions on their article as "pull requests" for instance). I don't want to sound hyperbolic but I think it also advances humanity (The information and the spirit behind it). Other companies do talks and papers too, but the Backblaze info seems really unfiltered and close to the real world.
- Ironchefpython 12y ago
  > the Backblaze info seems really unfiltered and close to the real world. Cloudflare seems to be a similar company. I recommend them both to anyone somewhat technical who needs a CDN. My only complaint about Backblaze is no Unix client. I thought about setting up a Windows VM to manage backups, but I was terrified that Gninjas would sneak into my house at night and shave my beard.
  - akgerber 12y ago
    They have a Unix client— probably just not the Unix you use (OS X).

tzz 12y ago

Backblaze should expand their service to a cheaper cloud storage service similar to Amazon S3. They already have the infrastructure and the know-how.

venomsnake 12y ago
Backblaze tech is not that impressive to give them competitive edge over the economy of scale of the big boys have. Amazon & co are not selling at cost, but at whatever the market with bear.
dsl 12y ago
The business model is based on only rarely having to retrieve old data. I suspect they spin down the drives (or possibly hibernate the whole box) when not in active "load" mode. You would need to compare the pricing here to Amazon Glacier, if anything.
- brianwski 12y ago
  Backblaze employee here. Our drives never spin down, although we have considered this because electricity is a large monthly bill to us. Because it is "backup" we are pretty relaxed about shutting down our pods to do maintenance, like replacing a drive. But that only takes 10 minutes, we try not to have machines offline more than that if possible, or it causes support issues from people trying to restore files.
  - rbanffy 12y ago
    Don't the writing patterns make old data tend to remain in older drives and newer data ending up in the newer devices? If that's the case, would reads to those devices become increasingly infrequent to the point it would be worth to spin them down? Are you worried about the stresses induced by spin up/down and thermal variations?
    - brianwski 12y ago
      In general that's my theory also (that after the initial fill, and maybe after an initial "data churn" we might power them down). However, a full pod often has at least one file from several hundred thousand individual customers. If any one of those customers prepares a restore, those drives have to be spinning. Also, we now have an iOS and Android app that you have access to all your files on your laptop from the convenience of your Smartphone - which again means the drives have to be spinning. > Are you worried about the stresses induced by spin up/down and thermal variations? Absolutely. Our datacenter techs are convinced that if they swap a drive QUICKLY the pod has a higher chance of coming back up without any problems, if they let the pod get entirely cool it statistically seems to have more problems when it comes back up - a full cool down and heat up seems to cause issues.

ck2 12y ago

Helium filled drives are literally a dead-end as their lifespan can be shorter than SSD

ghshephard 12y ago
Re: "Helium filled drives are a dead end" - Citation? Also, SSDs come with 10 year warranties [1], so, which exceeds anything you can get with a normal spinning HD. If anything, I'd have more confidence with my SSD than I would with a spinning HD - particularly as you don't have to worry about the drives failing to spin up over time. [1] http://www.amazon.com/SanDisk-Extreme-2-5-Inch-Warranty--SDSSDXPS-240G-G25/dp/B00KHRYRNM/ http://www.amazon.com/SanDisk-Extreme-2-5-Inch-Warranty--SDS...
- dsl 12y ago
  One legitimate argument against helium drives is they contribute to the depletion of the relatively scarce helium supply. http://www.boston.com/news/science/articles/2010/10/17/scientists_warn_worlds_supply_of_helium_close_to_depletion/ http://www.boston.com/news/science/articles/2010/10/17/scien...
  - Smegger 12y ago
    Helium wasted in hard drives though pale in comparison to that used in party balloons.
  - ghshephard 12y ago
    That article reads a lot like "Peak Oil" concerns. There is a lot of Helium available - [1] The global reserves of helium are known to be approximately 41 billion cubic meters. Most of them lie in Qatar, Algeria, the USA and Russia. Annual global production of helium is about 175 million cubic meters, and the USA remains the largest producer. " Regardless, this is something the market can correct for very, very easily. As helium supply becomes more scarce, the price will go up, resulting in greater supply. Most of it comes from natural gas, and is so cheap [2] it's not worth capturing. Oil trades for around $100/barrel. Helium trades at $100 for a thousand cubic feet (albeit in gas form) Some of the problems with Helium is that Physics experiments use a LOT, and previously was so cheap that it wasn't worth trying to conserve. That's changing - [3] A recycling system can recapture about $12,000/year of lost helium for a single scientist. From reading articles - apparently the problems isn't so much that the cost of helium is increasing - but that it's been so cheap because of the US Natural Reserves making it completely non-competitive to capture - they are basically giving the stuff away for next to nothing. And, putting it in a $100+ Hard Drive that will last a half decade is a FAR better use than putting 10x that much in a $0.50/balloon that will last 30 minutes. (Or Macy's parade which uses 400,000 cubic feet of helium) In fact, it may turn out to be the best conceivable use of Helium in terms of a value per m^3 equation. [1] http://www.gazprominfo.com/articles/helium/ http://www.gazprominfo.com/articles/helium/ [2] http://finance.yahoo.com/news/airgas-increase-prices-helium-20-164550580.html http://finance.yahoo.com/news/airgas-increase-prices-helium-... And on Friday, the bureau announced that it was raising the price for a thousand cubic feet for crude helium from $84 to $95 [3] http://www.nature.com/news/united-states-extends-life-of-helium-reserve-1.13819 http://www.nature.com/news/united-states-extends-life-of-hel...

unicornporn 12y ago

I'm so looking forward to the Seagate 8TB Archive HDD. Finally a drive that will hold all my photos and quite a bit more. It will make my backup procedure a whole lot easier.

medecau 12y ago
It's also a single basket for all your eggs.
- ghshephard 12y ago
  As the original poster noted, sometimes it's far better to have a single basket for your eggs, that you keep well backed up, than it is to have multiple baskets, none of which are backed up.
- deleted 12y ago
  [deleted]
pjc50 12y ago
Related ask HN to you and others: what software are you using to backup/sync to that drive? (What would you use on Windows?)
- flyinghamster 12y ago
  I can't speak for Windows, but the setup where I work involves four drives (currently 2 TB each), with three running as a triple-mirror ZFS pool. For backup, I split a drive off the pool, pull it, slide in the previous backup, and resilver the newly-inserted drive. Each time, I use a different drive bay. The ZFS pool itself provides backup for several other systems, and snapshots are made periodically by a cron job to provide a further line of defense.
- arm 12y ago
  I use OS X on my main computer, so I haven’t used this personally, but Bvckup 2 was mentioned on HN in the past: https://news.ycombinator.com/item?id=8027405 https://news.ycombinator.com/item?id=8027405

SideburnsOfDoom 12y ago

No failures recorded at all. This is pretty impressive for the drive manufacturers: "Let’s review the Seagate and Western Digital drives so far: Initial reliability (how many drives failed) – No failures. Running reliability (3 months) – No failures SMART Stats (3 months) – No error conditions recorded for the 5 stats that we utilize."

Cthulhu_ 12y ago
Of note is that since Backblaze is a backup company, their hard drives will probably not see a lot of churn / activity - write once, wait until something happens. Maybe refresh the data every once in a while. I'd like to see similar tests from companies like Amazon and Dropbox (although the latter probably uses the former), in various use cases.
- chrisan 12y ago
  Wouldnt they see the same amount of churn a typical user would see? I do not use them but they say "That’s why Backblaze backs up your data automatically and constantly looks for new and changed files to backup. Install Backblaze and never worry about losing a file again." Unless you are running a database it seems like the most common use case for files is write once, update every once in a while. Document creation (code for us or office docs non-coders) would be a bunch of writes up front then follow the same pattern of update every once in a while I suppose their reads are much less than a typical user, but I didnt think reads contributed much to eventual drive failure
  - prawks 12y ago
    Hypothesizing about how Backblaze stores data: because they likely keep a series of backups, it's more likely they write each delta out separately. I'd imagine it's likely they no not have whole copies of your latest backup on their drives, but they could reconstruct one at will if needed. If that were the case, it would indeed be close to write-once.
    - 12y ago

yc1010 12y ago

Does anyone know if there is a set of command line tools (linux) or an api for uploading, downloading and listing files on your account or do they specifically forbid this type of use and obfuscate?

e12e 12y ago
I've been hoping for Backblaze to launch a "pro"/"enterprise" tier that allowed for this kind of use. Unfortunately (for me) it would appear they prefer to stay within their chosen niche: home user backups. The service I'm aware of that is closest to what you describe is spideroak[1]. Last I used them (years ago) they were a little too slow, and had some other minor kinks -- but might be worth checking out again now. It looks like they've also finally made good on delivering Nimbus.io -- both as source and as a service (as I said, it's been years since I looked at Spideroak...): https://nimbus.io/ https://nimbus.io/ https://github.com/SpiderOak/nimbus.io https://github.com/SpiderOak/nimbus.io [1] https://spideroak.com/ https://spideroak.com/
- yc1010 12y ago
  So its not really "unlimited" storage they offer, there is a ceiling that matches the size of a users primary hard drive (max 6-8 TB at present?) with most users probably using laptops with small drives. For example I would love to backup 3-4TB on my NAS, and be able to occasionally access+download and check integrity
  - Xylakant 12y ago
    Incremental backups can easily exceed the size of your drive if you keep them around. I have 300-and-odd gigs of data on my drive, but my backup exceeds 1TB.
  - e12e 12y ago
    Are you reffering to backblaze? They do indeed offer "unlimited" backups AFAIK (to the degree that the phrase makes any sense, obviously). They just don't allow for clients that follow "enterprise usage patterns" so to speak. Another way to put it would be that they specialize in a simple services, with a simple UI -- with the inherit limitations that come with it. I'd personally love it if they (backblaze) started up a sister-service that catered more to what I want -- but I think the cold reality is that it's hard to compete with Amazon S3 on price -- and if you do, in a big way, they can just dump prices to sink your business. At any rate -- you usually don't want to compete on price unless you're the biggest service around -- you wan't the high-margin part of the market...

Tepix 12y ago

It's unfortunate that Backblaze feels the need to transfer the private key to their servers. Their knowledge base entry at https://help.backblaze.com/entries/20203731-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/entries/20203731-Can-you-tell-me-... doesn't clarify wether or not the secret private password is transmitted to their servers to decrypt the private keys there which would make the scheme rather weak. Ideally, the private keys would remain on the user's PCs.

deleted 12y ago
[deleted]
foldor 12y ago
While I do agree that private keys and transparency is important, I feel they almost need to store the private keys as well. But they should be password protected by something only the user knows, and the password shouldn't need to be transmitted over to their servers. The reason they would need the private keys should be obvious. They are a data disaster recovery service. In the event of a disaster where all of your drives fail, or are inaccessible you would still need access to your data. So if the only copy of the private key isn't on their servers you would be shit out of luck, not exactly a good situation for a company like theirs. But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear.
- twoodfin 12y ago
  But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear. I think even this is "too secure". I forget my 'secure passwords' regularly, and I'm sure I wouldn't be the only one. It's not a huge improvement over telling a user to print out a private key and keep it somewhere safe. Given the likely threats to Backblaze, I'd encrypt user backup private keys with a threshold or other multi-key scheme. No individual Backblaze employee's key would be sufficient to decrypt a backup; you'd need the active participation (and private knowledge) of at least a few. That prevents individuals from snooping, and requires compromising several employees to decrypt backup data.

plus 12y ago

Unfortunately not the most direct apples-to-apples comparison. The WD Red drives are designed for use in NAS systems, whereas the Seagates used are desktop drives. Seagate does have a brand of NAS drives (called simply enough Seagate NAS), though not apparently at 6TB.

sp332 12y ago
It's not saying that the WD drives are better in all cases. Backblaze has a specific use-case and the WD drives are the best fit.