8 ms·

Sun engineer responds to the Backblaze "Petabytes on a budget" design

169 points by bensummers 17y ago

jacquesm 17y ago
That's a pretty weird analysis, comparing a system that is special purpose with a general purpose one. He's concentrating on disk-to-board throughput, but this being network attached storage on a LAN, using the motherboards' single ethernet port the amount of data read or written to those drives per second will never exceed 1 Gbit/second anyway. I agree with the reliability analysis though, backblaze seems to be on fairly thin ice there. Still, it seems to work for backblaze, I'm taking it as read that they've lived through a bunch of drive failures by now and they seem to still be in business.
- deleted 17y ago
  [deleted]
  - jacquesm 17y ago
    > Given the reliability info he cites, the fact that backblaze is still in business must mean their usage-to-storage ratio is small. No, it probably means that they get their reliability at a different level, such as simply storing the data on a number of machines. Systems like glusterfs make this completely transparant.
    - abyssknight 17y ago
      Gluster is broken. Hadoop is pretty proven, but I haven't heard of single, non-academic Gluster deployment that worked. From personal experience I can honestly say that software cost us several customers, and caused a support nightmare for a good month before we bought a SAN instead.
      - jacquesm 17y ago
        > Gluster is broken Agreed, but I do know a few situations where it is being used, the 'stack' has been undressed to the minimum and tested to the hilt before deployment. Still, I would not risk my company on it. I've been a big fan of their architecture from pretty much day 1, but so far they seem to be feature oriented and not stability oriented.
      - bbgm 17y ago
        At least a couple of pharma companies actively use Gluster. Haven't used it myself, but have heard some stories (not positive) about the stability
    - notaddicted 17y ago
      ( ahhh I deleted my post before I saw the response ) From the numbers he cited I got the impression that the drives they are using would fail 3+ times more often than enterprise drives (lower MTTF + speced at low temps). The cost of replacing the drives (even if they are 1/2 the price) would make the system uneconomical. Also: because of their case design (non-hot-swappable) the cost of human replacement is higher too. Thus they must have a low usage rate to have a low failure rate for their system to make business sense.
      - jacquesm 17y ago
        I think their use case is sufficiently a-typical to get away with this. They're primarily write-only, only when a customer retrieves their data does stuff get read. If they spin down the drives when the volume is not being accessed the mttf for a single drive goes up a lot, possibly beyond the point where it matters.
        mattmcknight 17y ago
        That made me think of MAID (and COPAN), which is decent approach. http://www.copansystems.com/ http://www.copansystems.com/
        jrg 17y ago
        Floor loading (and rack size) is a problem with their 'very big box of disks'. That, and the fact that the company seems to be struggling to get sales.
      - rbanffy 17y ago
        Given proper redundancy and a adequately read/write unbalance, it's safe to power-down drives during read-heavy cycles and only power them up when you have some redundant data to write. But you will have to manage the cycles for redundancy, lifetime and power savings. The software to do it must be really clever.
        duskwuff 17y ago
- deleted 17y ago
  [deleted]
- sp332 17y ago
  "That's a pretty weird analysis, comparing a system that is special purpose with a general purpose one." I think that's why he says right at the beginning: This device is that cheap because it cuts several corners. That's okay for them. But for general purpose this creates problems. I want to share my concerns just to show you, that you can't compare this to a X4540 device.
  - jacquesm 17y ago
    And then he goes and does the comparison anyway...
    - sp332 17y ago
      No, he contrasts them :)
    - akamaka 17y ago
      Let's not forget that BackBlaze started this, by making a graph that compared their prices to S3, etc. (and were also criticised for that in the previous HN thread).
      - Retric 17y ago
        They are talking about cloud storage. Having 3 live copy's of your data in 3 locations is far more important than having lot's of redundancy at a single location. If you want redundancy just add more locations. Large scale company's like Google, Microsoft, and Yahoo understand this but sun is selling feel good hardware to companies that like to think they have unusual needs. As to speed, nothing you do is going to reduce a disk's low latency so talking about bandwidth is mostly a waste of time. Device bandwidth is rarely the limitation once you start to scale, because you can add devices to your network faster than you can build a better network.
- sh1mmer 17y ago
  I think the can of worms is that Backblaze themselves compared their system to more general solutions. While they were doing that for their own purpose they weren't so explicit with that point that many people (such as GigaOM, etc) took it as a general comparison.
  - jacquesm 17y ago
    They explicitly said that it was a storage unit/$ comparison, not throughput, not reliability.
    - Andys 17y ago
      It would have helped if they'd published graphs next to it showing the same list of systems sorted by throughput and another sorted by reliability!
      - jacquesm 17y ago
        For sure. I think one of the big problems with the original article is that the fabriacator of the project did the comparison and failed to see the suggestion that was being made. Probably not on purpose, but plenty of folks ran with it thinking they could have all of 'cheap, fast, reliable'. They just got 'cheap', reliable comes at the expense of using multiple of these units and fast looks like it is going to be impossible without striping across multiple setups like this. Interesting for an extremely narrow use case (long tail storage, think backup services, video sites, photo sites). Outside of that maybe useable with some adaptation but not 'as is', besides that it is not as though their machines are up for sale, the figures they quote are mostly after all the r&d is already factored out, if you had to do this for a one-off, just cutting and folding that case would cost a pretty penny. I'm fairly sure there is a market for a device that is a more robust version of this. Probably at a 50% or so premium compared to the one they showed you could get a wider spectrum of uses and a bit more robust construction. I would have bought one three months ago, now I had to settle for 20T or so in one 4U box, for a price a little over half of what they quoted in the article.
- buugs 17y ago
  He did a good job explaining why one should shop around and weigh their options... Such that it works for backblaze but that doesn't mean it will work on other applications especially concerning power failure and vibrations. He did a very good job of objecting to there message professionally regarding its use for others.
- anigbrowl 17y ago
  I think his 'DC-3' analogy is valid - BB has a decent system if you don't mind the risks, Sun is selling peace of mind.
  - Hoff 17y ago
    FUD and RAS being two of the basic tenets of marketing most anything in the enterprise IT space, after all.
    - rbanffy 17y ago
      There is a lot of good engineering in the Thumper. There is a lot of clever engineering in the Backblaze. The comparison states a lot of valid points on power distribution, drive reliability and RAID throughput. I see Backblaze will learn a lot from building their storage systems and future versions may be much better, but this one cuts too many corners for me to feel confident it will have the same kind of reliability I would recommend. What amazes me most is the use of desktop-grade hard drives. Not because they are sloppily built, but because their performance requirements are so different from a server environment. This makes for some nice reading: http://cacm.acm.org/magazines/2009/6/28493-hard-disk-drives-the-good-the-bad-and-the-ugly/fulltext http://cacm.acm.org/magazines/2009/6/28493-hard-disk-drives-...
  - uibuibk 17y ago
    >Sun is selling peace of mind Right upto the point where they are sold to a database company that isn't interested in selling disks. the nice thing about the blaze is that if seagate decide to double their prices or much you to a new expensive platform - you just buy toshiba instead.
  - sho 17y ago
    Ask Joyent about the "peace of mind" they got from their Thumper ...
    - duskwuff 17y ago
      Ref: http://www.joyent.com/joyeurblog/2008/01/16/strongspace-and-bingodisk-update/ http://www.joyent.com/joyeurblog/2008/01/16/strongspace-and-...
- onedognight 17y ago
  Backblaze is making a backup server. They don't need to handle power supply failures. If one goes down, you'll replace it long before someone needs their backup. They don't need to worry about disks failing while writing (zfs and battery backup). Hell they can restart from scratch any one backup that encounters disk failures. They have all the time they want to do a backup.
  - deleted 17y ago
    [deleted]
  - dlsspy 17y ago
    > you'll replace it long before someone needs their backup How do you know when I'll need my backup?
    - jacquesm 17y ago
      Because you can access it from that other server that bb kept, and maybe even those two other servers if they're halfway smart. Replication is what it is all about.
    - ntoshev 17y ago
      Statistically it would be a rare case for a customer to request a backup restore from a failed unit even if you don't replicate. Bb could handle these with polite mails explaining your data will be available in 24 hours and offering you some free service as compensation - it wouldn't hurt them, they do consumer backups.
- anigbrowl 17y ago
  Can anyone estimate the practical benefits (or not) of adding some extra Gbit network cards in the free motherboard slots on the BB device?
  - lsc 17y ago
    You would have to do work (either 'etherchannel' or similar bonding, or load balance requests over 2 IPs) but in theory, if you had a high-bandwidth bus like pci-express, you should be able to get more bandwidth out of a chassis by adding more ethernet cards. Me, I would add more cache at the same time. SATA tends to be really good for sequential, but be really, really bad for random. multiple sequential streams hitting the same disk start to look an awful lot like random access after a while.
  - jacquesm 17y ago
    They are using a consumer grade motherboard with one ethernet port, 3 PCI slots and one PCI express slot, the slots are all used up by the sata interfaces.
  - jsz0 17y ago
    Probably not much. The disk IO of this system is probably pretty bad. It might not be able to saturate a single GigE connection. Additionally the data is going mostly to consumer residential internet users. It's doubtful they need more than one GigE connection per box.
- uibuibk 17y ago
  Ironic though how Sun spent the last 20years trying to persuade companies that networks of Unix mini-computers were a cheap reliable alternative to mainframes - yet when it comes to putting disks in boxes you need to have everything stamped with the magic 'enterprise' badge. If you believed their spin presumably you would buy from a proper enterprise company like IBM or EDC - not some failed workstation vendor.
  - rythie 17y ago
    If you look at SANs in general, Sun undercuts that whole space, especially if you look at what they do with the 7000 series which combine SSDs and HDDs. http://www.sun.com/storage/disk_systems/unified_storage/ http://www.sun.com/storage/disk_systems/unified_storage/
    - russss 17y ago
      The 7000 series is an incredibly cool system, and it's definitely giving NetApp/EMC a run for their money. That said, it's still really expensive if you're willing to go that extra mile and design your own system for your needs. And it's really not that hard.
      - rythie 17y ago
        Yeah I totally agree it's expensive - though compared to the other enterprise storage solutions it's cheap. That said people should be making cheaper solutions when you don't that enterprise level system - especially if you can handle the failures in your software stack.
smanek 17y ago
I think everyone agrees the Backblaze (bb) has no performance - and that's fine. But a lot of his complaints are about reliability, and thats kind of a moot point since Backblaze (and anyone sane) stores every piece of data on at least two (and ideally 3 or more) separate bbs. I would argue if you don't need performance (i.e., no huge database), but you do need a lot of space, you're better off just setting up two 'mirrored' bbs (so that when one goes down, the other takes over) than almost any 'enterprise' solution. He does have a point about ZFS though - I'm sure at that kind of scale eventually the RAID 5/6 write-hole is going to bite you in the ass.
- abyssknight 17y ago
  I read that as "every piece of data on at least two (and ideally 3 or more) separate bbs", meaning BBS as in a bulletin board system. Wow, that caught me off guard.
skolor 17y ago
Fascinating read, and its nice to see he brought up a lot of the concerns that were mentioned here. A few things that I noticed: He mentioned that 1 Disk uses 120 MB/s, so 5 of them is 600 MB/s. He then says that converts to 6 GB/s. Now, I'm not all that familiar with speeds, but I know 1000 MB is 1 GB, so shouldn't 600 MB/s be .6 GB/s? Other than that, he raised a number of valid points. The Backblaze system is NOT general purpose. Its designed to take a bunch of data (Backups) and hold on to it for as long as possible. In that situation you don't need high Throughput, or even all that reliable hardware. Once the drives are full, their usage will drop, and as long as the RAID can be kept up to keep the data secure, as long as costs stay down they're fine to keep replacing hardware. One thing of note, and something I never saw in the Backblaze article: Their service promises to hold your data for 50 years. Assuming that one of their pods is filled, what is the estimated cost to keep that data over the course of 50 years? I would assume they calculated that out, and that the data showed to go with the consumer grade parts, but I would really like to see the numbers.
- jacquesm 17y ago
  You got confused about the GB/s because he uses the capital "B" at some point where it should have been the smaller 'b' for bits, not bytes. So, 600 MB/s is 600*8 = 4800 Mbps. I hope that clears it up. Please note that this is the guys private blog, he definitely does not speak for Sun in that spot and to label the article "SUN Engineer responds" makes it seem like the response is an official one but it very clearly isn't. The text is full of other mistakes as well.
- jeroen 17y ago
  From the article: "600 MByte/s, a little bit less than 6 GBit/s." bytes vs bits.
sophacles 17y ago
Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network filesystems if all data comes and goes via the internet? Will the hard drives see that much data I/O, or will they mostly just fill up and sit idling with the very occasional read for restore (i strongly suspect the later)[1]. Of course on the other hand, there are the poeple touting it as the end all be all, a solution to kill NetApp. I guess what Im wondering is: how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people? What am I missing? [1]A huge chunk of this article is about the hard drives and PSUs not being enterprise ready, but for an enterprise load, but I just don't see it. I bet a lot of these boxes run idle a large chunk of the time. I have 10 year old desktop hard drives running just fine in a file server, because it has a similar load: mostly idle most of the time.
- gaius 17y ago
  If "doing backups" is your business, then "look how cheap our kit is!" probably isn't the best marketing strategy. A "small business" is someone's bread-and-butter, they don't care that their backup needs aren't sexy to storage geeks.
  - sophacles 17y ago
    I dunno, it made me consider them. Instead of "we spend $2m on storage, and someday to hope to make enough money from people like you to not go out of business", its "look we aren't charging you much, and we are promising you 50 yrs of storage, this is how we can actually do that and stay in business".
    - sho 17y ago
      I agree. Do you remember a company called Bandwagon? http://ridethebandwagon.com/ http://ridethebandwagon.com/ They had a big splash a couple of years ago about how they were going to offer unlimited backup for all your mp3s in iTunes, all nicely and automatically synced, for about $10/month or something. They were going to do the storage on S3. They offered anyone who mentioned them on their blog a free years' service. http://blog.ridethebandwagon.com/post/37269480/free-accounts-for-bloggers http://blog.ridethebandwagon.com/post/37269480/free-accounts... I had about 400G of mp3s at the time I think. I thought, "OK!" Their service lasted about 2 days as they underestimated, seemingly by about an order of magnitude, how much data people actually had. http://delong.typepad.com/sdj/2007/02/adverse_selecti.html http://delong.typepad.com/sdj/2007/02/adverse_selecti.html Then they went back to the drawing board and came up with some lame service where you buy the storage space on S3 yourself and then they sell you the ability to access it or something. Lucky they didn't go out of business. Anyway, knowing the details of the kind of system Backblaze is using, and the price per gig it breaks down to for them, is evidence they're not overselling themselves, and gives me confidence that they'll still actually be there in a few years, unlike Bandwagon. I'm much more likely to use them because of that. Add in all the discussion they've generated, and it's been a very clever marketing event for them. Kudos!
psadauskas 17y ago
He also leaves out that you could buy 10 of these Backblaze things for the price of one Sun X4540. So with a distributed load, thats 10x the performance, 10x the reliability, etc... I'm not an expert on storage, but I'd be curious to see how that stacks up, never mind the comet-proof redundancy by placing your storage in datacenters across the world.
- yangyang 17y ago
  10x the rack space, 10x the cooling, 10x the power ...
  - modoc 17y ago
    Those costs are minor compared to the hardware solutions being discussed.
    - gaius 17y ago
      Haha! Unexpectedly needing a new cage at your DC, or an extra 50kW power, or being told you can't put anything more in your half-full racks because you've exceeded floor loading gets expensive real quick.
      - lsc 17y ago
        At svtix, a tier3 data center in downtown san jose, Not at the top end of the price scale, but not at the bottom, I can get 2 20a 120v circuits (4800w, that's 3360 usable.) and a full locking rack for around $1100 a month. I can buy two or three racks full of power. If I wanted to pull a gig connection to another nearby data center, say, because I couldn't get more at svtix, we are talking something like $500-$1000 a month, depending on how badly I get screwed by my poor negotiation skills. Just sayin, at those prices, it takes a long time to amortize out a six figure appliance.
        antonovka 17y ago
        At svtix, a tier3 data center in downtown san jose, Not at the top end of the price scale, but not at the bottom, I can get 2 20a 120v circuits and a full locking rack for around $1100 a month. I can buy two or three racks full of power. As long as that space is available. To guarantee business capacity you'll probably need to buy a large block of rackspace at one time. Of course, capacity availability varies based on the current local business climate, but that's something you have to plan for too. If I wanted to pull a gig connection to another nearby data center, say, because I couldn't get more at svtix, we are talking something like $500-$1000 a month Upwards of $1000, but note that GigE is a fairly small link for data center cross-connection and will still usually have a large lead-time. Not only do you need to be able to guarantee sufficient capacity either at svtix or at the other colo, but you need to be prepared to wait to get that capacity allocated for your use.
tlrobinson 17y ago
"It's the same with this storage, this hw needs the parachute in form of the software in front of the device." Yes, software on (relatively) unreliable hardware is exactly how Google has been able to scale to such an immense scale.
anarcticpuffin 17y ago
I agree that what the article misses is that Backblaze is spending the money on software rather than hardware, since the price difference is so huge (and the software solution is a fixed cost). What I'm more interested in is whether the lower MTBF of the cheap drives and home-brewed chassis ends up with a higher cost per year due to higher failure rates. If a desktop drive costs $100 and fails three times a year, but a server drive costs $200 and fails once every 2, the initial cost savings is moot.
- sho 17y ago
  Well, remember that if you were seriously on a budget you'd just return all the failed drives for warranty replacement (Seagate is 5yrs). The hardware costs would never add up to more than the enterprise, it's that the labour costs of dealing with it would be more, and worse - more likely to lose data. Enough more likely to justify the additional cost? Hm, I doubt it, not for this use case.
barryrandall 17y ago
At first don't compare the mentioned list prices with the street prices for components. This is why small companies, startups, and people new to the storage world hate working with storage vendors. I understand the economics, I get why they sell storage the way they do, and I like my 70% discount off list price, but it makes for a horrible user experience. If I were a cleverer person, I'd apply an every-day low price model to "enterprise" storage.
- lsc 17y ago
  I went with the reseller I did at svtix, primarily because they were upfront enough to list prices on their website. http://egihosting.com http://egihosting.com - I don't want to spend time fucking around over price, and I'm sure you don't want me to waste your time. Post the price, and I'll buy it or I won't. Seriously, If you say you want six figures for your product, I'm walking. I don't have that kind of scratch, and I'm not going to waste your time or my time lowballing you. If you really mean $30K, well, I'd have to scrimp and save for a few months, but it's doable if the product really does solve more than $30K worth of problems for me.
ShabbyDoo 17y ago
I used to work for a large consumer photo site, and we wished we could trade speed for lower cost or lower power consumption. Forget SCSI drives, SATA disks were 10x faster than what we needed for long-tail storage. If a photo became popular, we could cache it forward. The parato-ization of the photos was probably 99.9/0.1. I'm actually surprised that drive manufacturers haven't come out with a slower disk that is cheaper, lower power, and/or more reliable. We could find a zillion places to hide latency from our users.
- wmf 17y ago
  Seagate Barracuda LP and WD RE-GP are ~6000 RPM disks that are lower power than normal 7200 RPM SATA. Maybe next year they'll come out with ~5000 RPM disks.
  - ShabbyDoo 17y ago
    Cool. I wonder what sort of density could be achieved with slow solid state storage. The line between disk and tape continues to blur, I guess.
- siverson914 17y ago
  Check out Diomede Storage - it's a cloud storage service that allows you to specify the type of storage (Online, Nearline, Offline) for exactly this purpose. www.diomedestorage.com
- blasdel 17y ago
  Samsung's new generation of drives is currently only available in "EcoGreen" versions: 5400rpm. Doesn't particularly affect bandwidth when you're at 500gb/platter, though you lose on latency. http://www.newegg.com/Product/Product.aspx?Item=N82E16822152175 http://www.newegg.com/Product/Product.aspx?Item=N82E16822152...
rythie 17y ago
Even though Sun are at the cheaper end of the SAN market - I struggle to see why adding a network interface to hard drive still ends up as 10x the cost of the drive.
c00p3r 17y ago
zfs with OpenSolaris is a good tip. 2009.06 is a much less mess (leave alone that we-will-rewrite-everything-in-java svc:/* horror)
- kdw 17y ago
  zfs performance in 2009.06 has some very serious problems, particularly with zvols.
  - jrg 17y ago
    Do you have references and pointers? Is this going to hurt me when I put two non-storage servers live running 2009.06?
    - kdw 17y ago
      You can find a number of them on the opensolaris forums if you search on terms like 0906 iscsi, comstar 0906, etc... here's an example: http://opensolaris.org/jive/thread.jspa?threadID=104593&tstart=105 http://opensolaris.org/jive/thread.jspa?threadID=104593&... Our experience was that 0906 was utterly unusable if you're using comstar, with performance being mildly degraded for straight zfs usage.
thegoleffect 17y ago
Seems like he's just making it easier for Backblaze to match feature to feature by listing out the differences O_o.
- docmach 17y ago
  I'm pretty sure Sun isn't worried about Backblaze as competition in the enterprise storage market. Building cheap storage boxes is much different than building what Sun provides.
  - thegoleffect 17y ago
    That's usually how disruptive technologies start out though; the fact that Sun doesn't care makes Backblaze that more dangerous.
brisance 17y ago
How is MTBF derived? He claims "1.2 million hours normalized on 40 degrees", that's almost 140 years. It seems to me that it's just some extrapolated test statistic being bandied about with no relevance to real world conditions.

169 points by bensummers 17y ago

That's a pretty weird analysis, comparing a system that is special purpose with a general purpose one. He's concentrating on disk-to-board throughput, but this being network attached storage on a LAN, using the motherboards' single ethernet port the amount of data read or written to those drives per second will never exceed 1 Gbit/second anyway. I agree with the reliability analysis though, backblaze seems to be on fairly thin ice there. Still, it seems to work for backblaze, I'm taking it as read that they've lived through a bunch of drive failures by now and they seem to still be in business.

deleted 17y ago
[deleted]
- jacquesm 17y ago
  > Given the reliability info he cites, the fact that backblaze is still in business must mean their usage-to-storage ratio is small. No, it probably means that they get their reliability at a different level, such as simply storing the data on a number of machines. Systems like glusterfs make this completely transparant.
  - abyssknight 17y ago
    Gluster is broken. Hadoop is pretty proven, but I haven't heard of single, non-academic Gluster deployment that worked. From personal experience I can honestly say that software cost us several customers, and caused a support nightmare for a good month before we bought a SAN instead.
    - jacquesm 17y ago
      > Gluster is broken Agreed, but I do know a few situations where it is being used, the 'stack' has been undressed to the minimum and tested to the hilt before deployment. Still, I would not risk my company on it. I've been a big fan of their architecture from pretty much day 1, but so far they seem to be feature oriented and not stability oriented.
    - bbgm 17y ago
      At least a couple of pharma companies actively use Gluster. Haven't used it myself, but have heard some stories (not positive) about the stability
  - notaddicted 17y ago
    ( ahhh I deleted my post before I saw the response ) From the numbers he cited I got the impression that the drives they are using would fail 3+ times more often than enterprise drives (lower MTTF + speced at low temps). The cost of replacing the drives (even if they are 1/2 the price) would make the system uneconomical. Also: because of their case design (non-hot-swappable) the cost of human replacement is higher too. Thus they must have a low usage rate to have a low failure rate for their system to make business sense.
    - jacquesm 17y ago
      I think their use case is sufficiently a-typical to get away with this. They're primarily write-only, only when a customer retrieves their data does stuff get read. If they spin down the drives when the volume is not being accessed the mttf for a single drive goes up a lot, possibly beyond the point where it matters.
      - mattmcknight 17y ago
        That made me think of MAID (and COPAN), which is decent approach. http://www.copansystems.com/ http://www.copansystems.com/
        jrg 17y ago
        Floor loading (and rack size) is a problem with their 'very big box of disks'. That, and the fact that the company seems to be struggling to get sales.
    - rbanffy 17y ago
      Given proper redundancy and a adequately read/write unbalance, it's safe to power-down drives during read-heavy cycles and only power them up when you have some redundant data to write. But you will have to manage the cycles for redundancy, lifetime and power savings. The software to do it must be really clever.
      - duskwuff 17y ago
deleted 17y ago
[deleted]
sp332 17y ago
"That's a pretty weird analysis, comparing a system that is special purpose with a general purpose one." I think that's why he says right at the beginning: This device is that cheap because it cuts several corners. That's okay for them. But for general purpose this creates problems. I want to share my concerns just to show you, that you can't compare this to a X4540 device.
- jacquesm 17y ago
  And then he goes and does the comparison anyway...
  - sp332 17y ago
    No, he contrasts them :)
  - akamaka 17y ago
    Let's not forget that BackBlaze started this, by making a graph that compared their prices to S3, etc. (and were also criticised for that in the previous HN thread).
    - Retric 17y ago
      They are talking about cloud storage. Having 3 live copy's of your data in 3 locations is far more important than having lot's of redundancy at a single location. If you want redundancy just add more locations. Large scale company's like Google, Microsoft, and Yahoo understand this but sun is selling feel good hardware to companies that like to think they have unusual needs. As to speed, nothing you do is going to reduce a disk's low latency so talking about bandwidth is mostly a waste of time. Device bandwidth is rarely the limitation once you start to scale, because you can add devices to your network faster than you can build a better network.
sh1mmer 17y ago
I think the can of worms is that Backblaze themselves compared their system to more general solutions. While they were doing that for their own purpose they weren't so explicit with that point that many people (such as GigaOM, etc) took it as a general comparison.
- jacquesm 17y ago
  They explicitly said that it was a storage unit/$ comparison, not throughput, not reliability.
  - Andys 17y ago
    It would have helped if they'd published graphs next to it showing the same list of systems sorted by throughput and another sorted by reliability!
    - jacquesm 17y ago
      For sure. I think one of the big problems with the original article is that the fabriacator of the project did the comparison and failed to see the suggestion that was being made. Probably not on purpose, but plenty of folks ran with it thinking they could have all of 'cheap, fast, reliable'. They just got 'cheap', reliable comes at the expense of using multiple of these units and fast looks like it is going to be impossible without striping across multiple setups like this. Interesting for an extremely narrow use case (long tail storage, think backup services, video sites, photo sites). Outside of that maybe useable with some adaptation but not 'as is', besides that it is not as though their machines are up for sale, the figures they quote are mostly after all the r&d is already factored out, if you had to do this for a one-off, just cutting and folding that case would cost a pretty penny. I'm fairly sure there is a market for a device that is a more robust version of this. Probably at a 50% or so premium compared to the one they showed you could get a wider spectrum of uses and a bit more robust construction. I would have bought one three months ago, now I had to settle for 20T or so in one 4U box, for a price a little over half of what they quoted in the article.
buugs 17y ago
He did a good job explaining why one should shop around and weigh their options... Such that it works for backblaze but that doesn't mean it will work on other applications especially concerning power failure and vibrations. He did a very good job of objecting to there message professionally regarding its use for others.
anigbrowl 17y ago
I think his 'DC-3' analogy is valid - BB has a decent system if you don't mind the risks, Sun is selling peace of mind.
- Hoff 17y ago
  FUD and RAS being two of the basic tenets of marketing most anything in the enterprise IT space, after all.
  - rbanffy 17y ago
    There is a lot of good engineering in the Thumper. There is a lot of clever engineering in the Backblaze. The comparison states a lot of valid points on power distribution, drive reliability and RAID throughput. I see Backblaze will learn a lot from building their storage systems and future versions may be much better, but this one cuts too many corners for me to feel confident it will have the same kind of reliability I would recommend. What amazes me most is the use of desktop-grade hard drives. Not because they are sloppily built, but because their performance requirements are so different from a server environment. This makes for some nice reading: http://cacm.acm.org/magazines/2009/6/28493-hard-disk-drives-the-good-the-bad-and-the-ugly/fulltext http://cacm.acm.org/magazines/2009/6/28493-hard-disk-drives-...
- uibuibk 17y ago
  >Sun is selling peace of mind Right upto the point where they are sold to a database company that isn't interested in selling disks. the nice thing about the blaze is that if seagate decide to double their prices or much you to a new expensive platform - you just buy toshiba instead.
- sho 17y ago
  Ask Joyent about the "peace of mind" they got from their Thumper ...
  - duskwuff 17y ago
    Ref: http://www.joyent.com/joyeurblog/2008/01/16/strongspace-and-bingodisk-update/ http://www.joyent.com/joyeurblog/2008/01/16/strongspace-and-...
onedognight 17y ago
Backblaze is making a backup server. They don't need to handle power supply failures. If one goes down, you'll replace it long before someone needs their backup. They don't need to worry about disks failing while writing (zfs and battery backup). Hell they can restart from scratch any one backup that encounters disk failures. They have all the time they want to do a backup.
- deleted 17y ago
  [deleted]
- dlsspy 17y ago
  > you'll replace it long before someone needs their backup How do you know when I'll need my backup?
  - jacquesm 17y ago
    Because you can access it from that other server that bb kept, and maybe even those two other servers if they're halfway smart. Replication is what it is all about.
  - ntoshev 17y ago
    Statistically it would be a rare case for a customer to request a backup restore from a failed unit even if you don't replicate. Bb could handle these with polite mails explaining your data will be available in 24 hours and offering you some free service as compensation - it wouldn't hurt them, they do consumer backups.
anigbrowl 17y ago
Can anyone estimate the practical benefits (or not) of adding some extra Gbit network cards in the free motherboard slots on the BB device?
- lsc 17y ago
  You would have to do work (either 'etherchannel' or similar bonding, or load balance requests over 2 IPs) but in theory, if you had a high-bandwidth bus like pci-express, you should be able to get more bandwidth out of a chassis by adding more ethernet cards. Me, I would add more cache at the same time. SATA tends to be really good for sequential, but be really, really bad for random. multiple sequential streams hitting the same disk start to look an awful lot like random access after a while.
- jacquesm 17y ago
  They are using a consumer grade motherboard with one ethernet port, 3 PCI slots and one PCI express slot, the slots are all used up by the sata interfaces.
- jsz0 17y ago
  Probably not much. The disk IO of this system is probably pretty bad. It might not be able to saturate a single GigE connection. Additionally the data is going mostly to consumer residential internet users. It's doubtful they need more than one GigE connection per box.
uibuibk 17y ago
Ironic though how Sun spent the last 20years trying to persuade companies that networks of Unix mini-computers were a cheap reliable alternative to mainframes - yet when it comes to putting disks in boxes you need to have everything stamped with the magic 'enterprise' badge. If you believed their spin presumably you would buy from a proper enterprise company like IBM or EDC - not some failed workstation vendor.
- rythie 17y ago
  If you look at SANs in general, Sun undercuts that whole space, especially if you look at what they do with the 7000 series which combine SSDs and HDDs. http://www.sun.com/storage/disk_systems/unified_storage/ http://www.sun.com/storage/disk_systems/unified_storage/
  - russss 17y ago
    The 7000 series is an incredibly cool system, and it's definitely giving NetApp/EMC a run for their money. That said, it's still really expensive if you're willing to go that extra mile and design your own system for your needs. And it's really not that hard.
    - rythie 17y ago
      Yeah I totally agree it's expensive - though compared to the other enterprise storage solutions it's cheap. That said people should be making cheaper solutions when you don't that enterprise level system - especially if you can handle the failures in your software stack.

smanek 17y ago

I think everyone agrees the Backblaze (bb) has no performance - and that's fine. But a lot of his complaints are about reliability, and thats kind of a moot point since Backblaze (and anyone sane) stores every piece of data on at least two (and ideally 3 or more) separate bbs. I would argue if you don't need performance (i.e., no huge database), but you do need a lot of space, you're better off just setting up two 'mirrored' bbs (so that when one goes down, the other takes over) than almost any 'enterprise' solution. He does have a point about ZFS though - I'm sure at that kind of scale eventually the RAID 5/6 write-hole is going to bite you in the ass.

abyssknight 17y ago
I read that as "every piece of data on at least two (and ideally 3 or more) separate bbs", meaning BBS as in a bulletin board system. Wow, that caught me off guard.

skolor 17y ago

Fascinating read, and its nice to see he brought up a lot of the concerns that were mentioned here. A few things that I noticed: He mentioned that 1 Disk uses 120 MB/s, so 5 of them is 600 MB/s. He then says that converts to 6 GB/s. Now, I'm not all that familiar with speeds, but I know 1000 MB is 1 GB, so shouldn't 600 MB/s be .6 GB/s? Other than that, he raised a number of valid points. The Backblaze system is NOT general purpose. Its designed to take a bunch of data (Backups) and hold on to it for as long as possible. In that situation you don't need high Throughput, or even all that reliable hardware. Once the drives are full, their usage will drop, and as long as the RAID can be kept up to keep the data secure, as long as costs stay down they're fine to keep replacing hardware. One thing of note, and something I never saw in the Backblaze article: Their service promises to hold your data for 50 years. Assuming that one of their pods is filled, what is the estimated cost to keep that data over the course of 50 years? I would assume they calculated that out, and that the data showed to go with the consumer grade parts, but I would really like to see the numbers.

jacquesm 17y ago
You got confused about the GB/s because he uses the capital "B" at some point where it should have been the smaller 'b' for bits, not bytes. So, 600 MB/s is 600*8 = 4800 Mbps. I hope that clears it up. Please note that this is the guys private blog, he definitely does not speak for Sun in that spot and to label the article "SUN Engineer responds" makes it seem like the response is an official one but it very clearly isn't. The text is full of other mistakes as well.
jeroen 17y ago
From the article: "600 MByte/s, a little bit less than 6 GBit/s." bytes vs bits.

sophacles 17y ago

Ok I must be missing something. Everyone is knocking this backblaze thing because it doesnt do ZFS or because it is not a "super duper high end san replacement" or because the components are not rated for enterprise level work. It seems to me that a company whose product is personal and small business backups does not need any of those things. They have software to do mirroring. Why would they need any fast network filesystems if all data comes and goes via the internet? Will the hard drives see that much data I/O, or will they mostly just fill up and sit idling with the very occasional read for restore (i strongly suspect the later)[1]. Of course on the other hand, there are the poeple touting it as the end all be all, a solution to kill NetApp. I guess what Im wondering is: how did so many people get the idea that this article about a specific solution to a specific problem was actually some sort of general purpose solution attacking all the big name people? What am I missing? [1]A huge chunk of this article is about the hard drives and PSUs not being enterprise ready, but for an enterprise load, but I just don't see it. I bet a lot of these boxes run idle a large chunk of the time. I have 10 year old desktop hard drives running just fine in a file server, because it has a similar load: mostly idle most of the time.

gaius 17y ago
If "doing backups" is your business, then "look how cheap our kit is!" probably isn't the best marketing strategy. A "small business" is someone's bread-and-butter, they don't care that their backup needs aren't sexy to storage geeks.
- sophacles 17y ago
  I dunno, it made me consider them. Instead of "we spend $2m on storage, and someday to hope to make enough money from people like you to not go out of business", its "look we aren't charging you much, and we are promising you 50 yrs of storage, this is how we can actually do that and stay in business".
  - sho 17y ago
    I agree. Do you remember a company called Bandwagon? http://ridethebandwagon.com/ http://ridethebandwagon.com/ They had a big splash a couple of years ago about how they were going to offer unlimited backup for all your mp3s in iTunes, all nicely and automatically synced, for about $10/month or something. They were going to do the storage on S3. They offered anyone who mentioned them on their blog a free years' service. http://blog.ridethebandwagon.com/post/37269480/free-accounts-for-bloggers http://blog.ridethebandwagon.com/post/37269480/free-accounts... I had about 400G of mp3s at the time I think. I thought, "OK!" Their service lasted about 2 days as they underestimated, seemingly by about an order of magnitude, how much data people actually had. http://delong.typepad.com/sdj/2007/02/adverse_selecti.html http://delong.typepad.com/sdj/2007/02/adverse_selecti.html Then they went back to the drawing board and came up with some lame service where you buy the storage space on S3 yourself and then they sell you the ability to access it or something. Lucky they didn't go out of business. Anyway, knowing the details of the kind of system Backblaze is using, and the price per gig it breaks down to for them, is evidence they're not overselling themselves, and gives me confidence that they'll still actually be there in a few years, unlike Bandwagon. I'm much more likely to use them because of that. Add in all the discussion they've generated, and it's been a very clever marketing event for them. Kudos!

psadauskas 17y ago

He also leaves out that you could buy 10 of these Backblaze things for the price of one Sun X4540. So with a distributed load, thats 10x the performance, 10x the reliability, etc... I'm not an expert on storage, but I'd be curious to see how that stacks up, never mind the comet-proof redundancy by placing your storage in datacenters across the world.

yangyang 17y ago
10x the rack space, 10x the cooling, 10x the power ...
- modoc 17y ago
  Those costs are minor compared to the hardware solutions being discussed.
  - gaius 17y ago
    Haha! Unexpectedly needing a new cage at your DC, or an extra 50kW power, or being told you can't put anything more in your half-full racks because you've exceeded floor loading gets expensive real quick.
    - lsc 17y ago
      At svtix, a tier3 data center in downtown san jose, Not at the top end of the price scale, but not at the bottom, I can get 2 20a 120v circuits (4800w, that's 3360 usable.) and a full locking rack for around $1100 a month. I can buy two or three racks full of power. If I wanted to pull a gig connection to another nearby data center, say, because I couldn't get more at svtix, we are talking something like $500-$1000 a month, depending on how badly I get screwed by my poor negotiation skills. Just sayin, at those prices, it takes a long time to amortize out a six figure appliance.
      - antonovka 17y ago
        At svtix, a tier3 data center in downtown san jose, Not at the top end of the price scale, but not at the bottom, I can get 2 20a 120v circuits and a full locking rack for around $1100 a month. I can buy two or three racks full of power. As long as that space is available. To guarantee business capacity you'll probably need to buy a large block of rackspace at one time. Of course, capacity availability varies based on the current local business climate, but that's something you have to plan for too. If I wanted to pull a gig connection to another nearby data center, say, because I couldn't get more at svtix, we are talking something like $500-$1000 a month Upwards of $1000, but note that GigE is a fairly small link for data center cross-connection and will still usually have a large lead-time. Not only do you need to be able to guarantee sufficient capacity either at svtix or at the other colo, but you need to be prepared to wait to get that capacity allocated for your use.

tlrobinson 17y ago

"It's the same with this storage, this hw needs the parachute in form of the software in front of the device." Yes, software on (relatively) unreliable hardware is exactly how Google has been able to scale to such an immense scale.

anarcticpuffin 17y ago

I agree that what the article misses is that Backblaze is spending the money on software rather than hardware, since the price difference is so huge (and the software solution is a fixed cost). What I'm more interested in is whether the lower MTBF of the cheap drives and home-brewed chassis ends up with a higher cost per year due to higher failure rates. If a desktop drive costs $100 and fails three times a year, but a server drive costs $200 and fails once every 2, the initial cost savings is moot.

sho 17y ago
Well, remember that if you were seriously on a budget you'd just return all the failed drives for warranty replacement (Seagate is 5yrs). The hardware costs would never add up to more than the enterprise, it's that the labour costs of dealing with it would be more, and worse - more likely to lose data. Enough more likely to justify the additional cost? Hm, I doubt it, not for this use case.

barryrandall 17y ago

At first don't compare the mentioned list prices with the street prices for components. This is why small companies, startups, and people new to the storage world hate working with storage vendors. I understand the economics, I get why they sell storage the way they do, and I like my 70% discount off list price, but it makes for a horrible user experience. If I were a cleverer person, I'd apply an every-day low price model to "enterprise" storage.

lsc 17y ago
I went with the reseller I did at svtix, primarily because they were upfront enough to list prices on their website. http://egihosting.com http://egihosting.com - I don't want to spend time fucking around over price, and I'm sure you don't want me to waste your time. Post the price, and I'll buy it or I won't. Seriously, If you say you want six figures for your product, I'm walking. I don't have that kind of scratch, and I'm not going to waste your time or my time lowballing you. If you really mean $30K, well, I'd have to scrimp and save for a few months, but it's doable if the product really does solve more than $30K worth of problems for me.

ShabbyDoo 17y ago

I used to work for a large consumer photo site, and we wished we could trade speed for lower cost or lower power consumption. Forget SCSI drives, SATA disks were 10x faster than what we needed for long-tail storage. If a photo became popular, we could cache it forward. The parato-ization of the photos was probably 99.9/0.1. I'm actually surprised that drive manufacturers haven't come out with a slower disk that is cheaper, lower power, and/or more reliable. We could find a zillion places to hide latency from our users.

wmf 17y ago
Seagate Barracuda LP and WD RE-GP are ~6000 RPM disks that are lower power than normal 7200 RPM SATA. Maybe next year they'll come out with ~5000 RPM disks.
- ShabbyDoo 17y ago
  Cool. I wonder what sort of density could be achieved with slow solid state storage. The line between disk and tape continues to blur, I guess.
siverson914 17y ago
Check out Diomede Storage - it's a cloud storage service that allows you to specify the type of storage (Online, Nearline, Offline) for exactly this purpose. www.diomedestorage.com
blasdel 17y ago
Samsung's new generation of drives is currently only available in "EcoGreen" versions: 5400rpm. Doesn't particularly affect bandwidth when you're at 500gb/platter, though you lose on latency. http://www.newegg.com/Product/Product.aspx?Item=N82E16822152175 http://www.newegg.com/Product/Product.aspx?Item=N82E16822152...

rythie 17y ago

Even though Sun are at the cheaper end of the SAN market - I struggle to see why adding a network interface to hard drive still ends up as 10x the cost of the drive.

c00p3r 17y ago

zfs with OpenSolaris is a good tip. 2009.06 is a much less mess (leave alone that we-will-rewrite-everything-in-java svc:/* horror)

kdw 17y ago
zfs performance in 2009.06 has some very serious problems, particularly with zvols.
- jrg 17y ago
  Do you have references and pointers? Is this going to hurt me when I put two non-storage servers live running 2009.06?
  - kdw 17y ago
    You can find a number of them on the opensolaris forums if you search on terms like 0906 iscsi, comstar 0906, etc... here's an example: http://opensolaris.org/jive/thread.jspa?threadID=104593&tstart=105 http://opensolaris.org/jive/thread.jspa?threadID=104593&... Our experience was that 0906 was utterly unusable if you're using comstar, with performance being mildly degraded for straight zfs usage.

thegoleffect 17y ago

Seems like he's just making it easier for Backblaze to match feature to feature by listing out the differences O_o.

docmach 17y ago
I'm pretty sure Sun isn't worried about Backblaze as competition in the enterprise storage market. Building cheap storage boxes is much different than building what Sun provides.
- thegoleffect 17y ago
  That's usually how disruptive technologies start out though; the fact that Sun doesn't care makes Backblaze that more dangerous.

brisance 17y ago

How is MTBF derived? He claims "1.2 million hours normalized on 40 degrees", that's almost 140 years. It seems to me that it's just some extrapolated test statistic being bandied about with no relevance to real world conditions.