7 ms·

Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing

by jamwt 11y ago

Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing up to an order of magnitude more storage into 4U. And costs aren't close, let alone the addressable throughput given that you guys just went to 10G. Serious kudos though are due for the Backblaze blog. You're all at least talking about what you're up to, which is awesome both for education around building large storage systems and establishing transparency with your customer base. Unfortunately, a lot of the bigger clusters' teams won't/can't talk about what exactly they're doing and how they're doing it for trade secret reasons. If you want to learn about it, you have to come work for one of us. :-) Also, the orangered hardware looks awesome.

scurvy 11y ago
You're running 4U Quanta T21P-4U boxes but with 40gig OCP mezzanine networking now? I hope you guys aren't using SMR drives... Edit: Also, when are you going to move out of AWS? I really would like our/your clients to use IPv6.
budmang 11y ago
> packing up to an order of magnitude more storage into 4U Where are you buying 40 TB hard drives? I'd love to get some ;-) We actually don't claim these are 'state of the art', more 'state of the inexpensive'. You can buy faster performance storage servers, more redundant servers, etc. But I believe these are one of, if not the, lowest cost storage servers per GB. (And of course, open source.) BTW, I'm a fan of Dropbox (use it personally) and, in general, think there is a ton of value in developers, organizations, and companies sharing what they've learned about storage. The better and lower cost it gets, the more interesting things other people can do with it, the bigger the industry, and the more everyone benefits. (full disclosure - cofounder of Backblaze here)
- jamwt 11y ago
  Hi Gleb! > Where are you buying 40 TB hard drives? I'd love to get some ;-) Me too. :-) Not 40TB, but larger than 4T and quite a few more of them. > But I believe these are one of, if not the, lowest cost storage servers per GB. Yep, I meant price per GB as well. > I'm a fan of Dropbox (use it personally) Awesome, thanks! I've not used Backblaze personally, but I know some friends who do and they're happy customers. Your guys' blog and rapport with the technical community is amazing.
codemac 11y ago
What order of magnitude? There are 60 drive chassis in 4U you can get[0][1], but even a 4TB -> 6TB drive + 15 more drives isn't exactly an order of magnitude. And they have 2 10GiB ports, though I'm sure you can work with them and get more.. It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. [0]: http://www.newisys.com/Products/4600.shtml http://www.newisys.com/Products/4600.shtml [1]: http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H
- beachstartup 11y ago
  you can fit 72 x 8TB pretty easily into 4u with off-the-shelf hardware. with custom-built 2.5" ssd enclosures you could probably fit more in the same space.
  - codemac 11y ago
    Can you link to the shelf / hardware? I'm curious
    - beachstartup 11y ago
      http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
      - codemac 11y ago
        Surprised at the layout of the drives. Would make it a huge pain to actually replace failing disks - but I like that they put a decent Xeon on there. The connectivity to the drives is really quite bad, but now I'm just being picky. Thank you for the link, really appreciate the follow up!
        scurvy 11y ago
        This is the Quanta version, which is a bit better from a build standpoint: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan...
atYevP 11y ago
Yev here -> #TeamOrangeRed
twelvenmonkeys 11y ago
I thought Dropbox uses Amazon S3 not their own storage?
- johansch 11y ago
  I guess they realized S3 was too expensive, eventually? :)
josefresco 11y ago
Now that's a backhanded compliment.
johansch 11y ago
Put up or shut up. ;)
noja 11y ago
But Backblaze is for backup. Dropbox.. isn't.
- josefresco 11y ago
  I concur. I tried on several occasions to use Dropbox, and Google Drive for my backup needs, but the initial sync was so large (numerous files) that the Windows software would crash (memory). No 64-bit client on the horizon so... I found Backblaze and never looked back.
stanmancan 11y ago
You guys are able to fit 1.8PB into a single 4U? Seriously impressive..!
- arca_vorago 11y ago
  Only way I know how to do that is all flash storage...
rakoo 11y ago
> these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing up to an order of magnitude more storage into 4U. Blog post or it didn't happen :) Seriously though, there seems to be some really cool engineering at Dropbox, first with Go usage (https://twitter.com/jamwt/status/629727590782099456 https://twitter.com/jamwt/status/629727590782099456) and now with your storage capacity. We're all craving for more infrastructure details, see what you consider as the state of the art and how you use the different pieces of software/language !
BinaryIdiot 11y ago
> Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. So basically you decided to go into this thread, post something that isn't possible for anyone here to corroborate and call the original post "far from state of the art"...to what end? What was the purpose of your posting? Just to toot your own horn? I don't get it. If you want to talk about how awesome or state of the art your company does things then great do that. But your actual post? It just comes off as childish like someone showing off they can do something and you're that person that has to stand in between them and the people watching them to say you can do it too and better. Surely we've all seen those very awkward moments in sitcoms. > We're (Dropbox) packing up to an order of magnitude more storage into 4U. Are you sure? They can pack up to 180TB into 4U so you're saying DropBox can pack up to 1,800TB into 4U?
- keypusher 11y ago
  I don't work at Dropbox, but I do work in storage. I've seen similar systems pack ~250-500TB into a 4U box. 60 drives x 6TB drives = 360TB. Not sure how you would get anywhere over 1000TB with today's spinning drives. There is a vendor linked in Backblaze blog post (45drives) which sells 60 drive enclosures, which they somehow list at 480TB but I'm not sure how the math works out there. However, keep in mind that usable capacity is going to be less. Erasure coding has significantly less overhead than traditional raid, but there is still a cost (depends on chosen protection set). http://www.45drives.com/products/storage/xl60.php http://www.45drives.com/products/storage/xl60.php
  - qaq 11y ago
    supermicro has 72 drive server and 90 drives JBOD
    - dmourati 11y ago
      We used these and affectionately called them the mullet as in business in the front, party in the rear. http://www.supermicro.com/products/chassis/4u/847/sc847de26-r2k02jbod.cfm http://www.supermicro.com/products/chassis/4u/847/sc847de26-...
jacquesm 11y ago
That statement carries 0 value without any proof. Backblaze is doing an excellent job documenting their process and shouting 'we are doing 10 times better', 'costs aren't close' and 'far from the state of the art' is pretty lame if you're not willing to back those claims up. I'm a big fan of dropbox but this was below the belt.
- jamwt 11y ago
  Hmm, maybe I was being tone deaf? I've repeatedly tried to state in subsequent children in this thread that it seems as though Backblaze made the right set of tradeoffs given their previous gen, and their time to recoup their investment in this, etc. I was also (somewhat selfishly, I suppose?) trying to tap into the excitement of a group of readers who where thinking about big storage, and let them know that the possibilities were even great given greater investment. The responses I've been getting (including yours) lead me to believe that my comment is being seen as a slight against the set of tradeoffs that Backblaze has made, and that really wasn't how I intended it. Edit: I do disagree about it providing zero value, though. We've had a good subsequent exploratory discussion about ways to optimize storage density and (to some degree) cost. And I hope we're not at the point where everything said on the internet demands "proof"; I'm not sure a blog post would provide that, either!
  - jacquesm 11y ago
    The way to tap into that excitement is by making a blog post of your own highlighting what makes DB unique and special and how you do your thing, not by making disparaging comments about companies active in the same space. I'm sure your post would get just as much or even more attention as this one. It's very easy to do what you just did and quite hard to do what Backblaze did. Essentially, if you're not willing to do what Backblaze did here the graceful thing would be to either stay quiet, simply compliment them on their achievements or, and most useful, to be specific about what could be improved.
    - jamwt 11y ago
      Okay, fair point. Given empirical evidence wrt my blogging discipline (and the fact that I probably can't), it's unlikely that blog post will ever happen. I'll leave this thread alone because there's some useful technical information in here, and amend with an apology.
jamwt 11y ago
I can't edit the parent anymore, but it has been fairly pointed out that my tone of piggybacking on Backblaze's great blog post is a shameful bit of thunder-stealing. The technical community owes a great debt to them for their transparency about building storage clusters. I get excited about large storage systems since that's what I've been focusing on for the last few years, and my enthusiasm overran my tact/prudence. Sorry Backblaze.
mrmondo 11y ago
So what's your performance, public cost breakdown and where's your open sourced design available?
atYevP 11y ago
Hey! Yev from Backblaze here -> If you're still reading the thread, sorry things ended up getting a bit heated in the comments here. We like Dropbox! Our hope is that more and more companies will realize that it's a lot more entertaining to open up and chat about this stuff. Hoping Dropbox joins in soon! It'll make for a lot of interesting conversation. Either way, hope you don't feel too put out!

by jamwt 11y ago

scurvy 11y ago

You're running 4U Quanta T21P-4U boxes but with 40gig OCP mezzanine networking now? I hope you guys aren't using SMR drives... Edit: Also, when are you going to move out of AWS? I really would like our/your clients to use IPv6.

budmang 11y ago

> packing up to an order of magnitude more storage into 4U Where are you buying 40 TB hard drives? I'd love to get some ;-) We actually don't claim these are 'state of the art', more 'state of the inexpensive'. You can buy faster performance storage servers, more redundant servers, etc. But I believe these are one of, if not the, lowest cost storage servers per GB. (And of course, open source.) BTW, I'm a fan of Dropbox (use it personally) and, in general, think there is a ton of value in developers, organizations, and companies sharing what they've learned about storage. The better and lower cost it gets, the more interesting things other people can do with it, the bigger the industry, and the more everyone benefits. (full disclosure - cofounder of Backblaze here)

jamwt 11y ago
Hi Gleb! > Where are you buying 40 TB hard drives? I'd love to get some ;-) Me too. :-) Not 40TB, but larger than 4T and quite a few more of them. > But I believe these are one of, if not the, lowest cost storage servers per GB. Yep, I meant price per GB as well. > I'm a fan of Dropbox (use it personally) Awesome, thanks! I've not used Backblaze personally, but I know some friends who do and they're happy customers. Your guys' blog and rapport with the technical community is amazing.

codemac 11y ago

What order of magnitude? There are 60 drive chassis in 4U you can get[0][1], but even a 4TB -> 6TB drive + 15 more drives isn't exactly an order of magnitude. And they have 2 10GiB ports, though I'm sure you can work with them and get more.. It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. [0]: http://www.newisys.com/Products/4600.shtml http://www.newisys.com/Products/4600.shtml [1]: http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H

beachstartup 11y ago
you can fit 72 x 8TB pretty easily into 4u with off-the-shelf hardware. with custom-built 2.5" ssd enclosures you could probably fit more in the same space.
- codemac 11y ago
  Can you link to the shelf / hardware? I'm curious
  - beachstartup 11y ago
    http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
    - codemac 11y ago
      Surprised at the layout of the drives. Would make it a huge pain to actually replace failing disks - but I like that they put a decent Xeon on there. The connectivity to the drives is really quite bad, but now I'm just being picky. Thank you for the link, really appreciate the follow up!
      - scurvy 11y ago
        This is the Quanta version, which is a bit better from a build standpoint: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan...

atYevP 11y ago

Yev here -> #TeamOrangeRed

twelvenmonkeys 11y ago

I thought Dropbox uses Amazon S3 not their own storage?

johansch 11y ago
I guess they realized S3 was too expensive, eventually? :)

josefresco 11y ago

Now that's a backhanded compliment.

johansch 11y ago

Put up or shut up. ;)

noja 11y ago

But Backblaze is for backup. Dropbox.. isn't.

josefresco 11y ago
I concur. I tried on several occasions to use Dropbox, and Google Drive for my backup needs, but the initial sync was so large (numerous files) that the Windows software would crash (memory). No 64-bit client on the horizon so... I found Backblaze and never looked back.

stanmancan 11y ago

You guys are able to fit 1.8PB into a single 4U? Seriously impressive..!

arca_vorago 11y ago
Only way I know how to do that is all flash storage...

rakoo 11y ago

> these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing up to an order of magnitude more storage into 4U. Blog post or it didn't happen :) Seriously though, there seems to be some really cool engineering at Dropbox, first with Go usage (https://twitter.com/jamwt/status/629727590782099456 https://twitter.com/jamwt/status/629727590782099456) and now with your storage capacity. We're all craving for more infrastructure details, see what you consider as the state of the art and how you use the different pieces of software/language !

BinaryIdiot 11y ago

> Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. So basically you decided to go into this thread, post something that isn't possible for anyone here to corroborate and call the original post "far from state of the art"...to what end? What was the purpose of your posting? Just to toot your own horn? I don't get it. If you want to talk about how awesome or state of the art your company does things then great do that. But your actual post? It just comes off as childish like someone showing off they can do something and you're that person that has to stand in between them and the people watching them to say you can do it too and better. Surely we've all seen those very awkward moments in sitcoms. > We're (Dropbox) packing up to an order of magnitude more storage into 4U. Are you sure? They can pack up to 180TB into 4U so you're saying DropBox can pack up to 1,800TB into 4U?

keypusher 11y ago
I don't work at Dropbox, but I do work in storage. I've seen similar systems pack ~250-500TB into a 4U box. 60 drives x 6TB drives = 360TB. Not sure how you would get anywhere over 1000TB with today's spinning drives. There is a vendor linked in Backblaze blog post (45drives) which sells 60 drive enclosures, which they somehow list at 480TB but I'm not sure how the math works out there. However, keep in mind that usable capacity is going to be less. Erasure coding has significantly less overhead than traditional raid, but there is still a cost (depends on chosen protection set). http://www.45drives.com/products/storage/xl60.php http://www.45drives.com/products/storage/xl60.php
- qaq 11y ago
  supermicro has 72 drive server and 90 drives JBOD
  - dmourati 11y ago
    We used these and affectionately called them the mullet as in business in the front, party in the rear. http://www.supermicro.com/products/chassis/4u/847/sc847de26-r2k02jbod.cfm http://www.supermicro.com/products/chassis/4u/847/sc847de26-...

jacquesm 11y ago

That statement carries 0 value without any proof. Backblaze is doing an excellent job documenting their process and shouting 'we are doing 10 times better', 'costs aren't close' and 'far from the state of the art' is pretty lame if you're not willing to back those claims up. I'm a big fan of dropbox but this was below the belt.

jamwt 11y ago
Hmm, maybe I was being tone deaf? I've repeatedly tried to state in subsequent children in this thread that it seems as though Backblaze made the right set of tradeoffs given their previous gen, and their time to recoup their investment in this, etc. I was also (somewhat selfishly, I suppose?) trying to tap into the excitement of a group of readers who where thinking about big storage, and let them know that the possibilities were even great given greater investment. The responses I've been getting (including yours) lead me to believe that my comment is being seen as a slight against the set of tradeoffs that Backblaze has made, and that really wasn't how I intended it. Edit: I do disagree about it providing zero value, though. We've had a good subsequent exploratory discussion about ways to optimize storage density and (to some degree) cost. And I hope we're not at the point where everything said on the internet demands "proof"; I'm not sure a blog post would provide that, either!
- jacquesm 11y ago
  The way to tap into that excitement is by making a blog post of your own highlighting what makes DB unique and special and how you do your thing, not by making disparaging comments about companies active in the same space. I'm sure your post would get just as much or even more attention as this one. It's very easy to do what you just did and quite hard to do what Backblaze did. Essentially, if you're not willing to do what Backblaze did here the graceful thing would be to either stay quiet, simply compliment them on their achievements or, and most useful, to be specific about what could be improved.
  - jamwt 11y ago
    Okay, fair point. Given empirical evidence wrt my blogging discipline (and the fact that I probably can't), it's unlikely that blog post will ever happen. I'll leave this thread alone because there's some useful technical information in here, and amend with an apology.

jamwt 11y ago

I can't edit the parent anymore, but it has been fairly pointed out that my tone of piggybacking on Backblaze's great blog post is a shameful bit of thunder-stealing. The technical community owes a great debt to them for their transparency about building storage clusters. I get excited about large storage systems since that's what I've been focusing on for the last few years, and my enthusiasm overran my tact/prudence. Sorry Backblaze.

mrmondo 11y ago

So what's your performance, public cost breakdown and where's your open sourced design available?

atYevP 11y ago

Hey! Yev from Backblaze here -> If you're still reading the thread, sorry things ended up getting a bit heated in the comments here. We like Dropbox! Our hope is that more and more companies will realize that it's a lot more entertaining to open up and chat about this stuff. Hoping Dropbox joins in soon! It'll make for a lot of interesting conversation. Either way, hope you don't feel too put out!