6 ms·

What order of magnitude? There are 60 drive chassis in 4U you can get[0][1], but even a 4TB -> 6TB drive + 15 more drives isn't exactly an order of magnitude.

by codemac 11y ago

What order of magnitude? There are 60 drive chassis in 4U you can get[0][1], but even a 4TB -> 6TB drive + 15 more drives isn't exactly an order of magnitude. And they have 2 10GiB ports, though I'm sure you can work with them and get more.. It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. [0]: http://www.newisys.com/Products/4600.shtml http://www.newisys.com/Products/4600.shtml [1]: http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H http://www.aicipc.com/ProductDetail.aspx?ref=RSC-4H

beachstartup 11y ago
you can fit 72 x 8TB pretty easily into 4u with off-the-shelf hardware. with custom-built 2.5" ssd enclosures you could probably fit more in the same space.
- codemac 11y ago
  Can you link to the shelf / hardware? I'm curious
  - beachstartup 11y ago
    http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
    - codemac 11y ago
      Surprised at the layout of the drives. Would make it a huge pain to actually replace failing disks - but I like that they put a decent Xeon on there. The connectivity to the drives is really quite bad, but now I'm just being picky. Thank you for the link, really appreciate the follow up!
      - scurvy 11y ago
        This is the Quanta version, which is a bit better from a build standpoint: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan...
jamwt 11y ago
You can do a lot more than 60, and you can get a lot bigger than 6T. > It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. So, I never said it didn't meet their design goal; it looks like quite a nice system, and it's better than their previous gen. Looks like they're doing things right, looks like solid engineering. But I stand by the statement that, in the absolute sense, this is not quite the current benchmark on density or $/GB in the industry. That was the only point I was trying to make to the HN audience in general, and it's not intended as a critique of Backblaze specifically. But there are very different amounts of resources (I'm assuming) invested in achieving those benchmarks than Backblaze can likely commit. Ergo, Backblaze probably completely made the right decision for Backblaze. Both of these things are possible.
- codemac 11y ago
  > you can get a lot bigger than 6T Come on - most companies reasonably can't. There's a WD 10T smr/helium drive, and seagate is selling that crap 8T video drive. Unless you're talking ssd 3d/TLC density (more $), or the vapor-ware 20T drives that some storage vendors hint at and don't let me file a PO for, I literally don't know what you're referring to (now I'm getting where I can't name names) The reason I'm complaining is because of you don't state: - what the current state of the art is that they're not meeting - what they could do better with their design goals given that information. You merely state that it's just not state of the art, and to come work at dropbox. This is what I mean by being disingenuous.
  - jamwt 11y ago
    > You merely state that it's just not state of the art, and to come work at dropbox. That's about all I can do. However... > This is what I mean by being disingenuous. I'm don't think disingenuous is what I'm being. Maybe I'm not being as forthcoming as you'd like. Maybe my comment wasn't useful without details I'm not providing. But I'm not lying or misrepresenting anything. So let me try to be helpful without using proprietary information. Using information people have unearthed in this thread, we can do some math to prove I'm not completely bonkers: 96 unit chassis * 10T SMR = 960T, which is in the > 5x the density in 4U SMR $/GB is quite good if you work with the right vendor and make the right tradeoffs. If you could manage that 960T with the same compute resources, you'd save a lot on compute, since that's currently a ~30% of their cost. With custom enclosures and hardware, you can do more still. You can have deep relationships with vendors and have them customize firmware for your particular use case. You can take their discards that didn't quite meet spec. You can buy crazy cheap stock with high failure rates b/c your distributed system is insanely good at repairs. You get access to stuff that's not being sold yet. However, let me state again--this would take a lot of work. You need to mold your software around your hardware profile with very small tolerances. You might need to move some I/O stuff in-kernel, and/or move the IP stack out of kernel. You might need to reinforce your DC's raised floors to handle the weight (spoiler: you definitely do unless you planned for these types of machines ahead of time). You need great operations, and really thorough hardware quals b/c your failure domain is huge. Network traffic will go through the roof if a box fails, since you're repairing so much data. Dozens of people on a handfuls of teams will spend time attacking this problem from different angles. So, again--I don't necessarily think Backblaze did anything wrong here. Doing all this might be a crazy idea for them and not worth it. My statement was: it's not state of the art in terms of $/GB or density. But for some shops, they spend so much on storage they can afford invest really, really deeply on optimizing storage cost. That's where the state of the art is.
chx 11y ago
A 96 bay 4U chassis is readily available http://www.raidinc.com/products/object-storage/ability-4u-96-bay http://www.raidinc.com/products/object-storage/ability-4u-96... but that's still not a magnitude.
- codemac 11y ago
  Oh nice, I hadn't seen that! Need to look more at it's data sheet.
- atYevP 11y ago
  Heh, any place where I have to request a quote before they show me a price makes me nervous :D
  - epistasis 11y ago
    It should if you're thinking of buying one-offs. But at bigger scales like yours, you can either invest in crew to design chassis, or in purchasing people to deal with other sales people.
- codemac 11y ago
  Intel Atom is an odd choice, but liking the ethernet choice. Gonna look more into how they lay the disks out. I wonder what the serviceability of the drives is.
- hrez 11y ago
  That's just a storage enclosure. You'd need a separate server with SAS connectivity to use that. It still may be worth it $$/Gb wise depending on cost.
  - codemac 11y ago
    They have an Intel Atom addon module thingie, but the CPU can only really run 2x4Gb best case, almost all of these will need separate software running on Xeons.
- devonkim 11y ago
rstupek 11y ago
Maybe his definition of an order of magnitude is different than anyone elses? I'd like to hear how many drives and what size he's talking about
mSparks 11y ago
I'd say it's more Dropbox getting nervous. Dropbox wants all this trade secret non sense. And everyone else give them their data. That they promise they won't read. Give me a break. Bittorrent sync and blaze pods in house for the win.
michaelt 11y ago
What order of magnitude? Well, this article describes storage pods with 180TB of storage. HGST have a 10TB hard drive measuring 26.1mm * 101.6mm * 147mm [1] and a 4U chassis measures 178mm * 437mm * 521mm [2] If you had 4U of hard drives with no space for power, cooling, interconnect etc you cold fit 10 terabytes / (26.1mm * 101.6mm * 147mm) * (178mm * 437mm * 521mm) = 1.04 petabytes into 4U I don't know whether six times counts a a single order of magnitude, let alone multiple orders of magnitude, and the 10TB hard drives I linked are "enterprise" so there might be a substantial price premium. Samsung claim to have a 16TB 2.5" SSD in the works [2] - although details are pretty scarce at the moment. If its dimensions match normal 2.5" SSDs at around 100mm * 70mm * 6.9mm that would give 16 terabytes / (100mm * 70mm * 7mm) * (178mm * 437mm * 521mm) = 13.2 petabytes in 4U. Of course, the assumption you could use 100% of the space in a chassis for drives is pretty unrealistic. And nobody's seen the Samsung SSD working - could be a 16tb sticker on a 1tb ssd for all I know :) Filling it with 4TB SSDs would give you 3.35 PB and it actually seems possible to get those [4]. If you don't care about $/TB, it's probably possible to get density a single order of magnitude higher. [1] http://www.hgst.com/products/hard-drives/ultrastar-archive-ha10 http://www.hgst.com/products/hard-drives/ultrastar-archive-h... [2] http://www.supermicro.co.uk/products/chassis/4U/842/SC842TQ-865.cfm http://www.supermicro.co.uk/products/chassis/4U/842/SC842TQ-... [3] http://arstechnica.co.uk/gadgets/2015/08/samsung-unveils-2-5-inch-16tb-ssd-the-worlds-largest-hard-drive/ http://arstechnica.co.uk/gadgets/2015/08/samsung-unveils-2-5... [4] https://www.sandisk.com/business/datacenter/products/flash-devices/ssds/sas-ssd/optimus https://www.sandisk.com/business/datacenter/products/flash-d...
- kalleboo 11y ago
  > HGST have a 10TB hard drive "Archive drive" => shingled writes. Are people really using these things for online workloads? Might make sense for Backblaze's original "write and forget" backup workload but not very well for B2 or Dropbox

by codemac 11y ago

beachstartup 11y ago

you can fit 72 x 8TB pretty easily into 4u with off-the-shelf hardware. with custom-built 2.5" ssd enclosures you could probably fit more in the same space.

codemac 11y ago
Can you link to the shelf / hardware? I'm curious
- beachstartup 11y ago
  http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R72L.cfm http://www.supermicro.com/products/system/4U/6047/SSG-6047R-...
  - codemac 11y ago
    Surprised at the layout of the drives. Would make it a huge pain to actually replace failing disks - but I like that they put a decent Xeon on there. The connectivity to the drives is really quite bad, but now I'm just being picky. Thank you for the link, really appreciate the follow up!
    - scurvy 11y ago
      This is the Quanta version, which is a bit better from a build standpoint: http://www.qct.io/Product/Storage/Storage-Server/4U/New-QuantaPlex-T21P-4U-p291c77c71c150c222 http://www.qct.io/Product/Storage/Storage-Server/4U/New-Quan...

jamwt 11y ago

You can do a lot more than 60, and you can get a lot bigger than 6T. > It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. So, I never said it didn't meet their design goal; it looks like quite a nice system, and it's better than their previous gen. Looks like they're doing things right, looks like solid engineering. But I stand by the statement that, in the absolute sense, this is not quite the current benchmark on density or $/GB in the industry. That was the only point I was trying to make to the HN audience in general, and it's not intended as a critique of Backblaze specifically. But there are very different amounts of resources (I'm assuming) invested in achieving those benchmarks than Backblaze can likely commit. Ergo, Backblaze probably completely made the right decision for Backblaze. Both of these things are possible.

codemac 11y ago
> you can get a lot bigger than 6T Come on - most companies reasonably can't. There's a WD 10T smr/helium drive, and seagate is selling that crap 8T video drive. Unless you're talking ssd 3d/TLC density (more $), or the vapor-ware 20T drives that some storage vendors hint at and don't let me file a PO for, I literally don't know what you're referring to (now I'm getting where I can't name names) The reason I'm complaining is because of you don't state: - what the current state of the art is that they're not meeting - what they could do better with their design goals given that information. You merely state that it's just not state of the art, and to come work at dropbox. This is what I mean by being disingenuous.
- jamwt 11y ago
  > You merely state that it's just not state of the art, and to come work at dropbox. That's about all I can do. However... > This is what I mean by being disingenuous. I'm don't think disingenuous is what I'm being. Maybe I'm not being as forthcoming as you'd like. Maybe my comment wasn't useful without details I'm not providing. But I'm not lying or misrepresenting anything. So let me try to be helpful without using proprietary information. Using information people have unearthed in this thread, we can do some math to prove I'm not completely bonkers: 96 unit chassis * 10T SMR = 960T, which is in the > 5x the density in 4U SMR $/GB is quite good if you work with the right vendor and make the right tradeoffs. If you could manage that 960T with the same compute resources, you'd save a lot on compute, since that's currently a ~30% of their cost. With custom enclosures and hardware, you can do more still. You can have deep relationships with vendors and have them customize firmware for your particular use case. You can take their discards that didn't quite meet spec. You can buy crazy cheap stock with high failure rates b/c your distributed system is insanely good at repairs. You get access to stuff that's not being sold yet. However, let me state again--this would take a lot of work. You need to mold your software around your hardware profile with very small tolerances. You might need to move some I/O stuff in-kernel, and/or move the IP stack out of kernel. You might need to reinforce your DC's raised floors to handle the weight (spoiler: you definitely do unless you planned for these types of machines ahead of time). You need great operations, and really thorough hardware quals b/c your failure domain is huge. Network traffic will go through the roof if a box fails, since you're repairing so much data. Dozens of people on a handfuls of teams will spend time attacking this problem from different angles. So, again--I don't necessarily think Backblaze did anything wrong here. Doing all this might be a crazy idea for them and not worth it. My statement was: it's not state of the art in terms of $/GB or density. But for some shops, they spend so much on storage they can afford invest really, really deeply on optimizing storage cost. That's where the state of the art is.

chx 11y ago

A 96 bay 4U chassis is readily available http://www.raidinc.com/products/object-storage/ability-4u-96-bay http://www.raidinc.com/products/object-storage/ability-4u-96... but that's still not a magnitude.

codemac 11y ago
Oh nice, I hadn't seen that! Need to look more at it's data sheet.
atYevP 11y ago
Heh, any place where I have to request a quote before they show me a price makes me nervous :D
- epistasis 11y ago
  It should if you're thinking of buying one-offs. But at bigger scales like yours, you can either invest in crew to design chassis, or in purchasing people to deal with other sales people.
codemac 11y ago
Intel Atom is an odd choice, but liking the ethernet choice. Gonna look more into how they lay the disks out. I wonder what the serviceability of the drives is.
hrez 11y ago
That's just a storage enclosure. You'd need a separate server with SAS connectivity to use that. It still may be worth it $$/Gb wise depending on cost.
- codemac 11y ago
  They have an Intel Atom addon module thingie, but the CPU can only really run 2x4Gb best case, almost all of these will need separate software running on Xeons.
devonkim 11y ago

rstupek 11y ago

Maybe his definition of an order of magnitude is different than anyone elses? I'd like to hear how many drives and what size he's talking about

mSparks 11y ago

I'd say it's more Dropbox getting nervous. Dropbox wants all this trade secret non sense. And everyone else give them their data. That they promise they won't read. Give me a break. Bittorrent sync and blaze pods in house for the win.

michaelt 11y ago

What order of magnitude? Well, this article describes storage pods with 180TB of storage. HGST have a 10TB hard drive measuring 26.1mm * 101.6mm * 147mm [1] and a 4U chassis measures 178mm * 437mm * 521mm [2] If you had 4U of hard drives with no space for power, cooling, interconnect etc you cold fit 10 terabytes / (26.1mm * 101.6mm * 147mm) * (178mm * 437mm * 521mm) = 1.04 petabytes into 4U I don't know whether six times counts a a single order of magnitude, let alone multiple orders of magnitude, and the 10TB hard drives I linked are "enterprise" so there might be a substantial price premium. Samsung claim to have a 16TB 2.5" SSD in the works [2] - although details are pretty scarce at the moment. If its dimensions match normal 2.5" SSDs at around 100mm * 70mm * 6.9mm that would give 16 terabytes / (100mm * 70mm * 7mm) * (178mm * 437mm * 521mm) = 13.2 petabytes in 4U. Of course, the assumption you could use 100% of the space in a chassis for drives is pretty unrealistic. And nobody's seen the Samsung SSD working - could be a 16tb sticker on a 1tb ssd for all I know :) Filling it with 4TB SSDs would give you 3.35 PB and it actually seems possible to get those [4]. If you don't care about $/TB, it's probably possible to get density a single order of magnitude higher. [1] http://www.hgst.com/products/hard-drives/ultrastar-archive-ha10 http://www.hgst.com/products/hard-drives/ultrastar-archive-h... [2] http://www.supermicro.co.uk/products/chassis/4U/842/SC842TQ-865.cfm http://www.supermicro.co.uk/products/chassis/4U/842/SC842TQ-... [3] http://arstechnica.co.uk/gadgets/2015/08/samsung-unveils-2-5-inch-16tb-ssd-the-worlds-largest-hard-drive/ http://arstechnica.co.uk/gadgets/2015/08/samsung-unveils-2-5... [4] https://www.sandisk.com/business/datacenter/products/flash-devices/ssds/sas-ssd/optimus https://www.sandisk.com/business/datacenter/products/flash-d...

kalleboo 11y ago
> HGST have a 10TB hard drive "Archive drive" => shingled writes. Are people really using these things for online workloads? Might make sense for Backblaze's original "write and forget" backup workload but not very well for B2 or Dropbox