4 ms·

I'm looking at http://www.supermicro.com/products/chassis/4U/847/SC847E26-R1400U.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives

by lsc 15y ago

I'm looking at http://www.supermicro.com/products/chassis/4U/847/SC847E26-R1400U.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives, and room for a motherboard) or http://www.supermicro.com/products/chassis/4U/847/SC847E16-RJBOD1.cfm http://www.supermicro.com/products/chassis/4U/847/SC847E16-R... (45 drives, das. I have had serious outages caused by DAS back in the SCSI-3 VHDCI days, so I'm likely to start with the 36 bay system.) for my mass storage project. they are both well under $1500 with power supplies, backplanes and expanders taken care of. For me, the big thing is that with my storage model, I'm going to be replacing disks as they fail and rebuilding the RAID, so having easily accessible and easily swapped disks is worth paying a premium. (I am planning to have some cross-chassis redundancy by using zfs snapshots, but I'd rather just keep the nodes going as is.) Also, rack density? doesn't save you that much money. Most of what you are paying for in a data center is power. At the cheapest co-lo I'm in, here is the cost breakdown: 1 full cabinet (44u) with two twenty-amp 120v circuits: $875 1 full cabinet (44u) with one twenty-amp 120v circuit: $530 So, if I can double my density, I save $185 a month; and even at the disk density for my compute nodes (close to 100 disks in a rack) I get one disk failure maybe every two months per rack; so if I have to slide out the whole goddam computer, causing some chance of the power getting disconnected and downtime? yeah, with my model? it's probably worth paying the premium. I'm just saying; if you are small enough that paying five grand for a backblaze pod. Sure, at-scale, it's best to design your systems so that you can get zero downtime even when hardware fails. But, that's really difficult to do without introducing new failure modes; even amazon has trouble with it. My strategy is to accept that hardware failures mean a truck roll and downtime for customers on the hardware in question. As long as you don't have any one server go down more than once a year, (and a particular server failing once a year is pretty pathetic) and as long as you don't have a system where any one server brings down everyone, you are going to see pretty good reliability using this strategy.

driverdan 15y ago
Backblaze designed their system to be redundant across multiple systems with the ability for one to go down seamlessly. If losing one system from your network will cause unacceptable downtime then you shouldn't consider using Backblaze's design.
- lsc 15y ago
  clearly. I'm just saying, it's much easier to design a system where hardware failures cause downtime, and where that downtime is made acceptable because hardware with redundant drives doesn't fail that often than it is to design a system where hardware failures don't cause downtime (and where a failed hard drive causes a full node failure) Thus, to someone who isn't at scale, the supermicro systems are likely going to be cheaper, overall, than the backblaze pods. (It sounds like these people paid more per disk for the backblaze pods than I'm going to pay for the supermicro pods anyhow.)
nwilkens 15y ago
I just built two of these 36 drive systems. $9k for 108TB raw. Feel free to drop me a line, and I will share any details.
- lsc 15y ago
  sweet. what'd you use for your sas card?
  - ballen 15y ago
    I've also done three of these builds so far. Used the SC847A chassis with direct iPass cable access (i.e. no port multipliers), 4 drives per cable. Downside is you need 9x SFF-8087 connectors on controllers, and 9 iPass cables to somehow route. Don't get the SM iPass cables, TrendNet makes better ones. Upside is you have dedicated SAS2 bandwidth from the drive all the way though the controller and the PCIe bus. Likely overkill. Also a tip, you can mount 4x internal 2.5in or 2x 3.5in drives. SM has the part numbers for the brackets on the chassis' product page. Don't put anything you'd remotely want to hot swap in these brackets, they will be buried under the motherboard tray. I've used 4x LSI Logic 9211-8i controllers plus the onboard of the SM X8DTH-6F. Both the onboard and the 9211-8i use the LSI 2008 chipset. I have Solaris and ZFS setting on top of these, so I don't have hardware raid. This is actually very solid hardware so far. I had a PSU fail and thats it. Let me know if you have any questions.
e1ven 15y ago
Who would you buy through? Googling for the 45 drive version gives me slightly higher pricing. Also- The big advantage of the BackBlaze version is that people have done it before, and it mapped out. With the SM case, there's less community.. But it does look like a good solution. It looks like the drive bays are pre-wired up, so you wouldn't need to worry about that? What else would you need? Mboard/CPU/RAM, Raid cards, HDs and sleeves?
- lsc 15y ago
  Like most supermicro chassis, it comes with the drive caddies, backplane, and power supplies. All you need is the motherboard, cpu, ram, and a SAS card, well, and the drives. It's even got an expander built into the backplane. If you want h/w raid, you need to bring that as well. (I plan on using zraid2, most of the raid cards that cost less per port than the drives are not better than software raid.) I buy most of my supermicro stuff through kingstarusa.com - I know the site looks a little shady, and you have to email for quotes for almost everything, but they are good people. My office is actually above their warehouse; I'm unit C. Their price is usually a few dollars less than provantage, which is usually the next best retailer for supermicro chassis, and they don't do shady tax dodge bullshit, and I don't have to pay shipping. I could be misremembering on the exact price on the 45 bay. Most of the 'mapping out' done with the backblaze version is already done and tested on the supermicro.
  - wmf 15y ago
    most of the raid cards that cost less per port than the drives are not better than software raid Let's not get carried away; isn't a "gold standard" LSI controller only ~$1,000 ($27/drive)? But you certainly can buy a lot of Intel cores for that price.
    - lsc 15y ago
      yeah, I am exaggerating on the port cost some, but the big advantage of hardware raid over software isn't the hardware calculation of parity. A CPU can calculate parity so much faster than you can write to a drive that it doesn't matter at all that special hardware can do so even faster still. The advantage of the hardware raid card is the battery backed cache. If it doesn't have a BBU and a fair amount of cache, as far as I am concerned, you might as well be using MD. Hardware RAID cards have improved quite a lot recently; some of the stuff now has reasonably sized caches, so perhaps I should revisit my assumptions in this area. Of course, I'm planning on using ZFS on my storage servers, so even if hardware raid cards are now a reasonably good deal, they won't do me a whole lot of good.
serverascode 15y ago
This is a great thread here. I think there are a lot of people, even institutions like where I work, that are looking hard at storage that isn't block based storage from some huge vendor or storage that ends up being $5000+/TB (looking at you Isilon). Ok, maybe it's just me all alone here at my work place. :) Even running RAID6 or RAID10 over 3TB SAS drives on some largish enclosures (ie. 36,45,60 drives per 4U) will be very cost effective even just using md, lvm, and xfs. Or even better, use ZFS if possible. Not a huge fan of Solaris, but there is OpenIndiana, or FreeBSD has ZFS. And object storage such as openstack swift is really gaining momentum and will likely replace most storage systems for large amounts of data over the next couple/5 years. There are single orgs that have put out 5.5 petabytes of openstack swift storage! Right now!

by lsc 15y ago

driverdan 15y ago

Backblaze designed their system to be redundant across multiple systems with the ability for one to go down seamlessly. If losing one system from your network will cause unacceptable downtime then you shouldn't consider using Backblaze's design.

lsc 15y ago
clearly. I'm just saying, it's much easier to design a system where hardware failures cause downtime, and where that downtime is made acceptable because hardware with redundant drives doesn't fail that often than it is to design a system where hardware failures don't cause downtime (and where a failed hard drive causes a full node failure) Thus, to someone who isn't at scale, the supermicro systems are likely going to be cheaper, overall, than the backblaze pods. (It sounds like these people paid more per disk for the backblaze pods than I'm going to pay for the supermicro pods anyhow.)

nwilkens 15y ago

I just built two of these 36 drive systems. $9k for 108TB raw. Feel free to drop me a line, and I will share any details.

lsc 15y ago
sweet. what'd you use for your sas card?
- ballen 15y ago
  I've also done three of these builds so far. Used the SC847A chassis with direct iPass cable access (i.e. no port multipliers), 4 drives per cable. Downside is you need 9x SFF-8087 connectors on controllers, and 9 iPass cables to somehow route. Don't get the SM iPass cables, TrendNet makes better ones. Upside is you have dedicated SAS2 bandwidth from the drive all the way though the controller and the PCIe bus. Likely overkill. Also a tip, you can mount 4x internal 2.5in or 2x 3.5in drives. SM has the part numbers for the brackets on the chassis' product page. Don't put anything you'd remotely want to hot swap in these brackets, they will be buried under the motherboard tray. I've used 4x LSI Logic 9211-8i controllers plus the onboard of the SM X8DTH-6F. Both the onboard and the 9211-8i use the LSI 2008 chipset. I have Solaris and ZFS setting on top of these, so I don't have hardware raid. This is actually very solid hardware so far. I had a PSU fail and thats it. Let me know if you have any questions.

e1ven 15y ago

Who would you buy through? Googling for the 45 drive version gives me slightly higher pricing. Also- The big advantage of the BackBlaze version is that people have done it before, and it mapped out. With the SM case, there's less community.. But it does look like a good solution. It looks like the drive bays are pre-wired up, so you wouldn't need to worry about that? What else would you need? Mboard/CPU/RAM, Raid cards, HDs and sleeves?

lsc 15y ago
Like most supermicro chassis, it comes with the drive caddies, backplane, and power supplies. All you need is the motherboard, cpu, ram, and a SAS card, well, and the drives. It's even got an expander built into the backplane. If you want h/w raid, you need to bring that as well. (I plan on using zraid2, most of the raid cards that cost less per port than the drives are not better than software raid.) I buy most of my supermicro stuff through kingstarusa.com - I know the site looks a little shady, and you have to email for quotes for almost everything, but they are good people. My office is actually above their warehouse; I'm unit C. Their price is usually a few dollars less than provantage, which is usually the next best retailer for supermicro chassis, and they don't do shady tax dodge bullshit, and I don't have to pay shipping. I could be misremembering on the exact price on the 45 bay. Most of the 'mapping out' done with the backblaze version is already done and tested on the supermicro.
- wmf 15y ago
  most of the raid cards that cost less per port than the drives are not better than software raid Let's not get carried away; isn't a "gold standard" LSI controller only ~$1,000 ($27/drive)? But you certainly can buy a lot of Intel cores for that price.
  - lsc 15y ago
    yeah, I am exaggerating on the port cost some, but the big advantage of hardware raid over software isn't the hardware calculation of parity. A CPU can calculate parity so much faster than you can write to a drive that it doesn't matter at all that special hardware can do so even faster still. The advantage of the hardware raid card is the battery backed cache. If it doesn't have a BBU and a fair amount of cache, as far as I am concerned, you might as well be using MD. Hardware RAID cards have improved quite a lot recently; some of the stuff now has reasonably sized caches, so perhaps I should revisit my assumptions in this area. Of course, I'm planning on using ZFS on my storage servers, so even if hardware raid cards are now a reasonably good deal, they won't do me a whole lot of good.

serverascode 15y ago

This is a great thread here. I think there are a lot of people, even institutions like where I work, that are looking hard at storage that isn't block based storage from some huge vendor or storage that ends up being $5000+/TB (looking at you Isilon). Ok, maybe it's just me all alone here at my work place. :) Even running RAID6 or RAID10 over 3TB SAS drives on some largish enclosures (ie. 36,45,60 drives per 4U) will be very cost effective even just using md, lvm, and xfs. Or even better, use ZFS if possible. Not a huge fan of Solaris, but there is OpenIndiana, or FreeBSD has ZFS. And object storage such as openstack swift is really gaining momentum and will likely replace most storage systems for large amounts of data over the next couple/5 years. There are single orgs that have put out 5.5 petabytes of openstack swift storage! Right now!