7 ms·

Building a high performance SSD SAN, part 1

69 points by runarb 12y ago

jjoe 12y ago
I understand his decision to skip the HW RAID controller (I like mdadm too). But a BBU is definitely critical to this op. NVMe has so many more queues that on a busy box it's going to hold way too much uncommitted data and metadata. Now the only other reasonable option is a UPS but it adds significant RUs/cost to the setup.
- peterjs 12y ago
  I went down the same road - using mdadm and a UPS instead of a HW RAID controller with battery backup unit. LSI MegaRAID SAS 9261-8i SGL with a battery backup costs roughly 650 EUR; Eaton 9130i 1500VA Rack 2U UPS costs 850 EUR. So it's a bit more expensive, but you would be buying that UPS anyway.
- runarb 12y ago
  I am no expert, but are you sure that uncommitted data on the ssd is an issue? The Intel DC P3600 has a built in capacitor that should give it econoff backup power to commit the data. The SanDisk Extreme Pro disks don't have volatile cache, but instead uses SLC NAND flash for cashing, that will survive a power loss. There is of course still the issue that data send to the server and stored in main memory will be lost if you loses power, but a HW RAID controller with battery backup would not have prevented that (but an UPS of course might).
  - mrmondo 12y ago
    I tend to agree with you here runarb, I truly don't believe there is much that a hardware RAID contrôlée and battery will protect you from with modern flash storage, at this point I think time is better spent on improving / tuning the database and filesystem for resiliency.
  - otterley 12y ago
    It would definitely prevent an uncommitted data problem if fsync or O_DIRECT were used (which should always be used for critical writes). UPSes defend against power outages, but they're only one rung on the data-integrity ladder. Controller BBUs, on the other hand, protect against both power outages AND kernel panics.
    - mrmondo 12y ago
      Indeed we have fsync enabled on our database servers (PostgreSQL)
      - mrmondo 12y ago
        (And just wanted to point out that that is only one part of the equation)
- KaiserPro 12y ago
  UPSs all the way. With high random writes you'll get alot of buffering so even if you have a BBU for the raid, the data might still be in RAM. the order of shutdown is important too, should really be 1) consumers of storage (within the first minute or so) 2) storage (about 15 minutes after) if your network wobbles or the storage turns off too fast, everything hangs, and your in poostain pants town. Ideally your DNS/AD shouldn't ever willingly shutdown. or if it does it should be well after the last bit of your production system has turned off.
- fweespeech 12y ago
  I'd be willing to assume they have a UPS capable of handing the servers in each rack to allow for controlled shutdowns in the event of a power outage. Hell, I have a UPS in place for my desktop, modem, and router at home.
mrmondo 12y ago
I'm the author (Sam Mcleod) of this post and just saw this was posted on HN. I'm quite a bit further down the track now and should have a much better write up to post in the next week or so time providing. I can say that I'm hitting over 1,000,000 IOP/s on 4k random reads and around 850,000 IOP/s random 4k writes per 1U unit without cutting any corners with data safety. What I haven't finished yet is the cross-unit replication tuning and the load balancing / multipathing. So far everything has gone to plan and is progressing well as a serious replacement for our 'traditional' storage. Regardless, there's still quite a lot of testing to be done and once I have more of that completely I'll post an update with some serious numbers and observations.
- moe 12y ago
  cross-unit replication tuning and the load balancing / multipathing. That's going to be interesting. At those speeds (1.5GB/s, 850k IOPS) you must bump[1] into all kinds of DRBD bottlenecks? Also I'm very curious about your approach to redundancy/multipath. Dual-primary or failover? [1] http://blog.gmane.org/gmane.comp.linux.drbd/page=41 http://blog.gmane.org/gmane.comp.linux.drbd/page=41
  - mrmondo 12y ago
    I have no illusions that I'll be able to maintain such performance with DRBD replicating and iSCSI overheads. My current feeling on load balancing is to keep it simple, have half the clients treat node A as their active storage and the other half using node B. If node A becomes unavailable failover will occur and node B will serve all clients. As far as multipathing goes - my approach is that I honestly have no idea. That's something that's going to be learnt along the way and I'm in uncertain if it'll make it into the final build or not. Edit: I can't get that link to load on my connection at present but I've bookmarked to look at tomorrow. If you have any experience / advice I'm always very open to assistance!
    - FlyingAvatar 12y ago
      I was quite amused to see this posted as we just built nearly the exact same thing about 6 months ago (minus the NVMe drives which weren't available). We are using ZFSonLinux on top of DRBD instead of mdmadm, which has been surprisingly successful. The biggest frustration that we met was using LACP in trying to get a better data rate out of DRBD. Regardless of the LACP mode, we were not able to get DRBD to scale to full bandwidth (20gbps) and had to settle for using it for redundancy only. I imagine with your setup (read: much faster peak write capability), you might actually be able to bottleneck DRBD pretty well over a single 10GBE pipe if you were writing at peak capacity. I'll be interested to see if you happen upon a work-around. Load balancing as you've suggested is an intriguing compromise. Have you tried doing a two-way heartbeat setup where each both boxes have their own active IP and can fail over to each other?
engendered 12y ago
if you’ve got lots of money and you don’t care about how you spend it or translating those savings onto your customers This is a baseless, prejudiced claim, contrasting buying with some idealized scenario where your in-house crew can only possibly build better solutions cheaper. That there are no other outcomes. But they might also cost far more in manpower costs than any premium. They might give you an unreliable solution that costs you your entire business. Such an effort might distract from the core competencies of the organization. Operating that storage might end up dwarfing the up-front cost (it's easy to hire admins knowledgeable of EMC. Quite a different matter when it's your own home-brew solution). I understand the draw, but the "all upside" claim undermines the entire piece. There are enormous downsides, not least the reliability of your data. Companies like EMC and Nimble -- despite "great" editorial quotes added by some random person on Wikipedia -- base their entire existence on reliably serving your data, and things like multipathing and replication are the absolute minimum cost of entry in the market. Now add automatic tiering, thin provisioning, disk-deduplication and streaming hardware compression, etc, and the value starts to become evident. EDIT: The moderation through this is an abomination. HN shouldn't be overly critical, but nor should it pander patronizingly to some tripe because the author happened by.
- mrmondo 12y ago
  Hi there, I'm very open to constructive criticism on my project but I feel that your comment is more of a strawman attack than particularly useful in any way. I had a look back through your comments on other posts on HN and they seemed to have a similar tone. If you feel strongly about the topic one thing I could suggest that would help readers of your comment would be to include some details, facts or technical analysis that are relevant and insightful. Also I might suggest reading the recent hacker news post concerning community feedback: http://blog.ycombinator.com/new-hacker-news-guideline http://blog.ycombinator.com/new-hacker-news-guideline Edit: by the way, I do have compression, thin provisioning, replication and it also auto provisions storage volumes when new VMs are created. As I said, I didn't post my two month old blog here and there is plenty of detailed information for those that are genuinely interested on its way in the next few weeks.
  - 12y ago
KaiserPro 12y ago
I'm always interested in the outcomes of these types of experiments. One thing that I find interesting is no mention of what network card you are using? or switch setup, or ip config. I seem to remember in the dim and distant past that letting the storage layer handle the paths instead of LACP (which on some switches are active-passive) was better/faster/easier. (that could be bollocks though.) I used to work in VFX where performance and cost must be balanced. For the storage lumps we had a 1u dell with the fastest 10 core v3 processor in it, 384 gigs of ram (ram is a cheap and fast cache) tied to a dell 3060 (60 4tb disks in a 4u case, with up to 4 SAS connections out(well many other options it does iscsi too)) These topped out at about 2gigabytes/second (or about 1.3gigabytes with 100 concurrent fios from NFS clients). but there are 30+ of them so the aggregate was enough to dowse half the render farm in iops. A couple of things that stuck out for me: o 24/7 support contracts are amazing. o auto phone home disk replacements are also life savers o multipath iscsi can be painful unless you have decent switching (with ospf) and OS support o in short distances, SAS > FC >> iscsi for block storage. pretty much plug and play (multipath is easy too). o Replication needs a different path separate from the client o melenox or intel nics are the only 10gig cards worth paying for. (the only thing going for broadcom is that they are supported out of the box by vmware) o File systems always fail o for random IO; RAM > SSD by a massive factor. especially if your active dataset is less than ram cache. o drdb will be as slow as your slowest link in the network. o ethernet packet size really matters. (and depends on your workload) o synthetic benchmarks are the chocolate teapots of the storage world. o paying for decent support saved the (me and by extension the company(s)) at least twice. Yes, the dell/HP/whitebox route is more expensive in capital outlay. However you can if you so wish go the white box route. The dell raid enclosure is a rebadged netapp/lsi/enginio box. dothill do a similar one. The important thing to note though is the opex & RnD is far, far cheaper with prebuilt systems. With a click of a button you get replication. If you are using VMware (I'm sure KVM has a similar system) it can do it as well. The performance is predicatble, and the configuration and tuning has been done more or less for you (excluding the raid layout, FS tuning and network setup. but then you're a sysadmin who knows how to do all that right?) There is nothing worse than having to debug a storage system so unusual or highly configured that googling won't help. You feel terribly alone. Even more if you don't have decent backups. If you don't have those expertise, it'll cost you either time, or budget for a storage guy. (so just do some testing and buy the fastest prebuilt system for your budget.) For a previous place, I tested 7 different systems to get the best fit. (that included home rolled) A lot of the time was modelling the sotrage workload, and devising tests that would proxy that kind of load on the test systems. END_OF_RAMBLE
hengheng 12y ago
What is the reason to go with Raid1 for the intel SSDs? My guess would be that the rate of Raid1-catchable failures of these drives is not considerably higher than the rate of, say mainboard failures or other failures that render the whole unit out of operation.
- mrmondo 12y ago
  The only reason is to protect against per-device failures - the technology is new and while it's meant to have astounding resiliency and a long lifetime - I don't want to take that chance. Down the track when they have proved themselves to us I'll certainly be reconsidering this however.
- baruch 12y ago
  If a mainboard fails you replace it and get back up (with downtime), if an SSD fails you lost your data. If you care about your data put some redundancy in it. Even for the case that a motherboard fails having redundancy will help with the downtime, your software needs to be able to handle failover situations and that gets hard but sometimes worth it.
PaulHoule 12y ago
Enterprise storage systems made more sense back in the spinning disk era when the need to minimize latency caused by physical motion was acute. They're definitely struggling now to produce products that really add value to SSDs.
- KaiserPro 12y ago
  for Direct attached storage, possibly. But its simply not the case to say they are struggling. IBM yes. Netapp and EMC are still growing, just. Contrary to popular belief, spinning disks are here for the next few years at least. Especially when you compare disk performance with EBS "SSD" performance. 128gigs of "SSD" EBS storage gives you ~380 iops. The same performance as two 15k disks. We are very much in the era of spinning disk performance if Amazon can charge a premium for that kind of storage.
  - jbooth 12y ago
    What's the derivative of that growth look like, though? It could be that they just have enough customers who are either locked-in or on the back-half of the adoption curve, allowing them to limp in with some growth for a few more years before it's over. Amazon charges a premium for convenience and for the fact that there's 0-startup cost to get going with them. Why would someone every shell out the money for an enterprise 15k disk when they can buy an SSD instead?
    - KaiserPro 12y ago
      Good question. if you want an all in one file server and block storage device, then you only really have a few choices. (netapp, EMC, trintri) if you don't want to configure and support a custom filer and or block storage, then you're going to go to the likes of Netapp et al.
    - baruch 12y ago
      The 15k HDDs will likely be replaced by SSDs, the 7200 ones are likely to stay for a long while yet. Their capacity is yet unmatched, the SSDs are trying to catch up with higher densities but currently the HDDs are ahead of them on the curve of size and definitely ahead on the cost per Gb. I just recently switched my laptop to an SSD and the change is amazing but I'm not going to replace even the 1TB HDDs on my nas with equivalently sized SSDs. Both from a needs view and from a cost view.
StillBored 12y ago
I wrote a longer version of this, but I will summarize as not all the commercial flash systems are x86's driving SSDs. Disclaimer: I don't work for IBM, and have no commercial interest in what i'm about to say. I had the pleasure (a rare statement for me) of recently having access to a TMS/IBM flash system 840 for a year. Frankly, if the goal is shared network accessible performance, nothing you can build with PC's and SSDs will come close. The on the wire latency and bandwidth that even a single 2U (60TB) unit provides is fairly shocking. And that performance doesn't degrade over time, even under extreme load. The official performance numbers on the IBM site are actually conservative, which is unusual itself. The units are all custom (in a way that is a negative, cause they aren't cheap) and built like tanks. Although from an end user perspective they are very simplistic. AKA they provide a RAIDed volume(s) on the SAN and nothing else. Your on your own for replication and the like, although I think they have a canned solution for that now. Bottom line, they cost a mint, and are built like old school over engineered IBM hardware. But you won't be kicking yourself a few months down the line as you baby your custom storage solution. Normally i'm the guy building stuff like this, but sometimes I would rather just sleep at night than get called because the garbage collector kicked in on a SSD and it dropped out of the RAID in the middle of a big load. Plus, being IBM, you can probably get a unit with a POC agreement, just to see how it compares.
- johngalt 12y ago
  > you won't be kicking yourself a few months down the line as you baby your custom storage solution. Normally i'm the guy building stuff like this, but sometimes I would rather just sleep at night... "Even Southwest doesn't build their own airplanes." I have this argument with myself all the time. If you don't have an IT pro who works on storage every day, then buy something from someone who does work on storage every day. Their time is worth the price. Your time is worth it too.
  - mrmondo 12y ago
    I've taken this from where I replied to a comment on my blog: Support 4 hour on site response offered from vendors is not good enough for us and to improve on that is expensive. With using more standard components we are able to be on site at either our primary or secondary datacenter within 30 minutes with spare parts in hand if required. Choosing one vendor as an example, the support we've received from HP has been atrocious - they've caused more outages than they've be able to fix. The engineers they send to smaller organisations are generally relatively incompetent and they have certainly shown that they don't care about your uptime. Proprietary storage systems offered by HP, DELL and EMC are not only expensive to purchase and license but they're very time consuming to manage as they're essentially a 'snowflake' in your infrastructure. It's hard to make them integrate with modern automation tools such as Puppet and CI requirements and they all use their own management tools that are specific to the vendor or range of product - usually this involves having a Windows VM running Java or some equally frustrating technology to manage the system. Performing updates on proprietary systems can often be painful as hardware vendors generally are not very good at designing software. It's very hard to outsource quality and it comes at a large cost.
mrmondo 12y ago
It would be typical that this was posted just before I went to sleep last night! I've been flooded with messages, comments and feedback overnight and am slowly catching up on most of them. I wanted to say one thing: My blog post that was linked here was never intended to be much more than a brain dump at the start of a journey. It contains little-to-no useful information other than the idea itself and it doesn't have any technical information or nearly enough background as to why I'm exploring this option or why I think there is a valid use case for it. Give me a week or two and I'll have an updated post with a proper background story and experience with traditional solutions, my observations so far, some real technical content and of course some serious numbers. Thank you so much to all the people that have contacted me both from HN and directly to offered insight into their experiences.

69 points by runarb 12y ago

jjoe 12y ago

I understand his decision to skip the HW RAID controller (I like mdadm too). But a BBU is definitely critical to this op. NVMe has so many more queues that on a busy box it's going to hold way too much uncommitted data and metadata. Now the only other reasonable option is a UPS but it adds significant RUs/cost to the setup.

peterjs 12y ago
I went down the same road - using mdadm and a UPS instead of a HW RAID controller with battery backup unit. LSI MegaRAID SAS 9261-8i SGL with a battery backup costs roughly 650 EUR; Eaton 9130i 1500VA Rack 2U UPS costs 850 EUR. So it's a bit more expensive, but you would be buying that UPS anyway.
runarb 12y ago
I am no expert, but are you sure that uncommitted data on the ssd is an issue? The Intel DC P3600 has a built in capacitor that should give it econoff backup power to commit the data. The SanDisk Extreme Pro disks don't have volatile cache, but instead uses SLC NAND flash for cashing, that will survive a power loss. There is of course still the issue that data send to the server and stored in main memory will be lost if you loses power, but a HW RAID controller with battery backup would not have prevented that (but an UPS of course might).
- mrmondo 12y ago
  I tend to agree with you here runarb, I truly don't believe there is much that a hardware RAID contrôlée and battery will protect you from with modern flash storage, at this point I think time is better spent on improving / tuning the database and filesystem for resiliency.
- otterley 12y ago
  It would definitely prevent an uncommitted data problem if fsync or O_DIRECT were used (which should always be used for critical writes). UPSes defend against power outages, but they're only one rung on the data-integrity ladder. Controller BBUs, on the other hand, protect against both power outages AND kernel panics.
  - mrmondo 12y ago
    Indeed we have fsync enabled on our database servers (PostgreSQL)
    - mrmondo 12y ago
      (And just wanted to point out that that is only one part of the equation)
KaiserPro 12y ago
UPSs all the way. With high random writes you'll get alot of buffering so even if you have a BBU for the raid, the data might still be in RAM. the order of shutdown is important too, should really be 1) consumers of storage (within the first minute or so) 2) storage (about 15 minutes after) if your network wobbles or the storage turns off too fast, everything hangs, and your in poostain pants town. Ideally your DNS/AD shouldn't ever willingly shutdown. or if it does it should be well after the last bit of your production system has turned off.
fweespeech 12y ago
I'd be willing to assume they have a UPS capable of handing the servers in each rack to allow for controlled shutdowns in the event of a power outage. Hell, I have a UPS in place for my desktop, modem, and router at home.

mrmondo 12y ago

I'm the author (Sam Mcleod) of this post and just saw this was posted on HN. I'm quite a bit further down the track now and should have a much better write up to post in the next week or so time providing. I can say that I'm hitting over 1,000,000 IOP/s on 4k random reads and around 850,000 IOP/s random 4k writes per 1U unit without cutting any corners with data safety. What I haven't finished yet is the cross-unit replication tuning and the load balancing / multipathing. So far everything has gone to plan and is progressing well as a serious replacement for our 'traditional' storage. Regardless, there's still quite a lot of testing to be done and once I have more of that completely I'll post an update with some serious numbers and observations.

moe 12y ago
cross-unit replication tuning and the load balancing / multipathing. That's going to be interesting. At those speeds (1.5GB/s, 850k IOPS) you must bump[1] into all kinds of DRBD bottlenecks? Also I'm very curious about your approach to redundancy/multipath. Dual-primary or failover? [1] http://blog.gmane.org/gmane.comp.linux.drbd/page=41 http://blog.gmane.org/gmane.comp.linux.drbd/page=41
- mrmondo 12y ago
  I have no illusions that I'll be able to maintain such performance with DRBD replicating and iSCSI overheads. My current feeling on load balancing is to keep it simple, have half the clients treat node A as their active storage and the other half using node B. If node A becomes unavailable failover will occur and node B will serve all clients. As far as multipathing goes - my approach is that I honestly have no idea. That's something that's going to be learnt along the way and I'm in uncertain if it'll make it into the final build or not. Edit: I can't get that link to load on my connection at present but I've bookmarked to look at tomorrow. If you have any experience / advice I'm always very open to assistance!
  - FlyingAvatar 12y ago
    I was quite amused to see this posted as we just built nearly the exact same thing about 6 months ago (minus the NVMe drives which weren't available). We are using ZFSonLinux on top of DRBD instead of mdmadm, which has been surprisingly successful. The biggest frustration that we met was using LACP in trying to get a better data rate out of DRBD. Regardless of the LACP mode, we were not able to get DRBD to scale to full bandwidth (20gbps) and had to settle for using it for redundancy only. I imagine with your setup (read: much faster peak write capability), you might actually be able to bottleneck DRBD pretty well over a single 10GBE pipe if you were writing at peak capacity. I'll be interested to see if you happen upon a work-around. Load balancing as you've suggested is an intriguing compromise. Have you tried doing a two-way heartbeat setup where each both boxes have their own active IP and can fail over to each other?

engendered 12y ago

if you’ve got lots of money and you don’t care about how you spend it or translating those savings onto your customers This is a baseless, prejudiced claim, contrasting buying with some idealized scenario where your in-house crew can only possibly build better solutions cheaper. That there are no other outcomes. But they might also cost far more in manpower costs than any premium. They might give you an unreliable solution that costs you your entire business. Such an effort might distract from the core competencies of the organization. Operating that storage might end up dwarfing the up-front cost (it's easy to hire admins knowledgeable of EMC. Quite a different matter when it's your own home-brew solution). I understand the draw, but the "all upside" claim undermines the entire piece. There are enormous downsides, not least the reliability of your data. Companies like EMC and Nimble -- despite "great" editorial quotes added by some random person on Wikipedia -- base their entire existence on reliably serving your data, and things like multipathing and replication are the absolute minimum cost of entry in the market. Now add automatic tiering, thin provisioning, disk-deduplication and streaming hardware compression, etc, and the value starts to become evident. EDIT: The moderation through this is an abomination. HN shouldn't be overly critical, but nor should it pander patronizingly to some tripe because the author happened by.

mrmondo 12y ago
Hi there, I'm very open to constructive criticism on my project but I feel that your comment is more of a strawman attack than particularly useful in any way. I had a look back through your comments on other posts on HN and they seemed to have a similar tone. If you feel strongly about the topic one thing I could suggest that would help readers of your comment would be to include some details, facts or technical analysis that are relevant and insightful. Also I might suggest reading the recent hacker news post concerning community feedback: http://blog.ycombinator.com/new-hacker-news-guideline http://blog.ycombinator.com/new-hacker-news-guideline Edit: by the way, I do have compression, thin provisioning, replication and it also auto provisions storage volumes when new VMs are created. As I said, I didn't post my two month old blog here and there is plenty of detailed information for those that are genuinely interested on its way in the next few weeks.
- 12y ago

KaiserPro 12y ago

I'm always interested in the outcomes of these types of experiments. One thing that I find interesting is no mention of what network card you are using? or switch setup, or ip config. I seem to remember in the dim and distant past that letting the storage layer handle the paths instead of LACP (which on some switches are active-passive) was better/faster/easier. (that could be bollocks though.) I used to work in VFX where performance and cost must be balanced. For the storage lumps we had a 1u dell with the fastest 10 core v3 processor in it, 384 gigs of ram (ram is a cheap and fast cache) tied to a dell 3060 (60 4tb disks in a 4u case, with up to 4 SAS connections out(well many other options it does iscsi too)) These topped out at about 2gigabytes/second (or about 1.3gigabytes with 100 concurrent fios from NFS clients). but there are 30+ of them so the aggregate was enough to dowse half the render farm in iops. A couple of things that stuck out for me: o 24/7 support contracts are amazing. o auto phone home disk replacements are also life savers o multipath iscsi can be painful unless you have decent switching (with ospf) and OS support o in short distances, SAS > FC >> iscsi for block storage. pretty much plug and play (multipath is easy too). o Replication needs a different path separate from the client o melenox or intel nics are the only 10gig cards worth paying for. (the only thing going for broadcom is that they are supported out of the box by vmware) o File systems always fail o for random IO; RAM > SSD by a massive factor. especially if your active dataset is less than ram cache. o drdb will be as slow as your slowest link in the network. o ethernet packet size really matters. (and depends on your workload) o synthetic benchmarks are the chocolate teapots of the storage world. o paying for decent support saved the (me and by extension the company(s)) at least twice. Yes, the dell/HP/whitebox route is more expensive in capital outlay. However you can if you so wish go the white box route. The dell raid enclosure is a rebadged netapp/lsi/enginio box. dothill do a similar one. The important thing to note though is the opex & RnD is far, far cheaper with prebuilt systems. With a click of a button you get replication. If you are using VMware (I'm sure KVM has a similar system) it can do it as well. The performance is predicatble, and the configuration and tuning has been done more or less for you (excluding the raid layout, FS tuning and network setup. but then you're a sysadmin who knows how to do all that right?) There is nothing worse than having to debug a storage system so unusual or highly configured that googling won't help. You feel terribly alone. Even more if you don't have decent backups. If you don't have those expertise, it'll cost you either time, or budget for a storage guy. (so just do some testing and buy the fastest prebuilt system for your budget.) For a previous place, I tested 7 different systems to get the best fit. (that included home rolled) A lot of the time was modelling the sotrage workload, and devising tests that would proxy that kind of load on the test systems. END_OF_RAMBLE

hengheng 12y ago

What is the reason to go with Raid1 for the intel SSDs? My guess would be that the rate of Raid1-catchable failures of these drives is not considerably higher than the rate of, say mainboard failures or other failures that render the whole unit out of operation.

mrmondo 12y ago
The only reason is to protect against per-device failures - the technology is new and while it's meant to have astounding resiliency and a long lifetime - I don't want to take that chance. Down the track when they have proved themselves to us I'll certainly be reconsidering this however.
baruch 12y ago
If a mainboard fails you replace it and get back up (with downtime), if an SSD fails you lost your data. If you care about your data put some redundancy in it. Even for the case that a motherboard fails having redundancy will help with the downtime, your software needs to be able to handle failover situations and that gets hard but sometimes worth it.

PaulHoule 12y ago

Enterprise storage systems made more sense back in the spinning disk era when the need to minimize latency caused by physical motion was acute. They're definitely struggling now to produce products that really add value to SSDs.

KaiserPro 12y ago
for Direct attached storage, possibly. But its simply not the case to say they are struggling. IBM yes. Netapp and EMC are still growing, just. Contrary to popular belief, spinning disks are here for the next few years at least. Especially when you compare disk performance with EBS "SSD" performance. 128gigs of "SSD" EBS storage gives you ~380 iops. The same performance as two 15k disks. We are very much in the era of spinning disk performance if Amazon can charge a premium for that kind of storage.
- jbooth 12y ago
  What's the derivative of that growth look like, though? It could be that they just have enough customers who are either locked-in or on the back-half of the adoption curve, allowing them to limp in with some growth for a few more years before it's over. Amazon charges a premium for convenience and for the fact that there's 0-startup cost to get going with them. Why would someone every shell out the money for an enterprise 15k disk when they can buy an SSD instead?
  - KaiserPro 12y ago
    Good question. if you want an all in one file server and block storage device, then you only really have a few choices. (netapp, EMC, trintri) if you don't want to configure and support a custom filer and or block storage, then you're going to go to the likes of Netapp et al.
  - baruch 12y ago
    The 15k HDDs will likely be replaced by SSDs, the 7200 ones are likely to stay for a long while yet. Their capacity is yet unmatched, the SSDs are trying to catch up with higher densities but currently the HDDs are ahead of them on the curve of size and definitely ahead on the cost per Gb. I just recently switched my laptop to an SSD and the change is amazing but I'm not going to replace even the 1TB HDDs on my nas with equivalently sized SSDs. Both from a needs view and from a cost view.

StillBored 12y ago

I wrote a longer version of this, but I will summarize as not all the commercial flash systems are x86's driving SSDs. Disclaimer: I don't work for IBM, and have no commercial interest in what i'm about to say. I had the pleasure (a rare statement for me) of recently having access to a TMS/IBM flash system 840 for a year. Frankly, if the goal is shared network accessible performance, nothing you can build with PC's and SSDs will come close. The on the wire latency and bandwidth that even a single 2U (60TB) unit provides is fairly shocking. And that performance doesn't degrade over time, even under extreme load. The official performance numbers on the IBM site are actually conservative, which is unusual itself. The units are all custom (in a way that is a negative, cause they aren't cheap) and built like tanks. Although from an end user perspective they are very simplistic. AKA they provide a RAIDed volume(s) on the SAN and nothing else. Your on your own for replication and the like, although I think they have a canned solution for that now. Bottom line, they cost a mint, and are built like old school over engineered IBM hardware. But you won't be kicking yourself a few months down the line as you baby your custom storage solution. Normally i'm the guy building stuff like this, but sometimes I would rather just sleep at night than get called because the garbage collector kicked in on a SSD and it dropped out of the RAID in the middle of a big load. Plus, being IBM, you can probably get a unit with a POC agreement, just to see how it compares.

johngalt 12y ago
> you won't be kicking yourself a few months down the line as you baby your custom storage solution. Normally i'm the guy building stuff like this, but sometimes I would rather just sleep at night... "Even Southwest doesn't build their own airplanes." I have this argument with myself all the time. If you don't have an IT pro who works on storage every day, then buy something from someone who does work on storage every day. Their time is worth the price. Your time is worth it too.
- mrmondo 12y ago
  I've taken this from where I replied to a comment on my blog: Support 4 hour on site response offered from vendors is not good enough for us and to improve on that is expensive. With using more standard components we are able to be on site at either our primary or secondary datacenter within 30 minutes with spare parts in hand if required. Choosing one vendor as an example, the support we've received from HP has been atrocious - they've caused more outages than they've be able to fix. The engineers they send to smaller organisations are generally relatively incompetent and they have certainly shown that they don't care about your uptime. Proprietary storage systems offered by HP, DELL and EMC are not only expensive to purchase and license but they're very time consuming to manage as they're essentially a 'snowflake' in your infrastructure. It's hard to make them integrate with modern automation tools such as Puppet and CI requirements and they all use their own management tools that are specific to the vendor or range of product - usually this involves having a Windows VM running Java or some equally frustrating technology to manage the system. Performing updates on proprietary systems can often be painful as hardware vendors generally are not very good at designing software. It's very hard to outsource quality and it comes at a large cost.

mrmondo 12y ago

It would be typical that this was posted just before I went to sleep last night! I've been flooded with messages, comments and feedback overnight and am slowly catching up on most of them. I wanted to say one thing: My blog post that was linked here was never intended to be much more than a brain dump at the start of a journey. It contains little-to-no useful information other than the idea itself and it doesn't have any technical information or nearly enough background as to why I'm exploring this option or why I think there is a valid use case for it. Give me a week or two and I'll have an updated post with a proper background story and experience with traditional solutions, my observations so far, some real technical content and of course some serious numbers. Thank you so much to all the people that have contacted me both from HN and directly to offered insight into their experiences.