5 ms·

Lets say you enterprise division/company wants to store 100 Terabytes of data. You ring up Netapp and buy a solution for say $100,000 and plug it in. You get su

by slyall 11y ago

Lets say you enterprise division/company wants to store 100 Terabytes of data. You ring up Netapp and buy a solution for say $100,000 and plug it in. You get support, lots of documentation and it is usually fairly reliable. Now if you are Google or Facebook you want 100 Petabytes of storage but instead of paying Netapp $100 million you employee 20 really smart guys to build you something, you put it on cheap hardware and you build exactly what your software will talk to and what it needs. It costs you $10 million up front but the hardware cost is just $15 million because you use cheaper drives and build (thousands) of your own servers. Of course this doesn't mean that "Enterprise Storage" isn't going to be the next premium product hit by Open Source type solutions.

sre_ops 11y ago
Do notice a) you paid $1000 per TB b) you did not include the cost of support per year ( 30% or so ) c) you thought that support knows anything d) you presumed that in event of a failure Netpp would cover you.
mbubb 11y ago
I guess it depends how close to bare metal you want to get. Google/Facebook/AWS (and maybe a Backblaze) have architected their data center setup to not require the functionality that Netapp provides. If you buy a NetAPP and plug it in your netork it is because you cant architect a datacenter from the ground up... Facebook was a founding member of the Open Datacenter Alliance. They have a cool githup repo and put out interesting whitepapers like: http://www.opendatacenteralliance.org/docs/architecting_cloud_aware_applications.pdf http://www.opendatacenteralliance.org/docs/architecting_clou... It is not just cheaper but architected better. They would use Cisco and other enterprisey stuff if it were better but it is a case of "il y a moins bien mais c'est plus cher" as the French Linux motto goes... Which is not to say NetAPP or Cisco is bad - just that you can engineer the datacenter to make it work better without enterprise functionality. As an object lesson in failure, I had an experience with my first buildout of Hadoop back in 2011. Not so successful because I bought all the wrong components. Raid cards, 10G networking, HP and Bladenetworking rack switches, dual power supplies. It was hard to unlearn how I was used to setting up servers.
- maximilianburke 11y ago
  Off topic but how would you have done your Hadoop buildout differently?
  - mbubb 11y ago
    Wow - still a bit painful to remember my first go around with this. What i am about to admit is damning enough to keep me unemployed for the forseeable future so hopefully no hiring managers are reading this... I was a sysadmin and did the datacenter buildouts for the company. The task was to take the dev setup in Softlayer (on bare metal servers) and reproduce it in our racks. The error was to take this somewhat literally. The bare metal servers that we leased from Softlayer were 2u boxes with redundant power supplies, Raid controllers and 10g networking. I found a Supermicro configuration that almost identically matched and ordered 30 datanodes; 2 name nodes (this was back when Hadoop used the curiously named primary and secondary name nodes); and a server to launch jobs from. We already had a Netezza in place and since that used Bladenetwork switches I decided to use the same as TOR switches with a beefier on as the Aggregation switch. We decided to use Ubuntu as the OS and the Cloudera packages - but without the support or the console. Every single one of those choices were mistakes. It is remarkable in hindsight that it worked at all. The mistakes: 1) starting with the power supplies. Since they were redundant it dictated an A/B power setup in the racks. What this means is that you cannot use more than 50% of the power density because the entire rack is set to fail over. Each PDU has to be able ot keep the whole rack up so it alarms at 40% capacity. My using redundant powersupplies I was more than halving the amount of power i had at my disposal. 2) RAID. I hadnt yet read the excellent Hadoop Operations O'Reilly book which tells you why RAID is a bad idea for hdfs. Further buildouts included ripping out the RIAD card and JBODing the drives allowing Hadoop to properly use the raw disks. 3) 10G networking - because more is better right? Later after we hired talented networking folks I learned a bit about how important caching is in switches - particulary for Hadoop. My expensive monster Bladenetwork switches were falling over because the bursty traffic would saturate the switches. In addition I had the networking setup for a much more conventional network and we were doing the switch aware stuff in the config but had a wide open /16 as the network. I learned about properly segmenting to avoid uneccesary cross talk. 4) Ubuntu - never listen to developers. ;) Actually it was our default OS. CentOS works better for Hadoop I would later find out. 5) Lack of support. Cloudera is expensive but - unless you are already an expert - worth it. We should have gotten help early on. 6) Hardware choices. Supermicro was a bad choice. Penny wise / Pound foolish. They changed hardware config and it was hard to get replacemnts, etc. Cant say enough good things about working with PSSClabs for hardware though. I already mentioned the switches. I learned about Arista switches which are amazing for this application. By the time I left that company, I learned an enormous amount from really smart engineers that knew gobs more about Hadoop and netwrking than I did. I think the number of datanodes was up over 200 and we had a lean 1u datanode from PSSCLabs with single powersupplies 12 drives JBOD with onboard flash for the OS. Bonded 1G networking running up to Arista TORs and mutliple Arita Agg switches in a leaf / spine topology. It was a thing of beauty.
mwytock 11y ago
Makes sense. But I think the real question is should you pay $100K to Netapp or $3K/month to Amazon and store your data on S3. At least newer tech companies seem to strongly prefer the latter.
speakeron 11y ago
So move that 100 million/100 petabytes ratio down and you'll hit the price of one or two smart people. That's exactly what some companies are doing. There are some startup needs where you need at least an of magnitude reduction in OTS or cloud costs.

by slyall 11y ago

sre_ops 11y ago

Do notice a) you paid $1000 per TB b) you did not include the cost of support per year ( 30% or so ) c) you thought that support knows anything d) you presumed that in event of a failure Netpp would cover you.

mbubb 11y ago

I guess it depends how close to bare metal you want to get. Google/Facebook/AWS (and maybe a Backblaze) have architected their data center setup to not require the functionality that Netapp provides. If you buy a NetAPP and plug it in your netork it is because you cant architect a datacenter from the ground up... Facebook was a founding member of the Open Datacenter Alliance. They have a cool githup repo and put out interesting whitepapers like: http://www.opendatacenteralliance.org/docs/architecting_cloud_aware_applications.pdf http://www.opendatacenteralliance.org/docs/architecting_clou... It is not just cheaper but architected better. They would use Cisco and other enterprisey stuff if it were better but it is a case of "il y a moins bien mais c'est plus cher" as the French Linux motto goes... Which is not to say NetAPP or Cisco is bad - just that you can engineer the datacenter to make it work better without enterprise functionality. As an object lesson in failure, I had an experience with my first buildout of Hadoop back in 2011. Not so successful because I bought all the wrong components. Raid cards, 10G networking, HP and Bladenetworking rack switches, dual power supplies. It was hard to unlearn how I was used to setting up servers.

maximilianburke 11y ago
Off topic but how would you have done your Hadoop buildout differently?
- mbubb 11y ago
  Wow - still a bit painful to remember my first go around with this. What i am about to admit is damning enough to keep me unemployed for the forseeable future so hopefully no hiring managers are reading this... I was a sysadmin and did the datacenter buildouts for the company. The task was to take the dev setup in Softlayer (on bare metal servers) and reproduce it in our racks. The error was to take this somewhat literally. The bare metal servers that we leased from Softlayer were 2u boxes with redundant power supplies, Raid controllers and 10g networking. I found a Supermicro configuration that almost identically matched and ordered 30 datanodes; 2 name nodes (this was back when Hadoop used the curiously named primary and secondary name nodes); and a server to launch jobs from. We already had a Netezza in place and since that used Bladenetwork switches I decided to use the same as TOR switches with a beefier on as the Aggregation switch. We decided to use Ubuntu as the OS and the Cloudera packages - but without the support or the console. Every single one of those choices were mistakes. It is remarkable in hindsight that it worked at all. The mistakes: 1) starting with the power supplies. Since they were redundant it dictated an A/B power setup in the racks. What this means is that you cannot use more than 50% of the power density because the entire rack is set to fail over. Each PDU has to be able ot keep the whole rack up so it alarms at 40% capacity. My using redundant powersupplies I was more than halving the amount of power i had at my disposal. 2) RAID. I hadnt yet read the excellent Hadoop Operations O'Reilly book which tells you why RAID is a bad idea for hdfs. Further buildouts included ripping out the RIAD card and JBODing the drives allowing Hadoop to properly use the raw disks. 3) 10G networking - because more is better right? Later after we hired talented networking folks I learned a bit about how important caching is in switches - particulary for Hadoop. My expensive monster Bladenetwork switches were falling over because the bursty traffic would saturate the switches. In addition I had the networking setup for a much more conventional network and we were doing the switch aware stuff in the config but had a wide open /16 as the network. I learned about properly segmenting to avoid uneccesary cross talk. 4) Ubuntu - never listen to developers. ;) Actually it was our default OS. CentOS works better for Hadoop I would later find out. 5) Lack of support. Cloudera is expensive but - unless you are already an expert - worth it. We should have gotten help early on. 6) Hardware choices. Supermicro was a bad choice. Penny wise / Pound foolish. They changed hardware config and it was hard to get replacemnts, etc. Cant say enough good things about working with PSSClabs for hardware though. I already mentioned the switches. I learned about Arista switches which are amazing for this application. By the time I left that company, I learned an enormous amount from really smart engineers that knew gobs more about Hadoop and netwrking than I did. I think the number of datanodes was up over 200 and we had a lean 1u datanode from PSSCLabs with single powersupplies 12 drives JBOD with onboard flash for the OS. Bonded 1G networking running up to Arista TORs and mutliple Arita Agg switches in a leaf / spine topology. It was a thing of beauty.

mwytock 11y ago

Makes sense. But I think the real question is should you pay $100K to Netapp or $3K/month to Amazon and store your data on S3. At least newer tech companies seem to strongly prefer the latter.

speakeron 11y ago

So move that 100 million/100 petabytes ratio down and you'll hit the price of one or two smart people. That's exactly what some companies are doing. There are some startup needs where you need at least an of magnitude reduction in OTS or cloud costs.