6 ms·

"...since we were using low-end commodity storage that was by no means enterprise-quality." If Facebook (and I believe it's similar with other big players like

"...since we were using low-end commodity storage that was by no means enterprise-quality." If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? And more importantly, why? Why do "enterprise grade" products even exist if the largest and most deep-pocketed corporations don't find them to be good value propositions?

outside1234 11y ago
Because Facebook, Google, and Amazon have the engineering talent to build their own. No "ordinary" enterprise (basically everyone else) can just say "oh, mounting and unmounting is an issue? let's use a raw disk instead!" -- that is just well beyond their capabilities. Also, only Facebook / Google / Microsoft / Amazon have the scale that designing your own cold storage makes sense.
- fragmede 11y ago
  Not really. Skipping the filesystem layer and using the raw disk has been an optimization for databases (ms-sql, mysql, oracle db) has been a practice for quite a while.
  - xyzzyz 11y ago
    Microsoft and Oracle aren't ordinary enterprises either.
  - raverbashing 11y ago
    There's a joke that the O_DIRECT flag on Open(2) in Linux should be changed to O_RACLE
- pkz 11y ago
  This is very true. I work in the public sector and while we often need to store petabytes of data we have a hard time finding and keeping engineering talent to do stuff like this. Instead we buy expensive enterprise stuff. I find it frustrating.
  - cmurf 11y ago
    I agree. Public dollars should, as much as possible, be used to implement and advance open software and hardware. It's no longer just about "free" as in no cost, it's free as in no contract, no lock-in; and not "no one owns it" but rather "everyone owns it." There's a certain obligation and clear benefit for everyone to make it better, including easier to implement.
- mbubb 11y ago
  Yes it is a radical simplification of the data center setup that allows for the scaling to design this.
baoha 11y ago
If you're in the on-premises storage industry (aka NetApp, EMC), you will be amazed by the number of companies still buy networked storage arrays. They are ranging from Fortune 500 firms to extremely smaller ones like dental offices, school district, hospital... They all have one thing in common: they don't trust their data with public cloud infrastructure.
- deleted 11y ago
  [deleted]
- walterbell 11y ago
  What's the cheapest networked storage array that is trusted by on-premise buyers?
  - cnvogel 11y ago
    If you ask that way, I'd guess the answer is: For any network storage array there are at least a few on-premise buyer that trust this particular product. I guess that there are an awful lot of companies that, out of desire to save a few bucks, will go with home-NAS-types.
  - rythie 11y ago
    Ultimately the main difference between the vendors is performance and availability - you pay a lot more to increase either of those. If you don't need either even a consumer style synology nas can work.
- tacticus 11y ago
  And the hilarious part is how most of those implement a solution with less reliability and security.
slyall 11y ago
Lets say you enterprise division/company wants to store 100 Terabytes of data. You ring up Netapp and buy a solution for say $100,000 and plug it in. You get support, lots of documentation and it is usually fairly reliable. Now if you are Google or Facebook you want 100 Petabytes of storage but instead of paying Netapp $100 million you employee 20 really smart guys to build you something, you put it on cheap hardware and you build exactly what your software will talk to and what it needs. It costs you $10 million up front but the hardware cost is just $15 million because you use cheaper drives and build (thousands) of your own servers. Of course this doesn't mean that "Enterprise Storage" isn't going to be the next premium product hit by Open Source type solutions.
- sre_ops 11y ago
  Do notice a) you paid $1000 per TB b) you did not include the cost of support per year ( 30% or so ) c) you thought that support knows anything d) you presumed that in event of a failure Netpp would cover you.
- mbubb 11y ago
  I guess it depends how close to bare metal you want to get. Google/Facebook/AWS (and maybe a Backblaze) have architected their data center setup to not require the functionality that Netapp provides. If you buy a NetAPP and plug it in your netork it is because you cant architect a datacenter from the ground up... Facebook was a founding member of the Open Datacenter Alliance. They have a cool githup repo and put out interesting whitepapers like: http://www.opendatacenteralliance.org/docs/architecting_cloud_aware_applications.pdf http://www.opendatacenteralliance.org/docs/architecting_clou... It is not just cheaper but architected better. They would use Cisco and other enterprisey stuff if it were better but it is a case of "il y a moins bien mais c'est plus cher" as the French Linux motto goes... Which is not to say NetAPP or Cisco is bad - just that you can engineer the datacenter to make it work better without enterprise functionality. As an object lesson in failure, I had an experience with my first buildout of Hadoop back in 2011. Not so successful because I bought all the wrong components. Raid cards, 10G networking, HP and Bladenetworking rack switches, dual power supplies. It was hard to unlearn how I was used to setting up servers.
theVirginian 11y ago
Finance
e1g 11y ago
Choosing between "enterprise" vs "consumer" products is a lot like choosing between getting an Dell desktop vs building you own - a choice many of us made during our early IT years. The former gives you a reliable and predictable product, but it might be slightly lagging in tech or come at a significant premium. The latter gives you full control over your specs, but you carry the burden of fixing it and making drivers work together and that requires high tech skills and overheads. Enterprise hardware aims to be predictable: large businesses have to minimise risks so they value stability and clarity in performance, costs, and technical know-how. Enterprisey products are never cutting edge or best-in-breed or efficient, but they are predictable and that helps large businesses to have a reliable foundation for planning and executing IT projects over 3-5 years with known budgets, risks, support contracts etc. This predictability is important for businesses that are not based on IT (for example, making frozen foods or assembling tractors). Facebook/Google/Amazon and other software-centered businesses depend on agility and finely-tuned machines - they need their "custom-spec" equivalent, and they have the skills required to manage it. But many other companies have known needs and are happy with standard enterprise products - they are the equivalent of an office worker that needs a machine to run MS Office, and a standard machine from Dell/IBM/HP is a more appropriate choice for them (rather than tinker with building their own).
- frandroid 11y ago
  The original poster was asking, if I can reword properly: Who is using enterprise-grade hard drives instead of consumer-grade ones, if FB doesn't (and why)? Both consumer- and enterprise-grade hard drives use standard interfaces (SATA, etc) so all this "drivers" stuff you talked about doesn't apply to this particular hardware. Since even enterprise-grade hard drives will fail one day, one needs to have a proper backup strategy, which would backstop the needs of supporting of consumer-grade HDDs. So in light of this, why buy enterprise HDDs? (I'm not saying one shouldn't, I'm just not buying the explanation above :])
  - TheCondor 11y ago
    Most enterprise drives come off the exact same production line too.. Some have a different external interface but the important parts are all nearly the same, they might test them differently
ucaetano 11y ago
Because when using commodity hardware, strength is in numbers. Very few organizations need the computing power of Google, FB or Amazon, so they can't benefit from the "reliability at scale" that those companies can. Therefore, they need to buy enterprise-grade HW which fails far less often, even if disproportionately more expensive.
kalleboo 11y ago
> If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? The short answer is "everyone in between"
sre_ops 11y ago
Enterprise grade hardware in storage is a lie. It does not exist. Enterprise grade hard drives are hard drives with series and serial numbers programmed into the software to recognize by the "enterprise grade" software so you cannot buy the exact same drives at NewEgg for a fraction of the cost the likes of EMC are selling them to suckers while claiming there's some magic sauce in the drives.
dantillberg 11y ago
It depends on the applications you're going to use them for, and how much you're willing or able to invest in making your applications be able to take advantage of using a larger quantity of cheaper hardware. A lot of big companies run applications that are not nearly as resilient to failure or nearly as linearly scalable as the systems routinely built by the internet giants. I realized this before with regards to the ridiculously overpriced top-top-of-the-line CPUs. Let's say you're running an existing application at, say, a moderately small scale: 20 servers. Each server is only handling a couple dozen requests per minute, so the load is not very high, but the requests still take too long. You've hired a whole team of engineers to work on optimizing the code to make it run faster, and that costs you, say $10 million a year. You'd be happy to buy another hundred servers if that made it run any faster, but that's just not how it works -- the application is all single-threaded code, and maybe you can make that better, but that's what your $10 million engineering team is working on (maybe). So in the meantime, you spend whatever amount of money you can on getting the absolute fastest hardware you can find to throw in those servers -- fast CPUs, fast hard drives, fast network... and even if those servers cost $50k each, that's still just a fraction of what you're spending on the software engineering work.
ape4 11y ago
The value of the data matters. If Facebook loses some images that haven't been accessed for years maybe nobody will notice.
mbubb 11y ago
No I think that they are using drives with a specific purpose and do not need enterprisey functions. I think they mean drives that are the quality of the WD black drives, it looks like them in the one picture. The types of drives that you might build a Hadoop cluster with knowing that you will be replacing 1-2 drives per 100 every month. These drives (unless I misread) do not need to be hotswappable nor any other specialized setup. I would bet they are JBOD and directly attached with minimal controllers. If one drive per tray is powered on at a time then there is no RAID running across that tray... As a point of comparison an expensive enterprise drive that you might see in a Netezza appliance costs many times more and has specialized firmware to work in a specific application, with certain RAID controllers at specific firmware setings. From experience I do not think that they are much more reliable - they are specific to the appliance, or in some cases like Aerospike required SSDs specific to a software requirement. This storage design is stripped bare - the HDDs are off for most of their lifetime. I guess the biggest stress on them is spinning up and spinning down. They must focus on precise temperature/humidity control as it might be as much of a problem to have the drives too cold as too hot.
beachstartup 11y ago
short answer: people who don't have the scale of amazon/google/fb.
Kephael 11y ago
Facebook used to use Dell C series servers and HP servers, tons of Dell Custom Solutions C1100/C2100/C6100 and DL160 G6 servers showed up off lease years ago (DL160 racks were shown in a time magazine article of a FB datacenter). I'm fairly certain I've seen pictures of Dell C6220 servers at Facebook as well.
Spooky23 11y ago
It's actually starting to go that way in some enterprise environments, especially in storage where ditching the overhead of EMC/IBM/NetApp has a potentially high reward. But more conventionally, there is a different thought process in enterprise IT and what I call industrial IT. Enterprise IT implements systems around minimizing failure events and create bespoke environments to meet solution requirements. Industrial orgs acknowledge that these events are a fact of life build process and software to deal with them. They also tend to limit the options available to consume IT. Also, "at scale" cannot be achieved by an single entity outside of US Federal .gov. I work for a massive organization... 150k+ people. Facebook serves 10000x more users, and have the engineering resources to do amazing things as a result.

outside1234 11y ago

Because Facebook, Google, and Amazon have the engineering talent to build their own. No "ordinary" enterprise (basically everyone else) can just say "oh, mounting and unmounting is an issue? let's use a raw disk instead!" -- that is just well beyond their capabilities. Also, only Facebook / Google / Microsoft / Amazon have the scale that designing your own cold storage makes sense.

fragmede 11y ago
Not really. Skipping the filesystem layer and using the raw disk has been an optimization for databases (ms-sql, mysql, oracle db) has been a practice for quite a while.
- xyzzyz 11y ago
  Microsoft and Oracle aren't ordinary enterprises either.
- raverbashing 11y ago
  There's a joke that the O_DIRECT flag on Open(2) in Linux should be changed to O_RACLE
pkz 11y ago
This is very true. I work in the public sector and while we often need to store petabytes of data we have a hard time finding and keeping engineering talent to do stuff like this. Instead we buy expensive enterprise stuff. I find it frustrating.
- cmurf 11y ago
  I agree. Public dollars should, as much as possible, be used to implement and advance open software and hardware. It's no longer just about "free" as in no cost, it's free as in no contract, no lock-in; and not "no one owns it" but rather "everyone owns it." There's a certain obligation and clear benefit for everyone to make it better, including easier to implement.
mbubb 11y ago
Yes it is a radical simplification of the data center setup that allows for the scaling to design this.

baoha 11y ago

If you're in the on-premises storage industry (aka NetApp, EMC), you will be amazed by the number of companies still buy networked storage arrays. They are ranging from Fortune 500 firms to extremely smaller ones like dental offices, school district, hospital... They all have one thing in common: they don't trust their data with public cloud infrastructure.

deleted 11y ago
[deleted]
walterbell 11y ago
What's the cheapest networked storage array that is trusted by on-premise buyers?
- cnvogel 11y ago
  If you ask that way, I'd guess the answer is: For any network storage array there are at least a few on-premise buyer that trust this particular product. I guess that there are an awful lot of companies that, out of desire to save a few bucks, will go with home-NAS-types.
- rythie 11y ago
  Ultimately the main difference between the vendors is performance and availability - you pay a lot more to increase either of those. If you don't need either even a consumer style synology nas can work.
tacticus 11y ago
And the hilarious part is how most of those implement a solution with less reliability and security.

slyall 11y ago

Lets say you enterprise division/company wants to store 100 Terabytes of data. You ring up Netapp and buy a solution for say $100,000 and plug it in. You get support, lots of documentation and it is usually fairly reliable. Now if you are Google or Facebook you want 100 Petabytes of storage but instead of paying Netapp $100 million you employee 20 really smart guys to build you something, you put it on cheap hardware and you build exactly what your software will talk to and what it needs. It costs you $10 million up front but the hardware cost is just $15 million because you use cheaper drives and build (thousands) of your own servers. Of course this doesn't mean that "Enterprise Storage" isn't going to be the next premium product hit by Open Source type solutions.

sre_ops 11y ago
Do notice a) you paid $1000 per TB b) you did not include the cost of support per year ( 30% or so ) c) you thought that support knows anything d) you presumed that in event of a failure Netpp would cover you.
mbubb 11y ago
I guess it depends how close to bare metal you want to get. Google/Facebook/AWS (and maybe a Backblaze) have architected their data center setup to not require the functionality that Netapp provides. If you buy a NetAPP and plug it in your netork it is because you cant architect a datacenter from the ground up... Facebook was a founding member of the Open Datacenter Alliance. They have a cool githup repo and put out interesting whitepapers like: http://www.opendatacenteralliance.org/docs/architecting_cloud_aware_applications.pdf http://www.opendatacenteralliance.org/docs/architecting_clou... It is not just cheaper but architected better. They would use Cisco and other enterprisey stuff if it were better but it is a case of "il y a moins bien mais c'est plus cher" as the French Linux motto goes... Which is not to say NetAPP or Cisco is bad - just that you can engineer the datacenter to make it work better without enterprise functionality. As an object lesson in failure, I had an experience with my first buildout of Hadoop back in 2011. Not so successful because I bought all the wrong components. Raid cards, 10G networking, HP and Bladenetworking rack switches, dual power supplies. It was hard to unlearn how I was used to setting up servers.

theVirginian 11y ago

Finance

e1g 11y ago

Choosing between "enterprise" vs "consumer" products is a lot like choosing between getting an Dell desktop vs building you own - a choice many of us made during our early IT years. The former gives you a reliable and predictable product, but it might be slightly lagging in tech or come at a significant premium. The latter gives you full control over your specs, but you carry the burden of fixing it and making drivers work together and that requires high tech skills and overheads. Enterprise hardware aims to be predictable: large businesses have to minimise risks so they value stability and clarity in performance, costs, and technical know-how. Enterprisey products are never cutting edge or best-in-breed or efficient, but they are predictable and that helps large businesses to have a reliable foundation for planning and executing IT projects over 3-5 years with known budgets, risks, support contracts etc. This predictability is important for businesses that are not based on IT (for example, making frozen foods or assembling tractors). Facebook/Google/Amazon and other software-centered businesses depend on agility and finely-tuned machines - they need their "custom-spec" equivalent, and they have the skills required to manage it. But many other companies have known needs and are happy with standard enterprise products - they are the equivalent of an office worker that needs a machine to run MS Office, and a standard machine from Dell/IBM/HP is a more appropriate choice for them (rather than tinker with building their own).

frandroid 11y ago
The original poster was asking, if I can reword properly: Who is using enterprise-grade hard drives instead of consumer-grade ones, if FB doesn't (and why)? Both consumer- and enterprise-grade hard drives use standard interfaces (SATA, etc) so all this "drivers" stuff you talked about doesn't apply to this particular hardware. Since even enterprise-grade hard drives will fail one day, one needs to have a proper backup strategy, which would backstop the needs of supporting of consumer-grade HDDs. So in light of this, why buy enterprise HDDs? (I'm not saying one shouldn't, I'm just not buying the explanation above :])
- TheCondor 11y ago
  Most enterprise drives come off the exact same production line too.. Some have a different external interface but the important parts are all nearly the same, they might test them differently

ucaetano 11y ago

Because when using commodity hardware, strength is in numbers. Very few organizations need the computing power of Google, FB or Amazon, so they can't benefit from the "reliability at scale" that those companies can. Therefore, they need to buy enterprise-grade HW which fails far less often, even if disproportionately more expensive.

kalleboo 11y ago

> If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? The short answer is "everyone in between"

sre_ops 11y ago

Enterprise grade hardware in storage is a lie. It does not exist. Enterprise grade hard drives are hard drives with series and serial numbers programmed into the software to recognize by the "enterprise grade" software so you cannot buy the exact same drives at NewEgg for a fraction of the cost the likes of EMC are selling them to suckers while claiming there's some magic sauce in the drives.

dantillberg 11y ago

It depends on the applications you're going to use them for, and how much you're willing or able to invest in making your applications be able to take advantage of using a larger quantity of cheaper hardware. A lot of big companies run applications that are not nearly as resilient to failure or nearly as linearly scalable as the systems routinely built by the internet giants. I realized this before with regards to the ridiculously overpriced top-top-of-the-line CPUs. Let's say you're running an existing application at, say, a moderately small scale: 20 servers. Each server is only handling a couple dozen requests per minute, so the load is not very high, but the requests still take too long. You've hired a whole team of engineers to work on optimizing the code to make it run faster, and that costs you, say $10 million a year. You'd be happy to buy another hundred servers if that made it run any faster, but that's just not how it works -- the application is all single-threaded code, and maybe you can make that better, but that's what your $10 million engineering team is working on (maybe). So in the meantime, you spend whatever amount of money you can on getting the absolute fastest hardware you can find to throw in those servers -- fast CPUs, fast hard drives, fast network... and even if those servers cost $50k each, that's still just a fraction of what you're spending on the software engineering work.

ape4 11y ago

The value of the data matters. If Facebook loses some images that haven't been accessed for years maybe nobody will notice.

mbubb 11y ago

No I think that they are using drives with a specific purpose and do not need enterprisey functions. I think they mean drives that are the quality of the WD black drives, it looks like them in the one picture. The types of drives that you might build a Hadoop cluster with knowing that you will be replacing 1-2 drives per 100 every month. These drives (unless I misread) do not need to be hotswappable nor any other specialized setup. I would bet they are JBOD and directly attached with minimal controllers. If one drive per tray is powered on at a time then there is no RAID running across that tray... As a point of comparison an expensive enterprise drive that you might see in a Netezza appliance costs many times more and has specialized firmware to work in a specific application, with certain RAID controllers at specific firmware setings. From experience I do not think that they are much more reliable - they are specific to the appliance, or in some cases like Aerospike required SSDs specific to a software requirement. This storage design is stripped bare - the HDDs are off for most of their lifetime. I guess the biggest stress on them is spinning up and spinning down. They must focus on precise temperature/humidity control as it might be as much of a problem to have the drives too cold as too hot.

beachstartup 11y ago

short answer: people who don't have the scale of amazon/google/fb.

Kephael 11y ago

Facebook used to use Dell C series servers and HP servers, tons of Dell Custom Solutions C1100/C2100/C6100 and DL160 G6 servers showed up off lease years ago (DL160 racks were shown in a time magazine article of a FB datacenter). I'm fairly certain I've seen pictures of Dell C6220 servers at Facebook as well.

Spooky23 11y ago

It's actually starting to go that way in some enterprise environments, especially in storage where ditching the overhead of EMC/IBM/NetApp has a potentially high reward. But more conventionally, there is a different thought process in enterprise IT and what I call industrial IT. Enterprise IT implements systems around minimizing failure events and create bespoke environments to meet solution requirements. Industrial orgs acknowledge that these events are a fact of life build process and software to deal with them. They also tend to limit the options available to consume IT. Also, "at scale" cannot be achieved by an single entity outside of US Federal .gov. I work for a massive organization... 150k+ people. Facebook serves 10000x more users, and have the engineering resources to do amazing things as a result.