7 ms·

Three years later, Mr. Moore is still letting us punt on database sharding

127 points by jlangenauer 15y ago

alexjipark 15y ago
hmmm
groggles 15y ago
Pedantry -- that isn't Moore's Law related, aside from the loosest interpretation that computers as a whole get faster. This is so true, though. Much of the horizontally scalability need comes from the land of extremely underpowered VPS machines on platforms like AWS. Yet the mind-boggling scale of performance you can inexpensively* (*-term used relatively) acquire for database servers is astonishing. SSDs (and plug-in flash drives) and boundless memory have changed everything.
- masklinn 15y ago
  > Pedantry -- that isn't Moore's Law related Actually, RAM would be the perfect application of the exact meaning of Moore's law: more RAM is an almost direct function of more transistors, and Moore's law is: > the number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years Moore's law precisely predicts you can double your amount of RAM every two years for the exact same price. Which is pretty much what TFA is about. SSD storage size is the same thing.
- _delirium 15y ago
  Given all the problems they've had with AWS, I wonder what the overall cost/benefit would be for someone like Reddit moving from the flexed-EC2-servers model to just one high-powered db server and some caching webservers in front of it.
  - adestefan 15y ago
    Reddit's probably was using trying to use EBS as a backend for a transactional database. The issue is that EBS makes no guarantees on latency, only that they'll never lose your data.
    - bad_user 15y ago
      they'll never lose your data Amazon is not saying it, but EBS is probably lying when doing an fsync(). In this post [1] explaining why Reddit was down, one problem was that the DB slave got ahead of the master, with the most probable explanation being that the master flagged the data as being safe for replication, before committing it to disk and I trust PostgreSQL more than I trust EBS. In this forum answer Amazon is giving about this potential problem [2], they are dodging the question by saying that fsync() guarantees durability for instance failures, but not for volume failures, with the anual failure rate (AFR) being given as 0.1% - 0.5% for volumes (how accurate that is, it remains to be seen). So EBS is probably lying about fsync's success, especially since the behavior of fsync in virtual environments is always a surprise. So you can definitely lose your data more frequently than if you had your own hardware. [1] blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html [2] https://forums.aws.amazon.com/thread.jspa?threadID=27590 https://forums.aws.amazon.com/thread.jspa?threadID=27590
      - cperciva 15y ago
        Amazon is not saying it, but EBS is probably lying when doing an fsync(). Given that POSIX says "the nature of the transfer is implementation-defined" and they've defined what fsync does on their implementation: No, they're not lying. I wouldn't say that they're being misleading, either. On a local physical disk, once fsync returns your data is safely stored unless/until the disk dies. According to the forum post you linked to, semantics on EBS are exactly the same. fsync does not mean "has been written to spinning magnetic media".
    - deleted 15y ago
      [deleted]
    - pangram 15y ago
      Reddit also found EBS to be unreliable; they've moved all their high traffic data to ephemeral storage. They have an interesting summary of their technology stack here: http://blog.reddit.com/2012/01/january-2012-state-of-servers.html http://blog.reddit.com/2012/01/january-2012-state-of-servers...
  - foobarbazetc 15y ago
    Reddit could easily cut their spend on hosting by like 50-75% just by moving to a dedicated hosting provider. They'd also get bare metal speeds in the process. Who knows why they've stuck with AWS, though. It's possible Amazon is giving them a discount so they can be used as an example.
ohashi 15y ago
For some reason, I don't imagine BaseCamp's use is really exploding like some other startups might in terms of database usage. It could very well be a case of their growth is scaling in-line with Moore's law, so it just happens to work out well.
- moe 15y ago
  I agree. Without any figures this is a pretty meaningless blog-post...
- ams6110 15y ago
  I think most "other startups" would LOVE to be in the ballpark with BaseCamp in terms of usage. No BaseCamp is not big like Facebook, but you've heard of them right? Most "ordinary" business people who do project planning work have probably heard of BaseCamp too. Most people never hear about most startups, fewer try their services, fewer than that become regular users. The point of the post is that in many/most cases, it's still easier and cheaper to throw hardware at a performance problem than to devote scarce engineering effort to optimization. And it's only getting more and more so. If you are a startup and you can throw $10,000 of hardware at a problem you can then keep your $100K engineers working on things that hardware alone can't solve.
  - jonknee 15y ago
    > Most "ordinary" business people who do project planning work have probably heard of BaseCamp too. I don't think so. Maybe we just have a different opinion on ordinary, but I come across people all the time that would think Excel is the normal thing to use for this (and pretty much any other task...). Using web applications for day to day workflow is still alien to a lot of "ordinary" business types.
    - ams6110 15y ago
      Granted. s/most/some/
  - untog 15y ago
shanemhansen 15y ago
A lot of startups engage in a sort of cargo-cult architecture. Their reasoning goes something like this: 1. Amazon/Facebook/Google have a lot of traffic. 2. Amazon/Facebook/Google use X to scale horizontally. ergo: 3. My little startup should use X and scale horizontally. What they fail to realize is that most of these companies would be ecstatic if they could scale machines vertically, if they could focus on great user features instead of having to figure out how to shard in the application layer. You should never forget that Amazon, Facebook, and twitter all started out as pretty basic LAMP stacks and built the tools when it was obvious that no other tool would do. I think google's an exception because their MVP was in fact a web scale application. So by all means, vet your idea, get some customers, get traction, and scale the cheap way by buying more ram for as long as you possibly can.
- marshray 15y ago
  Yes. But let's look at the other side too. Before Google came along and showed the business people the benefits of horizontal scaling, any software engineer would be automatically considered crazy if they suggested an architecture that wasn't built on a central RDBMS. So you have to weigh it against the other cargo-cult. How many startups along the way have failed due the inability to scale horizontally? How many have failed due to too much cost and complexity associated with re-engineering an architecture in which the assumption of fully ACID transactions permeates the entire codebase? (While the phone is ringing off the hook because production systems are falling over under load.)
- j_baker 15y ago
  I realize this may not be a popular opinion on HN, but there's something to be said for planning ahead. I've seen this story before and already know how it ends: you wait until the last possible moment to switch to a more horizontally scalable system and next thing you know you're spending more time and money maintaining the "cheap" solution than it would have taken to switch to something like Cassandra beforehand. To make matters worse, your service is crashing and the short term fix takes a day and requires two or three people to do the replica switch shuffle. The long term fix will take a couple of weeks, if you have the time for it of course. Long story short, I have to grant that you shouldn't worry about scaling up too soon or too quickly. But that don't go to the opposite extreme by putting it off until the last possible moment.
krupan 15y ago
He mentions Fusion-io drives getting faster. What's interesting about iodrives is that you don't have to buy a new one to get more speed, just updating your CPU or RAM will speed up an iodrive: "Because VSL leverages powerful host CPU cores and memory for block mapping, ioMemory performance automatically improves as organizations upgrade host CPUs or memory" (http://www.fusionio.com/overviews/vsl-technical-overview/ http://www.fusionio.com/overviews/vsl-technical-overview/) VSL (Virtual Storage Layer) is the Fusion-io software that makes an iodrive look like a block device to the OS, among other things. (Yes, I'm an engineer at Fusion)
bitops 15y ago
What never seems to get mentioned is energy cost. Sure, you can keep throwing hardware at the problem, but eventually that will lead to a substantial power bill.
- rorrr 15y ago
  Sharding will lead to an even much higher power bill.
  - wildwood 15y ago
    Agreed, sharding takes "throwing hardware at the problem" to a whole new level.
    - bitops 15y ago
      I agree with what both of you have said. It's possible that, in the not-too-distant future, "green programming" will become a new field. If you can design an architecture (both hardware and software) that demonstrably reduces power costs, that's a valuable skill. And, I'd argue, on two fronts: lower costs for the company, plus good marketing. Every forward-thinking company (at least from what I've observed) are eager to brand themselves as "carbon neutral", "eco-friendly", etc.
wes-exp 15y ago
For those with custom hardware setups like this, how much time do you estimate you've spent designing, building, and maintaining custom servers? I can see how it would be less effort than sharding, but it seems like throwing hardware at the problem is not a completely "free lunch" when you get to the scale of having to custom design a server supercomputer and source specially made SSDs and terabytes of RAM.
- sciurus 15y ago
  You don't need to "custom design a server supercomputer". This stuff is as mainstream as Dell. You can go to dell.com and order a Poweredge R815 with 64 processor cores and 512GB of RAM for $16,000. If you want SSDs, you can purchase SATA or PCIe ones from Dell as well. http://www.dell.com/us/enterprise/p/poweredge-r815/pd http://www.dell.com/us/enterprise/p/poweredge-r815/pd http://www.dell.com/us/business/p/fusion-io-drive/pd http://www.dell.com/us/business/p/fusion-io-drive/pd
kyledrake 15y ago
I did a presentation on a game I made at Geoloqi that talks about the Fusion IO drives and why I think local hardware with SSD is the best solution for persistent data stores right now: http://www.slideshare.net/KyleDrake/building-mapattack http://www.slideshare.net/KyleDrake/building-mapattack
- kyledrake 15y ago
  The one warning I want to provide though is that not all SSDs are created equal: Make sure you get one that writes its cache to the disk on power failure, or you're going to be in a world of hurt.
  - krupan 15y ago
    Can you explain more? You aren't talking about writing data that is cached in the SSD to a regular hard-drive, are you?
    - tesseract 15y ago
      I think he's talking about writing data in the SSD's DRAM cache into the actual flash.
      - krupan 15y ago
        That's what I thought too. Fusion-io drives all flush their write buffer to the nand flash on a power cut event, I wasn't aware that some SSD's didn't. People use iodrives as a caching layer in conjunction with tradition spinning hard drives, so wondered if possibly he was referring to that kind of setup.
    - xolox 15y ago
      To guarantee on-disk consistency, programs like database servers call fsync for every transaction. By definition an fsync involves events on the hardware level and even in an SSD this gives a slowdown. The SSD can use a write cache to speed this up. However should you lose power and the write cache is not backed by a small battery, you lose your most recent write(s) even though the latest fsync call guaranteed that the changes were on disk. The battery allows the SSD to successfully flush the write cache to disk in the event that the computer shuts down.
bwooce 15y ago
Gah, stop abusing the word punt. The headline is nonsensical. Punt does not mean avoidance. It has much closer associations to "attempt" (punter, plus rugby usage). I totally agree with the rest of the article, why shard or otherwise distribute your database before being required? Although I think I would build my DB as shard 1 of 1 to allow for the future case.
- ellyagg 15y ago
  You're thinking of British English. In America, punt means avoid, and most people here probably think of an American football punt--which is the voluntary end of a failed offensive possession--so suggests you're giving up for now and will try again later. We probably bastardized it, sure, but that ship has sailed.
  - bwooce 15y ago
    Thanks. Such a great word, lost forever :) So if I said "take a punt", would that be to take, or not take? I should build a guide of "words not to use on presentations".
mhartl 15y ago
I think they mean "Dr. Moore". "Mr. Moore" is a hack documentarian. Dr. Moore, on the other hand, has a Ph.D. in Chemistry from Caltech.
dkhenry 15y ago
This is nothing more then trying to pay the interest on your technical debt instead of paying off the balance. The longer you wait the more it grows and the bigger the problem becomes down the road. Right now its 20K for a new system, And for your troubles you still get no redundancy. Yes ram is cheap and you can just buy a bigger system next year, but hopefully you will hit a limit (growth is like that) then you will need to scale your now large and complex system. Code will need to be reworked. Interfaces will need to be re architected. The guy who wrote that bit of code for you that runs your whole business he may not be with you any more, even if he is that hack he put in place because you had the horsepower to spare, he doesn't really remember why he did it at 3:00am that one morning when stuff got out of hand. Now you have downtime and huge development costs, because you ignored a problem and instead threw hardware at it.
wazoox 15y ago
One of the most common similar occurrence I meet regularly is people grossly underestimating how many bytes per second you can really get from basic hardware such as hard drives, gigabit ethernet cards, etc. People often still remember that in 1998 the fastest FC 10k rpm drives hardly topped 12 MB/s, and that you needed an 8 CPU Origin2000 to push 80 MB/s through a GigE link (given that you had a terrific RAID array at both ends). Nowadays, even the most basic PC can saturate a GigE link (115 MB/s), and the slowest hard drives go 100 MB/s. Any SSD sustains several thousands IOPS, and so on. One of the most unfortunate result is that people often buy hugely powerful hardware when very basic stuff would have done the job just fine. How many people have I seen running puny workloads on 100k bucks 3Par or EMC arrays, where a handful of SSDs in a server would have done as well or better? Pulling 40Gb fibre across the room when cat 6 cable would have sufficed?

127 points by jlangenauer 15y ago

hmmm

Pedantry -- that isn't Moore's Law related, aside from the loosest interpretation that computers as a whole get faster. This is so true, though. Much of the horizontally scalability need comes from the land of extremely underpowered VPS machines on platforms like AWS. Yet the mind-boggling scale of performance you can inexpensively* (*-term used relatively) acquire for database servers is astonishing. SSDs (and plug-in flash drives) and boundless memory have changed everything.

masklinn 15y ago
> Pedantry -- that isn't Moore's Law related Actually, RAM would be the perfect application of the exact meaning of Moore's law: more RAM is an almost direct function of more transistors, and Moore's law is: > the number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years Moore's law precisely predicts you can double your amount of RAM every two years for the exact same price. Which is pretty much what TFA is about. SSD storage size is the same thing.
_delirium 15y ago
Given all the problems they've had with AWS, I wonder what the overall cost/benefit would be for someone like Reddit moving from the flexed-EC2-servers model to just one high-powered db server and some caching webservers in front of it.
- adestefan 15y ago
  Reddit's probably was using trying to use EBS as a backend for a transactional database. The issue is that EBS makes no guarantees on latency, only that they'll never lose your data.
  - bad_user 15y ago
    they'll never lose your data Amazon is not saying it, but EBS is probably lying when doing an fsync(). In this post [1] explaining why Reddit was down, one problem was that the DB slave got ahead of the master, with the most probable explanation being that the master flagged the data as being safe for replication, before committing it to disk and I trust PostgreSQL more than I trust EBS. In this forum answer Amazon is giving about this potential problem [2], they are dodging the question by saying that fsync() guarantees durability for instance failures, but not for volume failures, with the anual failure rate (AFR) being given as 0.1% - 0.5% for volumes (how accurate that is, it remains to be seen). So EBS is probably lying about fsync's success, especially since the behavior of fsync in virtual environments is always a surprise. So you can definitely lose your data more frequently than if you had your own hardware. [1] blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html [2] https://forums.aws.amazon.com/thread.jspa?threadID=27590 https://forums.aws.amazon.com/thread.jspa?threadID=27590
    - cperciva 15y ago
      Amazon is not saying it, but EBS is probably lying when doing an fsync(). Given that POSIX says "the nature of the transfer is implementation-defined" and they've defined what fsync does on their implementation: No, they're not lying. I wouldn't say that they're being misleading, either. On a local physical disk, once fsync returns your data is safely stored unless/until the disk dies. According to the forum post you linked to, semantics on EBS are exactly the same. fsync does not mean "has been written to spinning magnetic media".
  - deleted 15y ago
    [deleted]
  - pangram 15y ago
    Reddit also found EBS to be unreliable; they've moved all their high traffic data to ephemeral storage. They have an interesting summary of their technology stack here: http://blog.reddit.com/2012/01/january-2012-state-of-servers.html http://blog.reddit.com/2012/01/january-2012-state-of-servers...
- foobarbazetc 15y ago
  Reddit could easily cut their spend on hosting by like 50-75% just by moving to a dedicated hosting provider. They'd also get bare metal speeds in the process. Who knows why they've stuck with AWS, though. It's possible Amazon is giving them a discount so they can be used as an example.

ohashi 15y ago

For some reason, I don't imagine BaseCamp's use is really exploding like some other startups might in terms of database usage. It could very well be a case of their growth is scaling in-line with Moore's law, so it just happens to work out well.

moe 15y ago
I agree. Without any figures this is a pretty meaningless blog-post...
ams6110 15y ago
I think most "other startups" would LOVE to be in the ballpark with BaseCamp in terms of usage. No BaseCamp is not big like Facebook, but you've heard of them right? Most "ordinary" business people who do project planning work have probably heard of BaseCamp too. Most people never hear about most startups, fewer try their services, fewer than that become regular users. The point of the post is that in many/most cases, it's still easier and cheaper to throw hardware at a performance problem than to devote scarce engineering effort to optimization. And it's only getting more and more so. If you are a startup and you can throw $10,000 of hardware at a problem you can then keep your $100K engineers working on things that hardware alone can't solve.
- jonknee 15y ago
  > Most "ordinary" business people who do project planning work have probably heard of BaseCamp too. I don't think so. Maybe we just have a different opinion on ordinary, but I come across people all the time that would think Excel is the normal thing to use for this (and pretty much any other task...). Using web applications for day to day workflow is still alien to a lot of "ordinary" business types.
  - ams6110 15y ago
    Granted. s/most/some/
- untog 15y ago

shanemhansen 15y ago

A lot of startups engage in a sort of cargo-cult architecture. Their reasoning goes something like this: 1. Amazon/Facebook/Google have a lot of traffic. 2. Amazon/Facebook/Google use X to scale horizontally. ergo: 3. My little startup should use X and scale horizontally. What they fail to realize is that most of these companies would be ecstatic if they could scale machines vertically, if they could focus on great user features instead of having to figure out how to shard in the application layer. You should never forget that Amazon, Facebook, and twitter all started out as pretty basic LAMP stacks and built the tools when it was obvious that no other tool would do. I think google's an exception because their MVP was in fact a web scale application. So by all means, vet your idea, get some customers, get traction, and scale the cheap way by buying more ram for as long as you possibly can.

marshray 15y ago
Yes. But let's look at the other side too. Before Google came along and showed the business people the benefits of horizontal scaling, any software engineer would be automatically considered crazy if they suggested an architecture that wasn't built on a central RDBMS. So you have to weigh it against the other cargo-cult. How many startups along the way have failed due the inability to scale horizontally? How many have failed due to too much cost and complexity associated with re-engineering an architecture in which the assumption of fully ACID transactions permeates the entire codebase? (While the phone is ringing off the hook because production systems are falling over under load.)
j_baker 15y ago
I realize this may not be a popular opinion on HN, but there's something to be said for planning ahead. I've seen this story before and already know how it ends: you wait until the last possible moment to switch to a more horizontally scalable system and next thing you know you're spending more time and money maintaining the "cheap" solution than it would have taken to switch to something like Cassandra beforehand. To make matters worse, your service is crashing and the short term fix takes a day and requires two or three people to do the replica switch shuffle. The long term fix will take a couple of weeks, if you have the time for it of course. Long story short, I have to grant that you shouldn't worry about scaling up too soon or too quickly. But that don't go to the opposite extreme by putting it off until the last possible moment.

krupan 15y ago

He mentions Fusion-io drives getting faster. What's interesting about iodrives is that you don't have to buy a new one to get more speed, just updating your CPU or RAM will speed up an iodrive: "Because VSL leverages powerful host CPU cores and memory for block mapping, ioMemory performance automatically improves as organizations upgrade host CPUs or memory" (http://www.fusionio.com/overviews/vsl-technical-overview/ http://www.fusionio.com/overviews/vsl-technical-overview/) VSL (Virtual Storage Layer) is the Fusion-io software that makes an iodrive look like a block device to the OS, among other things. (Yes, I'm an engineer at Fusion)

bitops 15y ago

What never seems to get mentioned is energy cost. Sure, you can keep throwing hardware at the problem, but eventually that will lead to a substantial power bill.

rorrr 15y ago
Sharding will lead to an even much higher power bill.
- wildwood 15y ago
  Agreed, sharding takes "throwing hardware at the problem" to a whole new level.
  - bitops 15y ago
    I agree with what both of you have said. It's possible that, in the not-too-distant future, "green programming" will become a new field. If you can design an architecture (both hardware and software) that demonstrably reduces power costs, that's a valuable skill. And, I'd argue, on two fronts: lower costs for the company, plus good marketing. Every forward-thinking company (at least from what I've observed) are eager to brand themselves as "carbon neutral", "eco-friendly", etc.

wes-exp 15y ago

For those with custom hardware setups like this, how much time do you estimate you've spent designing, building, and maintaining custom servers? I can see how it would be less effort than sharding, but it seems like throwing hardware at the problem is not a completely "free lunch" when you get to the scale of having to custom design a server supercomputer and source specially made SSDs and terabytes of RAM.

sciurus 15y ago
You don't need to "custom design a server supercomputer". This stuff is as mainstream as Dell. You can go to dell.com and order a Poweredge R815 with 64 processor cores and 512GB of RAM for $16,000. If you want SSDs, you can purchase SATA or PCIe ones from Dell as well. http://www.dell.com/us/enterprise/p/poweredge-r815/pd http://www.dell.com/us/enterprise/p/poweredge-r815/pd http://www.dell.com/us/business/p/fusion-io-drive/pd http://www.dell.com/us/business/p/fusion-io-drive/pd

kyledrake 15y ago

I did a presentation on a game I made at Geoloqi that talks about the Fusion IO drives and why I think local hardware with SSD is the best solution for persistent data stores right now: http://www.slideshare.net/KyleDrake/building-mapattack http://www.slideshare.net/KyleDrake/building-mapattack

kyledrake 15y ago
The one warning I want to provide though is that not all SSDs are created equal: Make sure you get one that writes its cache to the disk on power failure, or you're going to be in a world of hurt.
- krupan 15y ago
  Can you explain more? You aren't talking about writing data that is cached in the SSD to a regular hard-drive, are you?
  - tesseract 15y ago
    I think he's talking about writing data in the SSD's DRAM cache into the actual flash.
    - krupan 15y ago
      That's what I thought too. Fusion-io drives all flush their write buffer to the nand flash on a power cut event, I wasn't aware that some SSD's didn't. People use iodrives as a caching layer in conjunction with tradition spinning hard drives, so wondered if possibly he was referring to that kind of setup.
  - xolox 15y ago
    To guarantee on-disk consistency, programs like database servers call fsync for every transaction. By definition an fsync involves events on the hardware level and even in an SSD this gives a slowdown. The SSD can use a write cache to speed this up. However should you lose power and the write cache is not backed by a small battery, you lose your most recent write(s) even though the latest fsync call guaranteed that the changes were on disk. The battery allows the SSD to successfully flush the write cache to disk in the event that the computer shuts down.

bwooce 15y ago

Gah, stop abusing the word punt. The headline is nonsensical. Punt does not mean avoidance. It has much closer associations to "attempt" (punter, plus rugby usage). I totally agree with the rest of the article, why shard or otherwise distribute your database before being required? Although I think I would build my DB as shard 1 of 1 to allow for the future case.

ellyagg 15y ago
You're thinking of British English. In America, punt means avoid, and most people here probably think of an American football punt--which is the voluntary end of a failed offensive possession--so suggests you're giving up for now and will try again later. We probably bastardized it, sure, but that ship has sailed.
- bwooce 15y ago
  Thanks. Such a great word, lost forever :) So if I said "take a punt", would that be to take, or not take? I should build a guide of "words not to use on presentations".

mhartl 15y ago

I think they mean "Dr. Moore". "Mr. Moore" is a hack documentarian. Dr. Moore, on the other hand, has a Ph.D. in Chemistry from Caltech.

dkhenry 15y ago

This is nothing more then trying to pay the interest on your technical debt instead of paying off the balance. The longer you wait the more it grows and the bigger the problem becomes down the road. Right now its 20K for a new system, And for your troubles you still get no redundancy. Yes ram is cheap and you can just buy a bigger system next year, but hopefully you will hit a limit (growth is like that) then you will need to scale your now large and complex system. Code will need to be reworked. Interfaces will need to be re architected. The guy who wrote that bit of code for you that runs your whole business he may not be with you any more, even if he is that hack he put in place because you had the horsepower to spare, he doesn't really remember why he did it at 3:00am that one morning when stuff got out of hand. Now you have downtime and huge development costs, because you ignored a problem and instead threw hardware at it.

wazoox 15y ago

One of the most common similar occurrence I meet regularly is people grossly underestimating how many bytes per second you can really get from basic hardware such as hard drives, gigabit ethernet cards, etc. People often still remember that in 1998 the fastest FC 10k rpm drives hardly topped 12 MB/s, and that you needed an 8 CPU Origin2000 to push 80 MB/s through a GigE link (given that you had a terrific RAID array at both ends). Nowadays, even the most basic PC can saturate a GigE link (115 MB/s), and the slowest hard drives go 100 MB/s. Any SSD sustains several thousands IOPS, and so on. One of the most unfortunate result is that people often buy hugely powerful hardware when very basic stuff would have done the job just fine. How many people have I seen running puny workloads on 100k bucks 3Par or EMC arrays, where a handful of SSDs in a server would have done as well or better? Pulling 40Gb fibre across the room when cat 6 cable would have sufficed?