5 ms·

This is cool and all, but its part of an overall trend about aws/ec2 thats starting to get on my nerves. They're spending their time and resources working thei

by cagenut 16y ago

This is cool and all, but its part of an overall trend about aws/ec2 thats starting to get on my nerves. They're spending their time and resources working their way up the stack instead of improving the lower layers of the stack for which you use them in the first place. Right now ~2+ms inter-node latencies, arbitrary 200+ms hangups on EBS, and the general lack of an SSD option make AWS a non-starter for things that need to scale without being architecture-astronaut computer science projects. For many workloads I have to colo instead of use AWS because of the age old "spend 10K on hardware or hire an FTE developer" tradeoff.

gfodor 16y ago
Your claim is a false dilemma unless you can prove otherwise.
- cagenut 16y ago
  http://www.mysqlperformanceblog.com/2011/02/21/death-match-ebs-versus-ssd-price-performance-and-qos/ http://www.mysqlperformanceblog.com/2011/02/21/death-match-e... tl;dr - "price is in the same order of magnitude, but performance is two to three orders of magnitude different."
  - gfodor 16y ago
    Your claim was that Amazon, by working on features higher on the stack, is unable to work on improving lower performance. You need to prove that there is a lack of engineering resources at Amazon to work on both tasks at once.
nethergoat 16y ago
Designing for fault tolerance and scaling out instead of individual node durability and scaling up is, in most cases, a sounder approach. Since the vast majority of applications map well to this case, it makes sense for Amazon to focus on features that assume/assist scale-out. It's easier to design for scale-out than one might think; a great resource to get started is http://highscalability.com http://highscalability.com. Regarding EBS, I haven't seen the hangup issue you've described. Any data? As for low inter-node latencies, Amazon has an offering that specifically addresses that need: http://aws.amazon.com/ec2/hpc-applications/ http://aws.amazon.com/ec2/hpc-applications/ Overall, I think Amazon is making a lot of inroads in areas with specific hardware demands. They just launched GPU compute options, and I expect we'll even see SSDs soon.
- cagenut 16y ago
  Scaling out vs. scaling up is the false dilemma, the answer is both at the same time solving the bottlenecks you have using the time and resources you have. The "vast majority" of applications do not map well to an n-wide map-reduce-esque design, the "vast majority" of applications bottleneck on an authoritative datastore's I/O capacity. I'm the sysadmin (not the dba) for a website that does 30K http req/s at the edge, and our mysql cluster does about 3000 req/s during mid-day with most queries in the 1 - 5ms range. Highscalability is a terrible place to get started, its a great place to share notes but if you get started there you will waste insane amounts of time on architecture astronaut nonsense.
  - toddh 16y ago
    What do you consider architecture astronaut nonsense?
- travisglines 16y ago
  On the spectrum of scaling an application AWS has definitely leaned hard towards scaling out. (where the other end is scaling up) This means it caters a little more to denormalized databases and load balanced web servers than it does to one giant mysql database server or a cluster of a few large database servers. I agree with nethergoat in that if you need low latency communication between ec2 instances that you should be using their hpc instances with faster/optomized ethernet. I however disagree with nethergoat and agree with cagenut and think that the vast-majority of applications still currently depend more on a scale up paradigm. That is slowly moving towards scale out as people/businesses see its benefits and AWS is catering to this trend.

by cagenut 16y ago

gfodor 16y ago

Your claim is a false dilemma unless you can prove otherwise.

cagenut 16y ago
http://www.mysqlperformanceblog.com/2011/02/21/death-match-ebs-versus-ssd-price-performance-and-qos/ http://www.mysqlperformanceblog.com/2011/02/21/death-match-e... tl;dr - "price is in the same order of magnitude, but performance is two to three orders of magnitude different."
- gfodor 16y ago
  Your claim was that Amazon, by working on features higher on the stack, is unable to work on improving lower performance. You need to prove that there is a lack of engineering resources at Amazon to work on both tasks at once.

nethergoat 16y ago

Designing for fault tolerance and scaling out instead of individual node durability and scaling up is, in most cases, a sounder approach. Since the vast majority of applications map well to this case, it makes sense for Amazon to focus on features that assume/assist scale-out. It's easier to design for scale-out than one might think; a great resource to get started is http://highscalability.com http://highscalability.com. Regarding EBS, I haven't seen the hangup issue you've described. Any data? As for low inter-node latencies, Amazon has an offering that specifically addresses that need: http://aws.amazon.com/ec2/hpc-applications/ http://aws.amazon.com/ec2/hpc-applications/ Overall, I think Amazon is making a lot of inroads in areas with specific hardware demands. They just launched GPU compute options, and I expect we'll even see SSDs soon.

cagenut 16y ago
Scaling out vs. scaling up is the false dilemma, the answer is both at the same time solving the bottlenecks you have using the time and resources you have. The "vast majority" of applications do not map well to an n-wide map-reduce-esque design, the "vast majority" of applications bottleneck on an authoritative datastore's I/O capacity. I'm the sysadmin (not the dba) for a website that does 30K http req/s at the edge, and our mysql cluster does about 3000 req/s during mid-day with most queries in the 1 - 5ms range. Highscalability is a terrible place to get started, its a great place to share notes but if you get started there you will waste insane amounts of time on architecture astronaut nonsense.
- toddh 16y ago
  What do you consider architecture astronaut nonsense?
travisglines 16y ago
On the spectrum of scaling an application AWS has definitely leaned hard towards scaling out. (where the other end is scaling up) This means it caters a little more to denormalized databases and load balanced web servers than it does to one giant mysql database server or a cluster of a few large database servers. I agree with nethergoat in that if you need low latency communication between ec2 instances that you should be using their hpc instances with faster/optomized ethernet. I however disagree with nethergoat and agree with cagenut and think that the vast-majority of applications still currently depend more on a scale up paradigm. That is slowly moving towards scale out as people/businesses see its benefits and AWS is catering to this trend.