4 ms·

We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significa

by qhoxie 17y ago

We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significant effort trying to get our system stable and we ended up narrowing it down to EBS simply not being able to handle the IOPS we needed. We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter. There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly. If you want more information you are welcome to email me.

jedberg 17y ago
I'm beginning to thing that MySQL does something very different with disks than Postgres, because everyone I've talked to that said they had a problem had been running MySQL, and everyone who has been successful has been running Postgres (myself included).
- qhoxie 17y ago
  This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses. Our setup was surely not ideal, but even with significant tuning, it was not sufficient. It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
  - jedberg 17y ago
    We had some of databases at EC2 since November of 2008, and moved the rest of them over in May 2009.
- metajack 17y ago
  Maybe. I know when we migrated Chesspark from native machines to EC2, the postgresql database performance was at least an order of magnitude worse, and we spent two weeks doing caching and index hunting that we never had to worry about before.
bjclark 17y ago
Just to add to what QHoxie said (I worked with him on this project), the site is doing Alexa top 750 traffic. Every page view would hit the db atleast a few times. And it's not that the site just couldn't run off EBS volumes, but we would see massive performance spikes where EBS IOPS would slow to a crawl for a few minutes to even a couple hours.
- qhoxie 17y ago
  This is a good clarification to make. It was not that our entire application ground to a halt when we flipped the switch. We would experience intermittent and extreme performance issues. We dug deep with tools like cacti and ended up being able to point only at the 'unknowns' of the shared resources and EBS.
stephenjudkins 17y ago
The pain point in this particular case (I worked with qhoxie and bjclark on this) wasn't so much that the EBS performance was consistently bad, but that it was very unpredictable. Servers would suddenly, for no apparent reason, start performing very poorly. That's been a consistent theme I've seen with EC2 instances--it's hard to predict how fast something is going to be, and once it's running, you don't know if its performance is going to change. That's one reason to take benchmarks of EC2 (and possibly other cloud providers) with a grain of salt. I've heard anecdotes of people starting up n EC2 instances, running benchmarks on each, then killing all but the fastest ones.
- metajack 17y ago
  We don't go that far, but we did choose the data center that had the faster machines. It had upgraded hardware that the others didn't. It won't be the fastest forever, but you can certainly shop around internally to find faster machines.

by qhoxie 17y ago

jedberg 17y ago

I'm beginning to thing that MySQL does something very different with disks than Postgres, because everyone I've talked to that said they had a problem had been running MySQL, and everyone who has been successful has been running Postgres (myself included).

qhoxie 17y ago
This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses. Our setup was surely not ideal, but even with significant tuning, it was not sufficient. It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
- jedberg 17y ago
  We had some of databases at EC2 since November of 2008, and moved the rest of them over in May 2009.
metajack 17y ago
Maybe. I know when we migrated Chesspark from native machines to EC2, the postgresql database performance was at least an order of magnitude worse, and we spent two weeks doing caching and index hunting that we never had to worry about before.

bjclark 17y ago

Just to add to what QHoxie said (I worked with him on this project), the site is doing Alexa top 750 traffic. Every page view would hit the db atleast a few times. And it's not that the site just couldn't run off EBS volumes, but we would see massive performance spikes where EBS IOPS would slow to a crawl for a few minutes to even a couple hours.

qhoxie 17y ago
This is a good clarification to make. It was not that our entire application ground to a halt when we flipped the switch. We would experience intermittent and extreme performance issues. We dug deep with tools like cacti and ended up being able to point only at the 'unknowns' of the shared resources and EBS.

stephenjudkins 17y ago

The pain point in this particular case (I worked with qhoxie and bjclark on this) wasn't so much that the EBS performance was consistently bad, but that it was very unpredictable. Servers would suddenly, for no apparent reason, start performing very poorly. That's been a consistent theme I've seen with EC2 instances--it's hard to predict how fast something is going to be, and once it's running, you don't know if its performance is going to change. That's one reason to take benchmarks of EC2 (and possibly other cloud providers) with a grain of salt. I've heard anecdotes of people starting up n EC2 instances, running benchmarks on each, then killing all but the fastest ones.

metajack 17y ago
We don't go that far, but we did choose the data center that had the faster machines. It had upgraded hardware that the others didn't. It won't be the fastest forever, but you can certainly shop around internally to find faster machines.