6 ms·

Ask HN: Anyone used Amazon EC2 with a database or other high IO operation?

I'm thinking of using Amazon's EC2 to run my database and web farm but I'm concerned about the intense IO that's required to run the database. I've shared resources on a vps in the past and it's been a nightmare. I know they offer high IO servers that have limited sharing but it's still sharing. Has anyone had good or bad experience running a high IO operation using EC2?

hendler 17y ago
Good luck so far using EBS for MySQL, logs etc. I/O costs not significant yet. You can Raid EBS devices btw.
- MichaelApproved 17y ago
  Doesn't Amazon already RAID the storage they offer you? Is drive failure something I need to worry about? I thought they had a beefy RAID system already in place.
  - dryicerx 17y ago
    They do, but I think he's talking about doing a raid0 just to leech more io, not safety.
    - MichaelApproved 17y ago
      I see now with all the comments that everyone is referring to getting more platters spinning the data.
- rykov 17y ago
  RAID helps if disk is the bottleneck. If it's the network (EBS is network attached), then you're SOL and should probably try a bigger instance instead and hope fewer people share that physical host.
- grourk 17y ago
  If you enable striping (RAID0) for performance, how do you manage snapshots and backups? Seems like there would be consistency issues re-building the array from multiple snapshots.
bretthoerner 17y ago
You wouldn't want to use the virtual hard drive they provide you, but rather EBS (Elastic Block Storage). With that you can setup a mirrored RAID (redundancy is a given already) of as many devices as you please.
mark_l_watson 17y ago
I can't answer your question from personal experience because I always just use plain old EBS volumes with my EC2s. That said, here is a good writeup on RAID use with EBS by someone at Heroku: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...
- MichaelApproved 17y ago
  That's got some really good information. Most important to me is that it does confirm my fears of flaky IO performance. He does point out some ways to tune it but it still seems like a rough experience. These cloud services seem great for low IO apps but once you have to hit the database hard it seems like anyone's guess if your db will perform.
  - bretthoerner 17y ago
    That's why I'm a big fan of Rackspace. You can get bare metal servers for heavy I/O and simultaneously have "Cloud" servers on the same local network (example: Github).
    - modoc 17y ago
      FWIW: SoftLayer offers the same combo of dedicated servers and cloud offerings, and I prefer their turn around time, support, and pricing to RS.
    - MichaelApproved 17y ago
      That sounds like a good mix. I'll explore that option some more.
    - mrkurt 17y ago
      Are you actually using this sort of service from them? When I talked to them about 2 months ago, we couldn't get anything like that since their "cloud" stuff is in a datacenter you can't currently buy dedicated servers in. In theory, they're opening another next year that may solve the problem. It's hard to tell, though, since their sales people don't seem to have any experience selling a hybrid setup. My understanding is that Github virtualizes things themselves, they're not running Rackspace cloud servers.
garyrichardson 17y ago
I am using MySQL on EC2 c1.medium instances. I haven't noticed an IO limit yet, but that could be how I'm using it plus the data set is still under 2GB. I use the database as a raw data source and have a cron that generates read only data sets for the web servers to use. This works for my application, but probably not most.
deleted 17y ago
[deleted]
dryicerx 17y ago
Seconding the EBS Raid0 option. Some comparison here http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-linode/ http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-linode... Also, if you have a part of a DB that just requires unholy amounts IO, there is of course the new 68gig instances, move that part and run it purely on memory with a combination of frequent dumps to disk.
- MichaelApproved 17y ago
  Maybe a combination of DB file in RAM and transaction logs to disk?
scumola 17y ago
Just write to /mnt (150GB ramdrive on the small instance) then occasionally sync to an EBS volume. That'll probably give you the best I/O.
- jotto 17y ago
  /mnt is not a ram drive, it's ephemeral hard disk, and EBS has better IO than the /mnt. You only get 1.7GB RAM on the small instance.
charlesju 17y ago
With any high IO web application, it all comes down to horizontal sharding to make it work. But, we recently switched from a high powered DB to Amazon and we're doing fine.
qhoxie 17y ago
We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significant effort trying to get our system stable and we ended up narrowing it down to EBS simply not being able to handle the IOPS we needed. We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter. There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly. If you want more information you are welcome to email me.
- jedberg 17y ago
  I'm beginning to thing that MySQL does something very different with disks than Postgres, because everyone I've talked to that said they had a problem had been running MySQL, and everyone who has been successful has been running Postgres (myself included).
  - qhoxie 17y ago
    This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses. Our setup was surely not ideal, but even with significant tuning, it was not sufficient. It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
    - jedberg 17y ago
      We had some of databases at EC2 since November of 2008, and moved the rest of them over in May 2009.
  - metajack 17y ago
    Maybe. I know when we migrated Chesspark from native machines to EC2, the postgresql database performance was at least an order of magnitude worse, and we spent two weeks doing caching and index hunting that we never had to worry about before.
jedberg 17y ago
We run all of reddit on ec2, which includes a bunch of postgres servers. Each one is running with a single EBS. However, I've heard horror stories of people trying to run much less busy databases and having lots of problems, usually with MySQL. Those databases are all on XLarge instances, so there is minimal sharing, and we've also gone to great lengths to make sure all of our normal queries are in indexes, so the disk gets hit less. We also have a read slave for every database to alleviate read loads. One thing you might want to do is run 'iostat -xtc' on your current box and put that in a log file. Then go back and analyze it and see what your average and peak reads and writes are. Amazon's max for a single EBS appears to be about 1000 ops / second (at least, that is what we were doing when they told us we maxed out the performance of the disk). Good luck! Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).
- MichaelApproved 17y ago
  I was just talking to someone wondering how you guys at reddit were able to do it. The article about taking your last server offline really pushed me to move forward on this. I should mention that I'm running Windows with MSSQL. I imagine the performance should be the same but there must be some differences of course. I like the option of rackspace allowing for a dedicated server next to the cloud but I should probably give EC2 a few test runs of my own too. I think if EC2 is going to work it wont be an overnight solution. I'll have to move over slowly, learning to tune the box as I use it more.
  - jedberg 17y ago
    If you're using Windows, I would stay away from EC2 right now. They aren't quite mature enough for Windows yet. If you look at the EC2 forums, the majority of complaints are about Windows bugs. Actually, you might want to try and get in on Microsoft's Azure beta.
    - MichaelApproved 17y ago
JoeH 17y ago
Here at RescueTime we use EC2 for all of our servers, with the exception of one server outside EC2 for monitoring. All of our MySQL instances are currently each running on 4x EBS volumes using LVM stripped across all 4 volumes. Our primary transactional table is nearing a billion rows, and just this weekend we performed a migration that segmented this table into 40 partitions, containing data back to 2007. We are currently handling roughly 3,000,000 inserts and 2,500,000 updates on this one table daily. IOWait on our DB holds steady at about 5% throughout our peak (5am - 4pm PST). It is certainly not Fibre Channel backboned speeds, but the other benefits of using EBS CURRENTLY outweighs any IO performance hits we might have. The ability to backup production and refresh any of our other DB environments in less than 15 minutes, using EBS snapshots is worth it. Especially for the price we pay monthly for this infrastructure. If you need any more details on our setup or tips in troubleshooting/setting up your own, feel free to ping me.
jbellis 17y ago
I work on the Cassandra distributed database for Rackspace. A _lot_ of people start out trying Cassandra on EC2 but the universal conclusion has been that I/O performance is miserable. Anecdotally, Rackspace Cloud Servers I/O works much better, although the only serious head-to-head comparison I have seen is this one: http://pl.atyp.us/wordpress/?p=2240 http://pl.atyp.us/wordpress/?p=2240 (tl;dr: < 20MBps on writes with local storage or EBS on EC2 vs 100MBps+ on RCS)
- JoelPM 17y ago
  At OpenX we're using Cassandra on EC2 and it has served our needs well so far. Our use case is a little different in that we set hard timeouts on the client side reads and can tolerate a small percentage of requests that don't complete in time. According to Amazon the I/O depends on the instance type (http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/) so the OP may want to take that into consideration when picking an instance type.
  - paraschopra 17y ago
    I had heard you were using MySQL as database earlier. What made you switch to Cassandra? Just curious.
    - JoelPM 17y ago
      We're using both. Existing MySQL back-ends are still there, but for new development that can leverage a distributed key-value store we're using Cassandra.
  - JoelPM 17y ago
    Unfortunately, last night I completely failed to remember that Rackspace and EC2 are direct competitors in this space so what was intended to be a simple data-point may be construed as more than that - my apologies. What I intended to say was "you can make Cassandra work on EC2." However, if the I/O throughput of EC2 is a fraction that of some other provider, you'll need more machines in EC2 to achieve the same throughput. My personal site is hosted on the Rackspace Cloud and I've been very pleased with it, but my workload is different so I can't give a comparison. I also know that we've benefited directly from jbellis' support on IRC and the mailing list, so this should be considered an endorsement of Cassandra, if anything.
spudlyo 17y ago
It's worth noting that you can speed up write operations on EBS volumes by first initializing them by writing data to every block on the volume. Subsequent writes will be faster due to the way Amazon virtualizes disks. http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-storage.html http://docs.amazonwebservices.com/AWSEC2/2007-08-29/Develope...
nearestneighbor 17y ago
I'm curious about the economics of this. If you reserve an extra large high-CPU instance for 3 years, you end up paying $3,000 per year ($2,800 initial fee + hourly rate). Isn't it cheaper to have your own servers?
- jshen 17y ago
  cheaper but less flexible.
  - nearestneighbor 17y ago
    But you are already giving up flexibility with the 3 year term and reserved instances, or am I misunderstanding something?
    - mmt 17y ago
      I don't think you're misunderstanding anything at all My opinion of what constitutes "middle of nowhere" may differ from the average San Franciscan's, but I've had the opposite estimation of costs that jedberg did. That is, EC2 was 20% more expensive than operating ones own hardware in a South Bay datacenter for a year. This would have been about a year ago, but, IIRC, including operating costs, one would break even at around 10 months of unreserved EC2. Pricing has certainly changed in the meantime, so 30% in the other direction is well within the realm of possibility. Back then, it cost about $3 per year per watt. Now it's pushing $4. The trick is, in the case of ones own hardware, one usually pays for the available capacity, not just what one uses. On a 500W server, that's $2k annually. That's the flexibility that AWS offers: one doesn't pay for the operating capacity, only the use. It would seem, however, that one pays for the hardware up front either way. The great increase in flexibility of operating ones own hardware is not suffering from only ordering off an abbreviated menu. Germane to the OP's question, I have never found any virtual or dedicated server offerings come close (say, a factor of 10) to the I/O performance of something custom configured/assembled (even from inexpensive, commodity blocks). Whether this constitutes a problem is subject to interpretation In general, I remain skeptical that there exists in The Cloud cheap-enough commodity servers suited to even a majority of applications. Remember, even Google had custom hardware very early on.

hendler 17y ago

Good luck so far using EBS for MySQL, logs etc. I/O costs not significant yet. You can Raid EBS devices btw.

MichaelApproved 17y ago
Doesn't Amazon already RAID the storage they offer you? Is drive failure something I need to worry about? I thought they had a beefy RAID system already in place.
- dryicerx 17y ago
  They do, but I think he's talking about doing a raid0 just to leech more io, not safety.
  - MichaelApproved 17y ago
    I see now with all the comments that everyone is referring to getting more platters spinning the data.
rykov 17y ago
RAID helps if disk is the bottleneck. If it's the network (EBS is network attached), then you're SOL and should probably try a bigger instance instead and hope fewer people share that physical host.
grourk 17y ago
If you enable striping (RAID0) for performance, how do you manage snapshots and backups? Seems like there would be consistency issues re-building the array from multiple snapshots.

bretthoerner 17y ago

You wouldn't want to use the virtual hard drive they provide you, but rather EBS (Elastic Block Storage). With that you can setup a mirrored RAID (redundancy is a given already) of as many devices as you please.

mark_l_watson 17y ago

I can't answer your question from personal experience because I always just use plain old EBS volumes with my EC2s. That said, here is a good writeup on RAID use with EBS by someone at Heroku: http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...

MichaelApproved 17y ago
That's got some really good information. Most important to me is that it does confirm my fears of flaky IO performance. He does point out some ways to tune it but it still seems like a rough experience. These cloud services seem great for low IO apps but once you have to hit the database hard it seems like anyone's guess if your db will perform.
- bretthoerner 17y ago
  That's why I'm a big fan of Rackspace. You can get bare metal servers for heavy I/O and simultaneously have "Cloud" servers on the same local network (example: Github).
  - modoc 17y ago
    FWIW: SoftLayer offers the same combo of dedicated servers and cloud offerings, and I prefer their turn around time, support, and pricing to RS.
  - MichaelApproved 17y ago
    That sounds like a good mix. I'll explore that option some more.
  - mrkurt 17y ago
    Are you actually using this sort of service from them? When I talked to them about 2 months ago, we couldn't get anything like that since their "cloud" stuff is in a datacenter you can't currently buy dedicated servers in. In theory, they're opening another next year that may solve the problem. It's hard to tell, though, since their sales people don't seem to have any experience selling a hybrid setup. My understanding is that Github virtualizes things themselves, they're not running Rackspace cloud servers.

garyrichardson 17y ago

I am using MySQL on EC2 c1.medium instances. I haven't noticed an IO limit yet, but that could be how I'm using it plus the data set is still under 2GB. I use the database as a raw data source and have a cron that generates read only data sets for the web servers to use. This works for my application, but probably not most.

deleted 17y ago

[deleted]

dryicerx 17y ago

Seconding the EBS Raid0 option. Some comparison here http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-linode/ http://blog.mudy.info/2009/04/disk-io-ec2-vs-mosso-vs-linode... Also, if you have a part of a DB that just requires unholy amounts IO, there is of course the new 68gig instances, move that part and run it purely on memory with a combination of frequent dumps to disk.

MichaelApproved 17y ago
Maybe a combination of DB file in RAM and transaction logs to disk?

scumola 17y ago

Just write to /mnt (150GB ramdrive on the small instance) then occasionally sync to an EBS volume. That'll probably give you the best I/O.

jotto 17y ago
/mnt is not a ram drive, it's ephemeral hard disk, and EBS has better IO than the /mnt. You only get 1.7GB RAM on the small instance.

charlesju 17y ago

With any high IO web application, it all comes down to horizontal sharding to make it work. But, we recently switched from a high powered DB to Amazon and we're doing fine.

qhoxie 17y ago

We have a highly trafficked and large dataset that we aimed to move from our datacenter to EC2. This is all in MySQL. Upon doing so, we had to spend significant effort trying to get our system stable and we ended up narrowing it down to EBS simply not being able to handle the IOPS we needed. We explored all the options available and worked with both Percona and Amazon on the issues, but it was clear that EBS and EC2 just are not meant to handle such load. Our database is currently back running in our datacenter. There have been a number of reports with similar results using EBS. It all depends on your particular IO profile, but since you said it was high, I encourage you to tread lightly. If you want more information you are welcome to email me.

jedberg 17y ago
I'm beginning to thing that MySQL does something very different with disks than Postgres, because everyone I've talked to that said they had a problem had been running MySQL, and everyone who has been successful has been running Postgres (myself included).
- qhoxie 17y ago
  This really would not be all that surprising, but it also would not surprise me if the average Postgres setup was better tuned/architected than the majority case of MySQL uses. Our setup was surely not ideal, but even with significant tuning, it was not sufficient. It is really good to hear successes like Reddit's, though. How long have you folks had your databases in EC2?
  - jedberg 17y ago
    We had some of databases at EC2 since November of 2008, and moved the rest of them over in May 2009.
- metajack 17y ago
  Maybe. I know when we migrated Chesspark from native machines to EC2, the postgresql database performance was at least an order of magnitude worse, and we spent two weeks doing caching and index hunting that we never had to worry about before.

jedberg 17y ago

We run all of reddit on ec2, which includes a bunch of postgres servers. Each one is running with a single EBS. However, I've heard horror stories of people trying to run much less busy databases and having lots of problems, usually with MySQL. Those databases are all on XLarge instances, so there is minimal sharing, and we've also gone to great lengths to make sure all of our normal queries are in indexes, so the disk gets hit less. We also have a read slave for every database to alleviate read loads. One thing you might want to do is run 'iostat -xtc' on your current box and put that in a log file. Then go back and analyze it and see what your average and peak reads and writes are. Amazon's max for a single EBS appears to be about 1000 ops / second (at least, that is what we were doing when they told us we maxed out the performance of the disk). Good luck! Edit: I forgot to mention that on all the database disks, we use ext2 and noatime. Both decrease the total number of writes necessary, and have very little downside (the biggest being that you have to fsck on a crash).

MichaelApproved 17y ago
I was just talking to someone wondering how you guys at reddit were able to do it. The article about taking your last server offline really pushed me to move forward on this. I should mention that I'm running Windows with MSSQL. I imagine the performance should be the same but there must be some differences of course. I like the option of rackspace allowing for a dedicated server next to the cloud but I should probably give EC2 a few test runs of my own too. I think if EC2 is going to work it wont be an overnight solution. I'll have to move over slowly, learning to tune the box as I use it more.
- jedberg 17y ago
  If you're using Windows, I would stay away from EC2 right now. They aren't quite mature enough for Windows yet. If you look at the EC2 forums, the majority of complaints are about Windows bugs. Actually, you might want to try and get in on Microsoft's Azure beta.
  - MichaelApproved 17y ago

JoeH 17y ago

Here at RescueTime we use EC2 for all of our servers, with the exception of one server outside EC2 for monitoring. All of our MySQL instances are currently each running on 4x EBS volumes using LVM stripped across all 4 volumes. Our primary transactional table is nearing a billion rows, and just this weekend we performed a migration that segmented this table into 40 partitions, containing data back to 2007. We are currently handling roughly 3,000,000 inserts and 2,500,000 updates on this one table daily. IOWait on our DB holds steady at about 5% throughout our peak (5am - 4pm PST). It is certainly not Fibre Channel backboned speeds, but the other benefits of using EBS CURRENTLY outweighs any IO performance hits we might have. The ability to backup production and refresh any of our other DB environments in less than 15 minutes, using EBS snapshots is worth it. Especially for the price we pay monthly for this infrastructure. If you need any more details on our setup or tips in troubleshooting/setting up your own, feel free to ping me.

jbellis 17y ago

I work on the Cassandra distributed database for Rackspace. A _lot_ of people start out trying Cassandra on EC2 but the universal conclusion has been that I/O performance is miserable. Anecdotally, Rackspace Cloud Servers I/O works much better, although the only serious head-to-head comparison I have seen is this one: http://pl.atyp.us/wordpress/?p=2240 http://pl.atyp.us/wordpress/?p=2240 (tl;dr: < 20MBps on writes with local storage or EBS on EC2 vs 100MBps+ on RCS)

JoelPM 17y ago
At OpenX we're using Cassandra on EC2 and it has served our needs well so far. Our use case is a little different in that we set hard timeouts on the client side reads and can tolerate a small percentage of requests that don't complete in time. According to Amazon the I/O depends on the instance type (http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/) so the OP may want to take that into consideration when picking an instance type.
- paraschopra 17y ago
  I had heard you were using MySQL as database earlier. What made you switch to Cassandra? Just curious.
  - JoelPM 17y ago
    We're using both. Existing MySQL back-ends are still there, but for new development that can leverage a distributed key-value store we're using Cassandra.
- JoelPM 17y ago
  Unfortunately, last night I completely failed to remember that Rackspace and EC2 are direct competitors in this space so what was intended to be a simple data-point may be construed as more than that - my apologies. What I intended to say was "you can make Cassandra work on EC2." However, if the I/O throughput of EC2 is a fraction that of some other provider, you'll need more machines in EC2 to achieve the same throughput. My personal site is hosted on the Rackspace Cloud and I've been very pleased with it, but my workload is different so I can't give a comparison. I also know that we've benefited directly from jbellis' support on IRC and the mailing list, so this should be considered an endorsement of Cassandra, if anything.

spudlyo 17y ago

It's worth noting that you can speed up write operations on EBS volumes by first initializing them by writing data to every block on the volume. Subsequent writes will be faster due to the way Amazon virtualizes disks. http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/instance-storage.html http://docs.amazonwebservices.com/AWSEC2/2007-08-29/Develope...

nearestneighbor 17y ago

I'm curious about the economics of this. If you reserve an extra large high-CPU instance for 3 years, you end up paying $3,000 per year ($2,800 initial fee + hourly rate). Isn't it cheaper to have your own servers?

jshen 17y ago
cheaper but less flexible.
- nearestneighbor 17y ago
  But you are already giving up flexibility with the 3 year term and reserved instances, or am I misunderstanding something?
  - mmt 17y ago
    I don't think you're misunderstanding anything at all My opinion of what constitutes "middle of nowhere" may differ from the average San Franciscan's, but I've had the opposite estimation of costs that jedberg did. That is, EC2 was 20% more expensive than operating ones own hardware in a South Bay datacenter for a year. This would have been about a year ago, but, IIRC, including operating costs, one would break even at around 10 months of unreserved EC2. Pricing has certainly changed in the meantime, so 30% in the other direction is well within the realm of possibility. Back then, it cost about $3 per year per watt. Now it's pushing $4. The trick is, in the case of ones own hardware, one usually pays for the available capacity, not just what one uses. On a 500W server, that's $2k annually. That's the flexibility that AWS offers: one doesn't pay for the operating capacity, only the use. It would seem, however, that one pays for the hardware up front either way. The great increase in flexibility of operating ones own hardware is not suffering from only ordering off an abbreviated menu. Germane to the OP's question, I have never found any virtual or dedicated server offerings come close (say, a factor of 10) to the I/O performance of something custom configured/assembled (even from inexpensive, commodity blocks). Whether this constitutes a problem is subject to interpretation In general, I remain skeptical that there exists in The Cloud cheap-enough commodity servers suited to even a majority of applications. Remember, even Google had custom hardware very early on.