5 ms·

Make sure you explore the limits of EBS before assuming it's a perfect solution. We've found it to have incredibly slow throughput at times. Some reference lin

Make sure you explore the limits of EBS before assuming it's a perfect solution. We've found it to have incredibly slow throughput at times. Some reference links: - http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs - http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes - http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

gyardley 16y ago
We had these issues as well. I'd certainly try AWS, but I'd wait a couple of months before shelling out the upfront cost for the Reserved Instances.
- blantonl 16y ago
  EBS definitely has performance limitations, however creating striped RAID arrays of EBS volumes significantly helps with performance. This blog post walks through a bunch of things you can do to make the best out of the flexibility you get with EBS, and includes scheduler tuning, RAID block size planning, etc. http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs... My team runs a large AWS/EBS backed implementation and, while we've seen some of the EBS performance issues, we've certainly engineered past them to get us where we need to be.
  - chrisbolt 16y ago
    Wouldn't creating striped RAID arrays of EBS volumes add to the cost and remove the ability to do snapshots? How do you get a consistent snapshot of multiple striped EBS volumes?
    - spudlyo 16y ago
      It just makes it more complicated. You have lock and snap all of the volumes at once. The ec2-consistent-snapshot tool makes this easier to manage.
    - blantonl 16y ago
      EBS costs are pretty low. And my above post was directed to someone who is deploying a database on top of EBS. Sorry if I didn't make that clear. In our case, we replicate to set of non-RAID EBS instances where we pull snapshots.
    - imbriaco 16y ago
      Also keep in mind that creating a RAID0 of EBS volumes dramatically reduces your reliability too. If any one of the underlying EBS volumes misbehaves the entire RAID set has problems. Snapshots are still doable, they just require coordination of all the underlying devices to make sure they are all in a consistent state. The way you do this can vary but it's not particularly hard.
imbriaco 16y ago
As a heavy user of EBS, I heartily concur. The ability to move EBS volumes around is handy but you pay it back in inconsistent, poor-to-mediocre performance. Good luck.
recampbell 16y ago
I've also seen highly variable io performance (ie, slow/stuck) on EBS volumes. Sometimes, all reads/writes block for several second or minutes, with iowait going to 100% on an m1.small. example: http://developer.amazonwebservices.com/connect/message.jspa?messageID=201918#201918 http://developer.amazonwebservices.com/connect/message.jspa?... However, I've decided that the other benefits of EBS volumes are just to important to give up on (snapshotting, lazy-load, re-attachment). Instead, I plan to monitor for such situations and blow away the node when I detect it. But really, I don't understand why Amazon doesn't fix this. It's happened to me 3-4 times in a relatively small installation. Surely their monitoring can detect this?
- dasil003 16y ago
  Is it because of the network throughput on the small instance type? I haven't noticed anything nearly that severe with our 20-instance setup of larger instance sizes, but then again we've only run production on EC2 for a few months now...
- moe 16y ago
  But really, I don't understand why Amazon doesn't fix this. When you select your instance-type then you get to choose between "Low", "Moderate", "High" and "Very High" I/O performance. Want to take a guess at which instance types suffer first when there is high I/O demand or contention?
  - recampbell 16y ago
    I can handle suffering, but completely freezing?
    - moe 16y ago
      That is likely not intended so I guess it's probably inherent to how Xen and/or their I/O layer do the resource allocation. I, too, have seen these freezes occasionally on small instances. If you think that is harsh then try one of the new micro's. ;-) Anyways, I haven't had such a freeze last longer than a minute yet, and the instance would always recover. If yours does not recover then that would clearly be a bug.

gyardley 16y ago

We had these issues as well. I'd certainly try AWS, but I'd wait a couple of months before shelling out the upfront cost for the Reserved Instances.

blantonl 16y ago
EBS definitely has performance limitations, however creating striped RAID arrays of EBS volumes significantly helps with performance. This blog post walks through a bunch of things you can do to make the best out of the flexibility you get with EBS, and includes scheduler tuning, RAID block size planning, etc. http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/ http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs... My team runs a large AWS/EBS backed implementation and, while we've seen some of the EBS performance issues, we've certainly engineered past them to get us where we need to be.
- chrisbolt 16y ago
  Wouldn't creating striped RAID arrays of EBS volumes add to the cost and remove the ability to do snapshots? How do you get a consistent snapshot of multiple striped EBS volumes?
  - spudlyo 16y ago
    It just makes it more complicated. You have lock and snap all of the volumes at once. The ec2-consistent-snapshot tool makes this easier to manage.
  - blantonl 16y ago
    EBS costs are pretty low. And my above post was directed to someone who is deploying a database on top of EBS. Sorry if I didn't make that clear. In our case, we replicate to set of non-RAID EBS instances where we pull snapshots.
  - imbriaco 16y ago
    Also keep in mind that creating a RAID0 of EBS volumes dramatically reduces your reliability too. If any one of the underlying EBS volumes misbehaves the entire RAID set has problems. Snapshots are still doable, they just require coordination of all the underlying devices to make sure they are all in a consistent state. The way you do this can vary but it's not particularly hard.

imbriaco 16y ago

As a heavy user of EBS, I heartily concur. The ability to move EBS volumes around is handy but you pay it back in inconsistent, poor-to-mediocre performance. Good luck.

recampbell 16y ago

I've also seen highly variable io performance (ie, slow/stuck) on EBS volumes. Sometimes, all reads/writes block for several second or minutes, with iowait going to 100% on an m1.small. example: http://developer.amazonwebservices.com/connect/message.jspa?messageID=201918#201918 http://developer.amazonwebservices.com/connect/message.jspa?... However, I've decided that the other benefits of EBS volumes are just to important to give up on (snapshotting, lazy-load, re-attachment). Instead, I plan to monitor for such situations and blow away the node when I detect it. But really, I don't understand why Amazon doesn't fix this. It's happened to me 3-4 times in a relatively small installation. Surely their monitoring can detect this?

dasil003 16y ago
Is it because of the network throughput on the small instance type? I haven't noticed anything nearly that severe with our 20-instance setup of larger instance sizes, but then again we've only run production on EC2 for a few months now...
moe 16y ago
But really, I don't understand why Amazon doesn't fix this. When you select your instance-type then you get to choose between "Low", "Moderate", "High" and "Very High" I/O performance. Want to take a guess at which instance types suffer first when there is high I/O demand or contention?
- recampbell 16y ago
  I can handle suffering, but completely freezing?
  - moe 16y ago
    That is likely not intended so I guess it's probably inherent to how Xen and/or their I/O layer do the resource allocation. I, too, have seen these freezes occasionally on small instances. If you think that is harsh then try one of the new micro's. ;-) Anyways, I haven't had such a freeze last longer than a minute yet, and the instance would always recover. If yours does not recover then that would clearly be a bug.