4 ms·

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen. I

tldr: AWS EC2 has the concept of I/O credits for storage. If your instance runs out of credits, bad things, which may seem completely unrelated, will happen. I was having similar issues last week and did not consider I/O credits. I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm).

inferiorhuman 9y ago
Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive. The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.
- shusson 9y ago
  > EBS volumes are lazy loaded TIL, that probably explains other weird performance issues I've seen. further info: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initi...
- m00x 9y ago
  Should you ever assume IO to be deterministic? Hard drives can crash, they can be slow(especially magnetic). Networks can disconnect, become congested or have random interference. This is amplified on the cloud, but it should also be well known by anyone that works with the cloud. These things are well documented, although I could see why you don't really look for it until you have issues with it.
  - falcolas 9y ago
    I think the problem is that from your average application developer's point view - there's nothing that they can really do that they aren't already doing (unless they're blocking the main thread on disk IO, in which case they can do a little bit more).
- viraptor 9y ago
  Can't find it in the docs now, but the certification course I'm doing now claims the lazy loading is not happening anymore. Apparently prewarming volumes is not necessary these days unless you created the volume from a snapshot.
krallja 9y ago
> I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm). How would you configure these notifications? Who do they get sent to? How often? Do they escalate to someone else? When? Once you answer these questions, you have come up with ... CloudWatch.
- falcolas 9y ago
  Surfacing the metric on the host itself, and not buried in the EBS volume, would be a good start. WRT alerting - if you're at 0 credits; your performance on something obviously under load is now being degraded, why not surface an alert on it? AWS already has a configurable events system in place for EC2 - this seems like a good fit.

inferiorhuman 9y ago

Rather EBS performance is non-deterministic. This is definitely not new, but is an informative deep dive. The other big gotcha is that EBS volumes are lazy loaded. Not necessarily something that's going to bite you in a production environment regularly, but it's something that could very easily throw off your benchmarks and performance testing.

shusson 9y ago
> EBS volumes are lazy loaded TIL, that probably explains other weird performance issues I've seen. further info: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initi...
m00x 9y ago
Should you ever assume IO to be deterministic? Hard drives can crash, they can be slow(especially magnetic). Networks can disconnect, become congested or have random interference. This is amplified on the cloud, but it should also be well known by anyone that works with the cloud. These things are well documented, although I could see why you don't really look for it until you have issues with it.
- falcolas 9y ago
  I think the problem is that from your average application developer's point view - there's nothing that they can really do that they aren't already doing (unless they're blocking the main thread on disk IO, in which case they can do a little bit more).
viraptor 9y ago
Can't find it in the docs now, but the certification course I'm doing now claims the lazy loading is not happening anymore. Apparently prewarming volumes is not necessary these days unless you created the volume from a snapshot.

krallja 9y ago

> I think AWS could do better at notifying you if your EC2 instance gets into this state (without having to set up a cloud watch alarm). How would you configure these notifications? Who do they get sent to? How often? Do they escalate to someone else? When? Once you answer these questions, you have come up with ... CloudWatch.

falcolas 9y ago
Surfacing the metric on the host itself, and not buried in the EBS volume, would be a good start. WRT alerting - if you're at 0 credits; your performance on something obviously under load is now being degraded, why not surface an alert on it? AWS already has a configurable events system in place for EC2 - this seems like a good fit.