4 ms·

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fin

by 3np 6y ago

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota of the provisioned underlying EBS disks on the EC2 instances forming the ECS cluster had been depleted so disk performance completely tanked to the point the docker agent couldn’t be reached for 4 minutes. And here I thought the whole point was that these things would be abstracted away...

hashhar 6y ago
I've been bit by the same issue on ECS. Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough. And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message. Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.
- 3np 6y ago
  The icing on the cake here is that those IOPS from the docker agent is outside of your control. Before having to dive deeper into it, I would have assumed that only IOPS stemming from the workloads themselves would count against quotas. Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.
csnweb 6y ago
All these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.
- diveanon 6y ago
  Even more so when you factor in dev time.
andreareina 6y ago
We got bitten by this. Worst thing is that even with the help of an account specialist, we couldn't get a precise number for how much IOPS we needed, but had to look at the burst balance graph and calc^Wguesstimate the slope.

by 3np 6y ago

hashhar 6y ago

I've been bit by the same issue on ECS. Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough. And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message. Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.

3np 6y ago
The icing on the cake here is that those IOPS from the docker agent is outside of your control. Before having to dive deeper into it, I would have assumed that only IOPS stemming from the workloads themselves would count against quotas. Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.

csnweb 6y ago

All these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.

diveanon 6y ago
Even more so when you factor in dev time.

andreareina 6y ago

We got bitten by this. Worst thing is that even with the help of an account specialist, we couldn't get a precise number for how much IOPS we needed, but had to look at the burst balance graph and calc^Wguesstimate the slope.