6 ms·
I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a
by jiggawatts 6y ago
I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over.
My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular application stored only a few MB at the time, because it was brand new and hadn't accumulated anything yet. I had a very angry customer calling up asking me to please explain why they were seeing an average of 0.1 IOPS...
From: https://docs.aws.amazon.com/efs/latest/ug/performance.html https://docs.aws.amazon.com/efs/latest/ug/performance.html
"The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage).
AWS now provides a performance floor of 1 MiB/s, but at the time there was no floor. If I remember correctly, this application had something like 2 MIB of data, which was constantly being updated by various processes, so there was no quota being accumulated. The system performance went from something like 1 Gbps to 100 bytes per second instantly. It took 10 seconds for a 1 KiB I/O to complete. Fun times, fun times...
- crmd 6y agoThe linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.
- vlovich123 6y agoIt’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking it should work? Fixed cost per block transferred?
- freeone3000 6y agoFixed cost per IOPS allocated. Essentially, the same thing as before, but without the necessity of you storing large blank objects.
- vlovich123 6y agoHow would scheduling of concurrent I/O workloads work? If I’m paying some price per 1k IOPs and my service gets a spike, won’t I greedily take out the other services running on the same machine rather than getting throttled? Doesn’t this also penalize workloads that do lots of small I/Os rather than a few big ones even if the amount transferred is the same?
- freeone3000 6y ago1) By ensuring capacity for max allocated concurrent IOPS 2) Wrong relationship; you're paying for IOPS to the block store, so you'd be trampling on other accesses to the same block store. 3) This penalizes them less - in that workloads that do lots of small IO on small files will actually be able to request the IOPS they need, instead of IOPS being (wrongly) dynamically allocated.
- fuzzyengineer 6y agoCan't this be addressed by provisioned throughput on EFS?
- deleted 6y ago[deleted]
- pram 6y agoIt would make more sense to me to charge for the size of the provisioned share and performance profile, like EBS, instead of this weird performance tiering. It basically makes EFS useless for the bulk of use-cases I can personally imagine.
- 6y ago
- ahupp 6y agoI ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.
- jiggawatts 6y agoThe "pain to track down" is important because nothing failed. I didn't get an alert. There was nothing in the logs. There wasn't anything in the portal to indicate that something had changed. Everything was up and responding, just really, really slowly. The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time. After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on. Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.
- jhgg 6y agoTo be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
- jiggawatts 6y agoAhhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course. The golden rule of both backups and monitoring is: There are no time machines. It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once? Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured? That's not a rhetorical question! Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't. Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end. Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream! Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes? PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.
- pram 6y agoYep this bit me in the ass too. When I first used it I was expecting something like an NFS share from a NetApp or ZFS Filer. I see they offer provisioned iops now but still, why not offer a version without all the rigmarole?
- ineedasername 6y agoEven worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I kill the job, start again, and the problem is there from the very beginning. Kill again, review my code for an hour or two (python, grequest) tweak some parameters, start the job again and it seems fine, problem solved? Nope, halfway through the same issue occurs. More testing showed even that "halfway through" wasn't consistent. It's the hidden quota, and it wasn't even consistent: Sometimes it would go 10%, sometimes 70%. I don't know if they even had static limits or some type of dynamic system. Then you get throttled, only requests fail without any indicative error message except the "failed" or "unavailable" (I forget which) I determined the limit was tied to the API key. Given that error messages & the unknown limit made working within their hidden limits difficult, and support was neither sympathetic or forthcoming with details necessary to work within their limits, I simply created multiple API keys & rotated between them on requests. Probably not the best behavior for a tenant of a shared system, but I didn't see much choice when the alternative was to not use an essential advertised feature.
- dkarl 6y agoI had a similar experience with a SaaS. They let us run fine for months, then started degrading our service. When we contacted them, they tried to convince us we had a bug in our implementation. No, we confirmed we were using their service exactly as we intended to. They seemed to have designed their service with a particular use case in mind, and ours wasn't it. We tried multiple times to explain our use case and why their service was useful for us, because we didn't want them to think we were using the service in bad faith, but if they understood, they never let on. After much grumbling, they would lift the restrictions on us, but apparently not far, because we'd run into them again weeks or months later as our traffic increased. They were obviously unhappy with the way we were using their service and determined to cause issues for us, so we gave up and stopped using them.
- jiggawatts 6y ago
- tjbiddle 6y agoI remember having something similar years ago. Ended up just creating a 100GB file, or something, just to get the IOPS I needed until we could migrate off EFS.
- 3np 6y agoMy unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota of the provisioned underlying EBS disks on the EC2 instances forming the ECS cluster had been depleted so disk performance completely tanked to the point the docker agent couldn’t be reached for 4 minutes. And here I thought the whole point was that these things would be abstracted away...
- hashhar 6y agoI've been bit by the same issue on ECS. Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough. And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message. Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.
- 3np 6y agoThe icing on the cake here is that those IOPS from the docker agent is outside of your control. Before having to dive deeper into it, I would have assumed that only IOPS stemming from the workloads themselves would count against quotas. Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.
- csnweb 6y agoAll these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.
- mtsr 6y agoAnd this is why it’s not easy to use any of the cloud platforms parttime. They’re all full of these kinds of gotchas and your example is even a minor one since it really is quite clear from the documentation. There’s even a choice of Provisioned Throughput if you’re clicking this together in the Console, which should probably make you suspicious about what happens if you don’t use that. But a certified AWS Architect, especially at the higher levels will easily spot the big gotchas from just your architecture diagram. Consider getting certified yourself, if you want to be more confident around these in the future. But there really is no replacement for just putting in a lot of hours.
- bostonsre 6y agoDoes the certification cover all products? They seem to be releasing products more and more frequently.. it has to be tough to keep up to speed on everything to a degree where you don't need Google to explore options and plan projects.
- nix23 6y agoThis is why you should never use anyone else's server's for your software.
- jiggawatts 6y agoWhat value is the certification for the cloud? It's such an incredibly rapidly changing thing, anything you learn goes out-of-date in a matter of months. The two things I complained about in this thread are no longer issues, they got fixed. The project I'm working on right now is using a PaaS stack is a bit of a "moving target". I feel like those people running to catch a train that's still rolling along...
- waheoo 6y agoI've seen this same iops story play out 2 different times at two different companies both with the same story on production systems. Both were solved by the same thing, Provisioning a few terabytes of storage space with nothing on it.
- bogomipz 6y ago>""The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage)." Does the mean that baseline is only realized when at least a Tib of storage is actually being used then? In other words there was a distinction between how much you storage you were actually using vs how much storage you provisioned? Are there other services that use this same model?