3 ms·

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was qui

by ahupp 6y ago

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

jiggawatts 6y ago
The "pain to track down" is important because nothing failed. I didn't get an alert. There was nothing in the logs. There wasn't anything in the portal to indicate that something had changed. Everything was up and responding, just really, really slowly. The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time. After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on. Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.
- jhgg 6y ago
  To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
  - jiggawatts 6y ago
    Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course. The golden rule of both backups and monitoring is: There are no time machines. It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once? Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured? That's not a rhetorical question! Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't. Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end. Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream! Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes? PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.
    - bbarnett 6y ago
      I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better". Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS. And yes, that's with the hardware investment and wage costs rolled in. I feel like AWS and others created some sort of one ring, and just reeled admins in, pulling them from the wild, so that with a shortage of greybeards, people can't find those with the 'sysadmin temperament', and are therefore stuck with cloud. And it is indeed a specific temperament, to create stability, which means creating some constraint, yet to forge that constraint in the most effective way, whilst doing one's best to enable devs to work most effectively. AWS has constraints, but because it's a wall of "this is the way external corp does it", it gets far less flack than the guy that smells like onions, when he says "we can't do this safely" or "this policy must be followed". Yes, OK, this is a bit of a rant. Sorry.
      - jiggawatts 6y ago
        Hardware has the exact same monitoring issues, worse even. The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best. The cloud is eating their lunch for a reason. Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app. Everyone wants unified logging built-in, not Splunk on top of fifty different log formats, none of which can be easily correlated with each other. Everyone wants a unified IAM system, nobody likes to deal with expired SAML or LDAP certificates in the middle of a change. Etc, etc... Compared to having to deal with 10 different teams just to spin up a single VM and 15+ teams for a moderately complex application with HA/DR and monitoring, being able to simply click through a cloud portal is an absolute joy. It's the same thing that made VMware so popular. Nobody liked to have to get finance involved and order new kit six weeks ahead just to be able to spin up a tiny web server.
        bbarnett 6y ago
- dijit 6y ago
  FWIW these issues in ops are referred to as a “brown out”. Everything “works”, but at a huge reduced performance. Usually without alerts because nothing has completely failed.

by ahupp 6y ago

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

jiggawatts 6y ago

The "pain to track down" is important because nothing failed. I didn't get an alert. There was nothing in the logs. There wasn't anything in the portal to indicate that something had changed. Everything was up and responding, just really, really slowly. The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time. After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on. Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.

jhgg 6y ago
To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
- jiggawatts 6y ago
  Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course. The golden rule of both backups and monitoring is: There are no time machines. It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once? Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured? That's not a rhetorical question! Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't. Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end. Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream! Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes? PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.
  - bbarnett 6y ago
    I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better". Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS. And yes, that's with the hardware investment and wage costs rolled in. I feel like AWS and others created some sort of one ring, and just reeled admins in, pulling them from the wild, so that with a shortage of greybeards, people can't find those with the 'sysadmin temperament', and are therefore stuck with cloud. And it is indeed a specific temperament, to create stability, which means creating some constraint, yet to forge that constraint in the most effective way, whilst doing one's best to enable devs to work most effectively. AWS has constraints, but because it's a wall of "this is the way external corp does it", it gets far less flack than the guy that smells like onions, when he says "we can't do this safely" or "this policy must be followed". Yes, OK, this is a bit of a rant. Sorry.
    - jiggawatts 6y ago
      Hardware has the exact same monitoring issues, worse even. The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best. The cloud is eating their lunch for a reason. Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app. Everyone wants unified logging built-in, not Splunk on top of fifty different log formats, none of which can be easily correlated with each other. Everyone wants a unified IAM system, nobody likes to deal with expired SAML or LDAP certificates in the middle of a change. Etc, etc... Compared to having to deal with 10 different teams just to spin up a single VM and 15+ teams for a moderately complex application with HA/DR and monitoring, being able to simply click through a cloud portal is an absolute joy. It's the same thing that made VMware so popular. Nobody liked to have to get finance involved and order new kit six weeks ahead just to be able to spin up a tiny web server.
      - bbarnett 6y ago
dijit 6y ago
FWIW these issues in ops are referred to as a “brown out”. Everything “works”, but at a huge reduced performance. Usually without alerts because nothing has completely failed.