6 ms·

Eh, you're pulling a quote out of context. It's 27% reduction in EC2 use, which was only 18.5% total. So this only accounted for ~5% of total savings.

dannyw 6y ago
I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.
- brianwawok 6y ago
  More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.
  - billyhoffman 6y ago
    And doing this for months. Without noticing. Honestly this isn’t ultimately engineerings fault. This is a SaaS business Someone in their company is responsible for the COGS KPI. For that person to either not notice an increase in COGS, or to not be aggressively incentivizing engineering to reduce COGS, is giant red flag.
    - zaphirplane 6y ago
      Ouch this thread ruined some poor souls promotion / pay rise narrative
    - jinpan 6y ago
      The article did say that the motivation for doing so was because AWS credits were running out .. why prematurely optimize a free resource? :)
  - rlayton2 6y ago
    Perhaps more like ordering 100 lunches for your 100 employees and forgetting that 20 of them moved to night shift?
  - sankalp221 6y ago
    Fun fact, 30-40% of the food in our supply chain is wasted. So, we're actually not too far from that. Sure, the meals aren't directly wasted by the company and rather through the supply chain. But, the waste is still there and is surprisingly high IMO. Source: https://www.usda.gov/foodwaste/faqs https://www.usda.gov/foodwaste/faqs
- redis_mlc 6y ago
  > The whole point of AWS is to use services on demand That's a decade-old misconception about how people actually use AWS. Most servers I've seen in AWS are permanent. In fact, it's an anti-pattern to wait until you need more capacity to scale up, since those servers may not be available, especially in newer instance families. Even if the needed instances available, typically ASGs don't react the way you expect without a lot of experimentation (ie. outages.) An example is if traffic increases load, your health check may consider the servers to be unhealthy and start killing them, creating a death spiral.
  - tilolebo 6y ago
    This is why service health endpoints should be very carefully designed, with the proper balance between health inspection and performance penalty. That being said, it's true that an ALB doesn't offer throttling capability like a true reverse proxy such as HAProxy provides, were you can cap the number of concurrent requests and give a chance to your backend to avoid death by overload. I wish there would be a way for ASGs to at least make the distinction between an unhealthy instance and an overloaded one.
  - glogla 6y ago
    I think there's different levels of "on demand". In this case, I think this is much slower level of "on demand". I see two primary use cases of cloud: 1) You're a startup or just need something small, and want to focus on building your MVP, instead of messing around with colocated Linux servers. Cloud is much more expensive than those, but you don't care because you don't really need all that much, or maybe you are VC backed and have unlimited money. 2) You're a large company with broken internal processes. You can get server in company datacenter in three months after seven approvals (since it's capex), or you can spin up an EC2 instance. You don't care about cost since you have unlimited money. Those are kind of medium scale "on demand" - not "I need 100 new servers right this minute" but "I need server in ten minutes intead of 'when I get to buy one' or 'in three months and 37 forms'". In both cases, you're throwing money away, because time is more important for you than the extra money cloud costs.
- tjbiddle 6y ago
  I don't disagree with it that at all. By all means, we should be doing our best not to waste resources. Just stating that the waste was only 5% of the total savings.

dannyw 6y ago

I mean if a quarter of your EC2 instances were unused, that is absolutely an internal devops / IT mess. The whole point of AWS is to use services on demand; it's like buying 133 conference tickets for your 100 person company.

brianwawok 6y ago
More like ordering 133 lunches every day for your 100 employees and dumping 33 in the trash.
- billyhoffman 6y ago
  And doing this for months. Without noticing. Honestly this isn’t ultimately engineerings fault. This is a SaaS business Someone in their company is responsible for the COGS KPI. For that person to either not notice an increase in COGS, or to not be aggressively incentivizing engineering to reduce COGS, is giant red flag.
  - zaphirplane 6y ago
    Ouch this thread ruined some poor souls promotion / pay rise narrative
  - jinpan 6y ago
    The article did say that the motivation for doing so was because AWS credits were running out .. why prematurely optimize a free resource? :)
- rlayton2 6y ago
  Perhaps more like ordering 100 lunches for your 100 employees and forgetting that 20 of them moved to night shift?
- sankalp221 6y ago
  Fun fact, 30-40% of the food in our supply chain is wasted. So, we're actually not too far from that. Sure, the meals aren't directly wasted by the company and rather through the supply chain. But, the waste is still there and is surprisingly high IMO. Source: https://www.usda.gov/foodwaste/faqs https://www.usda.gov/foodwaste/faqs
redis_mlc 6y ago
> The whole point of AWS is to use services on demand That's a decade-old misconception about how people actually use AWS. Most servers I've seen in AWS are permanent. In fact, it's an anti-pattern to wait until you need more capacity to scale up, since those servers may not be available, especially in newer instance families. Even if the needed instances available, typically ASGs don't react the way you expect without a lot of experimentation (ie. outages.) An example is if traffic increases load, your health check may consider the servers to be unhealthy and start killing them, creating a death spiral.
- tilolebo 6y ago
  This is why service health endpoints should be very carefully designed, with the proper balance between health inspection and performance penalty. That being said, it's true that an ALB doesn't offer throttling capability like a true reverse proxy such as HAProxy provides, were you can cap the number of concurrent requests and give a chance to your backend to avoid death by overload. I wish there would be a way for ASGs to at least make the distinction between an unhealthy instance and an overloaded one.
- glogla 6y ago
  I think there's different levels of "on demand". In this case, I think this is much slower level of "on demand". I see two primary use cases of cloud: 1) You're a startup or just need something small, and want to focus on building your MVP, instead of messing around with colocated Linux servers. Cloud is much more expensive than those, but you don't care because you don't really need all that much, or maybe you are VC backed and have unlimited money. 2) You're a large company with broken internal processes. You can get server in company datacenter in three months after seven approvals (since it's capex), or you can spin up an EC2 instance. You don't care about cost since you have unlimited money. Those are kind of medium scale "on demand" - not "I need 100 new servers right this minute" but "I need server in ten minutes intead of 'when I get to buy one' or 'in three months and 37 forms'". In both cases, you're throwing money away, because time is more important for you than the extra money cloud costs.
tjbiddle 6y ago
I don't disagree with it that at all. By all means, we should be doing our best not to waste resources. Just stating that the waste was only 5% of the total savings.