4 ms·

+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. I

by harikb 6y ago

+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle of night but we wasted millions storing useless data Azure also has similar issues - deleting a VM sometimes doesn’t cleanup dependent resources and it is a mess to find and delete later - only because the dependent resources are deliberately not named with a matching tag.

threentaway 6y ago
This seems like you didn't have proper monitoring and alerting set up for your job, not sure how that is a downside of AWS.
- jjoonathan 6y ago
  AWS monitoring (and billing) is garbage because they make an extraordinary amount of money on unintentional spend. "But look at how many monitoring solutions they have in the dashboard! Why, just last re:invent they announced 20 new monitoring features!" They make a big fuss and show about improving monitoring but it's always crippled in some way that makes it easy to get wrong and time-consuming or expensive to get right.
  - catlifeonmars 6y ago
    I’m genuinely curious. Which parts of monitoring is crippled or difficult to use? Disclaimer: I work at AWS.
- milesvp 6y ago
  > Infinite scalability is also a curse This was the key sentence, I think. This type of problem actually shows up in other domains as well, queueing theory comes immediately to mind. Even the halting problem is only a problem with infinite tape, and becomes easier with (known?) limited resources. When you have some parameter that is unbounded you need to add extra checks to bound them yourself to some sane value. You are right, in that the parent failed to monitor some infrastructure, but if they were in their own datacenter, once they filled their NAS, I’m positive someone would have noticed, if only because other checks, like diskspace are less likely to be forgotten. Also, getting a huge surprise bill is a downside of any option, and the risk needs to be factored into the cost. I’m constantly paranoid when working in a cloud environment, even doing something as trivial as a directory listing from the command line on S3 costs money. I had a back and forths with AWS support just to be clear what the order of magnitude of the bill would be for a simple cleanup action since there were 2 documented ways to do what I needed, and one appeared to be easier, yet significantly more expensive.
zmmmmm 6y ago
> Infinite scalability is also a curse People don't like to admit it, but in many circumstances, having a service that is escalating to 10x or 100x its normal demand go off line is probably the desirable thing.
- Silhouette 6y ago
  There seem to be plenty of successful online businesses that share some information about their back end infrastructure, and when you look it turns out they just have a handful of web servers running their app behind a load balancer, enough DB servers for the throughput and redundancy they need and some sort of failover arrangements in case something dies, and some sort of cache and/or CDN arrangements just for efficiency. Even if they're running in the cloud, it's probably just a load of manually allocated EC2 instances, maybe RDS to save the hassle of managing the database manually, and maybe S3. I wonder how many businesses exist in the entire world that truly need to scale their server counts up or down by a factor of several so quickly that it has to be done automatically and not as a result of, say, daily human reviews of what's happening. I feel like even large businesses that need to run thousands of servers at all are probably relatively rare, as indeed large businesses themselves are. The number that might realistically need to scale from say 1,000 web servers to 1,500 within a matter of hours must be even rarer. But I have nothing even resembling a useful data source to tell whether this intuition is correct.

by harikb 6y ago

threentaway 6y ago

This seems like you didn't have proper monitoring and alerting set up for your job, not sure how that is a downside of AWS.

jjoonathan 6y ago
AWS monitoring (and billing) is garbage because they make an extraordinary amount of money on unintentional spend. "But look at how many monitoring solutions they have in the dashboard! Why, just last re:invent they announced 20 new monitoring features!" They make a big fuss and show about improving monitoring but it's always crippled in some way that makes it easy to get wrong and time-consuming or expensive to get right.
- catlifeonmars 6y ago
  I’m genuinely curious. Which parts of monitoring is crippled or difficult to use? Disclaimer: I work at AWS.
milesvp 6y ago
> Infinite scalability is also a curse This was the key sentence, I think. This type of problem actually shows up in other domains as well, queueing theory comes immediately to mind. Even the halting problem is only a problem with infinite tape, and becomes easier with (known?) limited resources. When you have some parameter that is unbounded you need to add extra checks to bound them yourself to some sane value. You are right, in that the parent failed to monitor some infrastructure, but if they were in their own datacenter, once they filled their NAS, I’m positive someone would have noticed, if only because other checks, like diskspace are less likely to be forgotten. Also, getting a huge surprise bill is a downside of any option, and the risk needs to be factored into the cost. I’m constantly paranoid when working in a cloud environment, even doing something as trivial as a directory listing from the command line on S3 costs money. I had a back and forths with AWS support just to be clear what the order of magnitude of the bill would be for a simple cleanup action since there were 2 documented ways to do what I needed, and one appeared to be easier, yet significantly more expensive.

zmmmmm 6y ago

> Infinite scalability is also a curse People don't like to admit it, but in many circumstances, having a service that is escalating to 10x or 100x its normal demand go off line is probably the desirable thing.

Silhouette 6y ago
There seem to be plenty of successful online businesses that share some information about their back end infrastructure, and when you look it turns out they just have a handful of web servers running their app behind a load balancer, enough DB servers for the throughput and redundancy they need and some sort of failover arrangements in case something dies, and some sort of cache and/or CDN arrangements just for efficiency. Even if they're running in the cloud, it's probably just a load of manually allocated EC2 instances, maybe RDS to save the hassle of managing the database manually, and maybe S3. I wonder how many businesses exist in the entire world that truly need to scale their server counts up or down by a factor of several so quickly that it has to be done automatically and not as a result of, say, daily human reviews of what's happening. I feel like even large businesses that need to run thousands of servers at all are probably relatively rare, as indeed large businesses themselves are. The number that might realistically need to scale from say 1,000 web servers to 1,500 within a matter of hours must be even rarer. But I have nothing even resembling a useful data source to tell whether this intuition is correct.