4 ms·

Yeah it's history repeating. I've said it before on here and I'll say it again I don't think running on AWS will ever be cheaper than DIY. It's kind of a fallac

Yeah it's history repeating. I've said it before on here and I'll say it again I don't think running on AWS will ever be cheaper than DIY. It's kind of a fallacy and a sales tactic by AWS to tell you how the elasticity of their offering allows you to "save money during the quite hours". I don't think I've ever seen a system that actually does that.

acdha 5y ago
The savings usually come from the corners being cut on DIY solutions: not accounting for the time spent on infrastructure services, skipping various security and monitoring features, sacrificing reliability, etc. You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.
- treis 5y ago
  The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.
  - acdha 5y ago
    > And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
    - treis 5y ago
      I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
      - fauigerzigerk 5y ago
        I think companies like Cloudflare and fly.io are trying to tackle some of those multi-datacenter complexity issues you're describing. Those ridiculous egress fees appear to have peaked as well since Cloudflare announced their zero egress S3 competitor. I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable. I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.
      - acdha 5y ago
        > Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES). Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions. > That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing. Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.
- Dave3of5 5y ago
  Cutting corners assumes not doing things that are necessary. What I've found is that systems in the cloud are often unnecessarily resilient in areas which they do not need to be. The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3. I've also found that just because you're paying over the odds for AWS's SLA doesn't mean jack to your systems actual uptime. All systems I've dealt with in AWS / Azure have had unplanned outages. I think almost all of these have been problems in the software itself nothing that AWS will help you with.
  - acdha 5y ago
    > Cutting corners assumes not doing things that are necessary. … The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3. … or which didn't seem necessary until something breaks. For example, the cost of using S3 for temporary files is often negative because in most cases your usage charges are going to be lower than keeping a sufficiently-large storage volume provisioned 24x7 for it, and if you know that you don't need geographic redundancy you can use the one zone storage for that. In all cases, you'll benefit from things like bitrot protection which that scratch storage space doesn't have built-in. > I think almost all of these have been problems in the software itself nothing that AWS will help you with. Definitely. What you get from a cloud environment is better tools for dealing with this — for example, shifting load is easy and making it trivial to get new resources makes it easy to do things like high-fidelity testing, blue-green deployments, rolling reboots, etc. which most organizations struggle to do efficiently in a traditional data center environment.
  - 8note 5y ago
    Optimistically, those unplanned outages are software errors, instead of data center management errors. There is an impact to your uptime by using AWS, but it's relative to having the same thing runing on not-AWS rather than an absolute value of uptime. The scope of what you can screw up is smaller

acdha 5y ago

The savings usually come from the corners being cut on DIY solutions: not accounting for the time spent on infrastructure services, skipping various security and monitoring features, sacrificing reliability, etc. You can definitely beat it with certain workloads (e.g. high-volume egress) but I’ve seen more cases where someone thought they had beaten it but was really comparing unlike things (e.g. comparing S3 to unreliable storage in a single location) or leaving out operator time.

treis 5y ago
The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.
- acdha 5y ago
  > And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
  - treis 5y ago
    I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
    - fauigerzigerk 5y ago
      I think companies like Cloudflare and fly.io are trying to tackle some of those multi-datacenter complexity issues you're describing. Those ridiculous egress fees appear to have peaked as well since Cloudflare announced their zero egress S3 competitor. I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable. I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.
    - acdha 5y ago
      > Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES). Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions. > That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing. Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.
Dave3of5 5y ago
Cutting corners assumes not doing things that are necessary. What I've found is that systems in the cloud are often unnecessarily resilient in areas which they do not need to be. The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3. I've also found that just because you're paying over the odds for AWS's SLA doesn't mean jack to your systems actual uptime. All systems I've dealt with in AWS / Azure have had unplanned outages. I think almost all of these have been problems in the software itself nothing that AWS will help you with.
- acdha 5y ago
  > Cutting corners assumes not doing things that are necessary. … The example of S3 is apt as often ephemeral files are kept in S3 in which case they don't need the reliability storage guarantees that you'll get with S3. … or which didn't seem necessary until something breaks. For example, the cost of using S3 for temporary files is often negative because in most cases your usage charges are going to be lower than keeping a sufficiently-large storage volume provisioned 24x7 for it, and if you know that you don't need geographic redundancy you can use the one zone storage for that. In all cases, you'll benefit from things like bitrot protection which that scratch storage space doesn't have built-in. > I think almost all of these have been problems in the software itself nothing that AWS will help you with. Definitely. What you get from a cloud environment is better tools for dealing with this — for example, shifting load is easy and making it trivial to get new resources makes it easy to do things like high-fidelity testing, blue-green deployments, rolling reboots, etc. which most organizations struggle to do efficiently in a traditional data center environment.
- 8note 5y ago
  Optimistically, those unplanned outages are software errors, instead of data center management errors. There is an impact to your uptime by using AWS, but it's relative to having the same thing runing on not-AWS rather than an absolute value of uptime. The scope of what you can screw up is smaller