4 ms·

The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle

by treis 5y ago

The counter to this is that companies rarely cut ops staff when moving to the cloud. The problem is that you still have security and monitoring knobs to fiddle with in a cloud. And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems.

acdha 5y ago
> And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”
- treis 5y ago
  I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
  - fauigerzigerk 5y ago
    I think companies like Cloudflare and fly.io are trying to tackle some of those multi-datacenter complexity issues you're describing. Those ridiculous egress fees appear to have peaked as well since Cloudflare announced their zero egress S3 competitor. I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable. I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.
  - acdha 5y ago
    > Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES). Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions. > That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing. Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.

by treis 5y ago

acdha 5y ago

> And the staff you had is worse at fiddling with these knobs than the old knobs that they knew well. Plus, you've now added a whole new category of cost optimization problems. More typically, it makes those problems visible — you get a bill from AWS every month in a way which you don't get a report that 70% of the hardware in your data center is either under-utilized or being used inefficiently. I think cloud migrations often suffer from that as a source of attribution error: “OMG, we don't have perfect security in the cloud!” “Aren't you running a production app on RHEL 6 under the account of someone who left 8 years ago?” “yeah, but we know about that!”

treis 5y ago
I'm talking about things like exorbitant egress fees changing how you have to architect your systems. Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. It also caused much more operational complexity. Being in 10 times as many regions meant that we got hit with data center issues 10 times as often. Similarly, we had to run 10x the instances instead of going with fewer bigger ones. Ultimately that means we got 10x the issues caused by random things like disk failures. Our alerting got much harder because the volume was smaller since it was split into 10 parts. The smaller the baseline is the harder it is to detect problematic variations. We frequently got paged because some customer in a small region did something unusual in the middle of the night or someone tried an outgoing spam attack. That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing.
- fauigerzigerk 5y ago
  I think companies like Cloudflare and fly.io are trying to tackle some of those multi-datacenter complexity issues you're describing. Those ridiculous egress fees appear to have peaked as well since Cloudflare announced their zero egress S3 competitor. I believe the way forward is not going back to self managed infrastructure. It looks like there will be sufficient competition to limit the most egregious price gauging. The way to avoid lock-in is to build on APIs that are at least somewhat portable. I know the portability devil is very much in the non functional details. But I think we'll see more of what happened with S3. Everybody will be adopting the more popular APIs. Over time we will converge on some new POSIX-like industry standard for the cloud.
- acdha 5y ago
  > Like at a previous job we had e-mail servers running in every AWS region our product ran partly because of egress fees. If we had run them in 1 (or a couple) regions we could have used a lot fewer machines due to demand smoothing and sharing things like Redis clusters. Were you either sending a huge amount of email or was this something related to the internal traffic needed to generate each message? I've definitely seen this in other contexts but am impressed to see email driving that (and, of course, preventing use of SES). Note that I said “more typical” — I would define “cutting corners” as something like having a production service running with only one instance but once I go greater than one I'd be investing in automation anyway for reliability, and that would have included things like health checks & automatic rebuilds. It sounds like your business was pretty aggressive in region usage and I'd be curious what was driving that many regions. > That's one example but there's a whole category of "avoid getting extorted" problems that arise because of AWS's predatory pricing. Don't take anything I'm saying as defense of their egress pricing. My point was just that cloud pricing is very visible and that can lead many places to focus on that without accurately assessing what they're currently paying for their own infrastructure.