3 ms·

I think FAANG centric folks underestimate how many fairly big orgs are in the same situation. Big top down push to move to public cloud, basically no thought t

I think FAANG centric folks underestimate how many fairly big orgs are in the same situation. Big top down push to move to public cloud, basically no thought to operating expenses.. and now theres a ton of low hanging fruit. Most of these same firms have very little ability to measure their costs in a timely, granular manner, and so put in only very blunt processes to try to limit costs. Last org I was at could probably re-allocate 10% of engineering staff using the cloud to a SWAT team finding cloud cost saves for 2 years before they ran out of stuff that costs 5x their salary to get fix/optimize/kill in the cloud. A lot of the same orgs CTOs sold themselves on the idea that they'd need less infra people in the cloud. This really has not been the case, and more often the cloud infra people are more expensive too. Meanwhile many need to permanently maintain an on-prem footprint for latency/legal/legacy/etc reasons.. so you end up with net more staff. I know a handful of Wall St firms that are moving a lot of "big data" type use cases back out of the cloud because its hideously expensive for their use cases.

lazystar 4y ago
well, its getting into the rearview mirror now, but spectre/meltdown-type vulnerabilities are a big motivator for leaving on-prem. even if you later leave the cloud for on-prem, the period without any phys infra means you can start on a clean slate with better equipment + better DC rental contracts. oh and it really sucks when your brand new SSD's arrive at your main DC, after a 4 month wait, you open the box and the boxes are empty - someone robbed ya. Those expenses are only linked to the infra team, but the effects reverberate through the engineering org.
sylens 4y ago
I wonder if we will also see a dropoff in attendance for big tech conferences like reInvent this year.
- steveBK123 4y ago
  Given how many devs were needlessly getting sent at the shops I worked (think 20% of dev team), yes...
Melatonic 4y ago
I have been saying this for years - hybrid cloud has always been the way to go. Put your highly variable workloads in the cloud and autoscale. And anything super GPU or compute hungry that are regular workloads should be run yourself. Bonus points if you can autoscale those same on prem workloads into the cloud provider of your choice seamlessly
- zmmmmm 4y ago
  It's the way to go but the battle you have to fight is maintaining internal competency to host the infra. Especially if you have any kind of security or regulatory overlay its going to mean almost full duplication of a giant stack of processes that parallel each other in cloud / on prem. I think the key as you have outlined is to break out the simple batch compute and CPU/memory intensive or data volume heavy tasks as these are what generate the cloud bills. In general these aren't the focus of your policy constraints, especially if you can claim the data is not "at rest" within these systems.
  - steveBK123 4y ago
    It pains me to tell you this, but "simple batch compute and CPU/memory intensive or data volume heavy tasks " were the first ones my firm has, unsuccessfully, tried to move to the cloud, no joke.
- markus_zhang 4y ago
  From my experience I think maybe internal reporting fits self hosting. You don't need to scale too much, it can tolerate some fails, etc. But I guess it's still complicated when you throw a wrench called BI into the machinery.
  - steveBK123 4y ago
    The thing is that self hosting kind of already has cost gates. Whenever I needed to buy another $100K of servers, we had to make a deck and have a skip level meeting 2-3 levels up to explain the capacity planning spend. On the other hand, in AWS, at my last shop.. we were just .. doing stuff. We at one point noticed a non-production RDS instance was going to run us $60k/year. This was for some small part of our processing, that in on-prem land would just occupy a slice of one of our $30k servers. But in cloud, we had the pleasure of $5k/month until we noticed & turned it off. At the scale our estate worked, we had easily 100 entities that could trigger this kind of spend.

lazystar 4y ago

well, its getting into the rearview mirror now, but spectre/meltdown-type vulnerabilities are a big motivator for leaving on-prem. even if you later leave the cloud for on-prem, the period without any phys infra means you can start on a clean slate with better equipment + better DC rental contracts. oh and it really sucks when your brand new SSD's arrive at your main DC, after a 4 month wait, you open the box and the boxes are empty - someone robbed ya. Those expenses are only linked to the infra team, but the effects reverberate through the engineering org.

sylens 4y ago

I wonder if we will also see a dropoff in attendance for big tech conferences like reInvent this year.

steveBK123 4y ago
Given how many devs were needlessly getting sent at the shops I worked (think 20% of dev team), yes...

Melatonic 4y ago

I have been saying this for years - hybrid cloud has always been the way to go. Put your highly variable workloads in the cloud and autoscale. And anything super GPU or compute hungry that are regular workloads should be run yourself. Bonus points if you can autoscale those same on prem workloads into the cloud provider of your choice seamlessly

zmmmmm 4y ago
It's the way to go but the battle you have to fight is maintaining internal competency to host the infra. Especially if you have any kind of security or regulatory overlay its going to mean almost full duplication of a giant stack of processes that parallel each other in cloud / on prem. I think the key as you have outlined is to break out the simple batch compute and CPU/memory intensive or data volume heavy tasks as these are what generate the cloud bills. In general these aren't the focus of your policy constraints, especially if you can claim the data is not "at rest" within these systems.
- steveBK123 4y ago
  It pains me to tell you this, but "simple batch compute and CPU/memory intensive or data volume heavy tasks " were the first ones my firm has, unsuccessfully, tried to move to the cloud, no joke.
markus_zhang 4y ago
From my experience I think maybe internal reporting fits self hosting. You don't need to scale too much, it can tolerate some fails, etc. But I guess it's still complicated when you throw a wrench called BI into the machinery.
- steveBK123 4y ago
  The thing is that self hosting kind of already has cost gates. Whenever I needed to buy another $100K of servers, we had to make a deck and have a skip level meeting 2-3 levels up to explain the capacity planning spend. On the other hand, in AWS, at my last shop.. we were just .. doing stuff. We at one point noticed a non-production RDS instance was going to run us $60k/year. This was for some small part of our processing, that in on-prem land would just occupy a slice of one of our $30k servers. But in cloud, we had the pleasure of $5k/month until we noticed & turned it off. At the scale our estate worked, we had easily 100 entities that could trigger this kind of spend.