5 ms·

Well, they aren't wrong about the bare metal either: Every organization ends up tied to their staff, and said staff was hired to work on the stack you are using

Well, they aren't wrong about the bare metal either: Every organization ends up tied to their staff, and said staff was hired to work on the stack you are using. People end up in quite the fights because their supposed experts are more fond of uniformity and learning nothing new. Many a company was stuck with a datacenter unit that was unresponsive to the company's needs, and people migrated to AWS to avoid dealing with them. This straight out happened in front of my eyes multiple times. At the same time, you also end up in AWS, or even within AWS, using tools that are extremely expensive, because the cost-benefit analysis for the individuals making the decision, who often don't know very much other than what they use right now, are just wrong for the company. The executive on top is often either not much of a technologist or 20 years out of date, so they have no way to discern the quality of their staff. Technical disagreements? They might only know who they like to hang out with, but that's where it ends. So for path dependent reasons, companies end up making a lot of decisions that in retrospect seem very poor. In startups if often just kills the company. Just don't assume the error is always in one direction.

baq 1y ago
> Many a company was stuck with a datacenter unit that was unresponsive to the company's needs I'd like to +1 here - it's an understated risk if you've got datacenter-scale workloads. But! You can host a lot of compute on a couple racks nowadays, so IMHO it's a problem only if you're too successful and get complacent. In the datacenter, creative destruction is a must and crucially finance must be made to understand this, or they'll give you budget targets which can only mean ossification.
- alemanek 1y ago
  In orgs I have seen this it is usually a symptom of the data center unit being starved of resources. It’s like they have only been given the choice of on prem but ridiculous paperwork and long lead times or pay 20x for cloud. Like can’t we just give the data center org more money and they can over provision hardware. Or can we not have them use that extra money to rent servers from OVH/Hetzner during the discovery phase to keep things going while we are waiting on things to get sized or arrive?
  - dumbledoren 1y ago
    > Or can we not have them use that extra money to rent servers from OVH/Hetzner Or just use Hetzner for major performance at low cost... Their apis and stuff make it look like its your datacenter.
  - LPisGood 1y ago
    The problem is if you over-provision and buy 2x as many resources as you need, this looks bad from a utilization standpoint. If you buy 2x as expensive cloud solutions and “auto scale” you will have a much higher utilization for the same coat.
  - Aeolun 1y ago
    I feel like companies are unreasonably afraid of cost up front, never mind that they’re going to pay more for cloud over the next 6 months, spending 6x monthly cloud cost on a single server makes them hesitate. It’s how they always refuse to spend half my monthly salary on the computer I work on, and instead insist I use an underpowered windows machine.
    - deleted 1y ago
      [deleted]
    - jrs235 1y ago
      Blame finance and accounting... Rent compute in the cloud can be immediately expensed against revenues. Purchasing equipment has to be depreciated over a few years. Also why spending $$$$$ on labor (salaries) to solve an ops issue rather than spending $$$$ on some software to do it happens. If the business relies on the software it looks like an ever ongoing cost of operating the business. Spending more on labor to juggle the craziness can "hide" that and make the business look more attractive to investors... Cutting labor costs is easier to improve the bottom line (in the short term).
    - vidarh 1y ago
      You also don't need to commit to upfront costs. You can easily rent, rent to own/lease these resources.
vidarh 1y ago
It's simple enough to hire people with experience with both, or pay someone else to do it for you. These skills aren't that hard to find. If you hire people that are not responsive to your needs, then, sure, that is a problem that will be a problem irrespective of what their pet stack is.
whstl 1y ago
Sure but I have seen the exact same thing happen with AWS. In a large company I worked the Ops team that had the keys to AWS was taking literal months to push things to the cloud, causing problems with bonuses and promotions. Security measures were not in place so there were cyberattacks. Passwords of critical services lapsed because they were not paying attention. At some point it got so bad that the entire team was demoted, lost privileges, and contractors had to jump in. The CTO was almost fired. It took months to recover and even to get to an acceptable state, because nothing was really documented.
- LPisGood 1y ago
  I can’t believe the CTO wasn’t fired for that.
  - whstl 1y ago
    The CTO was the one holding the bonus and promotions for tech, so he just shifted the blame down when it was "investigated". On the other hand it's not hard to believe that the CEO and the board are as sleepy as the CTO here. And the whole management team. The worst one was when a password for an integration with the judicial system expired. They asked the DevOps to open their email and there were daily alerts for six months. The only reason they found this happened was because a few low level operators made a big thing out of it. I don't like talking about "regulatory capture" but this is the only reason this company still exists. Easy market when there's almost no competition.
Edman274 1y ago
The entire value proposition of AWS vs running one's own server is basically this: is it easier to ask for permission, or forgiveness? You're asking for permission to get a million dollars worth of servers / hardware / power upgrades now, or you're asking for forgiveness for spending five million dollars in AWS after 10 months. Which will be easy: permission or forgiveness?
- chasemp 1y ago
  I had not thought of it this way, but interesting point. I have seen this as well.
embedding-shape 1y ago
> said staff was hired to work on the stack you are using Looking back at doing various hiring decisions at various levels of organizations, this is probably the single biggest mistake I've done multiple times, hiring specific people using specific technology because we were specifically using that. You'll end up with a team unwilling to change, because "you hired me for this, even if it's best for the business with something else, this is what I do". Once I and the organizations shifted our mindset to hiring people who are more flexible, even if they have expertise in one or two specific technologies, they won't put their head in the sand whenever changes come up, and everything became a lot easier.
- vidarh 1y ago
  Exactly. If someone has "Cloud Engineer" in the headline of their resume instead of "Devops Engineer" it's already warning and worth probing. If someone has "AWS|VMWare Engineer" in their bio, it's a giant red flag to me. Sometimes it's people just being aware where they'll find demand, but often it's indicative of someone who will push their pet stack - and it doesn't matter if it's VMWare on-prem or AWS (both purely as examples; it doesn't matter which specific tech it is), it's equally bad if they identify with a specific stack irrespective of what the stack is. I'll also tend to look closely at whether people have "gotten stuck" specialising in a single stack. It won't make me turn them down, but it will make me ask extra questions to determine how open they are to alternatives when suitable.
infecto 1y ago
Your comment also jogged my memory of how terrible bare metal days used to be. I think now with containers it can be better but the other reason so many switched to cloud is we don’t need to think about buying the bare metal ahead of time. We don’t need to justify it to a DevOps gatekeeper.
- vidarh 1y ago
  That so many people remember bare metal as of 20+ years ago is a large part of the problem. A modern server can be power cycled remotely, can be reinstalled remotely over networked media, can have its console streamed remotely, can have fans etc. checked remotely without access to the OS it's running etc. It's not very different from managing a cloud - any reasonable server hardware has management boards. Even if you rent space in a colo, most of the time you don't need to set foot there other than for an initial setup (and you can rent people to do that too). But for most people, bare metal will tend to mean renting bare metal servers already configured anyway. When the first thing you then tend to do is to deploy a container runtime and an orchestrator, you're effectively usually left with something more or less (depending on your needs) like a private cloud. As for "buying ahead of time", most managed server providers and some colo operators also offer cloud services, so that even if you don't want to deal with a multi-provider setup, you can still generally scale into cloud instances as needed if your provider can't bring new hardware up fast enough (but many managed server providers can do that in less than a day too). I never think about buying ahead of time. It hasn't been a thing I've had to worry about for a decade or more.
  - infecto 1y ago
    You are right but I just think people miss the history when we talk about moving to the cloud. It was not that long ago at a reasonable size Bay Area company, I would need to justify new metal to be provisioned to standup a service I was tasked with.
  - mmarq 1y ago
    > A modern server can be power cycled remotely, can be reinstalled remotely over networked media, can have its console streamed remotely, can have fans etc. checked remotely without access to the OS it's running etc. It's not very different from managing a cloud - any reasonable server hardware has management boards. Even if you rent space in a colo, most of the time you don't need to set foot there other than for an initial setup (and you can rent people to do that too). All of this was already possible 20 years ago, with iLO and DRAC cards.
YouAreWRONGtoo 1y ago
[dead]
hshdhdhj4444 1y ago
Considering the rapid shift from on Prem to the cloud I think it’s clearly false that the people who knew on Prem were fighting for their little area of expertise. In my experience, the ops folks were absolutely thrilled with the arrival of the cloud because with a trivial amount of training and a couple of certifications they had a pathway to get paid as much, if not more, than devs, especially if they rebranded as “devops engineers” instead of “ops guys”. The only pushback against the cloud, other than some of us engineers who actually were among the first to jump on the cloud, still really loved it, but also recognized that it wasn’t the best fit for all uses and carried significant risks, were people worried about data safety. The latter concern has largely turned out to not be a real one yet, but a decade and a half later people are finally realizing that actually there are many areas where the cloud may not be the best fit.

baq 1y ago

> Many a company was stuck with a datacenter unit that was unresponsive to the company's needs I'd like to +1 here - it's an understated risk if you've got datacenter-scale workloads. But! You can host a lot of compute on a couple racks nowadays, so IMHO it's a problem only if you're too successful and get complacent. In the datacenter, creative destruction is a must and crucially finance must be made to understand this, or they'll give you budget targets which can only mean ossification.

alemanek 1y ago
In orgs I have seen this it is usually a symptom of the data center unit being starved of resources. It’s like they have only been given the choice of on prem but ridiculous paperwork and long lead times or pay 20x for cloud. Like can’t we just give the data center org more money and they can over provision hardware. Or can we not have them use that extra money to rent servers from OVH/Hetzner during the discovery phase to keep things going while we are waiting on things to get sized or arrive?
- dumbledoren 1y ago
  > Or can we not have them use that extra money to rent servers from OVH/Hetzner Or just use Hetzner for major performance at low cost... Their apis and stuff make it look like its your datacenter.
- LPisGood 1y ago
  The problem is if you over-provision and buy 2x as many resources as you need, this looks bad from a utilization standpoint. If you buy 2x as expensive cloud solutions and “auto scale” you will have a much higher utilization for the same coat.
- Aeolun 1y ago
  I feel like companies are unreasonably afraid of cost up front, never mind that they’re going to pay more for cloud over the next 6 months, spending 6x monthly cloud cost on a single server makes them hesitate. It’s how they always refuse to spend half my monthly salary on the computer I work on, and instead insist I use an underpowered windows machine.
  - deleted 1y ago
    [deleted]
  - jrs235 1y ago
    Blame finance and accounting... Rent compute in the cloud can be immediately expensed against revenues. Purchasing equipment has to be depreciated over a few years. Also why spending $$$$$ on labor (salaries) to solve an ops issue rather than spending $$$$ on some software to do it happens. If the business relies on the software it looks like an ever ongoing cost of operating the business. Spending more on labor to juggle the craziness can "hide" that and make the business look more attractive to investors... Cutting labor costs is easier to improve the bottom line (in the short term).
  - vidarh 1y ago
    You also don't need to commit to upfront costs. You can easily rent, rent to own/lease these resources.

vidarh 1y ago

It's simple enough to hire people with experience with both, or pay someone else to do it for you. These skills aren't that hard to find. If you hire people that are not responsive to your needs, then, sure, that is a problem that will be a problem irrespective of what their pet stack is.

whstl 1y ago

Sure but I have seen the exact same thing happen with AWS. In a large company I worked the Ops team that had the keys to AWS was taking literal months to push things to the cloud, causing problems with bonuses and promotions. Security measures were not in place so there were cyberattacks. Passwords of critical services lapsed because they were not paying attention. At some point it got so bad that the entire team was demoted, lost privileges, and contractors had to jump in. The CTO was almost fired. It took months to recover and even to get to an acceptable state, because nothing was really documented.

LPisGood 1y ago
I can’t believe the CTO wasn’t fired for that.
- whstl 1y ago
  The CTO was the one holding the bonus and promotions for tech, so he just shifted the blame down when it was "investigated". On the other hand it's not hard to believe that the CEO and the board are as sleepy as the CTO here. And the whole management team. The worst one was when a password for an integration with the judicial system expired. They asked the DevOps to open their email and there were daily alerts for six months. The only reason they found this happened was because a few low level operators made a big thing out of it. I don't like talking about "regulatory capture" but this is the only reason this company still exists. Easy market when there's almost no competition.

Edman274 1y ago

The entire value proposition of AWS vs running one's own server is basically this: is it easier to ask for permission, or forgiveness? You're asking for permission to get a million dollars worth of servers / hardware / power upgrades now, or you're asking for forgiveness for spending five million dollars in AWS after 10 months. Which will be easy: permission or forgiveness?

chasemp 1y ago
I had not thought of it this way, but interesting point. I have seen this as well.

embedding-shape 1y ago

> said staff was hired to work on the stack you are using Looking back at doing various hiring decisions at various levels of organizations, this is probably the single biggest mistake I've done multiple times, hiring specific people using specific technology because we were specifically using that. You'll end up with a team unwilling to change, because "you hired me for this, even if it's best for the business with something else, this is what I do". Once I and the organizations shifted our mindset to hiring people who are more flexible, even if they have expertise in one or two specific technologies, they won't put their head in the sand whenever changes come up, and everything became a lot easier.

vidarh 1y ago
Exactly. If someone has "Cloud Engineer" in the headline of their resume instead of "Devops Engineer" it's already warning and worth probing. If someone has "AWS|VMWare Engineer" in their bio, it's a giant red flag to me. Sometimes it's people just being aware where they'll find demand, but often it's indicative of someone who will push their pet stack - and it doesn't matter if it's VMWare on-prem or AWS (both purely as examples; it doesn't matter which specific tech it is), it's equally bad if they identify with a specific stack irrespective of what the stack is. I'll also tend to look closely at whether people have "gotten stuck" specialising in a single stack. It won't make me turn them down, but it will make me ask extra questions to determine how open they are to alternatives when suitable.

infecto 1y ago

Your comment also jogged my memory of how terrible bare metal days used to be. I think now with containers it can be better but the other reason so many switched to cloud is we don’t need to think about buying the bare metal ahead of time. We don’t need to justify it to a DevOps gatekeeper.

vidarh 1y ago
That so many people remember bare metal as of 20+ years ago is a large part of the problem. A modern server can be power cycled remotely, can be reinstalled remotely over networked media, can have its console streamed remotely, can have fans etc. checked remotely without access to the OS it's running etc. It's not very different from managing a cloud - any reasonable server hardware has management boards. Even if you rent space in a colo, most of the time you don't need to set foot there other than for an initial setup (and you can rent people to do that too). But for most people, bare metal will tend to mean renting bare metal servers already configured anyway. When the first thing you then tend to do is to deploy a container runtime and an orchestrator, you're effectively usually left with something more or less (depending on your needs) like a private cloud. As for "buying ahead of time", most managed server providers and some colo operators also offer cloud services, so that even if you don't want to deal with a multi-provider setup, you can still generally scale into cloud instances as needed if your provider can't bring new hardware up fast enough (but many managed server providers can do that in less than a day too). I never think about buying ahead of time. It hasn't been a thing I've had to worry about for a decade or more.
- infecto 1y ago
  You are right but I just think people miss the history when we talk about moving to the cloud. It was not that long ago at a reasonable size Bay Area company, I would need to justify new metal to be provisioned to standup a service I was tasked with.
- mmarq 1y ago
  > A modern server can be power cycled remotely, can be reinstalled remotely over networked media, can have its console streamed remotely, can have fans etc. checked remotely without access to the OS it's running etc. It's not very different from managing a cloud - any reasonable server hardware has management boards. Even if you rent space in a colo, most of the time you don't need to set foot there other than for an initial setup (and you can rent people to do that too). All of this was already possible 20 years ago, with iLO and DRAC cards.

YouAreWRONGtoo 1y ago

[dead]

hshdhdhj4444 1y ago

Considering the rapid shift from on Prem to the cloud I think it’s clearly false that the people who knew on Prem were fighting for their little area of expertise. In my experience, the ops folks were absolutely thrilled with the arrival of the cloud because with a trivial amount of training and a couple of certifications they had a pathway to get paid as much, if not more, than devs, especially if they rebranded as “devops engineers” instead of “ops guys”. The only pushback against the cloud, other than some of us engineers who actually were among the first to jump on the cloud, still really loved it, but also recognized that it wasn’t the best fit for all uses and carried significant risks, were people worried about data safety. The latter concern has largely turned out to not be a real one yet, but a decade and a half later people are finally realizing that actually there are many areas where the cloud may not be the best fit.