4 ms·

Very heartening to know other people are thinking about this. For about 4 or 5 years I've been warning about an equivalent problem in the HPC space, whereby eve

Very heartening to know other people are thinking about this. For about 4 or 5 years I've been warning about an equivalent problem in the HPC space, whereby even if on-prem is cheaper, companies will be forced to adopt cloud due to the lack of skilled bare-metal sysadmins. I should note that these warnings aren't accompanied by much sympathy; companies and academic institutions both do a profoundly terrible job at incubating the skills they need, then bleat at their local governments about new graduates not having the skills to be employable.

PragmaticPulp 4y ago
> companies will be forced to adopt cloud due to the lack of skilled bare-metal sysadmins. More likely that they're just not paying enough, and those skilled sysadmins would rather go to any other company that will pay them appropriately for their skills. This is just another take on the "nobody wants to work anymore" rant that happens when companies can't understand why nobody is accepting their open positions.
- phpisthebest 4y ago
  I think both are actually true. Companies are not paying enough, but we also do not have as many opportunities (for SysAdmins) to learn things. Paid Internships, Jr Admin Roles, etc are fewer and far between, companies are looking for "experienced" people in a time where alot of the experience is retiring. Dont get me wrong there are plenty of us Gen X's out there with lots of experience but most of us are also massively over worked with limited opportunities given for us to train the next generation, so pay is only 1 part of this problem, head count is the other Even if you find a company that has good pay, they almost always under size their IT dept by 25-50%, and rarely promote internal training or continuing education, or offer entry level admin positions with the goal to skill up people.
  - hinkley 4y ago
    It's worse than that, he's dead, Jim. I think if you want to go back into history and look at things, we had sysadmins who were not paid enough, but fell into "give a man a little power" trap and showed a side of themselves that we would rather not see, fed and reinforced by getting blamed for anything bad that happened 'on their watch'. The cumulative generational trauma on both sides of that relationship sets us up for a situation where outsourcing IT to someone who can't sass back or harass us about our deployment plans seems like a good deal. Yeah it costs about the same and it'll interrupt our roadmap but I get to 'stick it' to the IT department. Fuck those guys. Now it's been a while and people are starting to feel nostalgia for the old days.
    - phpisthebest 4y ago
      That is the perception (an incorrect one) that alot of devs have, even though they have never done, and refuse to do any sysadmin work. It was never, has never, and could never be about "power". I have always been a generalist, with one foot in both roles of dev and admin. I am very good dev in 3 programming languages (not including scripting languages like powershell and bash.. php, C#, and JS are my strengths), and passable in about 8 or 9 others (go, python, learning rust now, etc). I hold multiple certs and years of experience in sysadmin roles. There are very few of us out there, people with extensive experience in both. I can tell you, once you have seen both sides of that wall you come way with the clear view that neither understand the other. DevOps was proposed as a solution to that, and if Devops would have been implemented as a Management sytle not a role it might have accomplished that goal but sadly most orgs just eliminated the valuable perspective of sysadmins, and as you said "outsourcing IT to someone who can't sass back or harass us about our deployment plans" which from my experience many devs need to have their plans pushed back on because often time they are widely inefficient, insecure, ill thought out, and expensive I have to wonder, if many of the cyber security incidents we have seen over the last 10 years could be avoided if Dev's lost the chip on their shoulder, and listened to the sysadmin more, instead of just rejecting their point of view as a "god complex" like you seem to
      - hinkley 4y ago
        It’s not incorrect. Inaccurate maybe, but not incorrect. You know why all internet traffic goes over port 80 and port 443? Entire generations of admins who learned that their job was to say “no” instead of “how”. By the time the fingers got peeled away from that issue the damage had already cemented. Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new. Today the priesthood of the DB is all but gone. Devs are expected to do those roles because otherwise nothing gets done. And similarly, the Cloud doesn’t say “no”, it says “how much”. Are bad things happening now? Bad things always happen. I read Applied Cryptography while I was still in college, and have worked at a lot of startups and in other small projects, so I am biased toward thinking devs should not be making other teams babysit them. Advice and control are very different things. Edit: I’ll note this isn’t my opinion, but my opinion plus decades of listening to devs vent about people out of earshot. Sometimes to the point of sedition. There are some big frustrations out there and creative people steer solutions away from frustrations. It’s not an accident we are where we are. People voted with their feet.
bazoom42 4y ago
> even if on-prem is cheaper Surely this calculation should include the cost of people with the required skills?
- EricE 4y ago
  To quote TFA: "Some things are simpler, others more complex, but on the whole, I've yet to hear of organizations at our scale being able to materially shrink their operations team, just because they moved to the cloud." Something I've observed for some time as well. If Ops was truly outsourced to the cloud service providers then why is devops such a buzzword/trend, hmm?
  - bazoom42 4y ago
    The complaint in the comment above was that on-prem was chaper but they couldnt hire people with the necessary skills. Obviously it is not actually cheaper if you don’t factor in hiring the necessary skills.
    - Moissanite 4y ago
      I suppose that is just another way of expressing my point; comparing hardware on a like-for-like basis, on-prem is cheaper. Once you factor in the hiring cost and adjust for the availability of those skills (since that is never truly reflected in the salary, in my experience) - then cloud becomes "cheaper" not in money but in opportunity-cost and reduced staffing-related business risk.
- icedchai 4y ago
  Generally, with cloud, you still need people with required skills. Engineers who write Terraform ("DevOps") aren't cheap. They certainly cost more than the old-school sysadmin who will set up a bare metal server with shell scripts.
  - l3uwin 4y ago
    You definitely don't want them using shell scripts. That's cloud-init/Ansible work. And you can do all of those through terraform as well.
c0l0 4y ago
It will get worse when the manufacturers of the best performing hardware will be the cloud providers themselves :) I am afraid Amazon will slowly drown out independent CPU (and eventually other gear, but with CPUs the playbook is most evident) manufacturers. Today, Intel (and probably also AMD et al.) offer them exclusive deals on and even exclusive models of their hardware, due to the insane volumes "hyper-scalers" procure. In parallel, Amazon offers their own CPU arch (Graviton) cheaper than x86 cores to users of their platform in order to make it more attractive - at the insane margin AWS operates at, it can make these instances a loss leader, no problem. As a result, those in the x86 market that do not have a cloud business attached will grow weaker and weaker in comparison, and most cloud-native developer mindshare will shift to the cheaper option (since most engineering work in cloud envs is about reducing cost, period), which is AWS/GCP/... custom CPUs. And you, the random shmuck with an appetite for powerful on-prem hardware, will be left with worse products to buy for a higher price.
- pclmulqdq 4y ago
  I'm not sure this is actually true. Intel and AMD are not thrilled that Amazon and Google have so much market power as their buyers. They are willing to give you a lot of other value-added products and a TON of engineering help along with their CPUs if you are buying a cluster. They will also give you a decent price (not as good as they have to give to hyperscalers, but good), and try to establish a long-term relationship with you. Also, cloud environments are rarely cost reductions. They are very frequently more expensive than on-prem deployments, but they are used because they are easy to spin up and down and they allow you to engineer things faster.
- hinkley 4y ago
  The 'cool' thing - from a vendor's perspective at least - is that running your applications locally gives you a lower confidence interval when production is a different chip architecture than developer machines, so you have to spin up more pre-prod machines to be really sure your code actually behaves. Or use some other strategy that all ends in the same place: more servers to concurrently run more revisions of your code.
fock 4y ago
curiously at the small chair I work, the professor goes against that flow and we do our small cluster all on our own. We also don't have much money (small), so dedicating probably half a year of state-funded PhD-time paid off (still quite academic around here) as we now have an environment newcomers can get work with in week 1. On top me and a colleague have a niche but interesting skillset pxe-booting a slurm cluster, running RoCE and NFS. Our softwaredistribution (as in "compiled sensibly"...) is arguably better than on some parts of the HPC-center (continent-wide academic TOP10 iirc) nearby. Also a first year PhD has some interest in that and will probably carry this on.
- WJW 4y ago
  I don't have a lot of insight in academic funding and I know the meme that PhDs don't get paid a lot, but is it really better to use presumably very highly trained PhDs to do the grunt work of systems administration rather than actual research? Honest question here, it just seems logical to hire dedicated technicians for the not-research work?
  - fock 4y ago
    there is no money for dedicated techs nowadays. Also hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging. And while it is grunt work sometimes (but more so is academic research in lab, which to 90% could be rationalized by a labtech...), it's not too stupid if the people programming the system know how the system works. Aside from racking severs, that's essentially what SRE does, no?
    - Moissanite 4y ago
      > hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging Anecdata: I have 2 customers who have had open roles of this type for more than a year each. I'm pleased to report that they are edging their salary offers up into "reasonable" territory - but the fact remains that someone with that skillset would ordinarily be able to get paid twice as much at any of the 3 major clouds.
hinkley 4y ago
Every architecture I've been responsible for has been better because of having conversations with operational people in which they tell me how divorced one of my ideas is from operational reality. Nobody would build a cluster that way because X. And then they started taking away my Ops people.

PragmaticPulp 4y ago

> companies will be forced to adopt cloud due to the lack of skilled bare-metal sysadmins. More likely that they're just not paying enough, and those skilled sysadmins would rather go to any other company that will pay them appropriately for their skills. This is just another take on the "nobody wants to work anymore" rant that happens when companies can't understand why nobody is accepting their open positions.

phpisthebest 4y ago
I think both are actually true. Companies are not paying enough, but we also do not have as many opportunities (for SysAdmins) to learn things. Paid Internships, Jr Admin Roles, etc are fewer and far between, companies are looking for "experienced" people in a time where alot of the experience is retiring. Dont get me wrong there are plenty of us Gen X's out there with lots of experience but most of us are also massively over worked with limited opportunities given for us to train the next generation, so pay is only 1 part of this problem, head count is the other Even if you find a company that has good pay, they almost always under size their IT dept by 25-50%, and rarely promote internal training or continuing education, or offer entry level admin positions with the goal to skill up people.
- hinkley 4y ago
  It's worse than that, he's dead, Jim. I think if you want to go back into history and look at things, we had sysadmins who were not paid enough, but fell into "give a man a little power" trap and showed a side of themselves that we would rather not see, fed and reinforced by getting blamed for anything bad that happened 'on their watch'. The cumulative generational trauma on both sides of that relationship sets us up for a situation where outsourcing IT to someone who can't sass back or harass us about our deployment plans seems like a good deal. Yeah it costs about the same and it'll interrupt our roadmap but I get to 'stick it' to the IT department. Fuck those guys. Now it's been a while and people are starting to feel nostalgia for the old days.
  - phpisthebest 4y ago
    That is the perception (an incorrect one) that alot of devs have, even though they have never done, and refuse to do any sysadmin work. It was never, has never, and could never be about "power". I have always been a generalist, with one foot in both roles of dev and admin. I am very good dev in 3 programming languages (not including scripting languages like powershell and bash.. php, C#, and JS are my strengths), and passable in about 8 or 9 others (go, python, learning rust now, etc). I hold multiple certs and years of experience in sysadmin roles. There are very few of us out there, people with extensive experience in both. I can tell you, once you have seen both sides of that wall you come way with the clear view that neither understand the other. DevOps was proposed as a solution to that, and if Devops would have been implemented as a Management sytle not a role it might have accomplished that goal but sadly most orgs just eliminated the valuable perspective of sysadmins, and as you said "outsourcing IT to someone who can't sass back or harass us about our deployment plans" which from my experience many devs need to have their plans pushed back on because often time they are widely inefficient, insecure, ill thought out, and expensive I have to wonder, if many of the cyber security incidents we have seen over the last 10 years could be avoided if Dev's lost the chip on their shoulder, and listened to the sysadmin more, instead of just rejecting their point of view as a "god complex" like you seem to
    - hinkley 4y ago
      It’s not incorrect. Inaccurate maybe, but not incorrect. You know why all internet traffic goes over port 80 and port 443? Entire generations of admins who learned that their job was to say “no” instead of “how”. By the time the fingers got peeled away from that issue the damage had already cemented. Similarly ORMs also got a healthy nudge from DBAs who decided it was their job to protect the business from ever doing anything new. Today the priesthood of the DB is all but gone. Devs are expected to do those roles because otherwise nothing gets done. And similarly, the Cloud doesn’t say “no”, it says “how much”. Are bad things happening now? Bad things always happen. I read Applied Cryptography while I was still in college, and have worked at a lot of startups and in other small projects, so I am biased toward thinking devs should not be making other teams babysit them. Advice and control are very different things. Edit: I’ll note this isn’t my opinion, but my opinion plus decades of listening to devs vent about people out of earshot. Sometimes to the point of sedition. There are some big frustrations out there and creative people steer solutions away from frustrations. It’s not an accident we are where we are. People voted with their feet.

bazoom42 4y ago

> even if on-prem is cheaper Surely this calculation should include the cost of people with the required skills?

EricE 4y ago
To quote TFA: "Some things are simpler, others more complex, but on the whole, I've yet to hear of organizations at our scale being able to materially shrink their operations team, just because they moved to the cloud." Something I've observed for some time as well. If Ops was truly outsourced to the cloud service providers then why is devops such a buzzword/trend, hmm?
- bazoom42 4y ago
  The complaint in the comment above was that on-prem was chaper but they couldnt hire people with the necessary skills. Obviously it is not actually cheaper if you don’t factor in hiring the necessary skills.
  - Moissanite 4y ago
    I suppose that is just another way of expressing my point; comparing hardware on a like-for-like basis, on-prem is cheaper. Once you factor in the hiring cost and adjust for the availability of those skills (since that is never truly reflected in the salary, in my experience) - then cloud becomes "cheaper" not in money but in opportunity-cost and reduced staffing-related business risk.
icedchai 4y ago
Generally, with cloud, you still need people with required skills. Engineers who write Terraform ("DevOps") aren't cheap. They certainly cost more than the old-school sysadmin who will set up a bare metal server with shell scripts.
- l3uwin 4y ago
  You definitely don't want them using shell scripts. That's cloud-init/Ansible work. And you can do all of those through terraform as well.

c0l0 4y ago

It will get worse when the manufacturers of the best performing hardware will be the cloud providers themselves :) I am afraid Amazon will slowly drown out independent CPU (and eventually other gear, but with CPUs the playbook is most evident) manufacturers. Today, Intel (and probably also AMD et al.) offer them exclusive deals on and even exclusive models of their hardware, due to the insane volumes "hyper-scalers" procure. In parallel, Amazon offers their own CPU arch (Graviton) cheaper than x86 cores to users of their platform in order to make it more attractive - at the insane margin AWS operates at, it can make these instances a loss leader, no problem. As a result, those in the x86 market that do not have a cloud business attached will grow weaker and weaker in comparison, and most cloud-native developer mindshare will shift to the cheaper option (since most engineering work in cloud envs is about reducing cost, period), which is AWS/GCP/... custom CPUs. And you, the random shmuck with an appetite for powerful on-prem hardware, will be left with worse products to buy for a higher price.

pclmulqdq 4y ago
I'm not sure this is actually true. Intel and AMD are not thrilled that Amazon and Google have so much market power as their buyers. They are willing to give you a lot of other value-added products and a TON of engineering help along with their CPUs if you are buying a cluster. They will also give you a decent price (not as good as they have to give to hyperscalers, but good), and try to establish a long-term relationship with you. Also, cloud environments are rarely cost reductions. They are very frequently more expensive than on-prem deployments, but they are used because they are easy to spin up and down and they allow you to engineer things faster.
hinkley 4y ago
The 'cool' thing - from a vendor's perspective at least - is that running your applications locally gives you a lower confidence interval when production is a different chip architecture than developer machines, so you have to spin up more pre-prod machines to be really sure your code actually behaves. Or use some other strategy that all ends in the same place: more servers to concurrently run more revisions of your code.

fock 4y ago

curiously at the small chair I work, the professor goes against that flow and we do our small cluster all on our own. We also don't have much money (small), so dedicating probably half a year of state-funded PhD-time paid off (still quite academic around here) as we now have an environment newcomers can get work with in week 1. On top me and a colleague have a niche but interesting skillset pxe-booting a slurm cluster, running RoCE and NFS. Our softwaredistribution (as in "compiled sensibly"...) is arguably better than on some parts of the HPC-center (continent-wide academic TOP10 iirc) nearby. Also a first year PhD has some interest in that and will probably carry this on.

WJW 4y ago
I don't have a lot of insight in academic funding and I know the meme that PhDs don't get paid a lot, but is it really better to use presumably very highly trained PhDs to do the grunt work of systems administration rather than actual research? Honest question here, it just seems logical to hire dedicated technicians for the not-research work?
- fock 4y ago
  there is no money for dedicated techs nowadays. Also hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging. And while it is grunt work sometimes (but more so is academic research in lab, which to 90% could be rationalized by a labtech...), it's not too stupid if the people programming the system know how the system works. Aside from racking severs, that's essentially what SRE does, no?
  - Moissanite 4y ago
    > hiring a competent sysadmin to do on-prem HPC-Linux might prove challenging Anecdata: I have 2 customers who have had open roles of this type for more than a year each. I'm pleased to report that they are edging their salary offers up into "reasonable" territory - but the fact remains that someone with that skillset would ordinarily be able to get paid twice as much at any of the 3 major clouds.

hinkley 4y ago

Every architecture I've been responsible for has been better because of having conversations with operational people in which they tell me how divorced one of my ideas is from operational reality. Nobody would build a cluster that way because X. And then they started taking away my Ops people.