11 ms·

The cloud skills shortage and the unemployed army of the certified

195 points by rajeshmr 8y ago

fma 8y ago
"The jobs opening up include a requirement to know all these skills in addition to everything else." I've been casually looking at job postings and sometimes wonder what drugs the job poster is on when writing job requirements.
- GoToRO 8y ago
  I you really want to have fun, just look at the job posting for your own role in the organization. Then, you can really compare stuff.
  - yjftsjthsd-h 8y ago
    Funny story: Once upon a time, I got promoted, and we needed to backfill my (old) position. They proceeded to post a job description that 1. I might not have qualified for, and 2. I never would have applied for.
- bsbechtel 8y ago
  That's partially because job postings are an SEO play, so that long list of requirements is more about trying to get as many eyeballs as possible on the page. Job postings used to be as short as possible because you were paying per line in the newspaper. Search engines changed that.
  - kozhevnikov 8y ago
    That doesn't make sense. They can just move all those requirements from 'required' to 'desired' section and get the same SEO out of it.
    - arvinsim 8y ago
      Small companies looking for a qualified person will bother. Large companies with hundreds of applicants won’t bother.
henrik_w 8y ago
"The Creeping IT Apoclypse" is on the same theme: https://news.ycombinator.com/item?id=18930781 https://news.ycombinator.com/item?id=18930781
deleted 8y ago
[deleted]
JVerstry 8y ago
Not so sure Docker is the only way forward when it comes to cloud scaling or deployment. VM templates are a very good alternative. They are more stable, more flexible/customizable and integrate more smoothly with CI.
- talltimtom 8y ago
  Docker, cloud hosted VM from templates, vagrant, Azure, AWS, it doesn’t really matter much to me. The important thing to me is that I no longer have any need for anyone else doing company wide infrastructure. We have several department heads engaged in a battle about who’s going to “own” our “cloud infrastructure” they seem Oblivious to the fact that the only thing we need from them is to negotiate an Azure or AWS subscription and after that they will loose any utility. A lot of talented people are going to find them self in a problematic situation because the area in which their talents lie is only going to be handled by low wage jobs at google, Azure and Amazon. Even big companies won’t bother setting up their own hardware because the people are costly even if they have slight savings on private hardware.
  - falcor84 8y ago
    > low wage jobs at google, Azure and Amazon I have no idea what makes you say that. As far as I'm familiar with these roles, none of them are low paying. These companies tend to pay very well because of the amazing scalability involved in these roles. Any engineering work that scales linearly with the number of users is automated almost immediately and the focus is generally on very high-level work.
    - talltimtom 8y ago
      When you go from 100 corporations. Each having infrastructure teams of 50+ people to one provider running the same thing with 20 people from that pool the wages go down. While there certainly are well paid individuals behind Azure or AWS, they are rare compared to “the people on the floor”, and those people used to be able to rise to senior specialist in companies before, now they just order services from the big players. Now don’t get me wrong there will always be avenues for the most talented players, but the crisis will come when the 45 out of the 50 strong infrastructure team at every larger corporation are no longer needed, and the last 5 will end up doing work that’s completely unrelated to their prior expertise.
angarg12 8y ago
The question: "Why it’s so hard to find roles in cloud technology, while jobs go unfilled." The answer: "The key thing here is that businesses are not hiring fresh-out-of-certification AWS experts to take over the infrastructure of their established operations teams. They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly. This often means insane job requirements [...] and an existing track-record of success." So jobs openings stay empty because companies have unrealistic expectations. Is that something new and surprising? This is the pattern in tech recruiting as far back as I can remember.
- wjnc 8y ago
  Wait. These aren't really unrealistic expectations. If you want transformation against a risk appetite of hardly any risk acceptable, you jack up requirements or no go. That sounds business wise to me. In my firm we've had many cloud related problems. The result was higher and higher demands in the direction of our suppliers. Obviously this limits the number of possible suppliers, and makes some suppliers unhappy, but it is proper business in line with the way we want to reach our goals.
  - mlthoughts2018 8y ago
    It becomes unrealistic if those people are in short supply but the companies also try to keep compensation low. If you need a special expert fast, but you don’t expect to pay a lot, then you are being unreasonable. Judging by the claimed urgency and how wisespread the need is becoming, not every company will be able to get an experienced person for this role.
    - jt2190 8y ago
      > If you need a special expert fast... A job posting doesn't necessarily mean that they need someone fast, just that they'd like someone, someday, for a certain price. Companies have many important but not urgent goals that can wait for the right people.
      - 8y ago
huffmsa 8y ago
> "10 years serverless" as a hard requirement... So what we have here is a failure in the hiring process. The people writing the jobs don't actually know the space, but are looking for candidates with literally impossible amounts of experience.
- dankohn1 8y ago
  We refuse to hire any Kubernetes developers without at least 6 years experience. https://landscape.cncf.io/selected=kubernetes https://landscape.cncf.io/selected=kubernetes
- tomjen3 8y ago
  Sometimes I think that it is a way to filter out those the company doesn't think are "go-getters", or to have something to hit back with when salaries are discussed.
- yjftsjthsd-h 8y ago
  Oh, that's easy;) See, "serverless" is just the new name for what we used to call CGI. There are people who've been doing it since the '90s! (Sure, we made it nicer since then, but if you squint just a little...)
theredbox 8y ago
I am sick of this non sense. It's because nobody knows what they want and those that should be the ones knowing it all also do not know what they know and what they need to learn. We built immensely complex systems that allow us to scale. The thing is we have increased the complexity so much that we no longer can justify hiring incompetent. Simplicity is beautiful and we will have a renaissance of simpler systems. Something along the lines of what Go brought to the game.
- sascha_sl 8y ago
  >Something along the lines of what Go brought to the game. Where the biggest adopter, Kubernetes, is ironically a way too complex system of automation routines that still regularly fail in production environments. And oh dear, if they fail, it's gonna be hard to figure out what exactly went wrong. We uncovered a bug in the job controller just yesterday that was probably not biting anyone else due to how a race condition usually plays out, but this kind of bug is way too common.
  - theredbox 8y ago
    Yeah that's the irony of it. But Go is just a tool. People were building monstrosities with bash before and will be long after we build the mythical AI Cloud.
    - apta 8y ago
      > But Go is just a tool. Yes, but there are better tools suited for the job. Ironically, or perhaps not, Java or .NET would have faired better due to maturity of debugging and introspection tools.
  - nova22033 8y ago
    >Where the biggest adopter, Kubernetes, is ironically a way too complex system of automation routines that still regularly fail in production environments. regularly fail? Do you have any data to back that up? No, a google search for one kubernetes failure doesn't count as proof of "regular failure"
empath75 8y ago
The real reason is that coding is hard and coders are hard to find and certifications are just an acknowledgment that you have domain knowledge. I don’t care if you know what an auto scaling group is. I care that you can write code that automates builds and deployments. The AWS part is easy and you can learn what you need to know on the job. In my experience, coding is something you either can do or you can’t, and if you come to me with a cert and no programming experience it’s very hard for me to know what you’re capable of. i spend like 5% of an interview asking about AWS stuff and like 50% talking about code and automation.
- theprotocol 8y ago
  >you can learn what you need to know on the job Maybe I've had bad luck, but I've encountered very few prospective employers who were OK with this. The ones I encountered expected to hire someone who would fit in like a jigsaw puzzle piece and they wouldn't settle for anything less. My confidence in my ability to pick up almost any tech quickly did not impress them, and they latched on to my lack of that particular bullet point and kept hammering on things like "oh, you don't know what a [platform-specific term for auto-scaling groups] is?..."
  - Pokepokalypse 8y ago
    Same. And I spent 10 years at a company where my job, literally, was new small projects, every 6-12 months, often radically different platforms and technologies, where I had to "come up to speed". (has it's plusses and minuses - you learn a lot of breadth, not a lot of depth). But interviewing for me, has been a very frustrating experience. When I did land a spot, they were desperate, as it turns out. And in interviewing and making hiring decisions for my team; I've had to make these same types of judgment calls. I look for candidates who have enough experience that I know they're not bullshitting. And I look for a work history that shows a progression of continually taking on new skills and roles. Because that's what this job really is. Anyone who's been doing this in the real world for at least 10 years knows this. But it's really puzzling to me how so many hiring managers at so many places just don't seem to understand this.
netheril96 8y ago
I'm guessing that another reason is that some of the existing IT personnel are resisting the change, and one of their methods is to give HR unrealistic requirements so no one will ever qualify.
HocusLocus 8y ago
"Hi! I'm your new cloud outsourcing specialist. I'm here to take your old IT structure and those servers in the closet devils-you-know with something so intangible and 'elsewhere', when it stops working some day you cannot possibly send an army to the closet to fix the problem. You can all just wallow in your helplessness and go home and wait for a text message. When do I start??"
- nostrebored 8y ago
  Yeah, except that team in the closet is now one of the most experienced teams in the world responsible for thousands of companies and managing a global backbone infrastructure that a smaller company couldn't possibly pay to maintain.
  - setquk 8y ago
    Unless it’s Azure as they make small company mistakes like forgetting to renew certs... YMMV for anything you can’t see or touch regardless of the size or experience or reputation so far.
paganel 8y ago
Just partially related to the article, but mentioned in it nonetheless, I just wanted to ask if “serverless” is now the new hype-word instead of containers/Docker/K8? I’m curious because I’ve started reading about it in different contexts more and more lately but afaik it doesn’t come with any clear advantages over the “classical” way of doing stuff.
- nostrebored 8y ago
  Not having to do any infrastructure management and being able to scale from 0 load paying for 0 load to peak traffic paying only for what you use is pretty incredible. There are definitely some workloads that lend themselves to serverless and some that don't -- it's not the silver bullet a lot of evangelists say it is -- but when it works it just works.
- dsr_ 8y ago
  Everything is a semi-rational reaction to something else. Companies used to own and run their own machines on-premises. But doing it properly (HVAC, power, raised floors, standardized racks, management, redundancy, preparedness...) is expensive and not actually most company's core competency. So they moved to colocated datacenters, where a competent company would take care of the infrastructure for a fee which would hopefully reflect a discount based on savings from scale. But managing generations of servers at colo datacenters takes manpower for hardware replacement, upgrades, cabling, and generally doing things right; the customer companies mostly don't have that as a business feature but see it as a cost. So managed infrastructure services came up, where the datacenter company leases you the hardware (standardized so they can have spares) and racks it for you and puts in network switches and remote services and gets it to the point where all you have to do is log in to a console server and install your OS and start deployment. But sysadmins who can keep track of security updates and package dependencies and keep an OS properly organized are relatively expensive, and what used to be "well it works on my laptop" becomes "well it works in my container" and the container itself gets shipped out. It's cheaper not to do security work, you know. So VM and container accepting services come up, where the datacenter company now runs a hypervisor and managed storage systems for you, in exchange for the salary of those sysadmins. The start-up costs can be much lower, to the point where it really doesn't make fiscal sense for any tiny company to do anything themselves except ship their containers/VMs out and puzzle over IP allocation schemes and load-balancing services. It's a trap, but a delicious one. So now your company is hooked on the ease and speed and cheapness of just spinning up another container or VM any time someone expresses a desire, or even automatically, when you get the bill at the end of the month and you managed to spend how much? That's ridiculous. Why are we getting billed for containers that we don't even need to run all the time? Along comes "serverless", which is a logical successor of inetd. Yes, inetd, the very old "internet super-server" which would read a table of ports and programs, open all the ports, and when a connection came in would run your program and connect stdin/out to the socket, isn't that great? Serverless is just a management system that spins up your very restricted single-function program on demand -- now it answers an HTTPS API instead of a raw socket -- and does the accounting work to make it profitable. Complexity is simultaneously the enemy of correctness and the source of profit.
taneq 8y ago
> QA has been gutted by TDD Wow. Just wow.
- rcaught 8y ago
  It is probably fair to say that CI/CD have gutted QA though.
  - pjmorris 8y ago
    It is probably fair to say that managers have gutted QA.
sailfast 8y ago
That S-Curve on cloud seems off. Most Federal agencies are on or moving to Cloud services right now. They are squarely toward the top right of the S. I don’t think you can say only “large trailblazers” are there. We can argue about how many services have moved and how it’s being used but the whole “let someone else manage it” argument is long done and normalized.
- laurentl 8y ago
  A lot of statements in the article seem off. The position on the S-curve, the claim that TDD replaced QA teams, even the fact that “serverless is coding for hardware engineer” or however it was formulated (the author probably meant “infrastructure as code”). With so many approximations in the argument, it’s hard to accept the conclusions of the article at face value.
- Pokepokalypse 8y ago
  right, but you basically can't get a job there, unless you live in Virginia.
hestefisk 8y ago
“server admin is very hard, and mostly gone with virtualization.“ I would say this is simply not true. For many companies, getting good admins and infrastructure experts this is extremely hard, at least in big corporate.
- ldoughty 8y ago
  Completely agree. Server admins are still super important. Not everything fits in a nice server-less box. Developers also tend to have issues with containers, and firewalls, and monitoring/troubleshooting (depending on their skills and interest). People whom enjoy the server admin side can still find rewarding work in a more serverless environment
- ldoughty 8y ago
  Most developers I know don't understand containers well... Or they understand containers but not the underlying implications. "What does yum/apt upgrade have to do with node? Why should I run that?".. hell developers typically need a hard-* server guy to remind them to run package updates/upgrades on their languages package manager.. I'm not much of a developer but I end up doing a lot of the Npm and pip updates.
andmarios 8y ago
Every year cloud providers announce tens of new services, poorly tested and far from production ready. Non-ops eng teams and management are always happy to use the latest buzzword out of AWS, then ask their ops team to make it somehow work. Certifications are moot. I don't care if you are certified, or if you have used a cloud (or kubernetes) following a tutorial. I do care if you have seen enough issues throughout your career, so you can debug every time (not when, nor if) mud hits the fan. I don't care if you can type a port in a yaml file or in your browser. I do care if you know how many ports are there, what's the ephemeral port range and so on. Systems are 75% experience, 25% basic knowledge, you can't get away just with certifications which matter only after you get experience and knowledge. If you are experienced in ops, I know you can learn and work out any cloud. If you claim you are experienced with a specific cloud though, then better have to share a ton of war stories or it will just be a huge red flag on your CV. Let's call it for what it is, clouds, like data-science are hot fields right now. It's only natural that many people will go after a good-paying job claiming expertise in the field.
- scarface74 8y ago
  If you have experience in ops and don’t know the specifics of how to do operations in the cloud and take that same mentality to A cloud provider, you are going to end up spending more and not get any benefits from being with AWS/GCP/Azure.
  - andmarios 8y ago
    Indeed, but ops people have to learn new technologies all the time, so it's not that big of a leap. When you talk about a cloud and see a sysadmin rolling their eyes, it's not that they have something specifically against the cloud, it's that they've seen so many diverse bugs and issues in their life, that they know you just exchange one set of problems for another. Also, a challenge to using clouds more effectively is you need to bring onboard your developers. Ops people often have to act as coaches and basic systems' knowledge is very important in this role.
    - scarface74 8y ago
heyjudy 8y ago
Disclaimer: ex preferred partner early AWS client-facing Fortune 200 consultant. The senior sysadmin at the DMDC around mid California coast: only one certification, VMware. A French offensive exploits broker, solid researcher and reeigne: no certs. I only have a few certs because work paid tens of thousands for offsite training. A few certs are okay because corporate HR wants staff to demonstrate continual learning and achievement, lots of low-quality ones (A+) are a negative signal. It's more important to keep up on news, tools and techniques on the front lines. My opinion is managing \aaS needs solid rack&stack ops experience at scale. For IT folks: either start huge if you can or start small and move towards huge. The biggest challenges are customer communication, managing expectations, anticipating and planning where possible and showing empathy... it's a 5% technical skill and 95% getting along productively game. Communicate the business impact, don't overwhelm customers with unnecessary* detail.
coleca 8y ago
I did some freelancing cloud work recently and was shocked at what I saw out there. From small one person operations to startups to large multinational Fortune 500 corporations I saw the same pattern repeated over and over. People using the cloud to spin up infrastructure that have no experience building out infrastructure and making really bad and dangerous mistakes. Amazon, Google, Microsoft and others make it simple to get going but the devil is in the details when it comes to building it right, safe and ready for production. Just a few examples: • Putting RDS SQL Servers on a public IP with no protection • No templating of servers so if it disappears you have no record of how to get a new one running again • Servers with SSH password authentication turned on with passwords of “Password”, because SSH key auth was “too hard for our devs” • No backups because “it’s in the cloud, isn’t the cloud backing it up for me, like iCloud?” I have an AWS Solution Architect Professional certification and agree you could get a cert and still not be qualified to run or design much on AWS but it does put you ahead of much of what I’ve seen out there.
- nicoburns 8y ago
  Oh wow. Those are some bad ones! I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).
  - ldoughty 8y ago
    Cert + personal project, that's my recommendation. Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS. For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions. I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)
lifeisstillgood 8y ago
I read this as "develop and sell a niche tool that's useful in cloud ops or transformations, and market it like hell"
halbritt 8y ago
I really appreciate this part: Some of these firms realize they need to renovate their IT but they attempt to do this through hiring “Digital Transformation” roles which do little but frustrate the revolving door of hires because they have no executive buy- in and budget. This kind of thing makes it easier for me to hire. My org swallowed the risk and moved our production infrastructure entirely to k8s. I've no expectation that I'm going to be able to hire engineers with years of experience with that, but my team has already cultivated those skills. Aside from all the benefits and pains of being on k8s has, it makes it easy to attract great engineers with lots of related experience looking to leave a company where they've heard they need to adopt something and fail to put the resources behind it.
ilaksh 8y ago
Really interesting article. This line was funny though: "can remember the differences between a stack and a heap".
mancerayder 8y ago
Typically we have shortening 5-year cycles in tech that cause clamoring for new skills. First it was the internet (“HTML! Perl! PHP!”) then mobile apps (“Android! Objective C! React Native! Flutter!”) and now cloud (“AWS Certification! DevOps! SecOps!”). IoT and machine learning are probably next. The technology itself doesn’t really matter — companies just want an army of tactical workers to build this stuff tirelessly. It's Kubernetes and microservices! Even though Kubernetes is what, four years old, where I am, numerous employers are chomping at the bit and demanding that you be an expert at these. Now, I'm a crusty old Linux sysadmin/programmer who puts "SRE" and "DevOps" on his resume, and I'm noticeably getting spurned because I don't have "5 years experience" with "containers". At first I was worried that my skills were obsoleted, having worked in large enterprise environments doing infrastructure engineering for over 15 years. I worried that I was too "systemsy" and not enough "cloud-y". Oh, and even though I spent years using TeamCity and Jenkins, I was also spurned because I didn't use the right lingo within "CI/CD pipelines" and demonstrate a Docker deployment system exactly like they used. Then I realized something. The job descriptions are asking for five years experience total, number one, and the second red flag: the salaries and contract rates are lower than the rates for large enterprise-y type jobs (minus management). Conclusion: hipster tech coincides with youth and coincides with companies following trends and buzzwords over wisdom and experience. Back to the point. Someone with a strong basis in the core disciplines of systems design, infrastructure automation, perimeter security and so forth are going to be able to adapt to AWS knowledge, as I have (for the record I have two AWS Associate certs and didn't find them hard). And as the author points out, there are these 5 year cycles where all we look for is experience in certain tech. Is it any wonder that there are horror stories about fragile and insecure cloud environments? It goes back to the incompetence of the hiring process which we see again and again discussed on HN.
RickJWagner 8y ago
29 year IT veteran here. I don't agree with a lot of this article. I remember panic-talk when outsourcing began. When offshoring arose. Etc. None of these extinguished the need for IT workers. I view the cloud as the next generation of the mainframe. By that, I mean your experience and understanding of the environment (nobody will master all of it) will be a career differentiator. Knowledge of Kubernetes, Istio, etc. will help you to navigate the next career paths. System admin skills (and associated extras, like networking) will likewise move you up the ladder. Bill Gates once said there is an infinite number of problems yet to be solved, an infinite amount of work to be done. I believe him.
tabtab 8y ago
This is an age-old problem in IT. Some new technology and/or fad comes along, and bunches of organizations suddenly want an expert in it. It's impossible to find enough experienced in the new field because it's so new and/or growing too fast. But the HR department or technically-inexperienced managers don't understand this and complain to politicians et. al. of a "shortage". (Those in know have seen the pattern multiple times.) The answer is simple: find a similar field and accept a learning curve for the employee to transition. If you need something quick, hire a consultant to both move the project along and to mentor the new employee.

195 points by rajeshmr 8y ago

fma 8y ago

"The jobs opening up include a requirement to know all these skills in addition to everything else." I've been casually looking at job postings and sometimes wonder what drugs the job poster is on when writing job requirements.

GoToRO 8y ago
I you really want to have fun, just look at the job posting for your own role in the organization. Then, you can really compare stuff.
- yjftsjthsd-h 8y ago
  Funny story: Once upon a time, I got promoted, and we needed to backfill my (old) position. They proceeded to post a job description that 1. I might not have qualified for, and 2. I never would have applied for.
bsbechtel 8y ago
That's partially because job postings are an SEO play, so that long list of requirements is more about trying to get as many eyeballs as possible on the page. Job postings used to be as short as possible because you were paying per line in the newspaper. Search engines changed that.
- kozhevnikov 8y ago
  That doesn't make sense. They can just move all those requirements from 'required' to 'desired' section and get the same SEO out of it.
  - arvinsim 8y ago
    Small companies looking for a qualified person will bother. Large companies with hundreds of applicants won’t bother.

henrik_w 8y ago

"The Creeping IT Apoclypse" is on the same theme: https://news.ycombinator.com/item?id=18930781 https://news.ycombinator.com/item?id=18930781

deleted 8y ago

[deleted]

JVerstry 8y ago

Not so sure Docker is the only way forward when it comes to cloud scaling or deployment. VM templates are a very good alternative. They are more stable, more flexible/customizable and integrate more smoothly with CI.

talltimtom 8y ago
Docker, cloud hosted VM from templates, vagrant, Azure, AWS, it doesn’t really matter much to me. The important thing to me is that I no longer have any need for anyone else doing company wide infrastructure. We have several department heads engaged in a battle about who’s going to “own” our “cloud infrastructure” they seem Oblivious to the fact that the only thing we need from them is to negotiate an Azure or AWS subscription and after that they will loose any utility. A lot of talented people are going to find them self in a problematic situation because the area in which their talents lie is only going to be handled by low wage jobs at google, Azure and Amazon. Even big companies won’t bother setting up their own hardware because the people are costly even if they have slight savings on private hardware.
- falcor84 8y ago
  > low wage jobs at google, Azure and Amazon I have no idea what makes you say that. As far as I'm familiar with these roles, none of them are low paying. These companies tend to pay very well because of the amazing scalability involved in these roles. Any engineering work that scales linearly with the number of users is automated almost immediately and the focus is generally on very high-level work.
  - talltimtom 8y ago
    When you go from 100 corporations. Each having infrastructure teams of 50+ people to one provider running the same thing with 20 people from that pool the wages go down. While there certainly are well paid individuals behind Azure or AWS, they are rare compared to “the people on the floor”, and those people used to be able to rise to senior specialist in companies before, now they just order services from the big players. Now don’t get me wrong there will always be avenues for the most talented players, but the crisis will come when the 45 out of the 50 strong infrastructure team at every larger corporation are no longer needed, and the last 5 will end up doing work that’s completely unrelated to their prior expertise.

angarg12 8y ago

The question: "Why it’s so hard to find roles in cloud technology, while jobs go unfilled." The answer: "The key thing here is that businesses are not hiring fresh-out-of-certification AWS experts to take over the infrastructure of their established operations teams. They want people with solid experience, a risk-free bet that the new hire can execute the tech flawlessly. This often means insane job requirements [...] and an existing track-record of success." So jobs openings stay empty because companies have unrealistic expectations. Is that something new and surprising? This is the pattern in tech recruiting as far back as I can remember.

wjnc 8y ago
Wait. These aren't really unrealistic expectations. If you want transformation against a risk appetite of hardly any risk acceptable, you jack up requirements or no go. That sounds business wise to me. In my firm we've had many cloud related problems. The result was higher and higher demands in the direction of our suppliers. Obviously this limits the number of possible suppliers, and makes some suppliers unhappy, but it is proper business in line with the way we want to reach our goals.
- mlthoughts2018 8y ago
  It becomes unrealistic if those people are in short supply but the companies also try to keep compensation low. If you need a special expert fast, but you don’t expect to pay a lot, then you are being unreasonable. Judging by the claimed urgency and how wisespread the need is becoming, not every company will be able to get an experienced person for this role.
  - jt2190 8y ago
    > If you need a special expert fast... A job posting doesn't necessarily mean that they need someone fast, just that they'd like someone, someday, for a certain price. Companies have many important but not urgent goals that can wait for the right people.
    - 8y ago

huffmsa 8y ago

> "10 years serverless" as a hard requirement... So what we have here is a failure in the hiring process. The people writing the jobs don't actually know the space, but are looking for candidates with literally impossible amounts of experience.

dankohn1 8y ago
We refuse to hire any Kubernetes developers without at least 6 years experience. https://landscape.cncf.io/selected=kubernetes https://landscape.cncf.io/selected=kubernetes
tomjen3 8y ago
Sometimes I think that it is a way to filter out those the company doesn't think are "go-getters", or to have something to hit back with when salaries are discussed.
yjftsjthsd-h 8y ago
Oh, that's easy;) See, "serverless" is just the new name for what we used to call CGI. There are people who've been doing it since the '90s! (Sure, we made it nicer since then, but if you squint just a little...)

theredbox 8y ago

I am sick of this non sense. It's because nobody knows what they want and those that should be the ones knowing it all also do not know what they know and what they need to learn. We built immensely complex systems that allow us to scale. The thing is we have increased the complexity so much that we no longer can justify hiring incompetent. Simplicity is beautiful and we will have a renaissance of simpler systems. Something along the lines of what Go brought to the game.

sascha_sl 8y ago
>Something along the lines of what Go brought to the game. Where the biggest adopter, Kubernetes, is ironically a way too complex system of automation routines that still regularly fail in production environments. And oh dear, if they fail, it's gonna be hard to figure out what exactly went wrong. We uncovered a bug in the job controller just yesterday that was probably not biting anyone else due to how a race condition usually plays out, but this kind of bug is way too common.
- theredbox 8y ago
  Yeah that's the irony of it. But Go is just a tool. People were building monstrosities with bash before and will be long after we build the mythical AI Cloud.
  - apta 8y ago
    > But Go is just a tool. Yes, but there are better tools suited for the job. Ironically, or perhaps not, Java or .NET would have faired better due to maturity of debugging and introspection tools.
- nova22033 8y ago
  >Where the biggest adopter, Kubernetes, is ironically a way too complex system of automation routines that still regularly fail in production environments. regularly fail? Do you have any data to back that up? No, a google search for one kubernetes failure doesn't count as proof of "regular failure"

empath75 8y ago

The real reason is that coding is hard and coders are hard to find and certifications are just an acknowledgment that you have domain knowledge. I don’t care if you know what an auto scaling group is. I care that you can write code that automates builds and deployments. The AWS part is easy and you can learn what you need to know on the job. In my experience, coding is something you either can do or you can’t, and if you come to me with a cert and no programming experience it’s very hard for me to know what you’re capable of. i spend like 5% of an interview asking about AWS stuff and like 50% talking about code and automation.

theprotocol 8y ago
>you can learn what you need to know on the job Maybe I've had bad luck, but I've encountered very few prospective employers who were OK with this. The ones I encountered expected to hire someone who would fit in like a jigsaw puzzle piece and they wouldn't settle for anything less. My confidence in my ability to pick up almost any tech quickly did not impress them, and they latched on to my lack of that particular bullet point and kept hammering on things like "oh, you don't know what a [platform-specific term for auto-scaling groups] is?..."
- Pokepokalypse 8y ago
  Same. And I spent 10 years at a company where my job, literally, was new small projects, every 6-12 months, often radically different platforms and technologies, where I had to "come up to speed". (has it's plusses and minuses - you learn a lot of breadth, not a lot of depth). But interviewing for me, has been a very frustrating experience. When I did land a spot, they were desperate, as it turns out. And in interviewing and making hiring decisions for my team; I've had to make these same types of judgment calls. I look for candidates who have enough experience that I know they're not bullshitting. And I look for a work history that shows a progression of continually taking on new skills and roles. Because that's what this job really is. Anyone who's been doing this in the real world for at least 10 years knows this. But it's really puzzling to me how so many hiring managers at so many places just don't seem to understand this.

netheril96 8y ago

I'm guessing that another reason is that some of the existing IT personnel are resisting the change, and one of their methods is to give HR unrealistic requirements so no one will ever qualify.

HocusLocus 8y ago

"Hi! I'm your new cloud outsourcing specialist. I'm here to take your old IT structure and those servers in the closet devils-you-know with something so intangible and 'elsewhere', when it stops working some day you cannot possibly send an army to the closet to fix the problem. You can all just wallow in your helplessness and go home and wait for a text message. When do I start??"

nostrebored 8y ago
Yeah, except that team in the closet is now one of the most experienced teams in the world responsible for thousands of companies and managing a global backbone infrastructure that a smaller company couldn't possibly pay to maintain.
- setquk 8y ago
  Unless it’s Azure as they make small company mistakes like forgetting to renew certs... YMMV for anything you can’t see or touch regardless of the size or experience or reputation so far.

paganel 8y ago

Just partially related to the article, but mentioned in it nonetheless, I just wanted to ask if “serverless” is now the new hype-word instead of containers/Docker/K8? I’m curious because I’ve started reading about it in different contexts more and more lately but afaik it doesn’t come with any clear advantages over the “classical” way of doing stuff.

nostrebored 8y ago
Not having to do any infrastructure management and being able to scale from 0 load paying for 0 load to peak traffic paying only for what you use is pretty incredible. There are definitely some workloads that lend themselves to serverless and some that don't -- it's not the silver bullet a lot of evangelists say it is -- but when it works it just works.
dsr_ 8y ago
Everything is a semi-rational reaction to something else. Companies used to own and run their own machines on-premises. But doing it properly (HVAC, power, raised floors, standardized racks, management, redundancy, preparedness...) is expensive and not actually most company's core competency. So they moved to colocated datacenters, where a competent company would take care of the infrastructure for a fee which would hopefully reflect a discount based on savings from scale. But managing generations of servers at colo datacenters takes manpower for hardware replacement, upgrades, cabling, and generally doing things right; the customer companies mostly don't have that as a business feature but see it as a cost. So managed infrastructure services came up, where the datacenter company leases you the hardware (standardized so they can have spares) and racks it for you and puts in network switches and remote services and gets it to the point where all you have to do is log in to a console server and install your OS and start deployment. But sysadmins who can keep track of security updates and package dependencies and keep an OS properly organized are relatively expensive, and what used to be "well it works on my laptop" becomes "well it works in my container" and the container itself gets shipped out. It's cheaper not to do security work, you know. So VM and container accepting services come up, where the datacenter company now runs a hypervisor and managed storage systems for you, in exchange for the salary of those sysadmins. The start-up costs can be much lower, to the point where it really doesn't make fiscal sense for any tiny company to do anything themselves except ship their containers/VMs out and puzzle over IP allocation schemes and load-balancing services. It's a trap, but a delicious one. So now your company is hooked on the ease and speed and cheapness of just spinning up another container or VM any time someone expresses a desire, or even automatically, when you get the bill at the end of the month and you managed to spend how much? That's ridiculous. Why are we getting billed for containers that we don't even need to run all the time? Along comes "serverless", which is a logical successor of inetd. Yes, inetd, the very old "internet super-server" which would read a table of ports and programs, open all the ports, and when a connection came in would run your program and connect stdin/out to the socket, isn't that great? Serverless is just a management system that spins up your very restricted single-function program on demand -- now it answers an HTTPS API instead of a raw socket -- and does the accounting work to make it profitable. Complexity is simultaneously the enemy of correctness and the source of profit.

taneq 8y ago

> QA has been gutted by TDD Wow. Just wow.

rcaught 8y ago
It is probably fair to say that CI/CD have gutted QA though.
- pjmorris 8y ago
  It is probably fair to say that managers have gutted QA.

sailfast 8y ago

That S-Curve on cloud seems off. Most Federal agencies are on or moving to Cloud services right now. They are squarely toward the top right of the S. I don’t think you can say only “large trailblazers” are there. We can argue about how many services have moved and how it’s being used but the whole “let someone else manage it” argument is long done and normalized.

laurentl 8y ago
A lot of statements in the article seem off. The position on the S-curve, the claim that TDD replaced QA teams, even the fact that “serverless is coding for hardware engineer” or however it was formulated (the author probably meant “infrastructure as code”). With so many approximations in the argument, it’s hard to accept the conclusions of the article at face value.
Pokepokalypse 8y ago
right, but you basically can't get a job there, unless you live in Virginia.

hestefisk 8y ago

“server admin is very hard, and mostly gone with virtualization.“ I would say this is simply not true. For many companies, getting good admins and infrastructure experts this is extremely hard, at least in big corporate.

ldoughty 8y ago
Completely agree. Server admins are still super important. Not everything fits in a nice server-less box. Developers also tend to have issues with containers, and firewalls, and monitoring/troubleshooting (depending on their skills and interest). People whom enjoy the server admin side can still find rewarding work in a more serverless environment
ldoughty 8y ago
Most developers I know don't understand containers well... Or they understand containers but not the underlying implications. "What does yum/apt upgrade have to do with node? Why should I run that?".. hell developers typically need a hard-* server guy to remind them to run package updates/upgrades on their languages package manager.. I'm not much of a developer but I end up doing a lot of the Npm and pip updates.

andmarios 8y ago

Every year cloud providers announce tens of new services, poorly tested and far from production ready. Non-ops eng teams and management are always happy to use the latest buzzword out of AWS, then ask their ops team to make it somehow work. Certifications are moot. I don't care if you are certified, or if you have used a cloud (or kubernetes) following a tutorial. I do care if you have seen enough issues throughout your career, so you can debug every time (not when, nor if) mud hits the fan. I don't care if you can type a port in a yaml file or in your browser. I do care if you know how many ports are there, what's the ephemeral port range and so on. Systems are 75% experience, 25% basic knowledge, you can't get away just with certifications which matter only after you get experience and knowledge. If you are experienced in ops, I know you can learn and work out any cloud. If you claim you are experienced with a specific cloud though, then better have to share a ton of war stories or it will just be a huge red flag on your CV. Let's call it for what it is, clouds, like data-science are hot fields right now. It's only natural that many people will go after a good-paying job claiming expertise in the field.

scarface74 8y ago
If you have experience in ops and don’t know the specifics of how to do operations in the cloud and take that same mentality to A cloud provider, you are going to end up spending more and not get any benefits from being with AWS/GCP/Azure.
- andmarios 8y ago
  Indeed, but ops people have to learn new technologies all the time, so it's not that big of a leap. When you talk about a cloud and see a sysadmin rolling their eyes, it's not that they have something specifically against the cloud, it's that they've seen so many diverse bugs and issues in their life, that they know you just exchange one set of problems for another. Also, a challenge to using clouds more effectively is you need to bring onboard your developers. Ops people often have to act as coaches and basic systems' knowledge is very important in this role.
  - scarface74 8y ago

heyjudy 8y ago

Disclaimer: ex preferred partner early AWS client-facing Fortune 200 consultant. The senior sysadmin at the DMDC around mid California coast: only one certification, VMware. A French offensive exploits broker, solid researcher and reeigne: no certs. I only have a few certs because work paid tens of thousands for offsite training. A few certs are okay because corporate HR wants staff to demonstrate continual learning and achievement, lots of low-quality ones (A+) are a negative signal. It's more important to keep up on news, tools and techniques on the front lines. My opinion is managing \aaS needs solid rack&stack ops experience at scale. For IT folks: either start huge if you can or start small and move towards huge. The biggest challenges are customer communication, managing expectations, anticipating and planning where possible and showing empathy... it's a 5% technical skill and 95% getting along productively game. Communicate the business impact, don't overwhelm customers with unnecessary* detail.

coleca 8y ago

I did some freelancing cloud work recently and was shocked at what I saw out there. From small one person operations to startups to large multinational Fortune 500 corporations I saw the same pattern repeated over and over. People using the cloud to spin up infrastructure that have no experience building out infrastructure and making really bad and dangerous mistakes. Amazon, Google, Microsoft and others make it simple to get going but the devil is in the details when it comes to building it right, safe and ready for production. Just a few examples: • Putting RDS SQL Servers on a public IP with no protection • No templating of servers so if it disappears you have no record of how to get a new one running again • Servers with SSH password authentication turned on with passwords of “Password”, because SSH key auth was “too hard for our devs” • No backups because “it’s in the cloud, isn’t the cloud backing it up for me, like iCloud?” I have an AWS Solution Architect Professional certification and agree you could get a cert and still not be qualified to run or design much on AWS but it does put you ahead of much of what I’ve seen out there.

nicoburns 8y ago
Oh wow. Those are some bad ones! I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).
- ldoughty 8y ago
  Cert + personal project, that's my recommendation. Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS. For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions. I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)

lifeisstillgood 8y ago

I read this as "develop and sell a niche tool that's useful in cloud ops or transformations, and market it like hell"

halbritt 8y ago

I really appreciate this part: Some of these firms realize they need to renovate their IT but they attempt to do this through hiring “Digital Transformation” roles which do little but frustrate the revolving door of hires because they have no executive buy- in and budget. This kind of thing makes it easier for me to hire. My org swallowed the risk and moved our production infrastructure entirely to k8s. I've no expectation that I'm going to be able to hire engineers with years of experience with that, but my team has already cultivated those skills. Aside from all the benefits and pains of being on k8s has, it makes it easy to attract great engineers with lots of related experience looking to leave a company where they've heard they need to adopt something and fail to put the resources behind it.

ilaksh 8y ago

Really interesting article. This line was funny though: "can remember the differences between a stack and a heap".

mancerayder 8y ago

Typically we have shortening 5-year cycles in tech that cause clamoring for new skills. First it was the internet (“HTML! Perl! PHP!”) then mobile apps (“Android! Objective C! React Native! Flutter!”) and now cloud (“AWS Certification! DevOps! SecOps!”). IoT and machine learning are probably next. The technology itself doesn’t really matter — companies just want an army of tactical workers to build this stuff tirelessly. It's Kubernetes and microservices! Even though Kubernetes is what, four years old, where I am, numerous employers are chomping at the bit and demanding that you be an expert at these. Now, I'm a crusty old Linux sysadmin/programmer who puts "SRE" and "DevOps" on his resume, and I'm noticeably getting spurned because I don't have "5 years experience" with "containers". At first I was worried that my skills were obsoleted, having worked in large enterprise environments doing infrastructure engineering for over 15 years. I worried that I was too "systemsy" and not enough "cloud-y". Oh, and even though I spent years using TeamCity and Jenkins, I was also spurned because I didn't use the right lingo within "CI/CD pipelines" and demonstrate a Docker deployment system exactly like they used. Then I realized something. The job descriptions are asking for five years experience total, number one, and the second red flag: the salaries and contract rates are lower than the rates for large enterprise-y type jobs (minus management). Conclusion: hipster tech coincides with youth and coincides with companies following trends and buzzwords over wisdom and experience. Back to the point. Someone with a strong basis in the core disciplines of systems design, infrastructure automation, perimeter security and so forth are going to be able to adapt to AWS knowledge, as I have (for the record I have two AWS Associate certs and didn't find them hard). And as the author points out, there are these 5 year cycles where all we look for is experience in certain tech. Is it any wonder that there are horror stories about fragile and insecure cloud environments? It goes back to the incompetence of the hiring process which we see again and again discussed on HN.

RickJWagner 8y ago

29 year IT veteran here. I don't agree with a lot of this article. I remember panic-talk when outsourcing began. When offshoring arose. Etc. None of these extinguished the need for IT workers. I view the cloud as the next generation of the mainframe. By that, I mean your experience and understanding of the environment (nobody will master all of it) will be a career differentiator. Knowledge of Kubernetes, Istio, etc. will help you to navigate the next career paths. System admin skills (and associated extras, like networking) will likewise move you up the ladder. Bill Gates once said there is an infinite number of problems yet to be solved, an infinite amount of work to be done. I believe him.

tabtab 8y ago

This is an age-old problem in IT. Some new technology and/or fad comes along, and bunches of organizations suddenly want an expert in it. It's impossible to find enough experienced in the new field because it's so new and/or growing too fast. But the HR department or technically-inexperienced managers don't understand this and complain to politicians et. al. of a "shortage". (Those in know have seen the pattern multiple times.) The answer is simple: find a similar field and accept a learning curve for the employee to transition. If you need something quick, hire a consultant to both move the project along and to mentor the new employee.