11 ms·

Is There a U.S. IT Worker Shortage?

21 points by vonmoltke 13y ago

gaius 13y ago
It is easy to tell if there is a shortage of ANY commodity, including workers, from the price. This is one of the most basic principles of capitalism. Similarly it is easy to tell if there is an oversupply, the price goes down... Which matches the observations of the state of the industry (allowing for inflation)?
- AnIrishDuck 13y ago
  This is not true. There are many different types of information encoded in a market clearing price (supply of inputs, the supply of inputs to their inputs, utility of the output, economies of scale, etc). You can't say that e.g. a rise in the price of pencils means it's "easy to tell" there's a shortage of pencils. It could mean that: 1) more people are getting better utility from pencils. 2) pencils are getting harder to produce at the current scale. 3) an input supply to pencil construction is rising in price Applying these lessons back to the market for developers, it's pretty easy to see parallels to ALL three of the above scenarios. Which explains the rising price. Unless you're using the word "shortage" to mean "you can't get what you want at the price you want", in which case the statement is almost tautologically true.
  - deleted 13y ago
    [deleted]
  - gaius 13y ago
    That is the way it is being used in this context, as in companies saying "I can't get an experienced developer for the price of a new graduate".
- EliRivers 13y ago
  This is one of the most basic principles of capitalism. And, like many basic economic principles, only works in the artificial environment of a fictional, deliberately simplified economy. House prices, for example, have a very strong dependency on the cost (and availability) of credit; people will spend as much money as the bank is willing to lend them. Turn on credit, price of houses goes up. Close off credit, price of houses goes down. This is but one example of many; the extraordinarily simple model of a commodity price, "supply and demand", is not inherently wrong, but is so simple that to rely on it alone in the complex economy we have is wrong.
  - don_draper 13y ago
    I don't know what you are talking about. If there is a shortage in housing, prices go up; if there is an oversupply, prices go down. Credit affects what people can afford and plays a factor, but the basic rules of economics are very much in play with housing.
    - EliRivers 13y ago
      I don't know what you are talking about. I'm talking about the fact that to apply the extraordinarily simple supply and demand model to the price of houses, and expect it to work, is wrong. Other factors are more important. If we were talking about something very cheap and fungible, it would be far more applicable. We're talking about houses. The majority of people cannot afford to buy a house. They don't have enough money. If there was no credit, we'd see a simple supply/demand model at work; I could take 20 percent off the cost of houses, and most people still wouldn't be able to buy them because they don't have enough money, but a few more would be able to, so the demand (or rather, the number of people who want one and can now buy one) would increase. However, there is credit. If I added 20 percent to the price and also arranged to lend people enough money to buy the newly expensive houses, they would sell. If I doubled house prices and lent people the money, they'd still sell. The price of houses is so dependent on the availability of credit that it swamps the simple supply and demand model. Credit doesn't just play a factor; it's the majority of it. They're also a veblen good; the simple supply and demand idea that as prices go up, the demand will drop, isn't just wrong - it's the complete opposite. As prices go up, people want them more; they fear that if they don't buy one now, it'll cost them more in the future ("don't miss the boat" is what often gets trotted out during the housing booms) and/or they think of it as some kind of investment and think they'll make more money since they're going up. The simple model that prices going up leads to demand going down is inverted. This works as they go down in price too; as prices drop, demand drops. Nobody wants to borrow money against something that will soon be worth less than they owe on it. Dropping prices cause their own positive feedback, every drop pushing demand lower, in turn dropping prices further and amplifying the effect; classic boom and bust behaviour. Houses are a veblen good, dependent on the availability of credit to buy. Simple supply and demand is utterly swamped. Look at the last housing boom (and indeed, the ones before it). Was there a massive reduction in the number of houses available? No; in fact, many housing companies were building them as fast as they could and the supply was increasing even as prices bubbled beyond sanity. Did hordes of people who had been living on the streets suddenly decide they wanted to live in houses? No. It was people suddenly able to buy one because someone was willing to lend them the money.
- skylan_q 13y ago
  And as far as I'm concerned, there's an oversupply as long as I'm competing for any job with anyone.
Irene 13y ago
2011 Report by McKinsey predicted a 50-60% shortage of "big data" talent in the US by 2018. Later Gartner projected 2/3 talent shortage. But as productivity is increasing, the gap between supply and demand is easily filled. As Forbes Adam Ozimek said, there is no shortage just “a large demand for more workers at current market clearing wages but not a penny more”
- debacle 13y ago
  I think "Big Data" demand is going to continue to spike for a while now, but a lot of the big data projects I see these days seem somewhat aimless: "We're going to aggregate and analyze a ton of data, and then somehow that's going to provide a revelation that we're going to act on and make/save a ton of money." We already know what happens when companies do this kind of introspection - obvious problems are confirmed, and in most instances are either too difficult to fix, or any change wouldn't be cost effective. In my opinion, the "Big Data" demand is simply being driven by modern business managers who have an unhealthy lust for statistics. While I can't deny that there are some amazing applications of "Big Data" out there, a lot of what I'm seeing makes me say to myself "You wouldn't know what to do with that information, even if you had it."
microarchitect 13y ago
Some thoughts on this issue. Some of these have been expressed by others like pg already. 1. Workers are not a commodity and not interchangeable. You can't a take a network-layer programmer who got laid-off from Cisco and ask that person to work on big data and reasonably expect success. 2. I suspect a lot of tech workers are not motivated by money as long as it's above a threshold. I know for a fact that I've taken jobs that paid me as much as 25% less because I got to work on "cooler" stuff. This also fits in with the stories of Google's early days when they actually paid less in dollar terms than MS but were able to lure workers away from MS because they had better "perks" and more interesting technology. 2a. If progammers were optimizing for money, most of us would be working for wall Street, and we'd all be contracting in our free time instead of building open source apps. Clearly this isn't happening. 3. Anecdotally, I know for a fact there is huge demand for competent programmers/engineers because I have standing offers from multiple employers. The reason for this is that a good programmer is orders of magnitude more productive than an average programmer so good managers will move heaven and earth to get their hire. On top of this, an average programmer in a good team will have effective negative productivity. So restricting your supply pool even a little bit can leave you with a drastically less productive team.
- debacle 13y ago
  > an average programmer in a good team will have effective negative productivity. I have a qualm with that statement, I think. Do you have anything to back that up? It doesn't make logical sense to say that "If I have four programmers and they give me nine megawarbles of productivity, and I add a fifth programmer who can only create one megawarble of productivity, I will have a team that only generates eight megawarbles of productivity." Too often, programmers are drawn into this romantic idea of being a 10x programmer, but I can honestly say that, in my experience, there are more 10x programmers out there than there are 10x programming problems. The IT world needs more data janitors than it needs data scientists.
  - BerislavLopac 13y ago
    Programming is knowledge work. If someone is "worse" than another, it means that they less knowledge. If you have four programmers with a similar amount of knowledge, and one with much less knowledge, to gain more knowledge this one will have to go to the four, who will then spend time transferring knowledge instead of using it effectively. That's your Fred Brooks 101.
debacle 13y ago
Salaries for developers in most parts of the country have been flat for the last several years. More teams are trying to do without development managers and with fewer project managers. Network operations salaries, from what I have seen, are actually lower now than they were when I was in college. Network architects and engineers still command pretty good salaries, probably because there are so many legacy networks out there that need to be replaced. In general, salaries in product oriented positions are going up, and salaries in service oriented positions are going down. IT has one of the highest unemployment percentages of any industry (14.9%, last I read). While I don't believe there is an actual, honest to goodness shortage, I do believe that there are three critical problems in the industry right now: 1. Hiring - Companies can't hire the right people, good people can't get past HR, companies are afraid that they're going to hire the wrong person, and in general there are no professional organizations that are making this any easier. This has resulted in ridiculous salaries (in both directions) because companies don't know what they're paying for when they first hire someone, and are desperate not to lose the people who they have hired that know what they're doing. In addition, positions stay open for an inordinate amount of time because of the difficulty of hiring, and overall employee mobility is reduced because the interview process is slowly moving from a multi-day process to a multi-week process. I got my first programming job on a phone screen and a lunch break interview. That would be ludicrous today. 2. Training - Companies don't want to hire people who have 85% of the skillset they're looking for, because they don't believe that training for the extra 15% is going to pay off. I think this is partially because the value proposition of a new graduate with six months of solid experience is probably double of the same graduate before their experience. I think this is something not strictly limited to IT at the moment, but I'm seeing it more in IT than in any other engineering discipline. 3. Placement - In short, people aren't getting positions that are most suited to their abilities. Unskilled, first-time "CIO"s are running startups into the ground, and former "CIO"s with plenty of experience are consulting because consulting, right now, actually seems more low-risk than being an employee. I see more and more of my friends moving over to consulting not because of the money, but because they were switching employers so much before they started consulting that it doesn't make sense not to go that route. Keep in mind that I am providing minimal evidence - although I know that the 14.9% employment rate is recently accurate, within a few weeks, but in general I'm seeing these trends everywhere and it's worrying as someone whose job it is to make systems function well simply because there is so much waste.
skylan_q 13y ago
This keeps coming up and I have to keep asking, what is a shortage? If we double the amount of IT workers available, do we still have a shortage? How do we determine what a shortage is? I think it's all perspective. From the manager/business owner who is looking for the lowest cost and highest-payout labor, there is a shortage. To the actual laborer who is competing with other laborers, there is an oversupply. So this talk about there being a shortage like it's an objective fact really strikes me as bizarre.

21 points by vonmoltke 13y ago

gaius 13y ago

It is easy to tell if there is a shortage of ANY commodity, including workers, from the price. This is one of the most basic principles of capitalism. Similarly it is easy to tell if there is an oversupply, the price goes down... Which matches the observations of the state of the industry (allowing for inflation)?

AnIrishDuck 13y ago
This is not true. There are many different types of information encoded in a market clearing price (supply of inputs, the supply of inputs to their inputs, utility of the output, economies of scale, etc). You can't say that e.g. a rise in the price of pencils means it's "easy to tell" there's a shortage of pencils. It could mean that: 1) more people are getting better utility from pencils. 2) pencils are getting harder to produce at the current scale. 3) an input supply to pencil construction is rising in price Applying these lessons back to the market for developers, it's pretty easy to see parallels to ALL three of the above scenarios. Which explains the rising price. Unless you're using the word "shortage" to mean "you can't get what you want at the price you want", in which case the statement is almost tautologically true.
- deleted 13y ago
  [deleted]
- gaius 13y ago
  That is the way it is being used in this context, as in companies saying "I can't get an experienced developer for the price of a new graduate".
EliRivers 13y ago
This is one of the most basic principles of capitalism. And, like many basic economic principles, only works in the artificial environment of a fictional, deliberately simplified economy. House prices, for example, have a very strong dependency on the cost (and availability) of credit; people will spend as much money as the bank is willing to lend them. Turn on credit, price of houses goes up. Close off credit, price of houses goes down. This is but one example of many; the extraordinarily simple model of a commodity price, "supply and demand", is not inherently wrong, but is so simple that to rely on it alone in the complex economy we have is wrong.
- don_draper 13y ago
  I don't know what you are talking about. If there is a shortage in housing, prices go up; if there is an oversupply, prices go down. Credit affects what people can afford and plays a factor, but the basic rules of economics are very much in play with housing.
  - EliRivers 13y ago
    I don't know what you are talking about. I'm talking about the fact that to apply the extraordinarily simple supply and demand model to the price of houses, and expect it to work, is wrong. Other factors are more important. If we were talking about something very cheap and fungible, it would be far more applicable. We're talking about houses. The majority of people cannot afford to buy a house. They don't have enough money. If there was no credit, we'd see a simple supply/demand model at work; I could take 20 percent off the cost of houses, and most people still wouldn't be able to buy them because they don't have enough money, but a few more would be able to, so the demand (or rather, the number of people who want one and can now buy one) would increase. However, there is credit. If I added 20 percent to the price and also arranged to lend people enough money to buy the newly expensive houses, they would sell. If I doubled house prices and lent people the money, they'd still sell. The price of houses is so dependent on the availability of credit that it swamps the simple supply and demand model. Credit doesn't just play a factor; it's the majority of it. They're also a veblen good; the simple supply and demand idea that as prices go up, the demand will drop, isn't just wrong - it's the complete opposite. As prices go up, people want them more; they fear that if they don't buy one now, it'll cost them more in the future ("don't miss the boat" is what often gets trotted out during the housing booms) and/or they think of it as some kind of investment and think they'll make more money since they're going up. The simple model that prices going up leads to demand going down is inverted. This works as they go down in price too; as prices drop, demand drops. Nobody wants to borrow money against something that will soon be worth less than they owe on it. Dropping prices cause their own positive feedback, every drop pushing demand lower, in turn dropping prices further and amplifying the effect; classic boom and bust behaviour. Houses are a veblen good, dependent on the availability of credit to buy. Simple supply and demand is utterly swamped. Look at the last housing boom (and indeed, the ones before it). Was there a massive reduction in the number of houses available? No; in fact, many housing companies were building them as fast as they could and the supply was increasing even as prices bubbled beyond sanity. Did hordes of people who had been living on the streets suddenly decide they wanted to live in houses? No. It was people suddenly able to buy one because someone was willing to lend them the money.
skylan_q 13y ago
And as far as I'm concerned, there's an oversupply as long as I'm competing for any job with anyone.

Irene 13y ago

2011 Report by McKinsey predicted a 50-60% shortage of "big data" talent in the US by 2018. Later Gartner projected 2/3 talent shortage. But as productivity is increasing, the gap between supply and demand is easily filled. As Forbes Adam Ozimek said, there is no shortage just “a large demand for more workers at current market clearing wages but not a penny more”

debacle 13y ago
I think "Big Data" demand is going to continue to spike for a while now, but a lot of the big data projects I see these days seem somewhat aimless: "We're going to aggregate and analyze a ton of data, and then somehow that's going to provide a revelation that we're going to act on and make/save a ton of money." We already know what happens when companies do this kind of introspection - obvious problems are confirmed, and in most instances are either too difficult to fix, or any change wouldn't be cost effective. In my opinion, the "Big Data" demand is simply being driven by modern business managers who have an unhealthy lust for statistics. While I can't deny that there are some amazing applications of "Big Data" out there, a lot of what I'm seeing makes me say to myself "You wouldn't know what to do with that information, even if you had it."

microarchitect 13y ago

Some thoughts on this issue. Some of these have been expressed by others like pg already. 1. Workers are not a commodity and not interchangeable. You can't a take a network-layer programmer who got laid-off from Cisco and ask that person to work on big data and reasonably expect success. 2. I suspect a lot of tech workers are not motivated by money as long as it's above a threshold. I know for a fact that I've taken jobs that paid me as much as 25% less because I got to work on "cooler" stuff. This also fits in with the stories of Google's early days when they actually paid less in dollar terms than MS but were able to lure workers away from MS because they had better "perks" and more interesting technology. 2a. If progammers were optimizing for money, most of us would be working for wall Street, and we'd all be contracting in our free time instead of building open source apps. Clearly this isn't happening. 3. Anecdotally, I know for a fact there is huge demand for competent programmers/engineers because I have standing offers from multiple employers. The reason for this is that a good programmer is orders of magnitude more productive than an average programmer so good managers will move heaven and earth to get their hire. On top of this, an average programmer in a good team will have effective negative productivity. So restricting your supply pool even a little bit can leave you with a drastically less productive team.

debacle 13y ago
> an average programmer in a good team will have effective negative productivity. I have a qualm with that statement, I think. Do you have anything to back that up? It doesn't make logical sense to say that "If I have four programmers and they give me nine megawarbles of productivity, and I add a fifth programmer who can only create one megawarble of productivity, I will have a team that only generates eight megawarbles of productivity." Too often, programmers are drawn into this romantic idea of being a 10x programmer, but I can honestly say that, in my experience, there are more 10x programmers out there than there are 10x programming problems. The IT world needs more data janitors than it needs data scientists.
- BerislavLopac 13y ago
  Programming is knowledge work. If someone is "worse" than another, it means that they less knowledge. If you have four programmers with a similar amount of knowledge, and one with much less knowledge, to gain more knowledge this one will have to go to the four, who will then spend time transferring knowledge instead of using it effectively. That's your Fred Brooks 101.

debacle 13y ago

Salaries for developers in most parts of the country have been flat for the last several years. More teams are trying to do without development managers and with fewer project managers. Network operations salaries, from what I have seen, are actually lower now than they were when I was in college. Network architects and engineers still command pretty good salaries, probably because there are so many legacy networks out there that need to be replaced. In general, salaries in product oriented positions are going up, and salaries in service oriented positions are going down. IT has one of the highest unemployment percentages of any industry (14.9%, last I read). While I don't believe there is an actual, honest to goodness shortage, I do believe that there are three critical problems in the industry right now: 1. Hiring - Companies can't hire the right people, good people can't get past HR, companies are afraid that they're going to hire the wrong person, and in general there are no professional organizations that are making this any easier. This has resulted in ridiculous salaries (in both directions) because companies don't know what they're paying for when they first hire someone, and are desperate not to lose the people who they have hired that know what they're doing. In addition, positions stay open for an inordinate amount of time because of the difficulty of hiring, and overall employee mobility is reduced because the interview process is slowly moving from a multi-day process to a multi-week process. I got my first programming job on a phone screen and a lunch break interview. That would be ludicrous today. 2. Training - Companies don't want to hire people who have 85% of the skillset they're looking for, because they don't believe that training for the extra 15% is going to pay off. I think this is partially because the value proposition of a new graduate with six months of solid experience is probably double of the same graduate before their experience. I think this is something not strictly limited to IT at the moment, but I'm seeing it more in IT than in any other engineering discipline. 3. Placement - In short, people aren't getting positions that are most suited to their abilities. Unskilled, first-time "CIO"s are running startups into the ground, and former "CIO"s with plenty of experience are consulting because consulting, right now, actually seems more low-risk than being an employee. I see more and more of my friends moving over to consulting not because of the money, but because they were switching employers so much before they started consulting that it doesn't make sense not to go that route. Keep in mind that I am providing minimal evidence - although I know that the 14.9% employment rate is recently accurate, within a few weeks, but in general I'm seeing these trends everywhere and it's worrying as someone whose job it is to make systems function well simply because there is so much waste.

skylan_q 13y ago

This keeps coming up and I have to keep asking, what is a shortage? If we double the amount of IT workers available, do we still have a shortage? How do we determine what a shortage is? I think it's all perspective. From the manager/business owner who is looking for the lowest cost and highest-payout labor, there is a shortage. To the actual laborer who is competing with other laborers, there is an oversupply. So this talk about there being a shortage like it's an objective fact really strikes me as bizarre.