5 ms·

It would be nice if the story touched on compensation levels. If all of those companies offered $400k/year, with guaranteed employment for 7 years, they would

It would be nice if the story touched on compensation levels. If all of those companies offered $400k/year, with guaranteed employment for 7 years, they would suddenly find no shortage of data scientists. Although other industries might suddenly wonder what happened to all their statisticians / physicists / etc. And two years later there would be a glut of data scientists who had just completed their master's degree.

wenc 8y ago
Not every labor shortage problem is solved by adjusting compensation levels upwards. The compensation has to match the value created.
- bobmarley1 8y ago
  Well not according to the people who constantly talk about Supply and Demand. If there's a massive supply shortage as implied there ought to be a corresponding increase in Compensation no?
- jlelonm 8y ago
  Why is that, economically speaking? If there's a labor shortage, doesn't that mean it's undervalued in the first place? Forgive me if that sounds naive, I'm no economist.
  - wenc 8y ago
    Because it makes the assumption that the system has no constraints, which is often untrue. First of all, you have to ask yourself why there is a labor shortage. If the underlying reason is insufficient pay, then sure, increasing pay will fix it. This is often the case for fungible talent -- incentivize something enough, and people will shift resources to it. However in many situations, certain types of work are not fungible. There are natural barriers to entry and qualification issues. Surgeons, for instance. You can incentivize and compensate all you want, but the fact is, not everyone is cut out to be a surgeon, so you have a funneling effect. It's not even a matter of pay. Then there's desirability issues. Deep sea welding is a highly specialized (and dangerous) trade that pays handsomely, but not everyone wants to do it. Some physically demanding jobs also have natural attrition issues in spite of compensation. When the inflow of talent is smaller than outflow over a long period of time, a shortage results. There's also a training and timing issues. Let's say it's 2011 you want someone who can build the kind of infrastructure that powers Netflix, in 6-12 months. The talent and experience pipeline would still have been brewing at the time, so there's going to be a temporary shortage of talent and experience until folks gain experience and mature in the field. In that instance, you can still hire and grow personnel over time, but there would still be a shortage of existing talent. Talent pipelines take time to build, are highly dependent on talent pools available and the ecosystems around it. Just throwing money at the problem doesn't always work. Compensation is just one end of it (the opportunity end); there are also significant long-term investments needed on the other end (the cradle end, which includes education, development, etc.). Texas Instruments did this -- they funded what eventually became UT Dallas... but it took many many years and the outcome was uncertain. There's geographic issues. To use an extremely unlikely example, let's say you wanted someone at the level of Jeff Dean or Sanjay Ghemawat, but you would need them to relocate to Podunk, Iowa. Very few people at that level would want to relocate for any amount of pay, hence a shortage. I realize this is a pretty extreme example, but I worked at a company in an undesirable part of the country and it was difficult to get truly talented folks to move out there even with significant premiums on compensation. Now you can keep increasing compensation until you get someone who's willing to move, but they're usually not the kind of talent you were looking for in the first place. Also there's a break-even point at which the compensation doesn't make sense for the value the position is likely to generate, so companies will just not hire. My point is there are all kinds of real and complex reasons why increasing compensation alone will not solve all shortage problems.
    - DoofusOfDeath 8y ago
      Thanks for the great rundown of some factors! I would like to take issue with just one part: > Very few people at that level would want to relocate for any amount of pay, hence a shortage. I'm not sure if you're using hyperbole or not. But I'm guessing that if you paid $1 million USD / year, there would be no practical shortage of data scientists willing to live in Podunk, Iowa.
      - wenc 8y ago
        There is a cost-benefit proposition to every hire. Unfortunately, the issue there is that the majority of data scientists don't generate $1 mil of value (exceptions exist of course), and so it's difficult to justify that level of compensation to management, which means the position may never get created in the first place. Absent an existential threat to their bottom lines, companies will just muddle on without hiring. This may be different in high growth companies, but in most traditional companies that aren't sloshing around in VC cash, to increase headcount you need to provide a value justification vis-a-vis salary (unless it's for cost center positions, but even then..) There's also an underlying assumption that talent flocks to the highest bidder.... but most HR folks will tell you that it's more complex than that. There are many quality of life issues that come into play, like weather, peer-group, spousal happiness, etc. Money doesn't buy everything, and humans aren't optimizers but satificers. I'm a sample of one, but for me, I would absolutely not move to Podunk, IA for a $1mil salary. I'm happy taking a lower salary living in a city and intellectual milieu that feeds me. The kind of person who would take the $1mil may likely not be the kind of talent you want.
      - rabidrat 8y ago
        It does not matter how much you pay, there are only so many 'qualified' data scientists to go around. If the wages are so ridiculously high, then there will be all manner of hucksters pitching themselves as data scientists with minimal training (but good enough to pass an interview), and they will give data scientists a bad name. This will make paying high-wages a super risky move, which almost never pays off, putting downward pressure on demand and wages. This is already happening.
  - deleted 8y ago
    [deleted]
- randomdata 8y ago
  > Not every labor shortage problem is solved by adjusting compensation levels upwards. Quite literally all shortages are solved by rising price. After all, the formal definition of a shortage is "a situation where an external mechanism, such as government intervention, prevents price from rising". If the price can rise, then a shortage cannot be. Which is the case here. If you have the budget for $1m/year compensation, it's yours to offer. Think about it this way: If there was only one person in the world capable of being a data scientist, each company in need of a data scientist would increase their offers to try and attract this person to their company. But no company in existence has an unlimited budget. Eventually all companies will be priced out of the market except for the company who ultimately employs this person. Since the other companies stop looking for data scientists beyond a certain price, the one position that remains in the market is perfectly filled with the one person available to do such work. To put it another way: No matter how much you want one, there isn't a shortage of Ferraris because you cannot afford one. You simply aren't in the market for one in the first place. Everyone who is in the market for a Ferrari can have one. Likewise, everyone who is in the market for a data scientist can have one. The mid-level exec with a data science poster on his wall, but a small budget for such work, is no more in the market for a data scientist than a teenager with a Ferrari poster on his wall is in the market for a Ferrari. That's just hopes and dreams. They don't count as anything more.
  - pyrale 8y ago
    Try to replace ferraris with food and explain that during a famine, most people are simply not in the market for eating. Your definition is a circular piece of logic defined to justify an ideology. The word 'shortage' has been used for a while. Creatin a definition that disqualifies its use is simply disingenuous.
    - dhillaoeu 8y ago
      Appealing to emotion doesn't work... for both of you. "Solving the problem" is not a well defined. Although it was a pleasure to read this exercise in persuasion.
      - 8y ago
- ummonk 8y ago
  If data scientists don't create a whole lot of value, then I wouldn't worry too much about a shortage in data scientists.

wenc 8y ago

Not every labor shortage problem is solved by adjusting compensation levels upwards. The compensation has to match the value created.

bobmarley1 8y ago
Well not according to the people who constantly talk about Supply and Demand. If there's a massive supply shortage as implied there ought to be a corresponding increase in Compensation no?
jlelonm 8y ago
Why is that, economically speaking? If there's a labor shortage, doesn't that mean it's undervalued in the first place? Forgive me if that sounds naive, I'm no economist.
- wenc 8y ago
  Because it makes the assumption that the system has no constraints, which is often untrue. First of all, you have to ask yourself why there is a labor shortage. If the underlying reason is insufficient pay, then sure, increasing pay will fix it. This is often the case for fungible talent -- incentivize something enough, and people will shift resources to it. However in many situations, certain types of work are not fungible. There are natural barriers to entry and qualification issues. Surgeons, for instance. You can incentivize and compensate all you want, but the fact is, not everyone is cut out to be a surgeon, so you have a funneling effect. It's not even a matter of pay. Then there's desirability issues. Deep sea welding is a highly specialized (and dangerous) trade that pays handsomely, but not everyone wants to do it. Some physically demanding jobs also have natural attrition issues in spite of compensation. When the inflow of talent is smaller than outflow over a long period of time, a shortage results. There's also a training and timing issues. Let's say it's 2011 you want someone who can build the kind of infrastructure that powers Netflix, in 6-12 months. The talent and experience pipeline would still have been brewing at the time, so there's going to be a temporary shortage of talent and experience until folks gain experience and mature in the field. In that instance, you can still hire and grow personnel over time, but there would still be a shortage of existing talent. Talent pipelines take time to build, are highly dependent on talent pools available and the ecosystems around it. Just throwing money at the problem doesn't always work. Compensation is just one end of it (the opportunity end); there are also significant long-term investments needed on the other end (the cradle end, which includes education, development, etc.). Texas Instruments did this -- they funded what eventually became UT Dallas... but it took many many years and the outcome was uncertain. There's geographic issues. To use an extremely unlikely example, let's say you wanted someone at the level of Jeff Dean or Sanjay Ghemawat, but you would need them to relocate to Podunk, Iowa. Very few people at that level would want to relocate for any amount of pay, hence a shortage. I realize this is a pretty extreme example, but I worked at a company in an undesirable part of the country and it was difficult to get truly talented folks to move out there even with significant premiums on compensation. Now you can keep increasing compensation until you get someone who's willing to move, but they're usually not the kind of talent you were looking for in the first place. Also there's a break-even point at which the compensation doesn't make sense for the value the position is likely to generate, so companies will just not hire. My point is there are all kinds of real and complex reasons why increasing compensation alone will not solve all shortage problems.
  - DoofusOfDeath 8y ago
    Thanks for the great rundown of some factors! I would like to take issue with just one part: > Very few people at that level would want to relocate for any amount of pay, hence a shortage. I'm not sure if you're using hyperbole or not. But I'm guessing that if you paid $1 million USD / year, there would be no practical shortage of data scientists willing to live in Podunk, Iowa.
    - wenc 8y ago
      There is a cost-benefit proposition to every hire. Unfortunately, the issue there is that the majority of data scientists don't generate $1 mil of value (exceptions exist of course), and so it's difficult to justify that level of compensation to management, which means the position may never get created in the first place. Absent an existential threat to their bottom lines, companies will just muddle on without hiring. This may be different in high growth companies, but in most traditional companies that aren't sloshing around in VC cash, to increase headcount you need to provide a value justification vis-a-vis salary (unless it's for cost center positions, but even then..) There's also an underlying assumption that talent flocks to the highest bidder.... but most HR folks will tell you that it's more complex than that. There are many quality of life issues that come into play, like weather, peer-group, spousal happiness, etc. Money doesn't buy everything, and humans aren't optimizers but satificers. I'm a sample of one, but for me, I would absolutely not move to Podunk, IA for a $1mil salary. I'm happy taking a lower salary living in a city and intellectual milieu that feeds me. The kind of person who would take the $1mil may likely not be the kind of talent you want.
    - rabidrat 8y ago
      It does not matter how much you pay, there are only so many 'qualified' data scientists to go around. If the wages are so ridiculously high, then there will be all manner of hucksters pitching themselves as data scientists with minimal training (but good enough to pass an interview), and they will give data scientists a bad name. This will make paying high-wages a super risky move, which almost never pays off, putting downward pressure on demand and wages. This is already happening.
- deleted 8y ago
  [deleted]
randomdata 8y ago
> Not every labor shortage problem is solved by adjusting compensation levels upwards. Quite literally all shortages are solved by rising price. After all, the formal definition of a shortage is "a situation where an external mechanism, such as government intervention, prevents price from rising". If the price can rise, then a shortage cannot be. Which is the case here. If you have the budget for $1m/year compensation, it's yours to offer. Think about it this way: If there was only one person in the world capable of being a data scientist, each company in need of a data scientist would increase their offers to try and attract this person to their company. But no company in existence has an unlimited budget. Eventually all companies will be priced out of the market except for the company who ultimately employs this person. Since the other companies stop looking for data scientists beyond a certain price, the one position that remains in the market is perfectly filled with the one person available to do such work. To put it another way: No matter how much you want one, there isn't a shortage of Ferraris because you cannot afford one. You simply aren't in the market for one in the first place. Everyone who is in the market for a Ferrari can have one. Likewise, everyone who is in the market for a data scientist can have one. The mid-level exec with a data science poster on his wall, but a small budget for such work, is no more in the market for a data scientist than a teenager with a Ferrari poster on his wall is in the market for a Ferrari. That's just hopes and dreams. They don't count as anything more.
- pyrale 8y ago
  Try to replace ferraris with food and explain that during a famine, most people are simply not in the market for eating. Your definition is a circular piece of logic defined to justify an ideology. The word 'shortage' has been used for a while. Creatin a definition that disqualifies its use is simply disingenuous.
  - dhillaoeu 8y ago
    Appealing to emotion doesn't work... for both of you. "Solving the problem" is not a well defined. Although it was a pleasure to read this exercise in persuasion.
    - 8y ago
ummonk 8y ago
If data scientists don't create a whole lot of value, then I wouldn't worry too much about a shortage in data scientists.