4 ms·

I recently did a study on how AI might be affecting the demand of software engineers. I took a dataset of 20M job postings from big and small startups. I then

I recently did a study on how AI might be affecting the demand of software engineers. I took a dataset of 20M job postings from big and small startups. I then used a ML classifier to classify each engineering job to the type of engineer it belonged to (ie backend, ML, date, mobile etc). In addition, I extracted the skills listed in each job using entity extraction. I indexed all of this to an elasticsearch cluster when I ran aggregation queries to see how the # of jobs changed for each type of engineer in the past year. Here were my findings: 1. The # of jobs for ML engineers and research scientists increased 70% from a year ago 2. # of jobs for frontend, mobile and data engineers all dropped 25% from a year ago, while backend engineers did a bit better dropping at 12% 4. Data scientist jobs and security engineer jobs did a bit better, both declining 5% or so 5. There seemed to be no correlation between tech layoffs and the demand for more AI engineers/talent. I published my findings in more detail in the article. Of course correlation != causation but just trying to make sense of the data as best as I could. Everything I wrote is still just an opinion, but backed by some data :)

snakeboy 3y ago
I wouldn't be surprised if the 5% drop in "Data Scientists" is from the same role being up-branded as something like "AI/ML Engineer".
- Xcelerate 3y ago
  Probably. My background is a PhD in the physical sciences + high performance computing, where I’ve pivoted to applying those techniques to machine learning. Depending on the current tech hype cycle, this background matches various job titles: data scientist, applied scientist, research scientist, quantitative engineer, machine learning engineer, research software engineer, scientific software engineer, etc. I typically go with whichever likely matches the most LinkedIn search filters of the day. Unfortunately, I can’t bring myself to put “AI” in my job title (despite the potential career benefits) because it just feels kind of icky. Maybe for a role that performs fundamental AGI research, but that’s about it.
qaq 3y ago
In a year of pretty sizable layoffs and hiring freezes that have 0 to do with AI ...
catchnear4321 3y ago
numbers always make things more fun. especially opinions. the fifth point tracks with subjective pulse checks in several places, which is roughly the following. “ai” isn’t at a point to replace developers, in fact it isn’t consistently enough to augment them. the squeeze is due to economic trends. market conditions. “ai” was seen as a potential solution. it is not snake oil, not yet panacea. so instead, time for everyone to get more efficient. do more with less. get 9h out of 8h days.
- beauzero 3y ago
  Anecdotal but we are in an AI freeze. I was, emphasize was, using copilot for over a quarter for development augmentation and seeing good velocity gains, our organization, now that it is all over the news (AI not our organization), has asked us to stop using everything except for one free search provider until they can come up with organizational rules. It is like walking through hip deep mud. The difference between this and dot com surge in 98-20 is that it happened within 6 months instead of 3-5 years.
blowski 3y ago
Fascinating! What numbers did you get in nominal terms?
jprete 3y ago
I think there genuinely isn't a correlation here. Big Internet tech has been close to saturation for a long time, we've been in a zero-interest-rate environment for a long time, and a lot of companies grossly overhired during COVID in the vain hope that the behavior changes were permanent. Shrinking the sector at this point is a natural response. GenAI finally showing potential for real value at the same time is not totally a coincidence; people who want high-yield investment opportunities need something to chase and it's natural to hype up whatever is available. Crypto was clearly on the downswing and GenAI was new and had potential. So of course the sector is going to get flooded with investment cash, both inside of companies and in the startup ecosystem.
wmoxam 3y ago
AI is hot. Its seems like every company wants to sprinkle some 'AI magic' to pump their valuations. Cheap money fueled SASS startup are dying, leading to reductions in demand for other roles. Larger more established SASS companies are shedding employees added during the pandemic too.

snakeboy 3y ago

I wouldn't be surprised if the 5% drop in "Data Scientists" is from the same role being up-branded as something like "AI/ML Engineer".

Xcelerate 3y ago
Probably. My background is a PhD in the physical sciences + high performance computing, where I’ve pivoted to applying those techniques to machine learning. Depending on the current tech hype cycle, this background matches various job titles: data scientist, applied scientist, research scientist, quantitative engineer, machine learning engineer, research software engineer, scientific software engineer, etc. I typically go with whichever likely matches the most LinkedIn search filters of the day. Unfortunately, I can’t bring myself to put “AI” in my job title (despite the potential career benefits) because it just feels kind of icky. Maybe for a role that performs fundamental AGI research, but that’s about it.

qaq 3y ago

In a year of pretty sizable layoffs and hiring freezes that have 0 to do with AI ...

catchnear4321 3y ago

numbers always make things more fun. especially opinions. the fifth point tracks with subjective pulse checks in several places, which is roughly the following. “ai” isn’t at a point to replace developers, in fact it isn’t consistently enough to augment them. the squeeze is due to economic trends. market conditions. “ai” was seen as a potential solution. it is not snake oil, not yet panacea. so instead, time for everyone to get more efficient. do more with less. get 9h out of 8h days.

beauzero 3y ago
Anecdotal but we are in an AI freeze. I was, emphasize was, using copilot for over a quarter for development augmentation and seeing good velocity gains, our organization, now that it is all over the news (AI not our organization), has asked us to stop using everything except for one free search provider until they can come up with organizational rules. It is like walking through hip deep mud. The difference between this and dot com surge in 98-20 is that it happened within 6 months instead of 3-5 years.

blowski 3y ago

Fascinating! What numbers did you get in nominal terms?

jprete 3y ago

I think there genuinely isn't a correlation here. Big Internet tech has been close to saturation for a long time, we've been in a zero-interest-rate environment for a long time, and a lot of companies grossly overhired during COVID in the vain hope that the behavior changes were permanent. Shrinking the sector at this point is a natural response. GenAI finally showing potential for real value at the same time is not totally a coincidence; people who want high-yield investment opportunities need something to chase and it's natural to hype up whatever is available. Crypto was clearly on the downswing and GenAI was new and had potential. So of course the sector is going to get flooded with investment cash, both inside of companies and in the startup ecosystem.

wmoxam 3y ago

AI is hot. Its seems like every company wants to sprinkle some 'AI magic' to pump their valuations. Cheap money fueled SASS startup are dying, leading to reductions in demand for other roles. Larger more established SASS companies are shedding employees added during the pandemic too.