4 ms·

Industry wide, not even in startups, the people who get screwed the most are the ones working under the hood. I am not even talking about the disparity between

Industry wide, not even in startups, the people who get screwed the most are the ones working under the hood. I am not even talking about the disparity between SWEs and C-suites. I'm talking within the engineering domain. The closer you are to the application side the more credit you get for the effort, even if it might have only been possible through the hard work of recent under the hood technological advances. A prime example is deep learning. The people getting all the credit are the AI researchers. Now I am not trying to discredit them. But a huge part of what made the recent training of large models possible are all the other advances. All the dedicated PyTorch/C++/CUDA/MPI code that is written just to be able to scale on a HPC cluster, for example. Those people are handsomely rewarded, but they aren't getting the Chief AI scientist recognition, despite it probably being just as hard if not harder to write all the optimized code to enable and facilitate LLMs in the first place. Before I got into deep learning I was into numerics. Numerics people get typical SWE salaries, but they aren't getting FANNG level salaries, even though their code is powering the simulations across whole industries that ultimately enable many of them to operate at all. Now I am sure somebody will debate me on the "getting screwed" part, but it sure feels as if, the ones who use a technical library and build an application on top of it, get the recognition both for their own work and also get part of the recognition that should due-fully be bestowed on the author of the library. But maybe this is how the entirety of society works, but one should still be conscious of this phenomenon. If we all stand on the shoulders of giants, then maybe we should also name the giants in the acknowledgements section.

dogman144 3y ago
I think this depends on what you view as “getting screwed.” If the under the hood line engineers aren’t getting wild compensation for products they built generating wild compensation for the engineers/researchers working on the end product, yes that is a fair point. If it’s these engineers not getting the recognition of Chief AI scientist and so on, that’s misguided I think. There is nothing worse than working under a technical SME who doesn’t know leadership, and also “product,” broadly defined. I’ve worked on both sides of the issue and org leadership is a defined skill set the same way optimized code is. Unless the backend engineers display that leadership ability, their closeness to the internals of the technology doesn’t justify them running how the technology gets deployed. It is the same as if you found a startup. During the ideation sessions with whatever VC buys into you, there is 5% discussion of tech, or better yet - checkbox the product mvp works. 95% who are you going to sell this to and have you spoken to them? Different skills. When tech is a business, the tech knowledge starts to matter less, or is only a portion of the picture. The individuals that grasp that do well, SWEs who ignore this don’t IMO.
- dauertewigkeit 3y ago
  I don't think a CUDA engineer, or a researcher working on optimization or distributed computing should get the same recognition as a Chief of anything. But dare I say that writing optimized code to run on clusters is just as hard as coming up with a new deep learning architecture? Actually having done both, it is harder, at least for me. The authors of "Attention is all you need" got all the recognition. The people who worked on scaling deep learning models to huge clusters got no recognition. But both were just as instrumental to the ultimate success of LLMs.
  - dogman144 3y ago
    Recognition and leadership roles isn’t based on hardness of the completed task. Never has been. A good leader will make sure the credit is spread out though, yes. Why: - Really hard or really easy things can generate the same revenue - Chief of X can screw up the leadership with hard or easy products if they can’t lead and explain to external audiences - external audiences deal with a million things to be an audience to. It’s wishing on a dream that that audience will dig into “well who actually built this” vs moving on to w/e other thing is trying to capture their attention. Put that together, recognition comes from glueing things and people together successfully, explaining how it works to outside world/clients, and steering the product away from an optimized elegant solution to w/e people will actually use. Ever since tech stopped being academic/open source and got revenue linked to it, “thems the rules.” Speaking of SWE survival guides, engineers often seem to ignore, hate, or argue this topic always, and meanwhile the smaller set of engineers who get on board with this (ugly, yes) reality tend to do well.
smokel 3y ago
Only a few people who were in the right place at the right time and struck AI gold got lucky. Many PhDs have been slaving away and metaphorically died in several harsh AI winters. Nothing to be jealous of. There are several examples of people who got lucky with infrastructure or more technical engineering, but it is not because of their perceived role or something superficial.
- dauertewigkeit 3y ago
  That is part of the hero narrative that I am also lamenting. If there are 100 possible research directions and only 1 proves successful. The people who travel along the other 99 get no recognition, despite expending the same effort and despite their contribution being just as important.
  - PH95VuimJjqBqy 3y ago
    why is their contribution important? Is it your supposition that the only reason the person who went down the successful path did so purely because they watched everyone else fail? That seems highly unfair to the successful person.
    - bmitc 3y ago
      Success doesn't always imply the better approach. Also, failure is an important part of science and technology, and success is often driven by insights derived from failures.
      - PH95VuimJjqBqy 3y ago
        I have a successful career and my neighbors failures are not related to that in any way. you'll need to give a more strict definition of the word better before this can be a useful conversation.
        bmitc 3y ago

dogman144 3y ago

I think this depends on what you view as “getting screwed.” If the under the hood line engineers aren’t getting wild compensation for products they built generating wild compensation for the engineers/researchers working on the end product, yes that is a fair point. If it’s these engineers not getting the recognition of Chief AI scientist and so on, that’s misguided I think. There is nothing worse than working under a technical SME who doesn’t know leadership, and also “product,” broadly defined. I’ve worked on both sides of the issue and org leadership is a defined skill set the same way optimized code is. Unless the backend engineers display that leadership ability, their closeness to the internals of the technology doesn’t justify them running how the technology gets deployed. It is the same as if you found a startup. During the ideation sessions with whatever VC buys into you, there is 5% discussion of tech, or better yet - checkbox the product mvp works. 95% who are you going to sell this to and have you spoken to them? Different skills. When tech is a business, the tech knowledge starts to matter less, or is only a portion of the picture. The individuals that grasp that do well, SWEs who ignore this don’t IMO.

dauertewigkeit 3y ago
I don't think a CUDA engineer, or a researcher working on optimization or distributed computing should get the same recognition as a Chief of anything. But dare I say that writing optimized code to run on clusters is just as hard as coming up with a new deep learning architecture? Actually having done both, it is harder, at least for me. The authors of "Attention is all you need" got all the recognition. The people who worked on scaling deep learning models to huge clusters got no recognition. But both were just as instrumental to the ultimate success of LLMs.
- dogman144 3y ago
  Recognition and leadership roles isn’t based on hardness of the completed task. Never has been. A good leader will make sure the credit is spread out though, yes. Why: - Really hard or really easy things can generate the same revenue - Chief of X can screw up the leadership with hard or easy products if they can’t lead and explain to external audiences - external audiences deal with a million things to be an audience to. It’s wishing on a dream that that audience will dig into “well who actually built this” vs moving on to w/e other thing is trying to capture their attention. Put that together, recognition comes from glueing things and people together successfully, explaining how it works to outside world/clients, and steering the product away from an optimized elegant solution to w/e people will actually use. Ever since tech stopped being academic/open source and got revenue linked to it, “thems the rules.” Speaking of SWE survival guides, engineers often seem to ignore, hate, or argue this topic always, and meanwhile the smaller set of engineers who get on board with this (ugly, yes) reality tend to do well.

smokel 3y ago

Only a few people who were in the right place at the right time and struck AI gold got lucky. Many PhDs have been slaving away and metaphorically died in several harsh AI winters. Nothing to be jealous of. There are several examples of people who got lucky with infrastructure or more technical engineering, but it is not because of their perceived role or something superficial.

dauertewigkeit 3y ago
That is part of the hero narrative that I am also lamenting. If there are 100 possible research directions and only 1 proves successful. The people who travel along the other 99 get no recognition, despite expending the same effort and despite their contribution being just as important.
- PH95VuimJjqBqy 3y ago
  why is their contribution important? Is it your supposition that the only reason the person who went down the successful path did so purely because they watched everyone else fail? That seems highly unfair to the successful person.
  - bmitc 3y ago
    Success doesn't always imply the better approach. Also, failure is an important part of science and technology, and success is often driven by insights derived from failures.
    - PH95VuimJjqBqy 3y ago
      I have a successful career and my neighbors failures are not related to that in any way. you'll need to give a more strict definition of the word better before this can be a useful conversation.
      - bmitc 3y ago