10 ms·

Why is machine learning 'hard'? (2016)

283 points by jxmorris12 3y ago

jruohonen 3y ago
"An aspect of this difficulty involves building an intuition for what tool should be leveraged to solve a problem." While I agree with the good point about debugging, like many others, I am rather worried that we're increasingly deploying AI/ML where we shouldn't be deploying it. Hence, the above quote.
- sjwhevvvvvsj 3y ago
  I’m old enough to have learned that the secret to success is much less knowing the tool of the moment than picking the right tool for a job. The right tool may in fact be the new one, and LLM do open a lot of doors with zero shot capabilities, but oftentimes they can underperform a well tuned heuristic. It’s the ability to pick the right tool that is key.
- wwarner 3y ago
  Want to agree with you, as so many ML apps seem to be solutions looking for problems. But I actually feel that we are rapidly deploying ML in a development context for vastly improved results. The way that good models are built relies on many ML steps, and when the results finally come together the result is superior to what could have been custom designed. Broad adoption of something like probabilistic programming is coming soon.
dang 3y ago
Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)
- epistasis 3y ago
  Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This embodies the very fundamental difference between science and engineering. With science, you make a discovery, but rarely do we ask "what was the magical combination that let me find the needle in the haystack today?" We instead just pass on the needle and show everyone we found it. Should we work on finding out the magic behind hyperparameters? In bioinformatics, the brilliant mathematician Lior Pachter once attacked the problem of sequence alignment using the tools of tropical algebra: what parameters to the alignment algorithms resulted in which regimes of solutions? It was beautiful. It was great to understand. But I'm not sure if it even ever got published (though it likely did). Having reasonable parameters is more important than understanding how to pick them from first principles, because even if you know all the possible output regimes for different segments of the hyper parameter space, really the only thing we care about is getting a functionally trained model at the end. Sometimes deeper understanding provides deeper insights to the problems at hand. But often, they don't, even when the deeper understanding is beautiful. If the hammer works when you hold it a certain way, that's great, but understanding all possible ways to hold a hammer doesn't always help get the nail in better.
  - amelius 3y ago
    The hammer analogy doesn't make much sense because for a hammer we can actually use our scientific knowledge to compute the best possible way to hold the tool, and we can make instruments that are better than hammers, like pneumatic hammers, pile drivers, etc. With your argument, we would be stuck with the good old, but basic hammer for the rest of time.
    - epistasis 3y ago
      That seems like a different analogy; making better hammers is a different thing than understanding why holding a hammer a certain way works well. We did eventually invent enough physics to understand why we hold hammers where we do, but we got really far just experimenting without first principles. And even if we use first principles, we are going to discover a lot more by actually using the modified hand-held hammer and testing it, than necessarily hitting it out of the park with great physical modeling of the hammer and the biomechanics of the human body. And in any case, I'm not saying we shouldn't search for deep understanding of what hyperparameters work on a first try, I'm just saying there's a good chance that even if the principles are fully discovered, it may be that calculating using those principles is more expensive than a bunch of experimentation and won't matter in the end. That's the trick about science, it's more about finding the right question to answer than how to find answers, and often times the best questions only become apparent afterwards.
  - sjwhevvvvvsj 3y ago
    I do a lot of model tuning and I’m almost ashamed to say I tell GPT what performance I’m aiming for and have it generate the hyper parameters (as in just literally give me a code block). Then I see what works, tell GPT, and try again. I’m deeply uncomfortable with such a method…but my models perform quite well. Note I spend a TON of time generating the right training data, so it’s not random.
    - danielmarkbruce 3y ago
      1/8th (soon to be 1/2) of the working world: "I do a lot of X and I'm almost ashamed to say I tell GPT Y then I see if it works and try again".
- naveen99 3y ago
  Similar: Machine learning is still too hard for software engineers - https://news.ycombinator.com/item?id=30432987 https://news.ycombinator.com/item?id=30432987 - 151 comments Machine learning is easier than it looks - https://news.ycombinator.com/item?id=6770785 https://news.ycombinator.com/item?id=6770785 167 comments
doctorM 3y ago
I'm a bit sceptical of the exponentially harder debugging claim. First it looks polynomially harder for the given example :p. Second other engineering domains arguably have additional dimensions which correspond to the machine learning ones mentioned in the article. The choice of which high level algorithm to implement is another dimension to traditional software engineering that seemingly exists and corresponds to the model dimension. This is often codified as 'design'. The data dimension often exists as well in standard learning software engineering. [Think of a system that is 'downstream' of other]. It's probably a lot simpler to deal with these dimensions in standard software engineering - but then this is what makes machine learning harder, not that there are simply 'more dimensions'. The delayed debugging cycles point seems a lot more valid.
- pizzaknife 3y ago
  i would subscribe to your newsletter if you offered one.
- jolt42 3y ago
  The article also pretends that there is only one correct answer, which seems atypical of the domain. The 1 green spot should extend somewhat fuzzily in each dimension in the ML case.
  - janalsncm 3y ago
    There isn’t only one correct answer. Quite the opposite actually, many configurations give you local maximas. The difficulty is that it can be hard to explain from first principles why one local maxima is good.
- grantpitt 3y ago
  I had similar thoughts on this. It looks polynomial because he classified errors into certain groups and spread them across an axis which we implicitly think is a dimension. But it's not... as he says, "Along any one dimension we might have a combination of issues (i.e. multiple implementation bugs)". So looking at the grid and thinking you're at a single point is wrong... but so is thinking you could be any configuration of locations (you couldn't be at points (1, 1) and (2, 2), you also need to be at points (1, 2) and (2, 1), i.e. the set of points you're at is transitive). Thus this is a bad way to "enumerate the failure cases" and makes his notion of "adding a dimension" pretty unintuitive. It makes more sense to see every point of his dimensions as itself a dimension. Each of these dimensions could be either correct or incorrect, 1 or 0. So you're configuration could be visualized as a string of 1's and 0's. Thus, the number of possible configurations grows exponentially (2^n).
PaulHoule 3y ago
The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)
- ramesh31 3y ago
  Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity.
  - rzzzz 3y ago
    I believe that Scale.ai was founded to do exactly this.
  - MichaelRo 3y ago
    >> Seems like a prime startup opportunity. Sometimes it's just ... hard. Apply some thought maybe before blindly parroting "profit!" Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to end World poverty?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to build a warp engine?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to wipe your ass using the left hand?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" You get the idea...
    - aleph_minus_one 3y ago
      > Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" What you want to optimize for is the money amount that you make at some quantile of the probablity distribution of the profits; say, the profits that are guaranteed in the best, say, 3 %, 5 %, 10 % or even 20 % of all possible outcomes. With a probablity of 97 % (if you choose the best 3 % of the outcomes), you won't make sufficient money if you attempt to cure cancer to be worth the risk, so the financial incentive is not there. TLDR: Financial incentives do matter, but work differently from how many people think that they are structured.
Cacti 3y ago
there are uncountable sets all over the place, and in practical terms, the repl loop may have a week long training lag after you hit enter. also, the data is almost always complete shit. lol. there’s no mystery why it’s hard.
phkahler 3y ago
>> It becomes essential to build an intuition for where something went wrong based on the signals available. This has always been my approach. I learned programming way before I had access to debuggers and other methods to dig in, set breakpoints and step through code to see where it was going wrong. As a result, when I got in the real world I kind of looked down on people using those tools (mostly because I hate tools actually). But then I saw people get to the root of problems that I don't think I ever could have solved, and I started to appreciate those tools and the detail you could get to. My preference is still to have a great understanding of how algorithms work, how the code is written, and what the problem is, and noodle out what and where things may be going wrong. I only switch to detailed monitoring of the insides when "thinking about it" fails. Maybe I should have gone into this ML stuff ;-)
deleted 3y ago
[deleted]
Xcelerate 3y ago
With regard to model selection, one thing I learned a long time ago that provides powerful intuitive guidance on which model to use is the question: "How could this be compressed further?" There are some deep connections between data compression and generalized learning, both at the statistical level and even lower at the algorithmic level (see Solomonoff induction). For a specific example at the statistical level, suppose you fit a linear trendline to some data points using OLS. Now compute the standard deviation of the residual terms for each data point, and using the CDF of the normal distribution, for each residual, map its value into the interval [0, 1]. Sum together the logarithms of the trendline coefficients, the standard deviation, and the normalized residuals. This value is approximately proportional to "sizeof(data | model) + sizeof(model)". It represents how well you compressed the data using the OLS model. But now suppose you plot the distribution of the residuals and find out that they do not in fact resemble a Gaussian distribution. This is a problem because our model assumed the error terms were distributed normally, and since this is not the case, our compression is suboptimal. So you back out some function f that closely maps between the uniform distribution on [0, 1] and the distribution that the residuals form and use this f to define a new model: yᵢ = m*xᵢ + b + εᵢ, with εᵢ distributed according to f(x;Θ), Θ being a parameter vector. When you sum the logarithms again, you will find that the new total is smaller than the original total obtained using OLS. The new trend line coefficients will slightly mess up the residual distribution again, so iterate on this process until you've converged on stable values for m, b, and Θ. At the algorithmic level, the recommendation to use compression as a model selection guide applies even to LLMs, but it's a bit harder to use in practice because "sizeof(data | model) + sizeof(model)" isn't the entire story here. Suppose you had a "perfect" language model. In this case, you would achieve minimization of K(data | training data), where K is Kolmogorov complexity. In practice, what is being minimized with each new LLM version is "sizeof(data | LLM model) + sizeof(training data | LLM model) + sizeof(LLM model)". You can assume "sizeof(LLM model)" is the smallest Turing machine equivalent to the LLM program.
sfink 3y ago
This article annoyed me. Not because I think it overestimates any of the difficulties, but it condescendingly compares against "standard software engineers" as if their problems are so much easier and lack additional dimensions beyond algorithm and implementation. It doesn't help that they call N^4 "exponential". Long debugging cycles are not new. In fact, the field started out that way, when computers were slow enough that even a single edit-compile-run cycle could take hours (or days and some political capital, if you go back far enough). Even today, long debugging cycles are far from being restricted to ML, especially for failures that are only seen intermittently and in production. Performance mattering is not unique to ML. Obviously. Data issues are not unique to ML. Anytime you need to run against a "representative workload", you'll bump into them. Heck, anytime you run against a large test suite you'll run into issues with the quality of the comparison data, especially if the tests are long in the tooth. Furthermore, anything you're doing statistics on in general is going to bump into data issues—unless doing statistics is automatically ML? ML lacks various other dimensions of difficulty. Distributed systems. Web browsers that need to execute arbitrary code that hasn't been written yet. Backwards compatibility. Forwards compatibility. In-production databases. Cross platform development. Power usage optimization. Concurrency (ML can have this, but it doesn't always need to, and it can be in the "embarrassingly parallel" bucket). Deep dependency graphs. The author thinks ML development is hard because that's what they work on. They are but a grasshopper.
- deleted 3y ago
  [deleted]
sackfield 3y ago
I think a big difference between ML and regular programming is how the components at scale make the systems viable. When I was learning computer science, it seemed quite intuitive to me that you would start out with assembly, then go to a C like compiler, then abstract that to a JIT/dynamic type language, and go from that to the UI. I could see how each step in the layer added value and presented its tradeoffs. Contrast that to ML and even though I have done a large amount of work in it (in both university and in industry), I still can't fully appreciate how the building blocks interact to form an entire system. I find that I use intuition from other systems I have read about or implemented (e.g. decision trees and tabular data, ReLUs and images) to reason about the results in new systems and guess at better configurations and architectures. Might say more about me, but I always found ML was a "start big and go backwards" deal whereas computer science was a "start small and go forwards" deal.
- barelyauser 3y ago
  When building models it is useful to spend some time finding out what you already know about the problem. Things you yet don't know you know. This kind of knowledge will greatly simplify the model. I see newcomers making this mistake very often. In industrial vision, for example, the newcomers like to create very complicated models. I then show them that the "box" you trained a entire model to recognize will actually always be there in this position for the camera, because it sits in a conveyor belt which restricts its lateral movement. The problem can simply be solved with simple image processing. Stuff like that happens all the time.
  - Cacti 3y ago
    You let newcomers dick around for months with a net on an industrial vision problem? This stuff was solved two decades ago. Why didn’t you just tell them?
    - barelyauser 3y ago
      They usually don't take months. Would be optimal if I could catch them at the get go. Not what happens most of the time. When that happen most of the times you will se a manager who is not technical leading a group selected by "professional" recruiters. There is a lot of waste out there.
Aunche 3y ago
I don't think machine learning is particularly hard. It just involves a lot of brute force work and most of the time, you get a middling result that isn't particularly exciting. One of my friends had no CS background whatsoever but managed to make a basic Runescape mining bot by manually labeling hundreds of the correct colored rocks. His account got banned after a couple days though.
adamnemecek 3y ago
It's harder because there are no unifying theories. The ones that exist are seriously lacking.
itissid 3y ago
Great teachers(if you can find them): Andrej Karpathy, Andrew Gelman & Ben Goodrich(Columbia), Subbarao Khambampati(ASU) to name a few I know of. Go Where the hard problems are (or find someone who is doing it): If you don't have a good intuition where to get good problems to practice on for a pay, choose a place to work where a Data Scientist is not just building dashboards/analytics but the company/team relies on them for answer to questions like: "What goals should we set for the next half based on what you see"? ML practitioner (read: use ML tech to do/debug X) different from ML Engineer (Read: Implement ML algorithm X e2e on data ) is different from Applied Statistician (think marketing sciences or powering experiments like A/B tests): All three areas of work in different areas of ML in one form or another. But make sure what you want to work in is clear in your head and your expectations from it. A lot of ML/Stats can be not with big data and yet really intuitive: I would say look for a problem domain in social/life/pharma/eco/political/survey/edtech sciences. They are full of intuitive models that need to be explainable and are often debuggable. An example here is usage of Stan software for Multilevel/Heirarchical Regression problems. Training here also makes you a great DS.
yshvrdhn 3y ago
I think one of the issues is that fixing a problem is a lot harder in ML than in software engineering. You know that the model fails on this particular data point. If you have identified a bug in the code and wrote a fix did a pull request as long you are able to test the code for conditions you failed on you would have solved the problem. With modern ml especially with nueral nets as long as you don't have a way to spin up a data engine to track the problems you are facing and collect similar points you problem is not fixed.
- nonethewiser 3y ago
  Certainly resonates with me. When the problem is something like a list being the wrong shape or two lists not maintaining a parallel order its basically invisible unless you load the mental model of the code and think deeply about it. Not like you’re going to notice your list of length 2,340,383 should start with [0.12, 1.67, 0.66 instead of [0.412, 0.567, 0.23
- pbh101 3y ago
  I remember running into a paper from Google circa 2017 IIRC discussing the maintainability issues with machine learning models but haven’t been able to track down since. Does anyone know which one this is and have a link?
  - hollerith 3y ago
    https://research.google/pubs/machine-learning-the-high-interest-credit-card-of-technical-debt/ https://research.google/pubs/machine-learning-the-high-inter...
    - pbh101 3y ago
      Looks like I found a similar one just now too from the group. Thanks! https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2015/file/8...
roenxi 3y ago
Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with. The major challenges in this youthful field of machine learning are building appropriate hardware and making it work. That, so far, has kept it the domain of the software engineer. As the situation continues to stabilise this is going to become the playground of statisticians and analysts. Or to put it another way - if you compare any field to software engineering, the problem is that other disciplines have a much harder time debugging things. Software is almost unique in that debugging is cheaper and quicker than building things right the first time.
- p-e-w 3y ago
  > Machine learning isn't comparable to software development. It is a statistical modelling exercise. It's neither of the two. Machine learning isn't comparable to any other human endeavor because in many cases, much more value comes out of the models than (seemingly) goes in. LLMs for example are punching way above their weight. The ideas underlying their software implementations are extremely simple compared to the incredibly complex behavior they produce. Take some neural networks that can be explained to a bright high schooler, add a few more relatively basic ML concepts, then push an unfiltered dump of half the Internet into it, and suddenly you get a machine that talks like a human. Obviously I'm simplifying here, but consider that state-of-the-art LLM architectures are still simple enough that they can be completely understood through a 10-hour online course, and can be implemented in a few hundred lines of PyTorch code. That's absolutely bananas considering that the end result is something that can write a poem about airplanes in the style of Beowulf.
  - eru 3y ago
    I'm not sure it's as simple as you make it sound. Lots of problems have very simple solutions. And progress often means finding simpler solutions over time. But coming up with those solutions, and debugging them, is what's hard. For a comparison, have a look at how pistols got simpler over the last two hundred years. Have a look at the intricate mechanism of the P08 Luger https://www.youtube.com/watch?v=9adOzT_qMq0 https://www.youtube.com/watch?v=9adOzT_qMq0 and compare it to a modern pistol of your choice. (And the P08 Luger is already pretty late invention.) Or have a look at modern Ikea furniture, which can be assembled and understood by untrained members of the general public. But designing new Ikea furniture is much harder.
jvans 3y ago
Machine learning systems are hard because your system can be badly under performing but still doing objectively great. I've seen systems that produce amazing improvements in core business metrics and then years later some subtle bug in the labeling process is accidentally uncovered and fixing it boosts performance by an additional 20%. A ton of time in ML is spent on minimizing the surface area for bugs because it can be difficult to even know they exist.
- nerdponx 3y ago
  It's hard because there's rarely one single correct solution, design, answer, etc. It's hard in the same way as any other open-ended research work is hard. It will never not be hard in that sense.
4death4 3y ago
I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea. Compare this to ML problems: you don't often know if your data or model selection can produce the results you want. On top of that, you don't know if you're not getting results because of a bug (i.e. the debugging issues mentioned by the author) or if because you have fundamental blocks elsewhere in the pipeline.
- owlninja 3y ago
  You nailed where I currently stand. At my company I've been a jack of all trades but mostly software/dba work. My boss and I were very excited about ML when the hype cycle was taking off several years ago and completed a successful project. Fast forward to today, I got loaned out to another team that lost their data scientist, and for the first time in my career I'm having to say - "I don't think we can do what you want." To me the "science" part really stands out. I have a decent grasp of methodologies and tools, but after weeks of dissecting the issue my conclusion is that they just don't have enough useful data...
  - throwaway8877 3y ago
    The situation is not bad then. Can they collect more data? Can they generate more data?
    - coldtea 3y ago
      A more relevant question would be: Is "not enough data" their problem, or the kind of data?
  - jxramos 3y ago
    It can be a very empirical art. If you can't generate more data at the time you can sometimes invest in reviewing the hand labeling ground truth to verify no false classifications slipped by.
epgui 3y ago
I find it difficult to buy the argument that “it’s not difficult because of the math”, at least in the way the author meant. I do however literally agree with the author: ML is not difficult because of the math. It’s difficult because for some reason people think the math is not important. But ML is math.
- DeathArrow 3y ago
  ML without math is like trying to play 5th Symphony without knowing notes, chords or any musical theory.
hooande 3y ago
Debugging is a problem. But the real problem I'm seeing is our expectations as software developers. We're used to being able to fix any problem that we see. If a div is misaligned or a column of numbers is wrong we can open the file, find the offending lines of code and FIX it. Machine learning is different because every implementation has a known error rate. If your application has a measured 80% accuracy then 20% of cases WILL have an error. You don't know which 20% and you don't get to choose. There's no way to notice a problem and immediately fix it, like you can with almost every other kind of engineering. At best you can expand your dataset, incorporate new models, fix actual bugs in the code. Doing those things could increase the accuracy up to, say, 85%. This means there will be fewer errors overall, but the one that you happened to notice may or may not still be there. There's no way to directly intervene. I see a lot of people who are new to the field struggle with this. There are many ways to improve models and handle edge cases. But not being able to fix a problem that's in front of you takes some getting used to.
crotchfire 3y ago
To be fair, chip design has both of these problems to a much greater degree than machine learning: - Exponentially Difficult Debugging - Delayed Debugging Cycles
nnevatie 3y ago
I found the explanation quite convoluted (pun intended). Any system with billions of parameters is bound to be difficult to reason about.
nazka 3y ago
For me it’s 2 things: math is already hard even with a teacher and after you found a job it’s just you and your screen. The other part is research moves at a crazy speed it’s very hard to keep up I think while having a job.
DeathArrow 3y ago
What do we even mean by "Machine Learning"? Understanding ML algorithms and the math behind, being able to change the algorithms and devise new ones? Using some ML libraries and frameworks? Taking an implemented solution and training it and fine tuning parameters?
DeathArrow 3y ago
As with everything, if you don't know the fundamentals, the basics, you are very limited on what you can achieve and find stuff difficult. To use a parallel from software if you just know a programming language, how to call libraries and APIs you can't compare to a guy with a solid CS background who also understands algorithms, data structures, complexity, math and knows how computers work down to NAND gates. Because I have studied some ML fundamentals during my batchelor and master degrees I know I am not competent in ML and AI. But I know how to become competent if I need or want to. Most people do believe finishing a bootcamp and calling some frameworks in Python makes them competent.
- jurgenaut23 3y ago
  Yes, bootcamps are really a great way to score high on the Dunning-Kruger score, but not so much on any other metric of value. You need lots of sweat and tears to become a capable and competent ML engineer, especially to tell apart intrinsic limitations (i.e., no signal in your data) vs extrinsic issues (e.g., a bug anywhere between ETL and prediction, or a methodological mistake).
- kick_in_the_dor 3y ago
  > and knows how computers work down to NAND gates. I lol'd
DeathArrow 3y ago
I don't think ML is harder than other technical and scientific fields. Being a good software developer, a good software scientist, a good mathematician, a good astronomer, a good physicist is hard. And of course, there are hard fields outside of science. Being good at any kind of art is hard. Just because some people believe they are good at something (because Dunning Krueger effect) doesn't mean that field is easier than ML. They assume ML is harder because they happen to know a few bits in a particular field and nothing about ML. Then, the same kind of people will learn some superficial bits on ML and think it's easy. Any human endeavor done properly means time, sweat, critical thinking, determination. And lots of it.
DeathArrow 3y ago
Let's draw a paralel. It's algebra hard? No if you want to solve simple equations. Yes if you want to achieve masters hip.
feverzsj 3y ago
It's not hard. It just doesn't work. It's modern alchemy.
jokoon 3y ago
Unless computer scientist really try to understand how actual brain build their networks, I don't think ML will really bring real advances. Computer science is pointless when it's not applied to another field of science.
mvkel 3y ago
This is a dumb question: could we not make an ML debugger out of ML? It seems ML, or at least model training and fine tuning, is all about pattern recognition. Could the same thing not be done where a model is trained on ML algos running correctly vs. not and it performs the pattern recognition (i.e. intuition) to find the root cause?
swingingFlyFish 3y ago
" Engineers specializing in machine learning continue to command a salary premium in the job market over standard software engineers" I still haven't really seen this. Data Scientist still seem to make about the same as an experienced Backend Engineer.

283 points by jxmorris12 3y ago

"An aspect of this difficulty involves building an intuition for what tool should be leveraged to solve a problem." While I agree with the good point about debugging, like many others, I am rather worried that we're increasingly deploying AI/ML where we shouldn't be deploying it. Hence, the above quote.

sjwhevvvvvsj 3y ago
I’m old enough to have learned that the secret to success is much less knowing the tool of the moment than picking the right tool for a job. The right tool may in fact be the new one, and LLM do open a lot of doors with zero shot capabilities, but oftentimes they can underperform a well tuned heuristic. It’s the ability to pick the right tool that is key.
wwarner 3y ago
Want to agree with you, as so many ML apps seem to be solutions looking for problems. But I actually feel that we are rapidly deploying ML in a development context for vastly improved results. The way that good models are built relies on many ML steps, and when the results finally come together the result is superior to what could have been custom designed. Broad adoption of something like probabilistic programming is coming soon.

dang 3y ago

Discussed at the time: Why is machine learning ‘hard’? - https://news.ycombinator.com/item?id=12936891 https://news.ycombinator.com/item?id=12936891 - Nov 2016 (88 comments)

epistasis 3y ago
Love that thread. The top comment is excellent: > Like picking hyperparamters - time and time again I've asked experts/trainers/colleagues: "How do I know what type of model to use? How many layers? How many nodes per layer? Dropout or not?" etc etc And the answer is always along the lines of "just try a load of stuff and pick the one that works best". > To me, that feels weird and worrying. It's like we don't yet understand ML properly yet to definitively say, for a given data set, what sort of model we'll need. This embodies the very fundamental difference between science and engineering. With science, you make a discovery, but rarely do we ask "what was the magical combination that let me find the needle in the haystack today?" We instead just pass on the needle and show everyone we found it. Should we work on finding out the magic behind hyperparameters? In bioinformatics, the brilliant mathematician Lior Pachter once attacked the problem of sequence alignment using the tools of tropical algebra: what parameters to the alignment algorithms resulted in which regimes of solutions? It was beautiful. It was great to understand. But I'm not sure if it even ever got published (though it likely did). Having reasonable parameters is more important than understanding how to pick them from first principles, because even if you know all the possible output regimes for different segments of the hyper parameter space, really the only thing we care about is getting a functionally trained model at the end. Sometimes deeper understanding provides deeper insights to the problems at hand. But often, they don't, even when the deeper understanding is beautiful. If the hammer works when you hold it a certain way, that's great, but understanding all possible ways to hold a hammer doesn't always help get the nail in better.
- amelius 3y ago
  The hammer analogy doesn't make much sense because for a hammer we can actually use our scientific knowledge to compute the best possible way to hold the tool, and we can make instruments that are better than hammers, like pneumatic hammers, pile drivers, etc. With your argument, we would be stuck with the good old, but basic hammer for the rest of time.
  - epistasis 3y ago
    That seems like a different analogy; making better hammers is a different thing than understanding why holding a hammer a certain way works well. We did eventually invent enough physics to understand why we hold hammers where we do, but we got really far just experimenting without first principles. And even if we use first principles, we are going to discover a lot more by actually using the modified hand-held hammer and testing it, than necessarily hitting it out of the park with great physical modeling of the hammer and the biomechanics of the human body. And in any case, I'm not saying we shouldn't search for deep understanding of what hyperparameters work on a first try, I'm just saying there's a good chance that even if the principles are fully discovered, it may be that calculating using those principles is more expensive than a bunch of experimentation and won't matter in the end. That's the trick about science, it's more about finding the right question to answer than how to find answers, and often times the best questions only become apparent afterwards.
- sjwhevvvvvsj 3y ago
  I do a lot of model tuning and I’m almost ashamed to say I tell GPT what performance I’m aiming for and have it generate the hyper parameters (as in just literally give me a code block). Then I see what works, tell GPT, and try again. I’m deeply uncomfortable with such a method…but my models perform quite well. Note I spend a TON of time generating the right training data, so it’s not random.
  - danielmarkbruce 3y ago
    1/8th (soon to be 1/2) of the working world: "I do a lot of X and I'm almost ashamed to say I tell GPT Y then I see if it works and try again".
naveen99 3y ago
Similar: Machine learning is still too hard for software engineers - https://news.ycombinator.com/item?id=30432987 https://news.ycombinator.com/item?id=30432987 - 151 comments Machine learning is easier than it looks - https://news.ycombinator.com/item?id=6770785 https://news.ycombinator.com/item?id=6770785 167 comments

doctorM 3y ago

I'm a bit sceptical of the exponentially harder debugging claim. First it looks polynomially harder for the given example :p. Second other engineering domains arguably have additional dimensions which correspond to the machine learning ones mentioned in the article. The choice of which high level algorithm to implement is another dimension to traditional software engineering that seemingly exists and corresponds to the model dimension. This is often codified as 'design'. The data dimension often exists as well in standard learning software engineering. [Think of a system that is 'downstream' of other]. It's probably a lot simpler to deal with these dimensions in standard software engineering - but then this is what makes machine learning harder, not that there are simply 'more dimensions'. The delayed debugging cycles point seems a lot more valid.

pizzaknife 3y ago
i would subscribe to your newsletter if you offered one.
jolt42 3y ago
The article also pretends that there is only one correct answer, which seems atypical of the domain. The 1 green spot should extend somewhat fuzzily in each dimension in the ML case.
- janalsncm 3y ago
  There isn’t only one correct answer. Quite the opposite actually, many configurations give you local maximas. The difficulty is that it can be hard to explain from first principles why one local maxima is good.
grantpitt 3y ago
I had similar thoughts on this. It looks polynomial because he classified errors into certain groups and spread them across an axis which we implicitly think is a dimension. But it's not... as he says, "Along any one dimension we might have a combination of issues (i.e. multiple implementation bugs)". So looking at the grid and thinking you're at a single point is wrong... but so is thinking you could be any configuration of locations (you couldn't be at points (1, 1) and (2, 2), you also need to be at points (1, 2) and (2, 1), i.e. the set of points you're at is transitive). Thus this is a bad way to "enumerate the failure cases" and makes his notion of "adding a dimension" pretty unintuitive. It makes more sense to see every point of his dimensions as itself a dimension. Each of these dimensions could be either correct or incorrect, 1 or 0. So you're configuration could be visualized as a string of 1's and 0's. Thus, the number of possible configurations grows exponentially (2^n).

PaulHoule 3y ago

The #1 thing that makes it ‘hard’ in real life is that nobody wants to make training and test sets. So we have 50,000 papers on the NIST digits but no insight into ‘would this work for a different problem?’ (Ironically the latter might have been exactly what academics would have needed to understand why these algorithms work!)

ramesh31 3y ago
Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity.
- rzzzz 3y ago
  I believe that Scale.ai was founded to do exactly this.
- MichaelRo 3y ago
  >> Seems like a prime startup opportunity. Sometimes it's just ... hard. Apply some thought maybe before blindly parroting "profit!" Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to end World poverty?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to build a warp engine?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" Reporter: "Why is it hard to wipe your ass using the left hand?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" You get the idea...
  - aleph_minus_one 3y ago
    > Reporter: "Why is it hard to cure cancer?". Crowd: "Would there be enough of a financial incentive to do so? Seems like a prime startup opportunity!" What you want to optimize for is the money amount that you make at some quantile of the probablity distribution of the profits; say, the profits that are guaranteed in the best, say, 3 %, 5 %, 10 % or even 20 % of all possible outcomes. With a probablity of 97 % (if you choose the best 3 % of the outcomes), you won't make sufficient money if you attempt to cure cancer to be worth the risk, so the financial incentive is not there. TLDR: Financial incentives do matter, but work differently from how many people think that they are structured.

Cacti 3y ago

there are uncountable sets all over the place, and in practical terms, the repl loop may have a week long training lag after you hit enter. also, the data is almost always complete shit. lol. there’s no mystery why it’s hard.

phkahler 3y ago

>> It becomes essential to build an intuition for where something went wrong based on the signals available. This has always been my approach. I learned programming way before I had access to debuggers and other methods to dig in, set breakpoints and step through code to see where it was going wrong. As a result, when I got in the real world I kind of looked down on people using those tools (mostly because I hate tools actually). But then I saw people get to the root of problems that I don't think I ever could have solved, and I started to appreciate those tools and the detail you could get to. My preference is still to have a great understanding of how algorithms work, how the code is written, and what the problem is, and noodle out what and where things may be going wrong. I only switch to detailed monitoring of the insides when "thinking about it" fails. Maybe I should have gone into this ML stuff ;-)

deleted 3y ago

[deleted]

Xcelerate 3y ago

With regard to model selection, one thing I learned a long time ago that provides powerful intuitive guidance on which model to use is the question: "How could this be compressed further?" There are some deep connections between data compression and generalized learning, both at the statistical level and even lower at the algorithmic level (see Solomonoff induction). For a specific example at the statistical level, suppose you fit a linear trendline to some data points using OLS. Now compute the standard deviation of the residual terms for each data point, and using the CDF of the normal distribution, for each residual, map its value into the interval [0, 1]. Sum together the logarithms of the trendline coefficients, the standard deviation, and the normalized residuals. This value is approximately proportional to "sizeof(data | model) + sizeof(model)". It represents how well you compressed the data using the OLS model. But now suppose you plot the distribution of the residuals and find out that they do not in fact resemble a Gaussian distribution. This is a problem because our model assumed the error terms were distributed normally, and since this is not the case, our compression is suboptimal. So you back out some function f that closely maps between the uniform distribution on [0, 1] and the distribution that the residuals form and use this f to define a new model: yᵢ = m*xᵢ + b + εᵢ, with εᵢ distributed according to f(x;Θ), Θ being a parameter vector. When you sum the logarithms again, you will find that the new total is smaller than the original total obtained using OLS. The new trend line coefficients will slightly mess up the residual distribution again, so iterate on this process until you've converged on stable values for m, b, and Θ. At the algorithmic level, the recommendation to use compression as a model selection guide applies even to LLMs, but it's a bit harder to use in practice because "sizeof(data | model) + sizeof(model)" isn't the entire story here. Suppose you had a "perfect" language model. In this case, you would achieve minimization of K(data | training data), where K is Kolmogorov complexity. In practice, what is being minimized with each new LLM version is "sizeof(data | LLM model) + sizeof(training data | LLM model) + sizeof(LLM model)". You can assume "sizeof(LLM model)" is the smallest Turing machine equivalent to the LLM program.

sfink 3y ago

This article annoyed me. Not because I think it overestimates any of the difficulties, but it condescendingly compares against "standard software engineers" as if their problems are so much easier and lack additional dimensions beyond algorithm and implementation. It doesn't help that they call N^4 "exponential". Long debugging cycles are not new. In fact, the field started out that way, when computers were slow enough that even a single edit-compile-run cycle could take hours (or days and some political capital, if you go back far enough). Even today, long debugging cycles are far from being restricted to ML, especially for failures that are only seen intermittently and in production. Performance mattering is not unique to ML. Obviously. Data issues are not unique to ML. Anytime you need to run against a "representative workload", you'll bump into them. Heck, anytime you run against a large test suite you'll run into issues with the quality of the comparison data, especially if the tests are long in the tooth. Furthermore, anything you're doing statistics on in general is going to bump into data issues—unless doing statistics is automatically ML? ML lacks various other dimensions of difficulty. Distributed systems. Web browsers that need to execute arbitrary code that hasn't been written yet. Backwards compatibility. Forwards compatibility. In-production databases. Cross platform development. Power usage optimization. Concurrency (ML can have this, but it doesn't always need to, and it can be in the "embarrassingly parallel" bucket). Deep dependency graphs. The author thinks ML development is hard because that's what they work on. They are but a grasshopper.

deleted 3y ago
[deleted]

sackfield 3y ago

I think a big difference between ML and regular programming is how the components at scale make the systems viable. When I was learning computer science, it seemed quite intuitive to me that you would start out with assembly, then go to a C like compiler, then abstract that to a JIT/dynamic type language, and go from that to the UI. I could see how each step in the layer added value and presented its tradeoffs. Contrast that to ML and even though I have done a large amount of work in it (in both university and in industry), I still can't fully appreciate how the building blocks interact to form an entire system. I find that I use intuition from other systems I have read about or implemented (e.g. decision trees and tabular data, ReLUs and images) to reason about the results in new systems and guess at better configurations and architectures. Might say more about me, but I always found ML was a "start big and go backwards" deal whereas computer science was a "start small and go forwards" deal.

barelyauser 3y ago
When building models it is useful to spend some time finding out what you already know about the problem. Things you yet don't know you know. This kind of knowledge will greatly simplify the model. I see newcomers making this mistake very often. In industrial vision, for example, the newcomers like to create very complicated models. I then show them that the "box" you trained a entire model to recognize will actually always be there in this position for the camera, because it sits in a conveyor belt which restricts its lateral movement. The problem can simply be solved with simple image processing. Stuff like that happens all the time.
- Cacti 3y ago
  You let newcomers dick around for months with a net on an industrial vision problem? This stuff was solved two decades ago. Why didn’t you just tell them?
  - barelyauser 3y ago
    They usually don't take months. Would be optimal if I could catch them at the get go. Not what happens most of the time. When that happen most of the times you will se a manager who is not technical leading a group selected by "professional" recruiters. There is a lot of waste out there.

Aunche 3y ago

I don't think machine learning is particularly hard. It just involves a lot of brute force work and most of the time, you get a middling result that isn't particularly exciting. One of my friends had no CS background whatsoever but managed to make a basic Runescape mining bot by manually labeling hundreds of the correct colored rocks. His account got banned after a couple days though.

adamnemecek 3y ago

It's harder because there are no unifying theories. The ones that exist are seriously lacking.

itissid 3y ago

Great teachers(if you can find them): Andrej Karpathy, Andrew Gelman & Ben Goodrich(Columbia), Subbarao Khambampati(ASU) to name a few I know of. Go Where the hard problems are (or find someone who is doing it): If you don't have a good intuition where to get good problems to practice on for a pay, choose a place to work where a Data Scientist is not just building dashboards/analytics but the company/team relies on them for answer to questions like: "What goals should we set for the next half based on what you see"? ML practitioner (read: use ML tech to do/debug X) different from ML Engineer (Read: Implement ML algorithm X e2e on data ) is different from Applied Statistician (think marketing sciences or powering experiments like A/B tests): All three areas of work in different areas of ML in one form or another. But make sure what you want to work in is clear in your head and your expectations from it. A lot of ML/Stats can be not with big data and yet really intuitive: I would say look for a problem domain in social/life/pharma/eco/political/survey/edtech sciences. They are full of intuitive models that need to be explainable and are often debuggable. An example here is usage of Stan software for Multilevel/Heirarchical Regression problems. Training here also makes you a great DS.

yshvrdhn 3y ago

I think one of the issues is that fixing a problem is a lot harder in ML than in software engineering. You know that the model fails on this particular data point. If you have identified a bug in the code and wrote a fix did a pull request as long you are able to test the code for conditions you failed on you would have solved the problem. With modern ml especially with nueral nets as long as you don't have a way to spin up a data engine to track the problems you are facing and collect similar points you problem is not fixed.

nonethewiser 3y ago
Certainly resonates with me. When the problem is something like a list being the wrong shape or two lists not maintaining a parallel order its basically invisible unless you load the mental model of the code and think deeply about it. Not like you’re going to notice your list of length 2,340,383 should start with [0.12, 1.67, 0.66 instead of [0.412, 0.567, 0.23
pbh101 3y ago
I remember running into a paper from Google circa 2017 IIRC discussing the maintainability issues with machine learning models but haven’t been able to track down since. Does anyone know which one this is and have a link?
- hollerith 3y ago
  https://research.google/pubs/machine-learning-the-high-interest-credit-card-of-technical-debt/ https://research.google/pubs/machine-learning-the-high-inter...
  - pbh101 3y ago
    Looks like I found a similar one just now too from the group. Thanks! https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2015/file/8...

roenxi 3y ago

Machine learning isn't comparable to software development. It is a statistical modelling exercise. This is like asking why advertising is hard - if a non-expert wades in to a different domain then they may find it has different challenges than what they are used to! This is just a specific case of the normal things that analysts routinely deal with. The major challenges in this youthful field of machine learning are building appropriate hardware and making it work. That, so far, has kept it the domain of the software engineer. As the situation continues to stabilise this is going to become the playground of statisticians and analysts. Or to put it another way - if you compare any field to software engineering, the problem is that other disciplines have a much harder time debugging things. Software is almost unique in that debugging is cheaper and quicker than building things right the first time.

p-e-w 3y ago
> Machine learning isn't comparable to software development. It is a statistical modelling exercise. It's neither of the two. Machine learning isn't comparable to any other human endeavor because in many cases, much more value comes out of the models than (seemingly) goes in. LLMs for example are punching way above their weight. The ideas underlying their software implementations are extremely simple compared to the incredibly complex behavior they produce. Take some neural networks that can be explained to a bright high schooler, add a few more relatively basic ML concepts, then push an unfiltered dump of half the Internet into it, and suddenly you get a machine that talks like a human. Obviously I'm simplifying here, but consider that state-of-the-art LLM architectures are still simple enough that they can be completely understood through a 10-hour online course, and can be implemented in a few hundred lines of PyTorch code. That's absolutely bananas considering that the end result is something that can write a poem about airplanes in the style of Beowulf.
- eru 3y ago
  I'm not sure it's as simple as you make it sound. Lots of problems have very simple solutions. And progress often means finding simpler solutions over time. But coming up with those solutions, and debugging them, is what's hard. For a comparison, have a look at how pistols got simpler over the last two hundred years. Have a look at the intricate mechanism of the P08 Luger https://www.youtube.com/watch?v=9adOzT_qMq0 https://www.youtube.com/watch?v=9adOzT_qMq0 and compare it to a modern pistol of your choice. (And the P08 Luger is already pretty late invention.) Or have a look at modern Ikea furniture, which can be assembled and understood by untrained members of the general public. But designing new Ikea furniture is much harder.

jvans 3y ago

Machine learning systems are hard because your system can be badly under performing but still doing objectively great. I've seen systems that produce amazing improvements in core business metrics and then years later some subtle bug in the labeling process is accidentally uncovered and fixing it boosts performance by an additional 20%. A ton of time in ML is spent on minimizing the surface area for bugs because it can be difficult to even know they exist.

nerdponx 3y ago
It's hard because there's rarely one single correct solution, design, answer, etc. It's hard in the same way as any other open-ended research work is hard. It will never not be hard in that sense.

4death4 3y ago

I used to work on an ML research team. In addition to what the author mentions, there is an entirely separate issue: whether or not what you're attempting to do is possible with the approach you've chosen. Consider making an iOS app. For the most part, an experienced software engineer can tell you if making a given app is possible, and they'll have a relatively clear idea about the steps required to realize the idea. Compare this to ML problems: you don't often know if your data or model selection can produce the results you want. On top of that, you don't know if you're not getting results because of a bug (i.e. the debugging issues mentioned by the author) or if because you have fundamental blocks elsewhere in the pipeline.

owlninja 3y ago
You nailed where I currently stand. At my company I've been a jack of all trades but mostly software/dba work. My boss and I were very excited about ML when the hype cycle was taking off several years ago and completed a successful project. Fast forward to today, I got loaned out to another team that lost their data scientist, and for the first time in my career I'm having to say - "I don't think we can do what you want." To me the "science" part really stands out. I have a decent grasp of methodologies and tools, but after weeks of dissecting the issue my conclusion is that they just don't have enough useful data...
- throwaway8877 3y ago
  The situation is not bad then. Can they collect more data? Can they generate more data?
  - coldtea 3y ago
    A more relevant question would be: Is "not enough data" their problem, or the kind of data?
- jxramos 3y ago
  It can be a very empirical art. If you can't generate more data at the time you can sometimes invest in reviewing the hand labeling ground truth to verify no false classifications slipped by.

epgui 3y ago

I find it difficult to buy the argument that “it’s not difficult because of the math”, at least in the way the author meant. I do however literally agree with the author: ML is not difficult because of the math. It’s difficult because for some reason people think the math is not important. But ML is math.

DeathArrow 3y ago
ML without math is like trying to play 5th Symphony without knowing notes, chords or any musical theory.

hooande 3y ago

Debugging is a problem. But the real problem I'm seeing is our expectations as software developers. We're used to being able to fix any problem that we see. If a div is misaligned or a column of numbers is wrong we can open the file, find the offending lines of code and FIX it. Machine learning is different because every implementation has a known error rate. If your application has a measured 80% accuracy then 20% of cases WILL have an error. You don't know which 20% and you don't get to choose. There's no way to notice a problem and immediately fix it, like you can with almost every other kind of engineering. At best you can expand your dataset, incorporate new models, fix actual bugs in the code. Doing those things could increase the accuracy up to, say, 85%. This means there will be fewer errors overall, but the one that you happened to notice may or may not still be there. There's no way to directly intervene. I see a lot of people who are new to the field struggle with this. There are many ways to improve models and handle edge cases. But not being able to fix a problem that's in front of you takes some getting used to.

crotchfire 3y ago

To be fair, chip design has both of these problems to a much greater degree than machine learning: - Exponentially Difficult Debugging - Delayed Debugging Cycles

nnevatie 3y ago

I found the explanation quite convoluted (pun intended). Any system with billions of parameters is bound to be difficult to reason about.

nazka 3y ago

For me it’s 2 things: math is already hard even with a teacher and after you found a job it’s just you and your screen. The other part is research moves at a crazy speed it’s very hard to keep up I think while having a job.

DeathArrow 3y ago

What do we even mean by "Machine Learning"? Understanding ML algorithms and the math behind, being able to change the algorithms and devise new ones? Using some ML libraries and frameworks? Taking an implemented solution and training it and fine tuning parameters?

DeathArrow 3y ago

As with everything, if you don't know the fundamentals, the basics, you are very limited on what you can achieve and find stuff difficult. To use a parallel from software if you just know a programming language, how to call libraries and APIs you can't compare to a guy with a solid CS background who also understands algorithms, data structures, complexity, math and knows how computers work down to NAND gates. Because I have studied some ML fundamentals during my batchelor and master degrees I know I am not competent in ML and AI. But I know how to become competent if I need or want to. Most people do believe finishing a bootcamp and calling some frameworks in Python makes them competent.

jurgenaut23 3y ago
Yes, bootcamps are really a great way to score high on the Dunning-Kruger score, but not so much on any other metric of value. You need lots of sweat and tears to become a capable and competent ML engineer, especially to tell apart intrinsic limitations (i.e., no signal in your data) vs extrinsic issues (e.g., a bug anywhere between ETL and prediction, or a methodological mistake).
kick_in_the_dor 3y ago
> and knows how computers work down to NAND gates. I lol'd

DeathArrow 3y ago

I don't think ML is harder than other technical and scientific fields. Being a good software developer, a good software scientist, a good mathematician, a good astronomer, a good physicist is hard. And of course, there are hard fields outside of science. Being good at any kind of art is hard. Just because some people believe they are good at something (because Dunning Krueger effect) doesn't mean that field is easier than ML. They assume ML is harder because they happen to know a few bits in a particular field and nothing about ML. Then, the same kind of people will learn some superficial bits on ML and think it's easy. Any human endeavor done properly means time, sweat, critical thinking, determination. And lots of it.

DeathArrow 3y ago

Let's draw a paralel. It's algebra hard? No if you want to solve simple equations. Yes if you want to achieve masters hip.

feverzsj 3y ago

It's not hard. It just doesn't work. It's modern alchemy.

jokoon 3y ago

Unless computer scientist really try to understand how actual brain build their networks, I don't think ML will really bring real advances. Computer science is pointless when it's not applied to another field of science.

mvkel 3y ago

This is a dumb question: could we not make an ML debugger out of ML? It seems ML, or at least model training and fine tuning, is all about pattern recognition. Could the same thing not be done where a model is trained on ML algos running correctly vs. not and it performs the pattern recognition (i.e. intuition) to find the root cause?

swingingFlyFish 3y ago

" Engineers specializing in machine learning continue to command a salary premium in the job market over standard software engineers" I still haven't really seen this. Data Scientist still seem to make about the same as an experienced Backend Engineer.