4 ms·

I'd argue software engineering is still too hard for ML engineers. Most of ML (not research but commercial applications of what's already been demonstrated) is

I'd argue software engineering is still too hard for ML engineers. Most of ML (not research but commercial applications of what's already been demonstrated) is now well within the realm of engineering, but there are few standard practices, bodies of knowledge, or agreed on processes for doing anything. These are problems that engineering formalism solve, not another auto-ml tool. Maybe I'm saying the same thing as the article, just framed differently?

yldedly 5y ago
In my experience, both are true. I'm more on the ML side, and I can tell I don't have the kind of routine and habits that good software engineers have, though I'm learning. But on the other hand, and I've seen this from software engineers who've made the transition to ML, and clearly have a good handle on the concepts (in one case even published papers in ML journals), they don't seem to have the intuition that allows them to select the right tool for the job, or understand when a given technique is appropriate; you could call that "modeling maturity" - a combination of mathematical maturity and the skill of relating domain knowledge to modeling choices.
- antman 5y ago
  Absolutely agree, the major challenge has been to explain to software engineers that the pick up an algo from github, monkey see monkey do a tutorial followed by automl or grid search is pretty irrelevant nowadays. If you can’t implement any existing algorithm from a scientific paper it is pretty hard to understand the flexibility that is required designing a training pipeline. In terms of tech stacks, cloud providers’ sample architectures and code samples are a base for cloud components and API understanding and not actual implementations.
- 7h3kk1d 5y ago
  I think the above commenter is more critiquing the notion of relying on intuition all together.
  - yldedly 5y ago
    Right, I see. That's not really possible imo. For things like mlops, sure. But model development, selection, evaluation? From what I've seen, it's exactly when engineers reach for standard tools without giving thought to how it applies to the given problem that they do a bad job.
  - jacobr1 5y ago
    Even in non-ML software engineering you still have architectural tradeoffs, that while you can in part make reasoned arguments about, you still are relying on the intuition of your technical leadership.
  - version_five 5y ago
    Intuition is always important, but it shouldn't be the last word in an engineering problem. I think there is room for a lot more rigour in how we build, optimize, and validate ML model performance, so less of it gets left to intuition. The discipline is becoming mature enough that this is possible, I think there is a lot of room to build out "standards" and a "body of knowledge" that can be applied to building ML models. We're seeing it in pockets, but in so many cases, it is still a dark art. And then from an actual software engineering perspective, so much ML code is just run-once jupyter notebook stuff... there is a lot we can do. I need to give this more thought, but I think it's acknowledged there is a big opportunity here
otras 5y ago
To add to your point, the Hidden Technical Debt in Machine Learning Systems paper: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896f...
- deleted 5y ago
  [deleted]
zcw100 5y ago
I'd argue software engineering is still too hard for software engineers.
- LadyCailin 5y ago
  I think industry and society just has too low a bar for software development. If the quality of civil engineers matched that of software engineers as a whole, we’d all be dead from the collapsed buildings and bridges. Of course, software isn’t usually such high stakes, but I don’t think it would hurt the industry to have some formal licensing. If you write a terrible API or whatever, then a review board decides if your license to practice should be removed. The field would suddenly get better, as developers would no longer care about the PMs artificial deadline, if they need more time to write quality code, they will take it, and the PM simply can’t find anyone else to cheap out on the work either. Now, how you enforce this, I don’t know, but this is my pie in the sky dream.
  - qazpot 5y ago
    Your suggestion is extremely laughable and naive and reeks of elitism. The domains where the quality of code has very high stakes do have very strict processes and have an extremely high bar. The domains where the stakes are not high, for ex - implementing an API to download user reports, have a very low bar. And your point about cancelling the license to code if you implement a bad API, that's like cancelling a journalist because they made a typo in their article and it went out.
  - neura 5y ago
    I hear what you're saying. I would argue that review boards, licensing, etc are only relevant because of the stakes. How to enforce it becomes irrelevant if you can't sell anybody on the idea that it's important in the first place. Even the strength of your statement "...I don't think it would hurt the industry to have..." doesn't sound like something that any part of government would be willing to get behind. There's just such a huge leap from "decisions civil engineers make can kill. people" to "decisions software engineers make can annoy people". Still, you're not wrong that the field might immediately get better in some ways, but I believe we could also go down a lot of paths that would lead to the widespread concept of "stifling innovation so that people are less likely to be annoyed by buggy or hard to use software". I'm sure there are also much stronger arguments in favor of your idea of a better world when you narrow the scope down to specific types of software or use cases.
Salgat 5y ago
Most folks don't realize that the vast majority of ML applications can be done with preexisting models, you don't need someone with a PhD to fine tune it unless you need to eek out a tiny bit of improvement. The real value is in the person preparing the data for training.
- eyegor 5y ago
  This is not true at all unless you're only referring to commodity applications. There's no preexisting model for general regression or seq to seq, for instance. You can say, oh, that's a good fit for an LSTM, but you still need to tune the individual layers to suit the application. Although I am slightly conflating ML and DL, since you mention training which heavily implies DL.
  - Salgat 5y ago
    Most folks can get away with logistic or linear regression. A tiny portion of those might need to move to random forest. And an even tinier portion may justify deep neural networks. The vast majority of ML is not deep learning, nor is it necessary. We're talking very boring typical business cases.
    - disgruntledphd2 5y ago
      Yeah, this fixation with deep learning is kinda weird, given that most companies don't have vast amounts of images/text lying around and a pressing business need to understand them. I'm a big fan of starting simple, which means linear/log regression (normally with lasso, as it does variable selection). Then, if you can prove business value, it may make sense to start trying to use more unstructured data.
      - graphpercolator 5y ago
        I think it is because linear regression is not science fiction enough even if that is the best that can be done on the data. We didn't hire this AI guy to do simple regressions!
saintarian 5y ago
Ha, I agree that software engineering is too hard for ML engineers, and even for software engineers like myself who have been doing it for 20 years like zcw100 said :). Author of the blog post here. It was definitely written from my narrow viewpoint and experience. Our goal is to make more solutions accessible to software developers and instinct was the same as yours - a lot of ML can be within the realm of engineering (even small / one-person teams) and that there are accidental complexities standing in the way of wider use. Our solution (AutoML+SaaS) def doesn't work for every situation. I'm curious to hear more of your thoughts on how ML can be made more accessible to Eng (and vice versa).

yldedly 5y ago

In my experience, both are true. I'm more on the ML side, and I can tell I don't have the kind of routine and habits that good software engineers have, though I'm learning. But on the other hand, and I've seen this from software engineers who've made the transition to ML, and clearly have a good handle on the concepts (in one case even published papers in ML journals), they don't seem to have the intuition that allows them to select the right tool for the job, or understand when a given technique is appropriate; you could call that "modeling maturity" - a combination of mathematical maturity and the skill of relating domain knowledge to modeling choices.

antman 5y ago
Absolutely agree, the major challenge has been to explain to software engineers that the pick up an algo from github, monkey see monkey do a tutorial followed by automl or grid search is pretty irrelevant nowadays. If you can’t implement any existing algorithm from a scientific paper it is pretty hard to understand the flexibility that is required designing a training pipeline. In terms of tech stacks, cloud providers’ sample architectures and code samples are a base for cloud components and API understanding and not actual implementations.
7h3kk1d 5y ago
I think the above commenter is more critiquing the notion of relying on intuition all together.
- yldedly 5y ago
  Right, I see. That's not really possible imo. For things like mlops, sure. But model development, selection, evaluation? From what I've seen, it's exactly when engineers reach for standard tools without giving thought to how it applies to the given problem that they do a bad job.
- jacobr1 5y ago
  Even in non-ML software engineering you still have architectural tradeoffs, that while you can in part make reasoned arguments about, you still are relying on the intuition of your technical leadership.
- version_five 5y ago
  Intuition is always important, but it shouldn't be the last word in an engineering problem. I think there is room for a lot more rigour in how we build, optimize, and validate ML model performance, so less of it gets left to intuition. The discipline is becoming mature enough that this is possible, I think there is a lot of room to build out "standards" and a "body of knowledge" that can be applied to building ML models. We're seeing it in pockets, but in so many cases, it is still a dark art. And then from an actual software engineering perspective, so much ML code is just run-once jupyter notebook stuff... there is a lot we can do. I need to give this more thought, but I think it's acknowledged there is a big opportunity here

otras 5y ago

To add to your point, the Hidden Technical Debt in Machine Learning Systems paper: https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896f...

deleted 5y ago
[deleted]

zcw100 5y ago

I'd argue software engineering is still too hard for software engineers.

LadyCailin 5y ago
I think industry and society just has too low a bar for software development. If the quality of civil engineers matched that of software engineers as a whole, we’d all be dead from the collapsed buildings and bridges. Of course, software isn’t usually such high stakes, but I don’t think it would hurt the industry to have some formal licensing. If you write a terrible API or whatever, then a review board decides if your license to practice should be removed. The field would suddenly get better, as developers would no longer care about the PMs artificial deadline, if they need more time to write quality code, they will take it, and the PM simply can’t find anyone else to cheap out on the work either. Now, how you enforce this, I don’t know, but this is my pie in the sky dream.
- qazpot 5y ago
  Your suggestion is extremely laughable and naive and reeks of elitism. The domains where the quality of code has very high stakes do have very strict processes and have an extremely high bar. The domains where the stakes are not high, for ex - implementing an API to download user reports, have a very low bar. And your point about cancelling the license to code if you implement a bad API, that's like cancelling a journalist because they made a typo in their article and it went out.
- neura 5y ago
  I hear what you're saying. I would argue that review boards, licensing, etc are only relevant because of the stakes. How to enforce it becomes irrelevant if you can't sell anybody on the idea that it's important in the first place. Even the strength of your statement "...I don't think it would hurt the industry to have..." doesn't sound like something that any part of government would be willing to get behind. There's just such a huge leap from "decisions civil engineers make can kill. people" to "decisions software engineers make can annoy people". Still, you're not wrong that the field might immediately get better in some ways, but I believe we could also go down a lot of paths that would lead to the widespread concept of "stifling innovation so that people are less likely to be annoyed by buggy or hard to use software". I'm sure there are also much stronger arguments in favor of your idea of a better world when you narrow the scope down to specific types of software or use cases.

Salgat 5y ago

Most folks don't realize that the vast majority of ML applications can be done with preexisting models, you don't need someone with a PhD to fine tune it unless you need to eek out a tiny bit of improvement. The real value is in the person preparing the data for training.

eyegor 5y ago
This is not true at all unless you're only referring to commodity applications. There's no preexisting model for general regression or seq to seq, for instance. You can say, oh, that's a good fit for an LSTM, but you still need to tune the individual layers to suit the application. Although I am slightly conflating ML and DL, since you mention training which heavily implies DL.
- Salgat 5y ago
  Most folks can get away with logistic or linear regression. A tiny portion of those might need to move to random forest. And an even tinier portion may justify deep neural networks. The vast majority of ML is not deep learning, nor is it necessary. We're talking very boring typical business cases.
  - disgruntledphd2 5y ago
    Yeah, this fixation with deep learning is kinda weird, given that most companies don't have vast amounts of images/text lying around and a pressing business need to understand them. I'm a big fan of starting simple, which means linear/log regression (normally with lasso, as it does variable selection). Then, if you can prove business value, it may make sense to start trying to use more unstructured data.
    - graphpercolator 5y ago
      I think it is because linear regression is not science fiction enough even if that is the best that can be done on the data. We didn't hire this AI guy to do simple regressions!

saintarian 5y ago

Ha, I agree that software engineering is too hard for ML engineers, and even for software engineers like myself who have been doing it for 20 years like zcw100 said :). Author of the blog post here. It was definitely written from my narrow viewpoint and experience. Our goal is to make more solutions accessible to software developers and instinct was the same as yours - a lot of ML can be within the realm of engineering (even small / one-person teams) and that there are accidental complexities standing in the way of wider use. Our solution (AutoML+SaaS) def doesn't work for every situation. I'm curious to hear more of your thoughts on how ML can be made more accessible to Eng (and vice versa).