4 ms·

> machine learning and software engineering are different disciplines entirely I don't think so. Or more precisely, they might look different academically but

by jstx1 5y ago

> machine learning and software engineering are different disciplines entirely I don't think so. Or more precisely, they might look different academically but you need to have both as a skill to build something useful.

uoaei 5y ago
In-built is the assumption that one person must have both competencies. This may be true for cash-strapped startups but this hardly plays well as general advice.
- jstx1 5y ago
  It's not an assumption, that's my main point - you need both skills in the same person. And it's true everywhere including in the biggest companies.
  - ngc248 5y ago
    Definitely not ... I am a SWE in a big company and working on ML projects for the past 4 years. We have a data scientist team which does the data exploration and comes up with a model. Their output would be a jupyter notebook. Then we have a team of "Applied ML practitioners", which I am a part of, we productionize the jupyter notebook, by setting up pipelines, services etc. We understand ML algos, stats, probability etc, but not as much as our data scientist team does. Having both in the same person would be good, but is not necessary.
    - jstx1 5y ago
      My point isn’t the separation of duties, my point is that the people who productionize it also know a reasonable amount of theory.
  - uoaei 5y ago
    Any good tech lead will remove that need, so no, still not good general advice.
    - disgruntledphd2 5y ago
      Regardless of how low you get your communication overhead, it still exists. It's rare to find people who can both run and test all the infrastructure and model code, and notice that the transformation you apply on line 34876 of file foobar_now_with_added_ml.py is statistically inappropriate for your problem. That's not even to mention the really hard part, selecting a good outcome variable and appropriate ways to measure the performance of your system once it hits prod. You can definitely split this stuff between people, but it gets super-linearly harder as you add more people, so it's really incredible to find people who can do both (and honestly, there aren't that many of them (I'd like to say us, but I'm probably not there yet)).
      - uoaei 5y ago
        > It's rare to find people who can both run and test all the infrastructure and model code It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines. How do you think any machine more complicated than a train car was designed? Do my mechanical engineers need to understand the intricacies of avionics? > it's really incredible to find people who can do both Absolutely, and I think you'd have a hard time finding someone who disagrees. But you made a very strong assertion about a "need" which requires much stronger arguments to support.
        disgruntledphd2 5y ago
        > It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines. You clearly work at much better run companies than I do ;)
        uoaei 5y ago
        I sincerely wish you luck! Yeah terrible managers are awful. The feedback loop of "my boss won't do shit to help me so I'll figure it out on myself" to "I'm competent in every single technology here and could run the show myself with enough time" to "management recognizes my skills and puts everything on my plate even though I have no time" is pretty dangerous.

by jstx1 5y ago

uoaei 5y ago

In-built is the assumption that one person must have both competencies. This may be true for cash-strapped startups but this hardly plays well as general advice.

jstx1 5y ago
It's not an assumption, that's my main point - you need both skills in the same person. And it's true everywhere including in the biggest companies.
- ngc248 5y ago
  Definitely not ... I am a SWE in a big company and working on ML projects for the past 4 years. We have a data scientist team which does the data exploration and comes up with a model. Their output would be a jupyter notebook. Then we have a team of "Applied ML practitioners", which I am a part of, we productionize the jupyter notebook, by setting up pipelines, services etc. We understand ML algos, stats, probability etc, but not as much as our data scientist team does. Having both in the same person would be good, but is not necessary.
  - jstx1 5y ago
    My point isn’t the separation of duties, my point is that the people who productionize it also know a reasonable amount of theory.
- uoaei 5y ago
  Any good tech lead will remove that need, so no, still not good general advice.
  - disgruntledphd2 5y ago
    Regardless of how low you get your communication overhead, it still exists. It's rare to find people who can both run and test all the infrastructure and model code, and notice that the transformation you apply on line 34876 of file foobar_now_with_added_ml.py is statistically inappropriate for your problem. That's not even to mention the really hard part, selecting a good outcome variable and appropriate ways to measure the performance of your system once it hits prod. You can definitely split this stuff between people, but it gets super-linearly harder as you add more people, so it's really incredible to find people who can do both (and honestly, there aren't that many of them (I'd like to say us, but I'm probably not there yet)).
    - uoaei 5y ago
      > It's rare to find people who can both run and test all the infrastructure and model code It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines. How do you think any machine more complicated than a train car was designed? Do my mechanical engineers need to understand the intricacies of avionics? > it's really incredible to find people who can do both Absolutely, and I think you'd have a hard time finding someone who disagrees. But you made a very strong assertion about a "need" which requires much stronger arguments to support.
      - disgruntledphd2 5y ago
        > It's also unnecessary to do so as long as your institutional processes are capable of synthesizing multiple peoples' competencies across multiple disciplines. You clearly work at much better run companies than I do ;)
        uoaei 5y ago
        I sincerely wish you luck! Yeah terrible managers are awful. The feedback loop of "my boss won't do shit to help me so I'll figure it out on myself" to "I'm competent in every single technology here and could run the show myself with enough time" to "management recognizes my skills and puts everything on my plate even though I have no time" is pretty dangerous.