4 ms·

The classic explanation is Lei Breiman's 'Two Cultures' paper. He was a statistics professor who left for industry, came back, and tried to get academics to ada

The classic explanation is Lei Breiman's 'Two Cultures' paper. He was a statistics professor who left for industry, came back, and tried to get academics to adapt industry approaches. The paper is very readable. http://www2.math.uu.se/~thulin/mm/breiman.pdf http://www2.math.uu.se/~thulin/mm/breiman.pdf An oversimplified version may be: Statistics focuses on fitting data to formally understandable models, whereas data science focuses on solving problems -- even if that means using techniques that aren't formally understood. Leo Breiman is also known for pioneering random forest and bootstrap aggregation. I think of machine learning as a subset of data science where you trick linear algebra into thinking.

deleted 7y ago
[deleted]
analog31 7y ago
This seems perfectly fair. And I think that historically there was plenty of use for statistics where people didn't care about the formal understanding, so they were doing crude machine learning before the term became widespread. I've sat through lengthy discussions of machine learning exercises, and could not silence the voice in my head, saying: "This is just curve fitting." Fitting data to an arbitrary curve, and then extrapolating the fitting function, is as old as the hills.
- AstralStorm 7y ago
  Formal understanding is critical to actually know the limitations of any given system. Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it. Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains.
  - TuringTest 7y ago
    > Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it. On the other hand, it can work indefinitely if it solves a problem well enough and it is used always on the same kind of problem. Sure, you will need to understand it better to apply it to new markets, but often that kind of research is outside the scope of a single business, and they don't need it.
    - AstralStorm 7y ago
      How do you know it solves the problem, instead of just tricking your measure? Do you know when it doesn't work? Business does not care about rigor, but then terrible things happen when your face recognition system happens to not work for people with dark skin tone. Even more terrible things happen when it detects cats as people, and worse once it's used to link data into a police database. Even pretty bad things happen when it's used for detecting potential (treated as absolute) copyright violations. Security holes when it's used for detecting security problems. Loss of business when it's used for ticket prioritization. Business world does not care that it sells a broken solution as long as it's not obvious and someone has been paid. Everyone else pays for the failures. And you cannot sue an ML system really.
      - dorgo 7y ago
        Why don't we do online learning? Each failure is a new training sample. Why dont't we do active learning? Or adversarial networks, or.... Sure it is easy to fail in machine learning. But it doesn't mean that we need to undstand how and what our models learn. That's the only big advantage of machine learning: the model learns, so I don't have to.
    - bumby 7y ago
      I don't necessarily disagree (in the sense that all models are wrong but some are useful) but to play devils advocate, its damn near impossible to claim a model will work indefinitely on a subset of problems if we don't know it's underpinnings. Its like saying software has been tested completely when we really should be saying its passed a subset of test cases out of the complete set of possible scenarios. The only way we can make that claim is if we can interpret the model in a mathematical proof.
  - sweeneyrod 7y ago
    > Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains. That's not my experience at all. What kind of ML requires something beyond basic linear algebra? Being comfortable manipulating matrices is certainly harder than plugging data into an sklearn function, but it's also significantly easier than understanding general relativity.
    - Retric 7y ago
      The NN complexity is not a question of doing the calculations. Go on a 5x5 board would be a very straightforward game, go in a 19x19 board is not. Similarly, understanding how NN fiction at a nuts and bolts level stops being enough as you build larger networks.
      - nitrogen 7y ago
        As a curious onlooker to this discussion, could you point to some references for learning the math that applies to emergent behavior in these larger networks?
      - dorgo 7y ago
        > Go on a 5x5 board would be a very straightforward game I think it is not a good comparison. A small board calls for simpler algorithms: just try out all possible games and choose the most favorable one. For larger boards this simple approach doesn't work any more and you start looking for something different. But the architecture of NN networks doesn't change at all if you move from small to large scales. Only the number and size of layers increases. Edit: actually, that's the reason they call it "deep learning". it has more layers..
        hadsed 7y ago
        The architecture doesn't change but the way it works often changes dramatically with scale.
      - sweeneyrod 7y ago
AstralStorm 7y ago
*Nonlinear algebra nowadays. With sprinkling of discrete algebra on top. (Gated and threshold units.)
beagle3 7y ago
Breiman is also responsible for ACE (Alternating Conditional Expectation) which is in many ways magical. https://en.wikipedia.org/wiki/Alternating_conditional_expectations https://en.wikipedia.org/wiki/Alternating_conditional_expect...

deleted 7y ago

[deleted]

analog31 7y ago

This seems perfectly fair. And I think that historically there was plenty of use for statistics where people didn't care about the formal understanding, so they were doing crude machine learning before the term became widespread. I've sat through lengthy discussions of machine learning exercises, and could not silence the voice in my head, saying: "This is just curve fitting." Fitting data to an arbitrary curve, and then extrapolating the fitting function, is as old as the hills.

AstralStorm 7y ago
Formal understanding is critical to actually know the limitations of any given system. Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it. Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains.
- TuringTest 7y ago
  > Treating it as a black box has short lifetime, as your method of analysis will miss key features of the system or oversimplify it. On the other hand, it can work indefinitely if it solves a problem well enough and it is used always on the same kind of problem. Sure, you will need to understand it better to apply it to new markets, but often that kind of research is outside the scope of a single business, and they don't need it.
  - AstralStorm 7y ago
    How do you know it solves the problem, instead of just tricking your measure? Do you know when it doesn't work? Business does not care about rigor, but then terrible things happen when your face recognition system happens to not work for people with dark skin tone. Even more terrible things happen when it detects cats as people, and worse once it's used to link data into a police database. Even pretty bad things happen when it's used for detecting potential (treated as absolute) copyright violations. Security holes when it's used for detecting security problems. Loss of business when it's used for ticket prioritization. Business world does not care that it sells a broken solution as long as it's not obvious and someone has been paid. Everyone else pays for the failures. And you cannot sue an ML system really.
    - dorgo 7y ago
      Why don't we do online learning? Each failure is a new training sample. Why dont't we do active learning? Or adversarial networks, or.... Sure it is easy to fail in machine learning. But it doesn't mean that we need to undstand how and what our models learn. That's the only big advantage of machine learning: the model learns, so I don't have to.
  - bumby 7y ago
    I don't necessarily disagree (in the sense that all models are wrong but some are useful) but to play devils advocate, its damn near impossible to claim a model will work indefinitely on a subset of problems if we don't know it's underpinnings. Its like saying software has been tested completely when we really should be saying its passed a subset of test cases out of the complete set of possible scenarios. The only way we can make that claim is if we can interpret the model in a mathematical proof.
- sweeneyrod 7y ago
  > Understanding modern ML algebra is really "general relativity hard" if not actually harder. Spiking NNs are "quantum physics hard". The math is very much translatable between these domains. That's not my experience at all. What kind of ML requires something beyond basic linear algebra? Being comfortable manipulating matrices is certainly harder than plugging data into an sklearn function, but it's also significantly easier than understanding general relativity.
  - Retric 7y ago
    The NN complexity is not a question of doing the calculations. Go on a 5x5 board would be a very straightforward game, go in a 19x19 board is not. Similarly, understanding how NN fiction at a nuts and bolts level stops being enough as you build larger networks.
    - nitrogen 7y ago
      As a curious onlooker to this discussion, could you point to some references for learning the math that applies to emergent behavior in these larger networks?
    - dorgo 7y ago
      > Go on a 5x5 board would be a very straightforward game I think it is not a good comparison. A small board calls for simpler algorithms: just try out all possible games and choose the most favorable one. For larger boards this simple approach doesn't work any more and you start looking for something different. But the architecture of NN networks doesn't change at all if you move from small to large scales. Only the number and size of layers increases. Edit: actually, that's the reason they call it "deep learning". it has more layers..
      - hadsed 7y ago
        The architecture doesn't change but the way it works often changes dramatically with scale.
    - sweeneyrod 7y ago

AstralStorm 7y ago

*Nonlinear algebra nowadays. With sprinkling of discrete algebra on top. (Gated and threshold units.)

beagle3 7y ago

Breiman is also responsible for ACE (Alternating Conditional Expectation) which is in many ways magical. https://en.wikipedia.org/wiki/Alternating_conditional_expectations https://en.wikipedia.org/wiki/Alternating_conditional_expect...