6 ms·

Two big challenges in machine learning [pdf]

56 points by jsnell 11y ago

syense 11y ago
Did I just read a powerpoint presentation?
- joe_the_user 11y ago
  Yeah, the presentation format is annoying but argument is good, in fact, the argument is really important. Machine learning is about approximate reasoning but with no real guarantees of the approximation. Even if the approximation is usually very good, being occasionally bad can be "deeply problematic" when we don't have control over exactly how exactly the bad recommendations are created.
  - shockzzz 11y ago
    Isn't that what confidence intervals are for?
    - RMarcus 11y ago
      I'd say that's more what cost functions are for, but that is neither here nor there. I think the biggest takeaways are... 1) Machine learning in large software projects is complex because decisions made by one algorithm can influence the data of other algorithms, creating massive biasing. 2) Simple cross-validation / hold back methodology is limited as we expand what we want machine learning to be able to handle. Reason: big data is too big and "correctness" is difficult to evaluate for things like Q/A systems.
    - joe_the_user 11y ago
      Indeed it is and machine learning not being "real statistics" but ad-hoc methodologies (based on statistics but not being statistics) doesn't give confidence intervals etc. Plus read the article/presentation. It goes into the issue with great depth and clarity.
    - xyzzy123 11y ago
      If we pick a silly thing like recommendations, they happen in a multidimensional space. Not only do we not know the right metrics, shape of the space or cost functions, our feedback is distorted by the actual recommendations we give!
joe_the_user 11y ago
It seems Léon Bottou is an important figure in Machine learning. His personal website has interesting stuff too. https://research.facebook.com/researchers/1558013787807218/leon-bottou/ https://research.facebook.com/researchers/1558013787807218/l... http://leon.bottou.org/ http://leon.bottou.org/
chestervonwinch 11y ago
I wasn't aware Leon Bouttou was with facebook as well now. They're building quite a research team. Interesting slides.
- joe_the_user 11y ago
  It's fascinating that now that things like deep learning have massive traction, people like LeCrun and Bouttou (both now at Facebook), who apparently pioneered the stuff, are taking a critical position on it - critical not being negative or dismissive but rather a "we have to see the limitations and good beyond them" approach. See: LeCrun's What Wrong With Deep Learning https://drive.google.com/file/d/0BxKBnD5y2M8NVHRiVXBnOVpiYUk/view?sle=true https://drive.google.com/file/d/0BxKBnD5y2M8NVHRiVXBnOVpiYUk...
  - chestervonwinch 11y ago
    To be fair, he's been talking about the relationship between theory and empiricism with regards to neural networks for some time now. See, for instance, page 12 here [1] (the titles of his talks all seem to be provocative questions; almost in a tongue-in-cheek sort of way). One of the problems (in my opinion) with networks is that the analysis has always been post hoc. Personally, I think it's more preferable to build a method starting from theory [2] which can then be tested empirically to see if the assumptions of the theory hold. Then augment the theory, the method, and experiment again. Now, there's nothing inherently wrong with post hoc analysis - it's just a different start point in the loop of science. However, because we didn't start from theory, the burden is then to extract some theory from empirical observation. Again IMO, this is can be problematic because: 1) It more easily allows for confirmation bias. 2) It leads to a multitude fragmented theories. The second is why everything surrounding neural networks seems so incredibly ad hoc. [1]: https://www.cs.nyu.edu/~yann/talks/lecun-20071207-nonconvex.pdf https://www.cs.nyu.edu/~yann/talks/lecun-20071207-nonconvex.... [2]: The principles could be based on statistical learning (see SVM), neurophysiology (see work by Poggio or Olshausen), mathematical invariants (see work by Mallat), etc...
    - joe_the_user 11y ago

56 points by jsnell 11y ago

syense 11y ago

Did I just read a powerpoint presentation?

joe_the_user 11y ago
Yeah, the presentation format is annoying but argument is good, in fact, the argument is really important. Machine learning is about approximate reasoning but with no real guarantees of the approximation. Even if the approximation is usually very good, being occasionally bad can be "deeply problematic" when we don't have control over exactly how exactly the bad recommendations are created.
- shockzzz 11y ago
  Isn't that what confidence intervals are for?
  - RMarcus 11y ago
    I'd say that's more what cost functions are for, but that is neither here nor there. I think the biggest takeaways are... 1) Machine learning in large software projects is complex because decisions made by one algorithm can influence the data of other algorithms, creating massive biasing. 2) Simple cross-validation / hold back methodology is limited as we expand what we want machine learning to be able to handle. Reason: big data is too big and "correctness" is difficult to evaluate for things like Q/A systems.
  - joe_the_user 11y ago
    Indeed it is and machine learning not being "real statistics" but ad-hoc methodologies (based on statistics but not being statistics) doesn't give confidence intervals etc. Plus read the article/presentation. It goes into the issue with great depth and clarity.
  - xyzzy123 11y ago
    If we pick a silly thing like recommendations, they happen in a multidimensional space. Not only do we not know the right metrics, shape of the space or cost functions, our feedback is distorted by the actual recommendations we give!

joe_the_user 11y ago

It seems Léon Bottou is an important figure in Machine learning. His personal website has interesting stuff too. https://research.facebook.com/researchers/1558013787807218/leon-bottou/ https://research.facebook.com/researchers/1558013787807218/l... http://leon.bottou.org/ http://leon.bottou.org/

chestervonwinch 11y ago

I wasn't aware Leon Bouttou was with facebook as well now. They're building quite a research team. Interesting slides.

joe_the_user 11y ago
It's fascinating that now that things like deep learning have massive traction, people like LeCrun and Bouttou (both now at Facebook), who apparently pioneered the stuff, are taking a critical position on it - critical not being negative or dismissive but rather a "we have to see the limitations and good beyond them" approach. See: LeCrun's What Wrong With Deep Learning https://drive.google.com/file/d/0BxKBnD5y2M8NVHRiVXBnOVpiYUk/view?sle=true https://drive.google.com/file/d/0BxKBnD5y2M8NVHRiVXBnOVpiYUk...
- chestervonwinch 11y ago
  To be fair, he's been talking about the relationship between theory and empiricism with regards to neural networks for some time now. See, for instance, page 12 here [1] (the titles of his talks all seem to be provocative questions; almost in a tongue-in-cheek sort of way). One of the problems (in my opinion) with networks is that the analysis has always been post hoc. Personally, I think it's more preferable to build a method starting from theory [2] which can then be tested empirically to see if the assumptions of the theory hold. Then augment the theory, the method, and experiment again. Now, there's nothing inherently wrong with post hoc analysis - it's just a different start point in the loop of science. However, because we didn't start from theory, the burden is then to extract some theory from empirical observation. Again IMO, this is can be problematic because: 1) It more easily allows for confirmation bias. 2) It leads to a multitude fragmented theories. The second is why everything surrounding neural networks seems so incredibly ad hoc. [1]: https://www.cs.nyu.edu/~yann/talks/lecun-20071207-nonconvex.pdf https://www.cs.nyu.edu/~yann/talks/lecun-20071207-nonconvex.... [2]: The principles could be based on statistical learning (see SVM), neurophysiology (see work by Poggio or Olshausen), mathematical invariants (see work by Mallat), etc...
  - joe_the_user 11y ago