3 ms·

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a

I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1. But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?

rumanator 7y ago
> Mind providing some concrete examples? It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications. To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
- JamesBarney 7y ago
  I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). But I was asking because I was specifically looking for concrete examples in deep learning.
  - ineedasername 7y ago
    Yep, this is why I talk about the virtuous feedback loop between these two modes. Empirical methods feed theory which feeds empirical methods ad infinitum. In the field of ML, a concrete example might be the tool Xgboost (#1) and the original work that led to and developed Gradient Boosting itself (#2), of which Xgboost is an implementation, and probably one that has helped refine the underlying theory as well. ML has lots of examples where the researcher(s) for #2 were also doing #1. A famous paper in NLP comes to mind as an excellent example of this overlap (PDF: https://www.csie.ntu.edu.tw/~b92b02053/print/good-turing-smoothing-without.pdf https://www.csie.ntu.edu.tw/~b92b02053/print/good-turing-smo...)
  - rumanator 7y ago
    > I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). You're confusing the occupation with the role. Just because your job title is professor of structural engineering it doesn't mean that you are not studying "matter, its motion and behavior through space and time, and the related entities of energy and force." https://en.wikipedia.org/wiki/Physics https://en.wikipedia.org/wiki/Physics
ineedasername 7y ago
>in my mind there is also a lot of overlap. Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2. [0] https://en.wikipedia.org/wiki/Leo_Breiman https://en.wikipedia.org/wiki/Leo_Breiman
- JamesBarney 7y ago
  Thanks!
majos 7y ago
One possible distinction is: does the work reveal anything beyond the solution itself? The work might, for example, give one instance of a class of problems for which the tool is useful (bonus points for a formal statement to that effect). Or improve the tool, or improve understanding of the tool’s strengths and weaknesses I think this is what we try to capture as “expanding human knowledge”. IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.
amilein7minutes 7y ago
I think the questions asked by researchers in #2 are very different from those by that of #1. The questions mostly surround the why's and how's of AI, i.e, mathematical questions. To take examples from deep learning, #2 might ask about the robustness and generalization of deep neural networks, applying dynamics/ODE theory to certain types of neural networks such as ResNets etc. #1 might ask about the performance of a deep neural network in approximating a given model in a specific application. Alphafold, on the front page currently, is an example of #1.

rumanator 7y ago

> Mind providing some concrete examples? It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications. To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.

JamesBarney 7y ago
I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). But I was asking because I was specifically looking for concrete examples in deep learning.
- ineedasername 7y ago
  Yep, this is why I talk about the virtuous feedback loop between these two modes. Empirical methods feed theory which feeds empirical methods ad infinitum. In the field of ML, a concrete example might be the tool Xgboost (#1) and the original work that led to and developed Gradient Boosting itself (#2), of which Xgboost is an implementation, and probably one that has helped refine the underlying theory as well. ML has lots of examples where the researcher(s) for #2 were also doing #1. A famous paper in NLP comes to mind as an excellent example of this overlap (PDF: https://www.csie.ntu.edu.tw/~b92b02053/print/good-turing-smoothing-without.pdf https://www.csie.ntu.edu.tw/~b92b02053/print/good-turing-smo...)
- rumanator 7y ago
  > I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical). You're confusing the occupation with the role. Just because your job title is professor of structural engineering it doesn't mean that you are not studying "matter, its motion and behavior through space and time, and the related entities of energy and force." https://en.wikipedia.org/wiki/Physics https://en.wikipedia.org/wiki/Physics

ineedasername 7y ago

>in my mind there is also a lot of overlap. Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2. [0] https://en.wikipedia.org/wiki/Leo_Breiman https://en.wikipedia.org/wiki/Leo_Breiman

JamesBarney 7y ago
Thanks!

majos 7y ago

One possible distinction is: does the work reveal anything beyond the solution itself? The work might, for example, give one instance of a class of problems for which the tool is useful (bonus points for a formal statement to that effect). Or improve the tool, or improve understanding of the tool’s strengths and weaknesses I think this is what we try to capture as “expanding human knowledge”. IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.

amilein7minutes 7y ago

I think the questions asked by researchers in #2 are very different from those by that of #1. The questions mostly surround the why's and how's of AI, i.e, mathematical questions. To take examples from deep learning, #2 might ask about the robustness and generalization of deep neural networks, applying dynamics/ODE theory to certain types of neural networks such as ResNets etc. #1 might ask about the performance of a deep neural network in approximating a given model in a specific application. Alphafold, on the front page currently, is an example of #1.