5 ms·

Studying Math at Harvard/MIT certainly puts you in a different category than the average software engineer. And if ML was still challenging to Greg, it is hones

Studying Math at Harvard/MIT certainly puts you in a different category than the average software engineer. And if ML was still challenging to Greg, it is honestly a bit discouraging.

gdb 7y ago
(I wrote the post.) If it's helpful, I dropped out of both schools — the vast majority of my knowledge is self taught!
- jcims 7y ago
  >I learn best when I have something specific in mind to build. This is so incredibly important for me and, based on my conversations, many others as well. The other thing I struggle with is the feeling that many of the problems I wish to solve are likely also solvable with simpler statistical methods and that I'm just being a poser by trying to pound them home with the ML hammer.
- applecrazy 7y ago
  Question: will the OpenAI fellows curriculum ever be released? I need a nice, structured intro to deep learning research and feel like the curriculum modules your company has developed would have the highest quality. (For reference, I’m an undergrad looking to get into this field)
- knbknb 7y ago
  How often do you go "back to basics" (read introductory book chapters, or classic papers, etc)?
streetcat1 7y ago
Maybe if you want to write tensorflow, but not if you want to use it.
- eanzenberg 7y ago
  If you want to write TF, then you are a ml engineer. If you want to use TF, then you are a scientist (data, research, whatever)
omarhaneef 7y ago
I think "doing" ML math is a little different than "doing" math math. With math, on paper say, it is hard to tell if you are doing it right or wrong. You can trick yourself quite easily. A compelling proof can have a huge hole. You can still trick yourself programming -- in a sense, that is what a bug is -- but it is much harder. The upshot is, I think it is easier to teach yourself math that is applied to a computer program than math on a piece of paper.
dhairya 7y ago
So there is a difference between ML research and application. Being a practitioner doesn't require deep math knowledge that perhaps research would. Jeremy Howard's fastai course is a great example of how someone with a solid programming background can effectively transition into being a deep learning practitioner. Given that production ml and deep learning is still the wild west, as a practioner, you can contribute also to the research around effective training, scaling, and application of these models. The math and intuition required are definitely acquirable. I think when you shift into pure research, yes a deep probability, information theory, linear algebra, and calculus background are needed. But at the level, you're rarely writing code and more likely working at theoretical level.
- jph00 7y ago
  Also, note that Greg's goal was to contribute to OpenAI's flagship project. That's a rather ambitious goal! Also, most folks I know that are making practical deep learning contributions are doing so by combining their pre-existing domain expertise with their new deep learning skills. E.g. a journalist analyzing a large corpus of text for a story, or an oil&gas analyst building models from well plots, etc.
  - dhairya 7y ago
    as a side note, I love that you highlight regex in your new NLP course. There is an inherent tension between the probabilistic nature of models and the need for deterministic outputs in most production settings. Often if we can uncover linguistics rules or regex patterns that guarantee minimal precision (or as our VP puts it - don't look stupid), we'll eschew the model in the short term or use the model to augment the rules. Also I really appreciated that on of the training goals for ULMfit was to be trainable on a single gpu. With these large-capacity models, training is getting crazy expensive and out of hand. Any chance that your future work will still keep the single gpu training goal?
- sevensor 7y ago
  I recently got the assignment to "do ML" on some data. I hadn't done anything in the area before, and a couple of things surprised me: 1. Most of your time is spent transforming data. Very little is spent building models. 2. Most of the eye-grabbing stuff that makes headlines is inapplicable. My application involves decisions that are expensive and can be safety critical. The models themselves have to be simple enough to be reasoned about, or they're no use. You might argue that this means what I'm actually doing is statistics.

Studying Math at Harvard/MIT certainly puts you in a different category than the average software engineer. And if ML was still challenging to Greg, it is honestly a bit discouraging.

gdb 7y ago

(I wrote the post.) If it's helpful, I dropped out of both schools — the vast majority of my knowledge is self taught!

jcims 7y ago
>I learn best when I have something specific in mind to build. This is so incredibly important for me and, based on my conversations, many others as well. The other thing I struggle with is the feeling that many of the problems I wish to solve are likely also solvable with simpler statistical methods and that I'm just being a poser by trying to pound them home with the ML hammer.
applecrazy 7y ago
Question: will the OpenAI fellows curriculum ever be released? I need a nice, structured intro to deep learning research and feel like the curriculum modules your company has developed would have the highest quality. (For reference, I’m an undergrad looking to get into this field)
knbknb 7y ago
How often do you go "back to basics" (read introductory book chapters, or classic papers, etc)?

streetcat1 7y ago

Maybe if you want to write tensorflow, but not if you want to use it.

eanzenberg 7y ago
If you want to write TF, then you are a ml engineer. If you want to use TF, then you are a scientist (data, research, whatever)

omarhaneef 7y ago

I think "doing" ML math is a little different than "doing" math math. With math, on paper say, it is hard to tell if you are doing it right or wrong. You can trick yourself quite easily. A compelling proof can have a huge hole. You can still trick yourself programming -- in a sense, that is what a bug is -- but it is much harder. The upshot is, I think it is easier to teach yourself math that is applied to a computer program than math on a piece of paper.

dhairya 7y ago

So there is a difference between ML research and application. Being a practitioner doesn't require deep math knowledge that perhaps research would. Jeremy Howard's fastai course is a great example of how someone with a solid programming background can effectively transition into being a deep learning practitioner. Given that production ml and deep learning is still the wild west, as a practioner, you can contribute also to the research around effective training, scaling, and application of these models. The math and intuition required are definitely acquirable. I think when you shift into pure research, yes a deep probability, information theory, linear algebra, and calculus background are needed. But at the level, you're rarely writing code and more likely working at theoretical level.

jph00 7y ago
Also, note that Greg's goal was to contribute to OpenAI's flagship project. That's a rather ambitious goal! Also, most folks I know that are making practical deep learning contributions are doing so by combining their pre-existing domain expertise with their new deep learning skills. E.g. a journalist analyzing a large corpus of text for a story, or an oil&gas analyst building models from well plots, etc.
- dhairya 7y ago
  as a side note, I love that you highlight regex in your new NLP course. There is an inherent tension between the probabilistic nature of models and the need for deterministic outputs in most production settings. Often if we can uncover linguistics rules or regex patterns that guarantee minimal precision (or as our VP puts it - don't look stupid), we'll eschew the model in the short term or use the model to augment the rules. Also I really appreciated that on of the training goals for ULMfit was to be trainable on a single gpu. With these large-capacity models, training is getting crazy expensive and out of hand. Any chance that your future work will still keep the single gpu training goal?
sevensor 7y ago
I recently got the assignment to "do ML" on some data. I hadn't done anything in the area before, and a couple of things surprised me: 1. Most of your time is spent transforming data. Very little is spent building models. 2. Most of the eye-grabbing stuff that makes headlines is inapplicable. My application involves decisions that are expensive and can be safety critical. The models themselves have to be simple enough to be reasoned about, or they're no use. You might argue that this means what I'm actually doing is statistics.