9 ms·

Jeff Dean’s Lecture for YC AI [video]

439 points by danicgross 9y ago

iandanforth 9y ago
The notion of running one giant model that has many sub-talents is epic. I can imagine that all the disparate models they run today could fuse into a giant network that melds predictions and guides computation as required by the task. That seems like a very Jeff Dean scale endeavor.
- cosminro 9y ago
  He is referring to this paper https://research.googleblog.com/2017/06/multimodel-multi-task-machine-learning.html https://research.googleblog.com/2017/06/multimodel-multi-tas... Where they used one model to do image recognition, speech recognition and translation in the same network.
  - deleted 9y ago
    [deleted]
  - shpx 9y ago
    He also mentions sparsely activating only the neurons that matter, which they explore in https://arxiv.org/pdf/1701.06538.pdf https://arxiv.org/pdf/1701.06538.pdf Personally I didn't find it very satisfying, I imagine something more fundamental and self referential.
- soVeryTired 9y ago
  Sounds like a joy to debug
- erikpukinskis 9y ago
  It's a great idea, and human beings are proof that conjoined colonies of intelligent agents are going to be smarter than individuals. But it also runs up against tradeoffs, as evidenced by the fact that many humans exist, and we didn't evolve into just one giant ur-human, even after we all got our internet implant. Sometimes a few agents with different strategies working independently is better than a single agent choosing amongst the strategies of several others.
sputknick 9y ago
If Tensorflow becomes the default library for Deep learning, is this a good thing or bad thing? Does it help in that all researchers can focus on what's important (the data and results) or does it hurt in that Google now controls an important paradigm for the next generation of computing?
- li4ick 9y ago
  Researchers use Tensorflow for backend only + Keras for implementation. However, PyTorch and Caffe are much more popular among researchers. I personally enjoy PyTorch a lot.
  - niyazpk 9y ago
    >> I personally enjoy PyTorch a lot Why?
- halflings 9y ago
  I fail to see what's wrong with this: it's open-source, and anybody can write different libraries, or libraries that use all the low-level work ($$$ paying experienced/skilled engineers) that Google pours into making tensorflow. Alternatives exist (PyTorch by facebook, theano, caffe, etc.) in any case, and the best solution wins. (in the same way scikit-learn once eclipsed all other ML libraries, but some would still use xgboost which had superior Gradient Boosting Trees).
  - sillysaurus3 9y ago
    the best solution wins Minor quibble, but: not really. The earliest solution wins, e.g. rails. It has to be good enough, but after a certain point "best" stops mattering.
    - omot 9y ago
      > The earliest solution wins Minor minor quibble, but: not really. The earliest solution wins, unless there is a 10x better solution, e.g Google. There were a lot of good enough options, but after a certain benchmark "earliest" stops mattering.
bluetwo 9y ago
If a doctor misdiagnosis an eye ailment, they might end up with a malpractice lawsuit. If an ML program misdiagnoses an eye ailment, what is going to happen?
- melling 9y ago
  If a car has to decide between killing ... The trolley problem: https://www.youtube.com/watch?v=21EiKfQYZXc&feature=youtu.be&t=1h25m50s https://www.youtube.com/watch?v=21EiKfQYZXc&feature=youtu.be...
- otoburb 9y ago
  The commercial entity backing the ML program should be responsible. If one doesn't exist then the comparison fails, as one would presumably not be willing to trust the competency of a public domain ML program without a true, commercially and medically backed "second" opinion (algorithmic or otherwise). Perhaps one day, ML programs running eye diagnostics will be as cheap and disposable to use as pregnancy stick tests: "Point and diagnose."
  - TuringNYC 9y ago
    Yes, the commercial entity backing the ML program, and possibly multiple associated entities would be held responsible, or at least dragged into litigation. You can get insurance companies to write custom policies for such things, but I dont think there has yet been a standards-setting case -- anyone know of existing or in-progress case law in this arena?
  - solomatov 9y ago
    Most likely all these models have TOS or EULAs which shift liability in such situations.
- dguaraglia 9y ago
  This is why these tools are generally described as "decision support" systems and not "diagnostics" systems. There are many reasons why you don't want to be classified as a diagnosis device (FDA approval is a big one), but ultimately letting the doctor make the final decision removes the ML controversy out of the picture. EDIT, for context: https://en.wikipedia.org/wiki/Clinical_decision_support_system https://en.wikipedia.org/wiki/Clinical_decision_support_syst...
deboflo 9y ago
Once, in early 2002, when the index servers went down, Jeff Dean answered user queries manually for two hours. Evals showed a quality improvement of 5 points.
- 1_2__4 9y ago
  I never thought Jeff Dean jokes would be a shibboleth but here we are.
  - nickelbox 9y ago
    I could imagine there being external Jeff Dean fans, but yeah, that jargon
    - ma2rten 9y ago
      https://www.quora.com/What-are-all-the-Jeff-Dean-facts https://www.quora.com/What-are-all-the-Jeff-Dean-facts
deleted 9y ago
[deleted]
hallman76 9y ago
As a ML enthusiast, this is incredible to watch! I'm completely blown away that Google was working on full-scale physical architectures that were optimized for these problems. Talk about being two steps ahead of the game!
- londons_explore 9y ago
  Smart people, combined with piles of cash, gets you ahead at almost anything... Sadly there's still a lot of smart people without access to cash (academics), and piles of cash without smart people to get the value out of it.
litzer 9y ago
As somebody who's recently starting to learn more about ML, a lot of the work of an ML engineer does seem to be automate-able (not doing research or pushing boundaries but just applying ML to some product need). For example, choosing hyperparameters, evaluating which features to collect, etc seem to be things that can be automated with very little human input. His slide on "learning to learn" has a goal of removing the ML expert in the equation. Can somebody who's more of an expert in the field comment on how plausible it is? Specifically, in the near future, will we only need ML people who do research, due to the application being so trivial to do once automated?
- halflings 9y ago
  You will still need data engineers to build the whole data ingestion and processing pipeline (although that can be easy if standardised tools are available, such as spark, it's still a challenge in many cases).
  - litzer 9y ago
    Right, but I'd consider that falling closer to the realm of general software engineering -- similar to tasks of collecting analytics of users or building infrastructure to get data from point A to point B. Maybe that currently is some parts of the job of an ML engineer. But if that's the only part, I don't think that role should be called one of ML engineer anymore
  - davedx 9y ago
    I am working on solving this problem at the moment - I'm building a product that lets anyone build the ETL pipelines that produce inputs for a ML model. If anyone's interested in beta access (coming month or two) let me know, davedx@gmail.com
- wenc 9y ago
  There is one job that is still difficult for a machine to do well (although machines are improving): feature engineering. ML works very well in bounded/closed domains like image and sound recognition. Open-domains are much more challenging. Building predictive models from data in specialized domains often require insight, which machines cannot provide. For instance, let's say you collect a bunch of data and are trying to predict sales. You need to apply domain knowledge, experience and intuition to know what variables are causal or correlative. If you just throw all the variables into the mix and build a model from that, you will end up with a model that overfits badly. There are automated "variable selection" techniques that can help to prevent overfit, but they are mostly imperfect because machines can only detect correlation and not causation. Also, many regression/classification techniques are easily fooled by noise and highly nonlinear relationships. We did some work a few years ago comparing predictive models built from a ton of sensor data (with automated variable selection) vs. one that was parsimonious that was built on select data that we knew accounted for 80% of the effect. The latter model was far superior. Noise/non-causal variables often don't just "wash out" even with very good variable selection algorithms. It takes domain knowledge to figure out what variables matter and what variables don't.

439 points by danicgross 9y ago

The notion of running one giant model that has many sub-talents is epic. I can imagine that all the disparate models they run today could fuse into a giant network that melds predictions and guides computation as required by the task. That seems like a very Jeff Dean scale endeavor.

cosminro 9y ago
He is referring to this paper https://research.googleblog.com/2017/06/multimodel-multi-task-machine-learning.html https://research.googleblog.com/2017/06/multimodel-multi-tas... Where they used one model to do image recognition, speech recognition and translation in the same network.
- deleted 9y ago
  [deleted]
- shpx 9y ago
  He also mentions sparsely activating only the neurons that matter, which they explore in https://arxiv.org/pdf/1701.06538.pdf https://arxiv.org/pdf/1701.06538.pdf Personally I didn't find it very satisfying, I imagine something more fundamental and self referential.
soVeryTired 9y ago
Sounds like a joy to debug
erikpukinskis 9y ago
It's a great idea, and human beings are proof that conjoined colonies of intelligent agents are going to be smarter than individuals. But it also runs up against tradeoffs, as evidenced by the fact that many humans exist, and we didn't evolve into just one giant ur-human, even after we all got our internet implant. Sometimes a few agents with different strategies working independently is better than a single agent choosing amongst the strategies of several others.

sputknick 9y ago

If Tensorflow becomes the default library for Deep learning, is this a good thing or bad thing? Does it help in that all researchers can focus on what's important (the data and results) or does it hurt in that Google now controls an important paradigm for the next generation of computing?

li4ick 9y ago
Researchers use Tensorflow for backend only + Keras for implementation. However, PyTorch and Caffe are much more popular among researchers. I personally enjoy PyTorch a lot.
- niyazpk 9y ago
  >> I personally enjoy PyTorch a lot Why?
halflings 9y ago
I fail to see what's wrong with this: it's open-source, and anybody can write different libraries, or libraries that use all the low-level work ($$$ paying experienced/skilled engineers) that Google pours into making tensorflow. Alternatives exist (PyTorch by facebook, theano, caffe, etc.) in any case, and the best solution wins. (in the same way scikit-learn once eclipsed all other ML libraries, but some would still use xgboost which had superior Gradient Boosting Trees).
- sillysaurus3 9y ago
  the best solution wins Minor quibble, but: not really. The earliest solution wins, e.g. rails. It has to be good enough, but after a certain point "best" stops mattering.
  - omot 9y ago
    > The earliest solution wins Minor minor quibble, but: not really. The earliest solution wins, unless there is a 10x better solution, e.g Google. There were a lot of good enough options, but after a certain benchmark "earliest" stops mattering.

bluetwo 9y ago

If a doctor misdiagnosis an eye ailment, they might end up with a malpractice lawsuit. If an ML program misdiagnoses an eye ailment, what is going to happen?

melling 9y ago
If a car has to decide between killing ... The trolley problem: https://www.youtube.com/watch?v=21EiKfQYZXc&feature=youtu.be&t=1h25m50s https://www.youtube.com/watch?v=21EiKfQYZXc&feature=youtu.be...
otoburb 9y ago
The commercial entity backing the ML program should be responsible. If one doesn't exist then the comparison fails, as one would presumably not be willing to trust the competency of a public domain ML program without a true, commercially and medically backed "second" opinion (algorithmic or otherwise). Perhaps one day, ML programs running eye diagnostics will be as cheap and disposable to use as pregnancy stick tests: "Point and diagnose."
- TuringNYC 9y ago
  Yes, the commercial entity backing the ML program, and possibly multiple associated entities would be held responsible, or at least dragged into litigation. You can get insurance companies to write custom policies for such things, but I dont think there has yet been a standards-setting case -- anyone know of existing or in-progress case law in this arena?
- solomatov 9y ago
  Most likely all these models have TOS or EULAs which shift liability in such situations.
dguaraglia 9y ago
This is why these tools are generally described as "decision support" systems and not "diagnostics" systems. There are many reasons why you don't want to be classified as a diagnosis device (FDA approval is a big one), but ultimately letting the doctor make the final decision removes the ML controversy out of the picture. EDIT, for context: https://en.wikipedia.org/wiki/Clinical_decision_support_system https://en.wikipedia.org/wiki/Clinical_decision_support_syst...

deboflo 9y ago

Once, in early 2002, when the index servers went down, Jeff Dean answered user queries manually for two hours. Evals showed a quality improvement of 5 points.

1_2__4 9y ago
I never thought Jeff Dean jokes would be a shibboleth but here we are.
- nickelbox 9y ago
  I could imagine there being external Jeff Dean fans, but yeah, that jargon
  - ma2rten 9y ago
    https://www.quora.com/What-are-all-the-Jeff-Dean-facts https://www.quora.com/What-are-all-the-Jeff-Dean-facts

deleted 9y ago

[deleted]

hallman76 9y ago

As a ML enthusiast, this is incredible to watch! I'm completely blown away that Google was working on full-scale physical architectures that were optimized for these problems. Talk about being two steps ahead of the game!

londons_explore 9y ago
Smart people, combined with piles of cash, gets you ahead at almost anything... Sadly there's still a lot of smart people without access to cash (academics), and piles of cash without smart people to get the value out of it.

litzer 9y ago

As somebody who's recently starting to learn more about ML, a lot of the work of an ML engineer does seem to be automate-able (not doing research or pushing boundaries but just applying ML to some product need). For example, choosing hyperparameters, evaluating which features to collect, etc seem to be things that can be automated with very little human input. His slide on "learning to learn" has a goal of removing the ML expert in the equation. Can somebody who's more of an expert in the field comment on how plausible it is? Specifically, in the near future, will we only need ML people who do research, due to the application being so trivial to do once automated?

halflings 9y ago
You will still need data engineers to build the whole data ingestion and processing pipeline (although that can be easy if standardised tools are available, such as spark, it's still a challenge in many cases).
- litzer 9y ago
  Right, but I'd consider that falling closer to the realm of general software engineering -- similar to tasks of collecting analytics of users or building infrastructure to get data from point A to point B. Maybe that currently is some parts of the job of an ML engineer. But if that's the only part, I don't think that role should be called one of ML engineer anymore
- davedx 9y ago
  I am working on solving this problem at the moment - I'm building a product that lets anyone build the ETL pipelines that produce inputs for a ML model. If anyone's interested in beta access (coming month or two) let me know, davedx@gmail.com
wenc 9y ago
There is one job that is still difficult for a machine to do well (although machines are improving): feature engineering. ML works very well in bounded/closed domains like image and sound recognition. Open-domains are much more challenging. Building predictive models from data in specialized domains often require insight, which machines cannot provide. For instance, let's say you collect a bunch of data and are trying to predict sales. You need to apply domain knowledge, experience and intuition to know what variables are causal or correlative. If you just throw all the variables into the mix and build a model from that, you will end up with a model that overfits badly. There are automated "variable selection" techniques that can help to prevent overfit, but they are mostly imperfect because machines can only detect correlation and not causation. Also, many regression/classification techniques are easily fooled by noise and highly nonlinear relationships. We did some work a few years ago comparing predictive models built from a ton of sensor data (with automated variable selection) vs. one that was parsimonious that was built on select data that we knew accounted for 80% of the effect. The latter model was far superior. Noise/non-causal variables often don't just "wash out" even with very good variable selection algorithms. It takes domain knowledge to figure out what variables matter and what variables don't.