5 ms·

Stanford Class on Deep Multi-Task and Meta-Learning

198 points by hamsterbooster 6y ago

panpanna 6y ago
Asking as someone who does not work with AI but has taken a couple of courses in ML: What new things will I be able to do after this course? (in a practical sense, the technical description on the course page I can read myself)
- Immortal333 6y ago
  At glance, It looks like ongoing research of Multi-Task and Meta-Learning will be discussed. Some new tricks on the architecture of NN, Few new methods to train, and some theoretical setting of the area. I will be looking forward to seeing whole series. Maybe share some notes or summary of videos.
317070 6y ago
The field of meta-learning is still very immature though. I can see why you would already want to start a scholarly discourse on the topic, but I am not sure how useful these techniques are for the students involved. They are still very ad hoc and often unprincipled. This is a good article on the topic: https://arxiv.org/abs/1902.03477 https://arxiv.org/abs/1902.03477
- amcoastal 6y ago
  This is Stanford, they aren't teaching the next generation of applied ML practicioners -- they are teaching the next generation of theorists. This is a perfect class for that and getting their students ahead of everyone else. I'm jealous of them.
  - frakt0x90 6y ago
    Assuming 330 is undergraduate level, they're teaching the full gambit of practitioners, theorists, and future drop outs.
    - goliathDown 6y ago
      300 series are advanced graduate classes. As an undergrad the sight of a 300 is horrifying.
      - dhosek 6y ago
        It's always amusing how different institutions number their classes. At the Claremont Colleges, I took a class Math 103 - Fundamentals of Mathematics which was an upper division course geared towards preparing students for analysis, abstract algebra, etc. Because I started college having completed my math coursework through linear algebra and differential equations, this was the lowest-numbered math course on my transcript from Claremont and when I started grad school for a teaching credential a couple decades later, the program director thought that it was a remedial math course.
        thebradbain 6y ago
        Recent Pomona College alum here checking in to say that the course numbering system has not changed (though Math 103 is now Intro to Combinatorics), and anecdotally it's still a point of confusion for those who go on to the grad schools the Colleges feed into. I've never before paid course numbers too much mind, but it does surprise me there's not yet some widespread standard of to help graduate admissions officers, graduate advisors, and grad students themselves when determining prerequisite eligibilty.
        dhosek 6y ago
        Yeah, I was looking for the class to see what the number was and it doesn't appear to exist anymore. I remember being amused that at Mudd, Calculus as Math 1a/b back in my day. It appears to have been renumbered a bit higher since then and now they only offer one semester of calculus (back in the 80s it was radical that Mudd did the Calculus sequence in two rather than three semesters, although I noticed that our local high school offers a third-year high school calculus class covering multivariable calculus).
      - r00fus 6y ago
- Eridrus 6y ago
  Another paper on the related topic of metric learning arguing that metric learning hasn't actually made any progress and is piggybacking on progress elsewhere: https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505
  - orange3xchicken 6y ago
    This is a pretty good paper, & they bring up many reasonable points, but I think it's important to distinguish deep metric learning from more traditionally formal ml methods for metric learning - there is plenty of progress being made in the context of scalable & provable metric learning algorithms that are robust to noise/corruption & missing data. Recommend work & talks by Anna Gilbert for anyone interested. Entertaining & good at distilling technical content. Here is her most recent one, but there are other good ones on youtube. https://www.youtube.com/watch?v=Sb1ZhtsZjyM https://www.youtube.com/watch?v=Sb1ZhtsZjyM
- devalgo 6y ago
  >This is a good article on the topic: https://arxiv.org/abs/1902.03477 https://arxiv.org/abs/1902.03477 Not to nitpick but that article is a year old and the field is moving at lightspeed
  - 317070 6y ago
    There have not really been a lot of major breakthroughs in meta-learning in the last year, as far as I am aware. The paper is basically saying there was not a lot of progress in the 3 years before that either. All in all, nobody really has a clue on how to do meta-learning right (or I am not aware of their work). There is progress being made on benchmarks, but some argue that progress is not really tackling the real issue at hand, i.e. learning to learn. Moreover, the current common benchmarks are not really good at untangling the progress in deep meta-learning from the progress in deep learning in general.
    - devalgo 6y ago
      Isn't GPT-3 exactly the kind of meta learner you're thinking of?
      - 317070 6y ago
        I would say it is exactly the opposite. :) It is showing how you can get drastically better at deep meta-learning by being better at deep learning. But it does not really show how you can be better at deep meta-learning outside of the improvements in deep learning. You can take any deep meta-learning algorithm, take the deep part in it, apply the improvements in deep learning from the last year and claim that you have improved on the deep meta-learning problem this year. Well yes, but actually also no. It's like trying to find a new antibiotic, and the solution is throwing more existing antibiotics into the same pill. Well yes, it works, but it is also not exactly the problem. Don't get me wrong though, GPT-3 is amazing work.
ArtWomb 6y ago
Humans observe an object once, such as a cup for drinking water, and we immediately grasp its "cupness". We can identify infinite varieties of cups despite variations in morphology, design, utility and context. Simply based on a single learning instance. This absence of any neural theory of inference is at the crux of the problem ;) Shortcut Learning in Deep Neural Networks https://arxiv.org/abs/2004.07780 https://arxiv.org/abs/2004.07780
- bulka 6y ago
  > Simply based on a single learning instance. Did not get this part. I have limited sample of two kids, but I would say it takes at least a year before humans understand "cupness"
  - ghaff 6y ago
    I assume the parent is referring to kids past some level of neurological development and, of course, it's not necessarily as simple as one and done. But, in general, deep learning requires far more examples to train image recognition and even then it's relatively fragile. (Not that humans can't be fooled but having models of the world in our brains help a lot. No, that's probably not a flying pig even though it looks like one.)
    - zamfi 6y ago
      > deep learning requires far more examples to train image recognition Than kids? Who have have video input 10h/day for years (~1B images) and can also choose their examples actively? There are many ways in which deep approaches differ from kids (understatement of the year?), but to say that kids don’t see a lot of data seems not quite right. They’ve got a huge “world model” to draw on by the time they are good at one- or few-shot learning.
      - sp332 6y ago
        I can go to Flickr and download 3 million photos labelled "cup". How many cups have you seen in your whole life? Probably fewer.
arkadyark 6y ago
This looks super cool! Prof. Finn has been doing a lot of interesting research in RL and meta-learning for several years, it's great to get a chance to learn this material directly from her.
mark_l_watson 6y ago
Good to see Chelsea Finn end up at Stanford. I had breakfast with her and her parents in 2013 when she was an undergraduate at MIT and it was fun to hear what options she was thinking about for her career. I took a look at the course outline, and except for AutoML, it appears to be a one stop shop for learning multi-task and meta-learning. I just bookmarked the lectures on youtube.

Asking as someone who does not work with AI but has taken a couple of courses in ML: What new things will I be able to do after this course? (in a practical sense, the technical description on the course page I can read myself)

Immortal333 6y ago
At glance, It looks like ongoing research of Multi-Task and Meta-Learning will be discussed. Some new tricks on the architecture of NN, Few new methods to train, and some theoretical setting of the area. I will be looking forward to seeing whole series. Maybe share some notes or summary of videos.

317070 6y ago

The field of meta-learning is still very immature though. I can see why you would already want to start a scholarly discourse on the topic, but I am not sure how useful these techniques are for the students involved. They are still very ad hoc and often unprincipled. This is a good article on the topic: https://arxiv.org/abs/1902.03477 https://arxiv.org/abs/1902.03477

amcoastal 6y ago
This is Stanford, they aren't teaching the next generation of applied ML practicioners -- they are teaching the next generation of theorists. This is a perfect class for that and getting their students ahead of everyone else. I'm jealous of them.
- frakt0x90 6y ago
  Assuming 330 is undergraduate level, they're teaching the full gambit of practitioners, theorists, and future drop outs.
  - goliathDown 6y ago
    300 series are advanced graduate classes. As an undergrad the sight of a 300 is horrifying.
    - dhosek 6y ago
      It's always amusing how different institutions number their classes. At the Claremont Colleges, I took a class Math 103 - Fundamentals of Mathematics which was an upper division course geared towards preparing students for analysis, abstract algebra, etc. Because I started college having completed my math coursework through linear algebra and differential equations, this was the lowest-numbered math course on my transcript from Claremont and when I started grad school for a teaching credential a couple decades later, the program director thought that it was a remedial math course.
      - thebradbain 6y ago
        Recent Pomona College alum here checking in to say that the course numbering system has not changed (though Math 103 is now Intro to Combinatorics), and anecdotally it's still a point of confusion for those who go on to the grad schools the Colleges feed into. I've never before paid course numbers too much mind, but it does surprise me there's not yet some widespread standard of to help graduate admissions officers, graduate advisors, and grad students themselves when determining prerequisite eligibilty.
        dhosek 6y ago
        Yeah, I was looking for the class to see what the number was and it doesn't appear to exist anymore. I remember being amused that at Mudd, Calculus as Math 1a/b back in my day. It appears to have been renumbered a bit higher since then and now they only offer one semester of calculus (back in the 80s it was radical that Mudd did the Calculus sequence in two rather than three semesters, although I noticed that our local high school offers a third-year high school calculus class covering multivariable calculus).
    - r00fus 6y ago
Eridrus 6y ago
Another paper on the related topic of metric learning arguing that metric learning hasn't actually made any progress and is piggybacking on progress elsewhere: https://arxiv.org/abs/2003.08505 https://arxiv.org/abs/2003.08505
- orange3xchicken 6y ago
  This is a pretty good paper, & they bring up many reasonable points, but I think it's important to distinguish deep metric learning from more traditionally formal ml methods for metric learning - there is plenty of progress being made in the context of scalable & provable metric learning algorithms that are robust to noise/corruption & missing data. Recommend work & talks by Anna Gilbert for anyone interested. Entertaining & good at distilling technical content. Here is her most recent one, but there are other good ones on youtube. https://www.youtube.com/watch?v=Sb1ZhtsZjyM https://www.youtube.com/watch?v=Sb1ZhtsZjyM
devalgo 6y ago
>This is a good article on the topic: https://arxiv.org/abs/1902.03477 https://arxiv.org/abs/1902.03477 Not to nitpick but that article is a year old and the field is moving at lightspeed
- 317070 6y ago
  There have not really been a lot of major breakthroughs in meta-learning in the last year, as far as I am aware. The paper is basically saying there was not a lot of progress in the 3 years before that either. All in all, nobody really has a clue on how to do meta-learning right (or I am not aware of their work). There is progress being made on benchmarks, but some argue that progress is not really tackling the real issue at hand, i.e. learning to learn. Moreover, the current common benchmarks are not really good at untangling the progress in deep meta-learning from the progress in deep learning in general.
  - devalgo 6y ago
    Isn't GPT-3 exactly the kind of meta learner you're thinking of?
    - 317070 6y ago
      I would say it is exactly the opposite. :) It is showing how you can get drastically better at deep meta-learning by being better at deep learning. But it does not really show how you can be better at deep meta-learning outside of the improvements in deep learning. You can take any deep meta-learning algorithm, take the deep part in it, apply the improvements in deep learning from the last year and claim that you have improved on the deep meta-learning problem this year. Well yes, but actually also no. It's like trying to find a new antibiotic, and the solution is throwing more existing antibiotics into the same pill. Well yes, it works, but it is also not exactly the problem. Don't get me wrong though, GPT-3 is amazing work.

ArtWomb 6y ago

Humans observe an object once, such as a cup for drinking water, and we immediately grasp its "cupness". We can identify infinite varieties of cups despite variations in morphology, design, utility and context. Simply based on a single learning instance. This absence of any neural theory of inference is at the crux of the problem ;) Shortcut Learning in Deep Neural Networks https://arxiv.org/abs/2004.07780 https://arxiv.org/abs/2004.07780

bulka 6y ago
> Simply based on a single learning instance. Did not get this part. I have limited sample of two kids, but I would say it takes at least a year before humans understand "cupness"
- ghaff 6y ago
  I assume the parent is referring to kids past some level of neurological development and, of course, it's not necessarily as simple as one and done. But, in general, deep learning requires far more examples to train image recognition and even then it's relatively fragile. (Not that humans can't be fooled but having models of the world in our brains help a lot. No, that's probably not a flying pig even though it looks like one.)
  - zamfi 6y ago
    > deep learning requires far more examples to train image recognition Than kids? Who have have video input 10h/day for years (~1B images) and can also choose their examples actively? There are many ways in which deep approaches differ from kids (understatement of the year?), but to say that kids don’t see a lot of data seems not quite right. They’ve got a huge “world model” to draw on by the time they are good at one- or few-shot learning.
    - sp332 6y ago
      I can go to Flickr and download 3 million photos labelled "cup". How many cups have you seen in your whole life? Probably fewer.

arkadyark 6y ago

This looks super cool! Prof. Finn has been doing a lot of interesting research in RL and meta-learning for several years, it's great to get a chance to learn this material directly from her.

mark_l_watson 6y ago

Good to see Chelsea Finn end up at Stanford. I had breakfast with her and her parents in 2013 when she was an undergraduate at MIT and it was fun to hear what options she was thinking about for her career. I took a look at the course outline, and except for AutoML, it appears to be a one stop shop for learning multi-task and meta-learning. I just bookmarked the lectures on youtube.