3 ms·

https://www.gwern.net/newsletter/2020/05#meta-learning https://www.gwern.net/newsletter/2020/05#meta-learning

by fossuser 6y ago

typon 6y ago
The site you linked asserts "meta-learning" is happening in GPT3 without any evidence of it
- fossuser 6y ago
  Isn't the capability to answer arithmetic prompts without the exact equation being present in the training data evidence? I'm not an AI researcher so I only have a lay-person's opinion from reading about this kind of thing via things like that blog post (I also don't have OpenAI access to play with the API myself), but if that is an accurate statement it seems like pretty good evidence of figuring out the underlying mathematical rules. I think this could be a path towards scaling being more effective than people think. I sense though that you're just arguing from a position of motivated reasoning based on a pre-existing conclusion rather than actually trying to look at new things that may contradict what you already believe to be true. Repeated one sentence responses arguing that none of this is evidence just isn't worth time to respond to. In the end we'll see what ends up happening.
  - deleted 6y ago
    [deleted]
YeGoblynQueenne 6y ago
Can I ask why you are not citing the GPT-3 paper itself? Why do we need to look somewhere else for evidence about anything to do with GPT-3 (or indeed any system) than the work of the people who have created it?
- fossuser 6y ago
  That's a fair criticism, original paper would be better. I linked to the blog because that's where I first read about this and it's more immediately accessible. [Edit: Paper Link, https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165] In fact after reading these sections of the actual paper, it's hard to believe that you could have read it yourself and taken away the idea that it was memorization? Pages 22-23 in particular. Part of the reason I tend to link to good blog posts instead of academic papers is when you link to papers nobody reads them. Often people linking them haven't read them either (I've only read small parts).
  - YeGoblynQueenne 6y ago
    I think you're saying I didn't read the paper, because I didn't draw the same conclusions from it as you did after reading a blog post about it? Again there's a simpler explanation: I read the paper and I drew different conclusions than you or your primary source. I give a very long explanation of why I drew those conclusions, above. To clarify, I discussed the exact same subject a couple of weeks ago, I think, after reading the GPT-3 paper. I made my original comment in this thread without re-reading the paper, so I misremembered what was in it. Then I re-read the relevant section (3.9.1. Arithmetic) to refresh my memory and made the very long comment above. That's in the intereste of full disclosure and so you don't have any reason to assume I didn't read the paper, which you shouldn't anyway.

https://www.gwern.net/newsletter/2020/05#meta-learning https://www.gwern.net/newsletter/2020/05#meta-learning

typon 6y ago

The site you linked asserts "meta-learning" is happening in GPT3 without any evidence of it

fossuser 6y ago
Isn't the capability to answer arithmetic prompts without the exact equation being present in the training data evidence? I'm not an AI researcher so I only have a lay-person's opinion from reading about this kind of thing via things like that blog post (I also don't have OpenAI access to play with the API myself), but if that is an accurate statement it seems like pretty good evidence of figuring out the underlying mathematical rules. I think this could be a path towards scaling being more effective than people think. I sense though that you're just arguing from a position of motivated reasoning based on a pre-existing conclusion rather than actually trying to look at new things that may contradict what you already believe to be true. Repeated one sentence responses arguing that none of this is evidence just isn't worth time to respond to. In the end we'll see what ends up happening.
- deleted 6y ago
  [deleted]

YeGoblynQueenne 6y ago

Can I ask why you are not citing the GPT-3 paper itself? Why do we need to look somewhere else for evidence about anything to do with GPT-3 (or indeed any system) than the work of the people who have created it?

fossuser 6y ago
That's a fair criticism, original paper would be better. I linked to the blog because that's where I first read about this and it's more immediately accessible. [Edit: Paper Link, https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165] In fact after reading these sections of the actual paper, it's hard to believe that you could have read it yourself and taken away the idea that it was memorization? Pages 22-23 in particular. Part of the reason I tend to link to good blog posts instead of academic papers is when you link to papers nobody reads them. Often people linking them haven't read them either (I've only read small parts).
- YeGoblynQueenne 6y ago
  I think you're saying I didn't read the paper, because I didn't draw the same conclusions from it as you did after reading a blog post about it? Again there's a simpler explanation: I read the paper and I drew different conclusions than you or your primary source. I give a very long explanation of why I drew those conclusions, above. To clarify, I discussed the exact same subject a couple of weeks ago, I think, after reading the GPT-3 paper. I made my original comment in this thread without re-reading the paper, so I misremembered what was in it. Then I re-read the relevant section (3.9.1. Arithmetic) to refresh my memory and made the very long comment above. That's in the intereste of full disclosure and so you don't have any reason to assume I didn't read the paper, which you shouldn't anyway.