3 ms·

There was a discussion about the paper[0] this was based on 7 days ago: https://news.ycombinator.com/item?id=35768824 https://news.ycombinator.com/item?id=35768

There was a discussion about the paper[0] this was based on 7 days ago: https://news.ycombinator.com/item?id=35768824 https://news.ycombinator.com/item?id=35768824 [0] https://arxiv.org/abs/2304.15004 https://arxiv.org/abs/2304.15004

dang 3y ago
Thanks! Macroexpanded: Are emergent abilities of large language models a mirage? - https://news.ycombinator.com/item?id=35768824 https://news.ycombinator.com/item?id=35768824 - May 2023 (126 comments)
Certhas 3y ago
Glad it came up again. I missed it the first time. I am a complex systems scientist and its hard to overstate the importance of this. The fact that there are reasonable metrics that make the emergence completely predictable as a simple threshold effect is massive. One alternative way to phrase this that I haven't seen in either thread yet (but I am on mobile and haven't looked thoroughly) would be: Human observers/users dramatically underestimate how much small language models already have learned, because we are very good at spotting the errors. Scaling up looks like emergence occurs because the last few errors get eliminated.
- krageon 3y ago
  If anything, it looks to me like the researchers are saying the opposite: We understand how this works, so it cannot be magic or special. The fact that you think it is means you are just wrong. Sadly it is possible I have arrived at this conclusion due to previous experience with the academic world, which is ironically full of magical thinkers that will take any sort of AI advancement and scoff because it may mean they're no longer as special as they feel they must be. It is a very hostile environment.
  - Certhas 3y ago
    They are saying correctly that we _can_ understand the learning of models in a continuous fashion, and that we do not need to assume that at some stage something different than at other stages happens. So no emergence. This is massive because it means, for example, that you can study what is happening in these models with small ones that perform worse but are simpler to understand, and you're not missing something fundamental. I don't think they claim we already understand what is happening I these models though. I don't know what exactly you are talking about with your academia bashing. If anything it seems to me academia is just as awash as everywhere else with barely substantiated AI hype. Disparaging and psychologizing critical voices just makes it seem to me like you might be missing the point of academia...
- _yb2s 3y ago
  Great summary, I think it clarifies an important point. It's not that these models aren't able to do the surprising things people are claiming they can do, but that in fact the smaller versions of the models had these type of capabilities as well, but to a lesser degree, and nobody was aware of it because we weren't even looking for it.
- mnky9800n 3y ago
  what does a complex systems scientist do?
  - lapama 3y ago
    See the world from a systemic relational approach, and publish and talk about it.

dang 3y ago

Thanks! Macroexpanded: Are emergent abilities of large language models a mirage? - https://news.ycombinator.com/item?id=35768824 https://news.ycombinator.com/item?id=35768824 - May 2023 (126 comments)

Certhas 3y ago

Glad it came up again. I missed it the first time. I am a complex systems scientist and its hard to overstate the importance of this. The fact that there are reasonable metrics that make the emergence completely predictable as a simple threshold effect is massive. One alternative way to phrase this that I haven't seen in either thread yet (but I am on mobile and haven't looked thoroughly) would be: Human observers/users dramatically underestimate how much small language models already have learned, because we are very good at spotting the errors. Scaling up looks like emergence occurs because the last few errors get eliminated.

krageon 3y ago
If anything, it looks to me like the researchers are saying the opposite: We understand how this works, so it cannot be magic or special. The fact that you think it is means you are just wrong. Sadly it is possible I have arrived at this conclusion due to previous experience with the academic world, which is ironically full of magical thinkers that will take any sort of AI advancement and scoff because it may mean they're no longer as special as they feel they must be. It is a very hostile environment.
- Certhas 3y ago
  They are saying correctly that we _can_ understand the learning of models in a continuous fashion, and that we do not need to assume that at some stage something different than at other stages happens. So no emergence. This is massive because it means, for example, that you can study what is happening in these models with small ones that perform worse but are simpler to understand, and you're not missing something fundamental. I don't think they claim we already understand what is happening I these models though. I don't know what exactly you are talking about with your academia bashing. If anything it seems to me academia is just as awash as everywhere else with barely substantiated AI hype. Disparaging and psychologizing critical voices just makes it seem to me like you might be missing the point of academia...
_yb2s 3y ago
Great summary, I think it clarifies an important point. It's not that these models aren't able to do the surprising things people are claiming they can do, but that in fact the smaller versions of the models had these type of capabilities as well, but to a lesser degree, and nobody was aware of it because we weren't even looking for it.
mnky9800n 3y ago
what does a complex systems scientist do?
- lapama 3y ago
  See the world from a systemic relational approach, and publish and talk about it.