3 ms·

As a side note, it's fascinating to read the comments on that thread when talking about RNNs and Deep Learning. So much has changed in the last 6 years and feel

by adlpz 4y ago

As a side note, it's fascinating to read the comments on that thread when talking about RNNs and Deep Learning. So much has changed in the last 6 years and feels so weird to read dismissive comments about the capabilities of these systems seeing what people are getting out of ChatGPT.

whatshisface 4y ago
ChatGPT hasn't overcome any of the fundamental issues, it's just a huge improvement on the things that the original GPTs were good at. Being able to stay coherent for a trained-in length that gets longer with larger models is different from the length-unlimited coherence that human beings can manage, spanning lifetimes of thought and multiple lifetimes of discourse.
- dragonwriter 4y ago
  > ChatGPT hasn't overcome any of the fundamental issues, it's just a huge improvement on the things that the original GPTs were good at. Being able to stay coherent for a trained-in length that gets longer with larger models is different from being internally coherent for 90 years by nature like people are. People are very often not internally coherent over periods much shorter than 90 years.
  - whatshisface 4y ago
    The hazily defined ideal of general intelligence - which everyone imagines that they are, but most of us do not live up to all the time, and nobody lives up to every waking moment (or at least before we've had our morning cup of coffee) isn't within the reach of present day transformer architectures because the length of text they can stay coherent over is limited by how far back they can look through their own output. Human beings can form new memories from their outputs that stay with them for the rest of their lives.
- IIAOPSW 4y ago
  >length-unlimited coherence that human beings can manage I've yet to meet a human that doesn't eventually run out of steam. You just aren't well situated to notice decoherence on time scales longer than your own coherence.
- nl 4y ago
  In 2016 Transformers didn't exist and the state of the art for neural network based NLP was using LSTMs that had a limit of maybe 100 words at most. With new implementations like xformers[1] and flash attention[2] it is unclear where the length limit is on modern transformer models. Flash Attention can currently scale up to 64,000 tokens on an A100. [1] https://github.com/facebookresearch/xformers/blob/main/HOWTO.md https://github.com/facebookresearch/xformers/blob/main/HOWTO... [2] https://github.com/HazyResearch/flash-attention https://github.com/HazyResearch/flash-attention
BlueTemplar 4y ago
Not really ? « Inflated » expectations doesn't mean NO expectations... People are still throwing « AI » around as a buzzword like it's something distinct from a computer program, in fact the situation got worse because non-neural network programs are somehow dismissed as « not AI » now. Autonomous cars are still nowhere to be seen, even more so for « General « AI » ». The singularity isn't much on track either looking at Kurzweil's predictions : we should have had molecular manufacturers by now, and nanobots connecting our brains to the Internet, extending our lives, and brain scanning people to recreate their avatars when they are dead, don't seem like they are going to happen by the 2030s either. (2045 is still far enough away that I wouldn't completely bet against a singularity by then.) (And Kurzweil doesn't get to blame it on misunderstanding the exponential function : how people have a too linear view of the future and tend to overestimate changes in the near future, but underestimate them in the long term !)
- alexheikel 4y ago
  [dead]

by adlpz 4y ago

whatshisface 4y ago

ChatGPT hasn't overcome any of the fundamental issues, it's just a huge improvement on the things that the original GPTs were good at. Being able to stay coherent for a trained-in length that gets longer with larger models is different from the length-unlimited coherence that human beings can manage, spanning lifetimes of thought and multiple lifetimes of discourse.

dragonwriter 4y ago
> ChatGPT hasn't overcome any of the fundamental issues, it's just a huge improvement on the things that the original GPTs were good at. Being able to stay coherent for a trained-in length that gets longer with larger models is different from being internally coherent for 90 years by nature like people are. People are very often not internally coherent over periods much shorter than 90 years.
- whatshisface 4y ago
  The hazily defined ideal of general intelligence - which everyone imagines that they are, but most of us do not live up to all the time, and nobody lives up to every waking moment (or at least before we've had our morning cup of coffee) isn't within the reach of present day transformer architectures because the length of text they can stay coherent over is limited by how far back they can look through their own output. Human beings can form new memories from their outputs that stay with them for the rest of their lives.
IIAOPSW 4y ago
>length-unlimited coherence that human beings can manage I've yet to meet a human that doesn't eventually run out of steam. You just aren't well situated to notice decoherence on time scales longer than your own coherence.
nl 4y ago
In 2016 Transformers didn't exist and the state of the art for neural network based NLP was using LSTMs that had a limit of maybe 100 words at most. With new implementations like xformers[1] and flash attention[2] it is unclear where the length limit is on modern transformer models. Flash Attention can currently scale up to 64,000 tokens on an A100. [1] https://github.com/facebookresearch/xformers/blob/main/HOWTO.md https://github.com/facebookresearch/xformers/blob/main/HOWTO... [2] https://github.com/HazyResearch/flash-attention https://github.com/HazyResearch/flash-attention

BlueTemplar 4y ago

Not really ? « Inflated » expectations doesn't mean NO expectations... People are still throwing « AI » around as a buzzword like it's something distinct from a computer program, in fact the situation got worse because non-neural network programs are somehow dismissed as « not AI » now. Autonomous cars are still nowhere to be seen, even more so for « General « AI » ». The singularity isn't much on track either looking at Kurzweil's predictions : we should have had molecular manufacturers by now, and nanobots connecting our brains to the Internet, extending our lives, and brain scanning people to recreate their avatars when they are dead, don't seem like they are going to happen by the 2030s either. (2045 is still far enough away that I wouldn't completely bet against a singularity by then.) (And Kurzweil doesn't get to blame it on misunderstanding the exponential function : how people have a too linear view of the future and tend to overestimate changes in the near future, but underestimate them in the long term !)

alexheikel 4y ago
[dead]