10 ms·

the joke explanations on page 38 of the full paper linked here are blowing my mind. it's crazy how far language models have come https://storage.googleapis.com

the joke explanations on page 38 of the full paper linked here are blowing my mind. it's crazy how far language models have come https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf https://storage.googleapis.com/pathways-language-model/PaLM-...

WithinReason 5y ago
I like how it appears that it had to convert from imperial to metric before it could make an inference: 300 miles per hour is about 480 km/h. This is about the speed of a commercial airplane. [...]
- sib 5y ago
  Also, that's a very slow commercial airplane. (Unless talking about an old turboprop?)
theincredulousk 5y ago
Haven't looked further, but I'm wondering about that. Is that the result of training to be able to explain that specific joke, or is it generalized? In the past these things have been misleading. Some impressive capability ends up being far more narrow than implied, so it's kind of like just storing information and retrieving it with extra steps.
- whimsicalism 5y ago
  From the example, it seems hard to imagine that it has been trained to explain this specific joke. I understand language model skepticism is very big on HN, but this is impressive.
  - mjburgess 5y ago
    How much of human written history can be compressed and aproximately stored in 504Bn parameters? It seems to me bascially certain that no compressed representation of text can be an understanding of langugae, so necessarily, any statistical algorithm here is always using coincidental tricks. That it takes 500bn parameters to do it, i think, is a clue that we dont even really need. Words mean what we do with them -- you need to be here in the world with us, to understand what we mean. There is nothing in the patterns of our usage of words which provides their semantics, so the whole field of distributional analysis precludes this superstition. You cannot, by mere statistical analysis of patterns in mere text, understand the nature of the world. But it is precisely this we communicate in text. We succeed because we are both in the world, not because "w" occuring before "d" somehow communicates anything. Apparent correlations in text are meaningful to us, because we created them, and we have their semantics. The system must by is nature be a mere remebering.
    - whimsicalism 5y ago
      > It seems to me bascially certain that no compressed representation of text can be an understanding of langugae, so necessarily, any statistical algorithm here is always using coincidental tricks. That it takes 500bn parameters to do it, i think, is a clue that we dont even really need. I think your premise contains your conclusion, which while common, is something you should strive to avoid. I do think your opinion is a good example of the prevailing sentiment on Hacker News. To me, it seems to come from a discomfort with the fact that even "we" emerge out of the basic interactions of basic building blocks. Our brain has been able to build world knowledge "merely by" analysis of electrical impulses being transmitted to it on wires.
      - mjburgess 5y ago
        I have no discomfort with the notion that our bodies, which grow in response to direct causal contact with our environment, contain in-their-structure the generative capbaility for knoweldge, imagination, skill, growth -- and so on. I have no discomfort with the basically schiozphrenic notion that the shapes of words have something to do with the nature of the world. I just think its a kind of insantity which absolutely destroys our ability to reason carefully about the use of these systems. That "tr" occurs before "ee" says as much about "trees" as "leaves are green" says -- it is only that *we* have the relevant semantics that the latter is meaningful when interpreted in the light of our "environmental history" recorded in our bodies, and given weight and utility by our imaginations. The structure of text is not the structure of the world. This thesis is mad. Its a scientific thesis. It is trivial to test it. It is trivial to wholey discred it. It's pseudoscience. No one here is a scientist and no one treats any of this as science. Where's the criteria for the emprical adequecy of NLP systems as models of language? Specifying any, conducting actual hypothesis tests, and establishing a theory of how NLP systems model language -- this would immediately reveal the smoke-and-mirros. The work to reveal the statistical tricks underneath them takes years, and no one has much motivation to do it. The money lies in this sales pitch, and this is no science. This is no scientific method.
        rafaelero 5y ago
nqzero 5y ago
this thing is already more human than i am
Xenixo 5y ago
Wow. The anti joke explanation was also very impressive.
- modeless 5y ago
  This may be the most impressive thing I've seen a language model do so far. That's incredible. The future is going to be very weird.
  - FeepingCreature 5y ago
    The future is going to be very short. What this says to me is that there is no general-intelligence task that language models cannot scale to.
  - visarga 5y ago
    Apparently Data (from Star Trek) was less capable of understanding jokes.
severine 5y ago
So... we are training models here on HN, specially if we follow the site's guidelines! Makes you think... which makes it think! Wow, interesting times, indeed.
TaylorAlexander 5y ago
Your comment prompted me to tweet an image of that section, complete with alt text (as much as can fit). If anyone cares to see it in tweet form. https://twitter.com/tlalexander/status/1511089810752126984 https://twitter.com/tlalexander/status/1511089810752126984

WithinReason 5y ago

I like how it appears that it had to convert from imperial to metric before it could make an inference: 300 miles per hour is about 480 km/h. This is about the speed of a commercial airplane. [...]

sib 5y ago
Also, that's a very slow commercial airplane. (Unless talking about an old turboprop?)

theincredulousk 5y ago

Haven't looked further, but I'm wondering about that. Is that the result of training to be able to explain that specific joke, or is it generalized? In the past these things have been misleading. Some impressive capability ends up being far more narrow than implied, so it's kind of like just storing information and retrieving it with extra steps.

whimsicalism 5y ago
From the example, it seems hard to imagine that it has been trained to explain this specific joke. I understand language model skepticism is very big on HN, but this is impressive.
- mjburgess 5y ago
  How much of human written history can be compressed and aproximately stored in 504Bn parameters? It seems to me bascially certain that no compressed representation of text can be an understanding of langugae, so necessarily, any statistical algorithm here is always using coincidental tricks. That it takes 500bn parameters to do it, i think, is a clue that we dont even really need. Words mean what we do with them -- you need to be here in the world with us, to understand what we mean. There is nothing in the patterns of our usage of words which provides their semantics, so the whole field of distributional analysis precludes this superstition. You cannot, by mere statistical analysis of patterns in mere text, understand the nature of the world. But it is precisely this we communicate in text. We succeed because we are both in the world, not because "w" occuring before "d" somehow communicates anything. Apparent correlations in text are meaningful to us, because we created them, and we have their semantics. The system must by is nature be a mere remebering.
  - whimsicalism 5y ago
    > It seems to me bascially certain that no compressed representation of text can be an understanding of langugae, so necessarily, any statistical algorithm here is always using coincidental tricks. That it takes 500bn parameters to do it, i think, is a clue that we dont even really need. I think your premise contains your conclusion, which while common, is something you should strive to avoid. I do think your opinion is a good example of the prevailing sentiment on Hacker News. To me, it seems to come from a discomfort with the fact that even "we" emerge out of the basic interactions of basic building blocks. Our brain has been able to build world knowledge "merely by" analysis of electrical impulses being transmitted to it on wires.
    - mjburgess 5y ago
      I have no discomfort with the notion that our bodies, which grow in response to direct causal contact with our environment, contain in-their-structure the generative capbaility for knoweldge, imagination, skill, growth -- and so on. I have no discomfort with the basically schiozphrenic notion that the shapes of words have something to do with the nature of the world. I just think its a kind of insantity which absolutely destroys our ability to reason carefully about the use of these systems. That "tr" occurs before "ee" says as much about "trees" as "leaves are green" says -- it is only that *we* have the relevant semantics that the latter is meaningful when interpreted in the light of our "environmental history" recorded in our bodies, and given weight and utility by our imaginations. The structure of text is not the structure of the world. This thesis is mad. Its a scientific thesis. It is trivial to test it. It is trivial to wholey discred it. It's pseudoscience. No one here is a scientist and no one treats any of this as science. Where's the criteria for the emprical adequecy of NLP systems as models of language? Specifying any, conducting actual hypothesis tests, and establishing a theory of how NLP systems model language -- this would immediately reveal the smoke-and-mirros. The work to reveal the statistical tricks underneath them takes years, and no one has much motivation to do it. The money lies in this sales pitch, and this is no science. This is no scientific method.
      - rafaelero 5y ago

nqzero 5y ago

this thing is already more human than i am

Xenixo 5y ago

Wow. The anti joke explanation was also very impressive.

modeless 5y ago
This may be the most impressive thing I've seen a language model do so far. That's incredible. The future is going to be very weird.
- FeepingCreature 5y ago
  The future is going to be very short. What this says to me is that there is no general-intelligence task that language models cannot scale to.
- visarga 5y ago
  Apparently Data (from Star Trek) was less capable of understanding jokes.

severine 5y ago

So... we are training models here on HN, specially if we follow the site's guidelines! Makes you think... which makes it think! Wow, interesting times, indeed.

TaylorAlexander 5y ago

Your comment prompted me to tweet an image of that section, complete with alt text (as much as can fit). If anyone cares to see it in tweet form. https://twitter.com/tlalexander/status/1511089810752126984 https://twitter.com/tlalexander/status/1511089810752126984