3 ms·

These and similar attempts are like those "perpetual motion machines" that captivated people's imagination for a long time. That the machine will generate ener

by wg0 3y ago

These and similar attempts are like those "perpetual motion machines" that captivated people's imagination for a long time. That the machine will generate energy and that would consume energy and thus will keep moving forever. Similar hopes here and everywhere. A model will generate its own input and will watch it's own output and in process, will become more intelligent than it really is.

phreeza 3y ago
This is also the percept and possibly fallacy that underlies the fast takeoff AI singularity thinking.
andruby 3y ago
I see the resemblance, but for perpetual motion machines, we have an actual physical law, the 2nd law of thermodynamics, that makes it impossible. As far as I know, we can't yet disprove that a self-improving AI will become more intelligent than it starts as. I'm not trying to say that it is likely or that I believe it will happen. Just that we can't disprove it yet, unlike perpetual motion devices. EDIT: corrected from 3rd to 2nd law of thermodynamics.
- majoe 3y ago
  Small nitpick: I think you mean the second law of thermodynamics. Also the second law of thermodynamics is an empirical law. It is based on observations and not proven. To say, that a perpetual motion machine can't exist because of the 2nd law, is circular logic, since the 2nd law was established precisely because a perpetual motion machine was never observed. That's from the POV of thermodynamics, though, you can show with statistical mechanics, that the probability for a process that defies this laws goes to zero.
travisjungroth 3y ago
That's not a fitting analogy because there is no equivalent to the laws of thermodynamics that says a system can't be self improving. You're dismissing a technological possibility without much reason. We are extremely similar genetically to our ancestors of 100k years ago. The big difference is cultural inheritance. We come up with ideas and objects and pass them down. Humans have improved the capabilities of humans. There's no reason to think computer programs aren't capable of the same.
- yreg 3y ago
  We also have proven self improving AIs - GANs. Or even evolutionary algorithms.
  - phreeza 3y ago
    I would not say GANs are self-improving in ways that vanilla NNs are not.
- geraneum 3y ago
  It’s not that straightforward. Quoting Yann LeCun from a LinkedIn post [1]: > I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes. Here is the argument: Let e be the probability that any generated token exits the tree of "correct" answers. Then the probability that an answer of length n is correct is (1-e)^n. > Errors accumulate. The probability of correctness decreases exponentially. One can mitigate the problem by making e smaller (through training) but one simply cannot eliminate the problem entirely. A solution would require to make LLMs non auto-regressive while preserving their fluency. Moreover, I think the human analogy doesn’t quite fit here. There’s no evidence that today’s human is cognitively more capable than their human ancestors. It’s a conflation of technology and transference of knowledge with intelligence. 1: https://www.linkedin.com/posts/yann-lecun_i-have-claimed-that-auto-regressive-llms-activity-7045908925660950528-hJGk https://www.linkedin.com/posts/yann-lecun_i-have-claimed-tha...
  - spi 3y ago
    I guess I should be wiser than contradicting LeCun on a public forum, but his math doesn't really work out. It only works if there is a unique correct answer to any question, in which case e=1/dict_size which is clearly false - even humans can just put "Let me think..." in front of anything else and get logically "the same" answer; after how many such filler words do you deem it "wrong"? Even taking colloquial speech aside, in Math, there is apparently a book that collects 367 different proofs of the Pythagorean Theorem; that tree of correct answers is certainly quite complicated. You can't approximate it with a fixed probability and take the exponential - the number of possibly correct next tokens varies very strongly depending on the previous string. LLMs and autoregression are very good at avoiding the 99.999[...]% of the strings that are simple gibberish. If you were to generate a string made of 20 random tokens from GPT2 tokenizer, you would get something like: "automakersGrand carries liberties Occupations ongoingOULDessing heartbeat Pillar intrigued Trotskymediatelyearable founding examinations lavAg redesign folds" and of course any half decent language model does much better than that. If the "paths to truth" were as unlikely as LeCun puts it there would be no hope. A non-autoregressive model would certainly be "better" because it would be faster, which is where language models started from (BERT & co.), it just doesn't seem to work as well... similarly to how a human sometimes needs to write something down and only realizes the correct answer to a complicated question on the go. If anything, we'd need to allow LLMs to realize mistakes and correct themselves out of them, i.e. making the generation non-linear. If you ask GPT4 something complicated (like math) it's not rare at all that it logically contradicts itself in their answers. I would be surprised if, somewhere deep in the model, it doesn't "realize" this, but it can't fix it, so it falls back to what humans do at an exam or interview that started badly: try to bullshit their way out of the thing, sweeping the inconsistency under the carpet, unless you explicitly point it to them (and often even after that, both GPT4 and humans). P.S. Mathematician rant: who on Earth calls a probability "e"??
deleted 3y ago
[deleted]
conjectures 3y ago
> A model will generate its own input and will watch it's own output and in process, will become more intelligent than it really is. You're aware this is basically how AlphaGo was trained - playing against itself?

by wg0 3y ago

phreeza 3y ago

This is also the percept and possibly fallacy that underlies the fast takeoff AI singularity thinking.

andruby 3y ago

I see the resemblance, but for perpetual motion machines, we have an actual physical law, the 2nd law of thermodynamics, that makes it impossible. As far as I know, we can't yet disprove that a self-improving AI will become more intelligent than it starts as. I'm not trying to say that it is likely or that I believe it will happen. Just that we can't disprove it yet, unlike perpetual motion devices. EDIT: corrected from 3rd to 2nd law of thermodynamics.

majoe 3y ago
Small nitpick: I think you mean the second law of thermodynamics. Also the second law of thermodynamics is an empirical law. It is based on observations and not proven. To say, that a perpetual motion machine can't exist because of the 2nd law, is circular logic, since the 2nd law was established precisely because a perpetual motion machine was never observed. That's from the POV of thermodynamics, though, you can show with statistical mechanics, that the probability for a process that defies this laws goes to zero.

travisjungroth 3y ago

That's not a fitting analogy because there is no equivalent to the laws of thermodynamics that says a system can't be self improving. You're dismissing a technological possibility without much reason. We are extremely similar genetically to our ancestors of 100k years ago. The big difference is cultural inheritance. We come up with ideas and objects and pass them down. Humans have improved the capabilities of humans. There's no reason to think computer programs aren't capable of the same.

yreg 3y ago
We also have proven self improving AIs - GANs. Or even evolutionary algorithms.
- phreeza 3y ago
  I would not say GANs are self-improving in ways that vanilla NNs are not.
geraneum 3y ago
It’s not that straightforward. Quoting Yann LeCun from a LinkedIn post [1]: > I have claimed that Auto-Regressive LLMs are exponentially diverging diffusion processes. Here is the argument: Let e be the probability that any generated token exits the tree of "correct" answers. Then the probability that an answer of length n is correct is (1-e)^n. > Errors accumulate. The probability of correctness decreases exponentially. One can mitigate the problem by making e smaller (through training) but one simply cannot eliminate the problem entirely. A solution would require to make LLMs non auto-regressive while preserving their fluency. Moreover, I think the human analogy doesn’t quite fit here. There’s no evidence that today’s human is cognitively more capable than their human ancestors. It’s a conflation of technology and transference of knowledge with intelligence. 1: https://www.linkedin.com/posts/yann-lecun_i-have-claimed-that-auto-regressive-llms-activity-7045908925660950528-hJGk https://www.linkedin.com/posts/yann-lecun_i-have-claimed-tha...
- spi 3y ago
  I guess I should be wiser than contradicting LeCun on a public forum, but his math doesn't really work out. It only works if there is a unique correct answer to any question, in which case e=1/dict_size which is clearly false - even humans can just put "Let me think..." in front of anything else and get logically "the same" answer; after how many such filler words do you deem it "wrong"? Even taking colloquial speech aside, in Math, there is apparently a book that collects 367 different proofs of the Pythagorean Theorem; that tree of correct answers is certainly quite complicated. You can't approximate it with a fixed probability and take the exponential - the number of possibly correct next tokens varies very strongly depending on the previous string. LLMs and autoregression are very good at avoiding the 99.999[...]% of the strings that are simple gibberish. If you were to generate a string made of 20 random tokens from GPT2 tokenizer, you would get something like: "automakersGrand carries liberties Occupations ongoingOULDessing heartbeat Pillar intrigued Trotskymediatelyearable founding examinations lavAg redesign folds" and of course any half decent language model does much better than that. If the "paths to truth" were as unlikely as LeCun puts it there would be no hope. A non-autoregressive model would certainly be "better" because it would be faster, which is where language models started from (BERT & co.), it just doesn't seem to work as well... similarly to how a human sometimes needs to write something down and only realizes the correct answer to a complicated question on the go. If anything, we'd need to allow LLMs to realize mistakes and correct themselves out of them, i.e. making the generation non-linear. If you ask GPT4 something complicated (like math) it's not rare at all that it logically contradicts itself in their answers. I would be surprised if, somewhere deep in the model, it doesn't "realize" this, but it can't fix it, so it falls back to what humans do at an exam or interview that started badly: try to bullshit their way out of the thing, sweeping the inconsistency under the carpet, unless you explicitly point it to them (and often even after that, both GPT4 and humans). P.S. Mathematician rant: who on Earth calls a probability "e"??

deleted 3y ago

[deleted]

conjectures 3y ago

> A model will generate its own input and will watch it's own output and in process, will become more intelligent than it really is. You're aware this is basically how AlphaGo was trained - playing against itself?