4 ms·

> You can perfectly try things and learn without being embodied. A brilliant 'brain in a vat' can come up with novel answers to questions but outside of narrow

> You can perfectly try things and learn without being embodied. A brilliant 'brain in a vat' can come up with novel answers to questions but outside of narrow categories like pure mathematics and logic which can be internally validated, the brain can't know how correct or incorrect its novel answers are without some way to objectively test, observe and validate the correctness of its answers in the relevant domain (ie 'the real world'). A model can only be as useful as its parameters correctly reflect the modeled target. Even very complex and detailed simulations tend to quickly de-correlate when they repeatedly run untethered from ground truth. Games like Go have clear rule sets. Reality doesn't.

api 1y ago
That was better said in some ways than my own comment.
- mrandish 1y ago
  Thanks! And I just saw in a parallel post currently on the HN home page, John Carmack also said a similar thing in his lecture notes. > "offline training can bootstrap itself off into a coherent fantasy untested by reality." "a coherent fantasy untested by reality" is a lovely turn of phrase.
  - api 1y ago
    The Yudkowskiites are all about coherent fantasies untested by reality, as are… really… a lot of philosophers throughout history. Maybe most. I fundamentally do not believe in knowing without sensing or learning without experiencing. Of course it need not be direct experience. You can “download” information. But that information had to be gathered somehow at some point. There is only so much training data. As I said — I don’t dismiss the idea of AI challenging human intellect or replacing human jobs. An AI “merely” as smart as a human but tireless and faster could seem superhuman. I am just intensely skeptical of the idea of something learning to self improve and then magically taking off into some godlike superintelligence realm far beyond what is latent in or implied by its training data. That would be an informatic perpetual motion machine. It would, in fact, be magic, in the fantasy sense.

api 1y ago

That was better said in some ways than my own comment.

mrandish 1y ago
Thanks! And I just saw in a parallel post currently on the HN home page, John Carmack also said a similar thing in his lecture notes. > "offline training can bootstrap itself off into a coherent fantasy untested by reality." "a coherent fantasy untested by reality" is a lovely turn of phrase.
- api 1y ago
  The Yudkowskiites are all about coherent fantasies untested by reality, as are… really… a lot of philosophers throughout history. Maybe most. I fundamentally do not believe in knowing without sensing or learning without experiencing. Of course it need not be direct experience. You can “download” information. But that information had to be gathered somehow at some point. There is only so much training data. As I said — I don’t dismiss the idea of AI challenging human intellect or replacing human jobs. An AI “merely” as smart as a human but tireless and faster could seem superhuman. I am just intensely skeptical of the idea of something learning to self improve and then magically taking off into some godlike superintelligence realm far beyond what is latent in or implied by its training data. That would be an informatic perpetual motion machine. It would, in fact, be magic, in the fantasy sense.