4 ms·

It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics o

It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one, mostly based on repeating whatever we have read about in popular science literature without any intuitive understanding. GPT-3 understanding of physical world is similarly bad. It's just simply a modality not available to the model. But GPT-3 is able to do few-shots learning. If you show it couple of times how to do something it will try to repeat after you showing some rudimentary understanding of what you are trying to do. It's even able to learn to do simple analogies [0]. GPT-3 is able to do very, very simple arthritics like addition and subtractions with 2-3 numbers. Actually even a lot smaller models trained on arthritics are able to do impressive feats like symbolic integration and solving differential equations. > Deep Learning for Symbolic Mathematics [1]: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica. What is true is that GPT-3 is not very good at following instructions. It would not be able to execute an algorithm. So, it would not learn arithmetic from instructions if you would not include explicit examples. [0] https://medium.com/@melaniemitchell.me/can-gpt-3-make-analogies-16436605c446 https://medium.com/@melaniemitchell.me/can-gpt-3-make-analog... [1] https://arxiv.org/abs/1912.01412 https://arxiv.org/abs/1912.01412

mrfusion 6y ago
So you’re saying ai needs to be grounded to the real world to be effective?
- ccozan 6y ago
  Yes. Specifically the way human brain interacts with environment, the same way any AGI will need to sense and react to an environment, the more real the better. An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). So it can possibly learn from a human what is good or bad in that book, etc. Then both AI and humans can speak the same language ( on terms.of "feeling"/sensing ) and eventually learn the proper context, so less parameters are needed.
  - mikkom 6y ago
    > An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). But isn't that exactly what has been done with GPT-3? AFAIK it has been trained with both facts (wikipedia and such) and web-crawled text content that contains both content and critique of content (book reviews, movie reviews etc.)?
    - ccozan 6y ago
      I know that, but does the network perceive the critique as environmental reaction? It means if it can distinguish between them. Putting them in the same basket not sure if it helps.
mellosouls 6y ago
It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one [...] GPT-3 understanding of physical world is similarly bad. No, GPT-3 has no "understanding" whatsoever - we at least have a conscious engagement with the topics you cite and a genuine understanding that we lack a directly relatable perspective. Our understanding in many circumstances - like those you mention, may be severely constrained, but it exists as part of a sentience that there is no reason to believe GPT-3 or any other current AI tool - possesses.
- sdenton4 6y ago
  Do we really understand? What's the mechanism for that? My thinking is that we humans have some ability to do things by rote until we have a good statistical understanding, at which point we use the statistical understanding mostly, and fall back to rote where we have to be careful. But, mostly, we're stats machines too.
  - tsimionescu 6y ago
    We obviously have a working model of the world that we draw facts from, that is partly built-in (trained by evolutionary processes) and partly refined with personal learning. By contrast, GPT-3 only has a knowledge of text. To the extent you could say that it thinks, it thinks in terms of textual symbols that it has seen before - that is its world. It is theoretically possible that if we found a way to encode experiences of the world of the world in a dataset and used the same training mechanisms and models that we used for GPT-3, we would get an AI with a decent model of the world. But it is magical thinking to imagine that we can get there by training a model on raw text.

mrfusion 6y ago

So you’re saying ai needs to be grounded to the real world to be effective?

ccozan 6y ago
Yes. Specifically the way human brain interacts with environment, the same way any AGI will need to sense and react to an environment, the more real the better. An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). So it can possibly learn from a human what is good or bad in that book, etc. Then both AI and humans can speak the same language ( on terms.of "feeling"/sensing ) and eventually learn the proper context, so less parameters are needed.
- mikkom 6y ago
  > An interesting example would be if the NN learns something from a book, also fed it with the all the critique on that book ( the environment reaction ). But isn't that exactly what has been done with GPT-3? AFAIK it has been trained with both facts (wikipedia and such) and web-crawled text content that contains both content and critique of content (book reviews, movie reviews etc.)?
  - ccozan 6y ago
    I know that, but does the network perceive the critique as environmental reaction? It means if it can distinguish between them. Putting them in the same basket not sure if it helps.

mellosouls 6y ago

It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one [...] GPT-3 understanding of physical world is similarly bad. No, GPT-3 has no "understanding" whatsoever - we at least have a conscious engagement with the topics you cite and a genuine understanding that we lack a directly relatable perspective. Our understanding in many circumstances - like those you mention, may be severely constrained, but it exists as part of a sentience that there is no reason to believe GPT-3 or any other current AI tool - possesses.

sdenton4 6y ago
Do we really understand? What's the mechanism for that? My thinking is that we humans have some ability to do things by rote until we have a good statistical understanding, at which point we use the statistical understanding mostly, and fall back to rote where we have to be careful. But, mostly, we're stats machines too.
- tsimionescu 6y ago
  We obviously have a working model of the world that we draw facts from, that is partly built-in (trained by evolutionary processes) and partly refined with personal learning. By contrast, GPT-3 only has a knowledge of text. To the extent you could say that it thinks, it thinks in terms of textual symbols that it has seen before - that is its world. It is theoretically possible that if we found a way to encode experiences of the world of the world in a dataset and used the same training mechanisms and models that we used for GPT-3, we would get an AI with a decent model of the world. But it is magical thinking to imagine that we can get there by training a model on raw text.