5 ms·

Yes. In Elazar et al. (2019) "How Large Are Lions? Inducing Distributions over Quantitative Attributes" (https://arxiv.org/pdf/1906.01327.pdf https://arxiv.org

Yes. In Elazar et al. (2019) "How Large Are Lions? Inducing Distributions over Quantitative Attributes" (https://arxiv.org/pdf/1906.01327.pdf https://arxiv.org/pdf/1906.01327.pdf), it requires 100M English triples to roughly induce the answer to this question. How many images do you think a model needs to see in order to answer this question? Of course, there are fact books that contain the size of the lion. But fact books are non-exhaustive and don't contain much of the information that can quickly be gleaned through other forms of perception. Additionally, multimodal learning simply learns faster. What could be a long slog through a shallow gradient trough of linguistic information can instead become a simple decisive step in a multimodal space. If you're interested to read more about this [WARNING: SELF CITE], see Bisk et al. 2020, "Experience Grounds Language" (https://arxiv.org/pdf/2004.10151.pdf https://arxiv.org/pdf/2004.10151.pdf).

IIAOPSW 4y ago
No. But with the caveat that it can only truly grasp things that are self contained systems of text. Allow me to give examples: Transit wayfinding: A train service is nothing more than a list of stations it goes to, a station is nothing more than a list of train services that stop there. Nothing about the physical nature of trains or commuting is needed to have a discussion about train lines or to answer questions like "how many transfers does it take from x to y?". You could never have seen a train in your life or even a map of a train, if you've studied the dual lists of stations and services there's nothing more to learn. Chess (and other board games): Standard Chess notation is a list of sequential move made by each player in the format [piece name][x coordinate][y coordinate]. Eg RB6 represents moving the Rook to column B row 6. Chess can be understood entirely in terms of a game of passing a paper back and forth, appending a new token each time, and the rules of the game expressed entirely in terms of how the next token must match the previous part of the list. At no point is it needed to have actually seen the physical board based representation of the game. The machine is grounded in a text based existence. Conversational and linguistic objects are its literal physical objects. Anything outside of that, well it can understand lions and "large" the same we understand atoms and "eigen-states". Experience may ground language, but you are free to ground language in a different reality with different basic constituents and relations. If AGI were to emerge out of trading bots, they would have a language grounded in money and trades. If it emerged out of a bot made to play Diplomacy, it would be a consciousness with the body of a nation state and the atoms of its world would be bits of plastic on a map of Europe. Grounding in our particular reality isn't strictly necessary, but it is helpful if the goal is to make something adept at the modalities of our particular existence and conversational form.
- bravura 4y ago
  I acknowledge your point that there might be rich ineffable depths that could be present in the game of Go that are beyond the ken of human understanding or even imagination. But I fail to see how an AI whose entire domain of knowledge is limited to board games, or financial trading, would qualify as artificial general intelligence. Unless you take a radical decentering viewpoint that human experience is just one kind of intelligence, and aren't all kinds of intelligence equally valid? That would be glib. Of course we want superhuman drug discovery algorithms that don't understand the pleasure of a sunny day. But most people, when they speak of the topic of AGI, mean: encompassing the scope of human experience.
  - IIAOPSW 4y ago
    >But I fail to see how an AI whose entire domain of knowledge is limited to board games, or financial trading, would qualify as artificial general intelligence. Let me recast this: >I fail to see how an I whose entire domain of knowledge is limited to low energy physics, or the large mass limit, would ever qualify as general intelligence. You see the problem. You can extrapolate just fine beyond the reality of Newtonian physics and Euclidean geometry which is native to you. A trading bot AGI could talk about things that aren't money, it would just be using money related objects with money related properties to tell this narrative picture of a world which isn't money. Not too different from the sort of analogies we use all the time to map hard concepts back to the space we find natural. So let me answer your question with a question. What makes a substrate of properties like "size" and "color" and "texture" a better reality to be grounded in than properties like "buy price", "sell price", "dividend"? Is the former uniquely suited for intelligence to generalize beyond that given domain, or is it possible that all groundings are good as any other?
    - r_hoods_ghost 4y ago
      Very interesting question! I would argue that concepts like "size", "colour", "texture" are grounded in a more base level of reality than concepts such as "buy price", "sell price", "dividend" etc. The former set of concepts relate to aspects of the physical world as perceived by creatures that inhabit that world, while the latter refer to social constructs of the human socio-economic world that are themselves reliant on abstract human concepts of utility, value, the market etc. This thought is not fully developed but I'm drawn to the idea that if an intelligence is grounded in an understanding of a more base level of reality it will find it easier to generalize beyond any one given domain. I could be entirely wrong of course.
ummonk 4y ago
Your comment seems to suggest the answer is no, and that LLMs can indeed learn sensory-grounded information, but it's just orders of magnitude less efficient to train them on text rather than on multimodal data.

IIAOPSW 4y ago

No. But with the caveat that it can only truly grasp things that are self contained systems of text. Allow me to give examples: Transit wayfinding: A train service is nothing more than a list of stations it goes to, a station is nothing more than a list of train services that stop there. Nothing about the physical nature of trains or commuting is needed to have a discussion about train lines or to answer questions like "how many transfers does it take from x to y?". You could never have seen a train in your life or even a map of a train, if you've studied the dual lists of stations and services there's nothing more to learn. Chess (and other board games): Standard Chess notation is a list of sequential move made by each player in the format [piece name][x coordinate][y coordinate]. Eg RB6 represents moving the Rook to column B row 6. Chess can be understood entirely in terms of a game of passing a paper back and forth, appending a new token each time, and the rules of the game expressed entirely in terms of how the next token must match the previous part of the list. At no point is it needed to have actually seen the physical board based representation of the game. The machine is grounded in a text based existence. Conversational and linguistic objects are its literal physical objects. Anything outside of that, well it can understand lions and "large" the same we understand atoms and "eigen-states". Experience may ground language, but you are free to ground language in a different reality with different basic constituents and relations. If AGI were to emerge out of trading bots, they would have a language grounded in money and trades. If it emerged out of a bot made to play Diplomacy, it would be a consciousness with the body of a nation state and the atoms of its world would be bits of plastic on a map of Europe. Grounding in our particular reality isn't strictly necessary, but it is helpful if the goal is to make something adept at the modalities of our particular existence and conversational form.

bravura 4y ago
I acknowledge your point that there might be rich ineffable depths that could be present in the game of Go that are beyond the ken of human understanding or even imagination. But I fail to see how an AI whose entire domain of knowledge is limited to board games, or financial trading, would qualify as artificial general intelligence. Unless you take a radical decentering viewpoint that human experience is just one kind of intelligence, and aren't all kinds of intelligence equally valid? That would be glib. Of course we want superhuman drug discovery algorithms that don't understand the pleasure of a sunny day. But most people, when they speak of the topic of AGI, mean: encompassing the scope of human experience.
- IIAOPSW 4y ago
  >But I fail to see how an AI whose entire domain of knowledge is limited to board games, or financial trading, would qualify as artificial general intelligence. Let me recast this: >I fail to see how an I whose entire domain of knowledge is limited to low energy physics, or the large mass limit, would ever qualify as general intelligence. You see the problem. You can extrapolate just fine beyond the reality of Newtonian physics and Euclidean geometry which is native to you. A trading bot AGI could talk about things that aren't money, it would just be using money related objects with money related properties to tell this narrative picture of a world which isn't money. Not too different from the sort of analogies we use all the time to map hard concepts back to the space we find natural. So let me answer your question with a question. What makes a substrate of properties like "size" and "color" and "texture" a better reality to be grounded in than properties like "buy price", "sell price", "dividend"? Is the former uniquely suited for intelligence to generalize beyond that given domain, or is it possible that all groundings are good as any other?
  - r_hoods_ghost 4y ago
    Very interesting question! I would argue that concepts like "size", "colour", "texture" are grounded in a more base level of reality than concepts such as "buy price", "sell price", "dividend" etc. The former set of concepts relate to aspects of the physical world as perceived by creatures that inhabit that world, while the latter refer to social constructs of the human socio-economic world that are themselves reliant on abstract human concepts of utility, value, the market etc. This thought is not fully developed but I'm drawn to the idea that if an intelligence is grounded in an understanding of a more base level of reality it will find it easier to generalize beyond any one given domain. I could be entirely wrong of course.

ummonk 4y ago

Your comment seems to suggest the answer is no, and that LLMs can indeed learn sensory-grounded information, but it's just orders of magnitude less efficient to train them on text rather than on multimodal data.