3 ms·

Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers. It will need a

Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers. It will need actual "sense-data" to do that. For instance, to get a model that generates image captions, you need to train it with actual images. If the much-vaunted "general intelligence" consists in both vague and precise causal reasoning and optimal control with respect to objects in the real world, then no, it obviously cannot be done with mere language. At least one sensor and one effector will be needed to train an ML/AI model to perform those tasks.

Houshalter 10y ago
There is nothing magical about "sense data". A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. The internet is so large and so comprehensive (especially if you include digitized books and papers, e.g. libgen or google books) that I doubt any important information that can be learned through video data, can't be obtained through text data.
- eli_gottlieb 10y ago
  >A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. Uhhhh... there's this thing called entropy. A corpus of video data is vastly higher in entropy content than text data, and probably a good deal more tightly correlated too, making it much easier to learn from. Remember, for a human, the whole point of speech and text is to function as a robust, efficient code (in the information-theoretic sense) for activating generative models we already have in our heads when we acquire the words. This is why we have such a hard time with very abstract concepts like mathematical ones: the causal-role concepts (easy to theorize how to encode those as generative models) are difficult to acquire from nothing but the usage statistics of the words and symbols, in contrast to "concrete", sense-grounded concepts, which have large amounts of high-dimensional data to fuel the Blessing of Abstraction. Nevermind, I should probably just get someone to let me into a PhD program so I can publish this stuff. If only they'd consider the first paper novel enough already!
  - Houshalter 10y ago
    I think you mean redundancy. And yes videos are highly redundant. But I don't see how that's any kind of advantage. Text has all the relevant information contained within it, with a ton of irrelevant information discarded. But there are still plenty of learnable patterns. Even trivial algorithms like word2vec can glean a huge amount of semantic information (much easier than is possible with video, currently.) I don't know if humans have generative models in their heads. There are people who have no ability to form mental images, and they function fine. Regardless, an AI should be able to get around that by learning our common patterns. It need not mimic our internal states, only our external behavior.
    - eli_gottlieb 10y ago
      >I think you mean redundancy. No, I meant a pair of specific information-theoretical quantities I've been studying. >And yes videos are highly redundant. But I don't see how that's any kind of advantage. Representations are easier to learn for highly-correlated data. Paper forthcoming, but conceptually so obvious that quantifying it is (apparently) non-novel. >I don't know if humans have generative models in their heads. The best available neuroscience and computational cognitive science says we do. >Regardless, an AI should be able to get around that by learning our common patterns. It need not mimic our internal states, only our external behavior. Our external behavior is determined by the internal states, insofar as those internal states are functions which map sensory (including proprioceptive and interoceptive) statistics to distributions over actions. If you want your robots to function in society, at least well enough to take it over and kill everyone, they need a good sense of context and good representations for structured information, behavior, and goals. Further, most of the context to our external behavior is nonverbal. You know how it's difficult to detect sarcasm over the internet? That's because you're trying to serialize a tightly correlated high-dimensional data-stream into a much lower-dimensional representation, and losing some of the variance (thus, some of the information content) along the way. Humans know enough about natural speech that we can usually, mostly reconstruct the intended meaning, but even then, we've had to develop a separate art of good writing to make our written representations conducive to easy reconstruction of actual speech. Deep learning can't do this stuff right now. Objectively speaking, it's kinda primitive, actually. OTOH, realizing the implications of our best theories about neuroscience and information theory as good computational theories for cognitive science and ML/AI is going to take a while!
- jdmichal 10y ago
  A video is encoded to a series of ones and zeros by a codex. That codex determines the interpretation of that data. The codex basically becomes a sensor -- it serves the same purpose as the eyes, which is taking raw data (photon excitements or binary string) and turning it into meaningful data (an image). And without that codex, the information is basically meaningless.

Houshalter 10y ago

There is nothing magical about "sense data". A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. The internet is so large and so comprehensive (especially if you include digitized books and papers, e.g. libgen or google books) that I doubt any important information that can be learned through video data, can't be obtained through text data.

eli_gottlieb 10y ago
>A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. Uhhhh... there's this thing called entropy. A corpus of video data is vastly higher in entropy content than text data, and probably a good deal more tightly correlated too, making it much easier to learn from. Remember, for a human, the whole point of speech and text is to function as a robust, efficient code (in the information-theoretic sense) for activating generative models we already have in our heads when we acquire the words. This is why we have such a hard time with very abstract concepts like mathematical ones: the causal-role concepts (easy to theorize how to encode those as generative models) are difficult to acquire from nothing but the usage statistics of the words and symbols, in contrast to "concrete", sense-grounded concepts, which have large amounts of high-dimensional data to fuel the Blessing of Abstraction. Nevermind, I should probably just get someone to let me into a PhD program so I can publish this stuff. If only they'd consider the first paper novel enough already!
- Houshalter 10y ago
  I think you mean redundancy. And yes videos are highly redundant. But I don't see how that's any kind of advantage. Text has all the relevant information contained within it, with a ton of irrelevant information discarded. But there are still plenty of learnable patterns. Even trivial algorithms like word2vec can glean a huge amount of semantic information (much easier than is possible with video, currently.) I don't know if humans have generative models in their heads. There are people who have no ability to form mental images, and they function fine. Regardless, an AI should be able to get around that by learning our common patterns. It need not mimic our internal states, only our external behavior.
  - eli_gottlieb 10y ago
    >I think you mean redundancy. No, I meant a pair of specific information-theoretical quantities I've been studying. >And yes videos are highly redundant. But I don't see how that's any kind of advantage. Representations are easier to learn for highly-correlated data. Paper forthcoming, but conceptually so obvious that quantifying it is (apparently) non-novel. >I don't know if humans have generative models in their heads. The best available neuroscience and computational cognitive science says we do. >Regardless, an AI should be able to get around that by learning our common patterns. It need not mimic our internal states, only our external behavior. Our external behavior is determined by the internal states, insofar as those internal states are functions which map sensory (including proprioceptive and interoceptive) statistics to distributions over actions. If you want your robots to function in society, at least well enough to take it over and kill everyone, they need a good sense of context and good representations for structured information, behavior, and goals. Further, most of the context to our external behavior is nonverbal. You know how it's difficult to detect sarcasm over the internet? That's because you're trying to serialize a tightly correlated high-dimensional data-stream into a much lower-dimensional representation, and losing some of the variance (thus, some of the information content) along the way. Humans know enough about natural speech that we can usually, mostly reconstruct the intended meaning, but even then, we've had to develop a separate art of good writing to make our written representations conducive to easy reconstruction of actual speech. Deep learning can't do this stuff right now. Objectively speaking, it's kinda primitive, actually. OTOH, realizing the implications of our best theories about neuroscience and information theory as good computational theories for cognitive science and ML/AI is going to take a while!
jdmichal 10y ago
A video is encoded to a series of ones and zeros by a codex. That codex determines the interpretation of that data. The codex basically becomes a sensor -- it serves the same purpose as the eyes, which is taking raw data (photon excitements or binary string) and turning it into meaningful data (an image). And without that codex, the information is basically meaningless.