8 ms·

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billio

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vector "king" minus the vector for "man" plus "woman" equals "queen". It's learning the meaning of words, and the relationships between them. Recurrent NNs, using similar techniques, but with much more complexity, can learn to predict the next word in a sentence very well. They can learn to write responses that are almost indistinguishable from humans. And it's incredible this works at all, given RNNs have only a few thousand neurons at most, and a few days of training, compared to humans' billions of neurons trained over a lifetime. All of the information of our world is contained in text. Humans have produced billions of books, papers, articles, and internet comments. Billions of times more information than any human could read in their entire lifetime. Any information you can imagine is contained in text somewhere. I don't think it's necessary for AIs to be able to see, or interact with the world in any way. If you can predict the word a human would say next, with enough accuracy, then you could also produce answers indistinguishable from theirs. Meaning you could pass the Turing test, and perform any language task they could do just as well. So language prediction alone may be sufficient for AGI. This is the theory behind the Hutter Prize, which proposes that predicting (compressing) wikipedia's text is a measure of AI progress. The Hutter Prize isn't perfect (it's only a sample of wikipedia, which is very small compared to all the text humans have produced), but the idea is solid.

AsyncAwait 10y ago
If we ever want 'true AI' then interacting with the world would certainly be a crucial component of building one. Text is a crucial component, but ultimately being able to 'see' gives one an entirely new perspective on what words mean. There's a reason after all, for why toddlers touch hot surfaces even after repeatedly being told that it will hurt - until they experience it for themselves, the word 'hot' doesn't quite connect even if they heard it dozens of times and have a very good abstract sense of what it means.
kafkaesq 10y ago
It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations between words, when the signal is strong enough. But to say that it's "learning" the "meaning" of these words is really quite a stretch.
- Houshalter 10y ago
  Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the relationship between countries, celebrities, etc. All that information is contained in a few hundred dimensions, which is tiny compared to the neurons in the brain.
  - mcphage 10y ago
    > What is the meaning of a word, if not the relationship it has to other words? There's also the relationship it has with the world.
    - Houshalter 10y ago
      Well in my example the AI doesn't have to interact with the world at all. To pass the Turing test simply requires imitating a human, predicting what words they would say. You only need to know the relationships between words.
      - dwaltrip 10y ago
        Only knowing the relationships between words alone would just be a poor proxy for knowing the meanings of the words, e.g. what real world concepts the words attempt to represent. You might be able to get pretty far with this technique, but I would bet a lot of money you would not be able to get reliable, in-depth human level communication. The system needs to have an understanding of the world. And then there is the fundamentally dynamic aspect of language, which strengthens the need for a rich understanding of the world that words describe and convey.
melloclello 10y ago
I can't say anything about the feasibility of what you describe, but something about the idea of being a being composed of pure text, no ability to perceive or visualise the world around me except through the medium of text, utterly horrifies me
erikpukinskis 10y ago
> All of the information of our world is contained in text. That's a bit overstated. I think maybe you mean to say "all of the information you'd need to be an effective citizen of our world is contained in text"? Or something similar? I think even that is too strong a claim, but it's at least understandable. As stated the assertion doesn't make any sense. There is more information in a glass of milk than could be stored on all of the computers on earth, and Heisenberg showed that it's impossible to even record all of the information about a single particle. Certainly there is no textual information available about my grandfather's eyes, but that information is accessible in the world for those who'll look. You seem to be underestimating the quantity of information you absorbed as a baby, just by reacting robotically to the people and events around you and absorbing the relationships between percepts and feelings.
- Houshalter 10y ago
  Well of course that's what I meant. Any information an average person knows is contained in text, somewhere. That is all common sense knowledge. Everything from detailed descriptions of trees, to the color of the sky, to the shape of the human face, etc. But also much more, like all of our scientific knowledge and written history. Billions of things the average person doesn't know.
hacker42 10y ago
I agree that learning to reason about the world likely does not require experience with motor control and proprioception (i.e. literally how babies do it), though I do think that you either need at least some sort of tempo-spatial experience (e.g. visual). Tempo-spatial representations are just extremely hard to convey by text only. You might get the idea of closeness by saying 'close is when two words are close in a sequence of words' and ordering by saying 'this word comes after that word', but I think it would be very difficult to extrapolate that concept to more than one dimension, and dimensions that actually have not just an ordering but also a metric (just think about our inability to reason about just the fourth dimension). You need rich representations of our 3+1 dimensional world to be able to reason about it and text only gives you perhaps "0.5" dimensions (because it lacks a metric, i.e. it does not convey durations in terms of the ticks of the recurrent network). But I doubt, too, that interaction with the world is necessary, in fact I think it would be rather easy for an AI to simply write motor programs in a programming language given unrestricted and noiseless memory once it has learned to reason about tempo-spatial patterns from just observing them and identifying them with our language-coded shared concept space. It is not constrained to real-time performance of actions as humans are, therefore it can take the much easier way of programming any interaction with the world as needed, on the fly. Our shared concept space likely sufficiently conveys our general (common sense) knowledge how these patterns are known to interact and evolve over time once rudimentary tempo-spatial representations are in place.
- hacker42 10y ago
  tl;dr I think you need a small set of tempo-spatially grounded meanings (though not necessarily agent-related) and you can bootstrap everything from that using only textual knowledge.
dmreedy 10y ago
> All of the information of our world is contained in text Even if this were a true statement, it's still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. While it may be the case that certain varieties of intelligence are learnable tabula rasa from a powerful neural net, the surface form of human natural language (the part your recommending measuring) may simply not have enough information in it to decode the whole picture. It is entirely possible that you need to supply some of your own information as well to the picture, in a specific manner so as to act as a kind of decryption key. A record needs a record player, even if you can make similar sounds with cassettes and CDs. And so, I'm willing to bet that you simply cannot, using raw, uninformed statistical techniques, predict what word a human would say next. You need to understand more of the underlying structure of humans first. I will agree, however, that the success towards the Hutter Prize is a valuable demonstration of AI progress. Simply because I believe that maximal compression and the kind of intelligence I'm talking about are one and the same thing. You need to offload as much of the semantic weight of the corpus into the encryption algorithm as you can. That means building a very complex model of natural language. And if you accept the premise that this model is not simply learnable by observing the surface form, then that means building Strong AI
- Houshalter 10y ago
  >There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even markov chains. There are tons of statistical patterns in text that can be learned by computers.
  - dmreedy 10y ago
    It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.
jacobrobbins 10y ago
> All of the information of our world is contained in text. This statement is false. There is a well known thought experiment called Mary’s Room the gist of which is that knowing all conceivable scientific knowledge about how humans perceive color is still not a substitute for being a human and perceiving the color red: https://philosophynow.org/issues/99/What_Did_Mary_Know https://philosophynow.org/issues/99/What_Did_Mary_Know The experience of seeing red is an example of what is called “qualia”. In Google AI systems that identify cats, birds, etc it is reasonable to imagine AI technology evolving towards systems that can discuss those objects at the level of a typical person. However with an AI based on text only there is no possibility of that. It would be like discussing color with a blind person or sound with a deaf person.
- sgnelson 10y ago
  To further this comment, also see Tacit Knowledge, of which the very definition is essentially knowledge that cannot or is extremely difficult to transfer through words alone. https://en.wikipedia.org/wiki/Tacit_knowledge https://en.wikipedia.org/wiki/Tacit_knowledge
- Houshalter 10y ago
  Mary's Room and qualia is totally irrelevant. I'm not asking if the computer will "feel" "redness", simply if it can pretend to do so through text. If it can talk about the color red, in a way indistinguishable from any other human talking about red. In any case, at some level everything is symbols. A video is just a bunch of 1's and 0's, as is text, and everything else. A being raised on only text input would have qualia just like a being raised on video input. It would just be different qualia.
  - Natanael_L 10y ago
    Like explained elsewhere here, it may be that our brains share a genetically coded "decryption key", and that many of the things we talk about are too poorly and noisily expressed for a purely text based computer AI to ever truly replicate the processes going on inside our brains. Insufficient data, simply put, and no way to get it. It may sure look indistinguishable, but on the inside it just wouldn't be the same.
orthoganol 10y ago
I think if AGI were possible from the basic statistical NLP techniques outlined in most advanced NLP textbooks, it would have already happened a decade ago.
- Houshalter 10y ago
  I'm not saying it is possible from just basic statistical NLP techniques. It may take much more advanced techniques. And it may take much more computing power than we have even now. But I do believe it is possible, someday. Probably within our lifetime.
  - orthoganol 10y ago
    I certainly think AGI is possible. I just don't think Word2vec, RNN's, i.e. stuff from the NLP textbooks, is in the same ballpark as what it will take to achieve. Edit - To be more clear I also agree AGI could be possible with text as the only input. I just think we need a new paradigm. Ostensibly AGI is meant to mimic human intelligence (minus the pitfalls), so IMO, the best approach will be that which mimics the underlying processes of human intelligence - not just the results. Traditional statistical NLP methods will probably have some role in this final system, but not the heart of it, as far as mimicking intelligence by mimicking intelligence's underlying processes goes.
Joof 10y ago
The problem seems to be in understanding the overall meaning over long periods of time. We can understand general structure quite easily and a single sentence might make sense, but overall it never ends up cohesive or meaningful. Maybe we can do that with just words, but humans certainly don't. Words are related to concepts first, then we figure out the meaning of the rest.
Cybiote 10y ago
I agree that AI is possible from text alone but only with the added stipulation that full understanding of text requires a very sophisticated learner. In order to predict what a human will say next, you need the ability to maintain extended context, use world models plus deduction to narrow and maintain multiple possibilities; all while being able to infer the state of the thing you are trying to predict (which means the AI itself has complex internal state far beyond anything an RNN or Neural Turing machine could manage today). If I said "That is one huge fan", could you predict what my next word will be? It would depend a lot on context and the ability to reason within a complex world model. Depending on whether I had gone to a concert or to a wind tunnel, your distribution over guesses would alter. If I had gone whale watching you might even suspect I made a typo. Changing huge to large would lead to major to no adjustments, depending on each guess. So while I agree an AI could emerge from text alone, it would have to be very sophisticated to do this.
wlievens 10y ago
So a related question becomes, can you learn to understand and thus predict physics (the way a child does - I'm not talking about quantum mechanics) from literature only, without interacting in space?
- knodi 10y ago
  Yes, the child doesn't learn physics he or she learns motor control of its body. Baby learning to talk is more about the brain learning to control its body through throwing its neurological system. Look at animal kingdom some animals are walking about 5 minutes of being born and running in hours.
- xapata 10y ago
  > from literature only How about when you include multimedia recordings? Or give the machine a camera and wheels?
  - wlievens 10y ago
    Giving the machine a camera and wheels is basically embedding it in the real world. It goes against the spirit of my question, and since I was asking it skeptically it's actually the thing I intuitively (without any relevant expertise!) expect to be required.
hellameta 10y ago
> It's learning the meaning of words, and the relationships between them. It's learning a meaning, not the meaning. It's just a probabilistic model for the occurrence of a word based on the words that surround it. This should not serve as a base for the rest of your claims. Anyway -- the improvements gained by multi-modal systems essentially disprove your thesis. Which is a good news! We're making progress.
ACow_Adonis 10y ago
> All of the information of our world is contained in text Communicate the concept of "green" to me in text. The sound of a dog barking, a motor turning over, a sonic boom, or the experience of a Doppler shift. Beethoven's symphony. Sour. Sweet. What does "mint" taste like? Shame. Merit. Learn facial and object recognition via text. Vertigo. Tell a boxer how to box by reading? Hand eye coordination, bodies in 3 dimensional space. Look, I love text, maybe even more than yourself. But all these things imbibe, structure and influence or text, but are not contained in them. To make substantial inroads to something that looks like human esque AI, text is not enough. The division of these fields are artificial and based on our current limited tech and the specialisation of our researchers, faculties and limitations. When we read, we play back memories, visions, sounds, feelings, etc, and inherent ideas gained through experience of ourselves as physical bodies in space. Strong AI, at least to be vaguely recognised as such, must work with algorithms and machinery that understand these things, but which then works at that next level of abstraction to combine them into proper human type concepts. Of course, there is the question about why we would want to create a human like AI, it's my contention that human like AI isn't actually what many of us would want, but that's another topic...
- rictic 10y ago
  I think you're over-romanticizing the problem. I won't touch the qualia aspect, but everything necessary to flawlessly pretend to understand the color green, the sound of a dog's bark, or the experience of hearing a sonic boom can be represented with text. As an existence proof, you could encode an adult human as a serial stream of characters.
- Houshalter 10y ago
  Are blind or deaf people not intelligent? But if you must pretend to be sighted and hearing, there are many descriptions of green, of dogs barking, of motors, etc, scattered through the many books written in English (and other languages.) Are these descriptions perfect? Maybe not. But they are sufficient to mimic or communicate with humans through text. It's sufficient to beat a Turing test, to answer questions intelligently, to write books and novels, and political arguments, etc. If that's not AGI, I don't know what is.
eli_gottlieb 10y ago
Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers. It will need actual "sense-data" to do that. For instance, to get a model that generates image captions, you need to train it with actual images. If the much-vaunted "general intelligence" consists in both vague and precise causal reasoning and optimal control with respect to objects in the real world, then no, it obviously cannot be done with mere language. At least one sensor and one effector will be needed to train an ML/AI model to perform those tasks.
- Houshalter 10y ago
  There is nothing magical about "sense data". A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. The internet is so large and so comprehensive (especially if you include digitized books and papers, e.g. libgen or google books) that I doubt any important information that can be learned through video data, can't be obtained through text data.
  - eli_gottlieb 10y ago
    >A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. Uhhhh... there's this thing called entropy. A corpus of video data is vastly higher in entropy content than text data, and probably a good deal more tightly correlated too, making it much easier to learn from. Remember, for a human, the whole point of speech and text is to function as a robust, efficient code (in the information-theoretic sense) for activating generative models we already have in our heads when we acquire the words. This is why we have such a hard time with very abstract concepts like mathematical ones: the causal-role concepts (easy to theorize how to encode those as generative models) are difficult to acquire from nothing but the usage statistics of the words and symbols, in contrast to "concrete", sense-grounded concepts, which have large amounts of high-dimensional data to fuel the Blessing of Abstraction. Nevermind, I should probably just get someone to let me into a PhD program so I can publish this stuff. If only they'd consider the first paper novel enough already!
MichaelMoser123 10y ago
> Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words doesn't that depend on the syntax of a language? my guess is that it would not work for a language that has a more flexible word order than English; so it wouldn't quite work for such languages as Russian, Turkish and Finnish (and other languages).

AsyncAwait 10y ago

If we ever want 'true AI' then interacting with the world would certainly be a crucial component of building one. Text is a crucial component, but ultimately being able to 'see' gives one an entirely new perspective on what words mean. There's a reason after all, for why toddlers touch hot surfaces even after repeatedly being told that it will hurt - until they experience it for themselves, the word 'hot' doesn't quite connect even if they heard it dozens of times and have a very good abstract sense of what it means.

kafkaesq 10y ago

It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations between words, when the signal is strong enough. But to say that it's "learning" the "meaning" of these words is really quite a stretch.

Houshalter 10y ago
Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the relationship between countries, celebrities, etc. All that information is contained in a few hundred dimensions, which is tiny compared to the neurons in the brain.
- mcphage 10y ago
  > What is the meaning of a word, if not the relationship it has to other words? There's also the relationship it has with the world.
  - Houshalter 10y ago
    Well in my example the AI doesn't have to interact with the world at all. To pass the Turing test simply requires imitating a human, predicting what words they would say. You only need to know the relationships between words.
    - dwaltrip 10y ago
      Only knowing the relationships between words alone would just be a poor proxy for knowing the meanings of the words, e.g. what real world concepts the words attempt to represent. You might be able to get pretty far with this technique, but I would bet a lot of money you would not be able to get reliable, in-depth human level communication. The system needs to have an understanding of the world. And then there is the fundamentally dynamic aspect of language, which strengthens the need for a rich understanding of the world that words describe and convey.

melloclello 10y ago

I can't say anything about the feasibility of what you describe, but something about the idea of being a being composed of pure text, no ability to perceive or visualise the world around me except through the medium of text, utterly horrifies me

erikpukinskis 10y ago

> All of the information of our world is contained in text. That's a bit overstated. I think maybe you mean to say "all of the information you'd need to be an effective citizen of our world is contained in text"? Or something similar? I think even that is too strong a claim, but it's at least understandable. As stated the assertion doesn't make any sense. There is more information in a glass of milk than could be stored on all of the computers on earth, and Heisenberg showed that it's impossible to even record all of the information about a single particle. Certainly there is no textual information available about my grandfather's eyes, but that information is accessible in the world for those who'll look. You seem to be underestimating the quantity of information you absorbed as a baby, just by reacting robotically to the people and events around you and absorbing the relationships between percepts and feelings.

Houshalter 10y ago
Well of course that's what I meant. Any information an average person knows is contained in text, somewhere. That is all common sense knowledge. Everything from detailed descriptions of trees, to the color of the sky, to the shape of the human face, etc. But also much more, like all of our scientific knowledge and written history. Billions of things the average person doesn't know.

hacker42 10y ago

I agree that learning to reason about the world likely does not require experience with motor control and proprioception (i.e. literally how babies do it), though I do think that you either need at least some sort of tempo-spatial experience (e.g. visual). Tempo-spatial representations are just extremely hard to convey by text only. You might get the idea of closeness by saying 'close is when two words are close in a sequence of words' and ordering by saying 'this word comes after that word', but I think it would be very difficult to extrapolate that concept to more than one dimension, and dimensions that actually have not just an ordering but also a metric (just think about our inability to reason about just the fourth dimension). You need rich representations of our 3+1 dimensional world to be able to reason about it and text only gives you perhaps "0.5" dimensions (because it lacks a metric, i.e. it does not convey durations in terms of the ticks of the recurrent network). But I doubt, too, that interaction with the world is necessary, in fact I think it would be rather easy for an AI to simply write motor programs in a programming language given unrestricted and noiseless memory once it has learned to reason about tempo-spatial patterns from just observing them and identifying them with our language-coded shared concept space. It is not constrained to real-time performance of actions as humans are, therefore it can take the much easier way of programming any interaction with the world as needed, on the fly. Our shared concept space likely sufficiently conveys our general (common sense) knowledge how these patterns are known to interact and evolve over time once rudimentary tempo-spatial representations are in place.

hacker42 10y ago
tl;dr I think you need a small set of tempo-spatially grounded meanings (though not necessarily agent-related) and you can bootstrap everything from that using only textual knowledge.

dmreedy 10y ago

> All of the information of our world is contained in text Even if this were a true statement, it's still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. While it may be the case that certain varieties of intelligence are learnable tabula rasa from a powerful neural net, the surface form of human natural language (the part your recommending measuring) may simply not have enough information in it to decode the whole picture. It is entirely possible that you need to supply some of your own information as well to the picture, in a specific manner so as to act as a kind of decryption key. A record needs a record player, even if you can make similar sounds with cassettes and CDs. And so, I'm willing to bet that you simply cannot, using raw, uninformed statistical techniques, predict what word a human would say next. You need to understand more of the underlying structure of humans first. I will agree, however, that the success towards the Hutter Prize is a valuable demonstration of AI progress. Simply because I believe that maximal compression and the kind of intelligence I'm talking about are one and the same thing. You need to offload as much of the semantic weight of the corpus into the encryption algorithm as you can. That means building a very complex model of natural language. And if you accept the premise that this model is not simply learnable by observing the surface form, then that means building Strong AI

Houshalter 10y ago
>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even markov chains. There are tons of statistical patterns in text that can be learned by computers.
- dmreedy 10y ago
  It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.

jacobrobbins 10y ago

> All of the information of our world is contained in text. This statement is false. There is a well known thought experiment called Mary’s Room the gist of which is that knowing all conceivable scientific knowledge about how humans perceive color is still not a substitute for being a human and perceiving the color red: https://philosophynow.org/issues/99/What_Did_Mary_Know https://philosophynow.org/issues/99/What_Did_Mary_Know The experience of seeing red is an example of what is called “qualia”. In Google AI systems that identify cats, birds, etc it is reasonable to imagine AI technology evolving towards systems that can discuss those objects at the level of a typical person. However with an AI based on text only there is no possibility of that. It would be like discussing color with a blind person or sound with a deaf person.

sgnelson 10y ago
To further this comment, also see Tacit Knowledge, of which the very definition is essentially knowledge that cannot or is extremely difficult to transfer through words alone. https://en.wikipedia.org/wiki/Tacit_knowledge https://en.wikipedia.org/wiki/Tacit_knowledge
Houshalter 10y ago
Mary's Room and qualia is totally irrelevant. I'm not asking if the computer will "feel" "redness", simply if it can pretend to do so through text. If it can talk about the color red, in a way indistinguishable from any other human talking about red. In any case, at some level everything is symbols. A video is just a bunch of 1's and 0's, as is text, and everything else. A being raised on only text input would have qualia just like a being raised on video input. It would just be different qualia.
- Natanael_L 10y ago
  Like explained elsewhere here, it may be that our brains share a genetically coded "decryption key", and that many of the things we talk about are too poorly and noisily expressed for a purely text based computer AI to ever truly replicate the processes going on inside our brains. Insufficient data, simply put, and no way to get it. It may sure look indistinguishable, but on the inside it just wouldn't be the same.

orthoganol 10y ago

I think if AGI were possible from the basic statistical NLP techniques outlined in most advanced NLP textbooks, it would have already happened a decade ago.

Houshalter 10y ago
I'm not saying it is possible from just basic statistical NLP techniques. It may take much more advanced techniques. And it may take much more computing power than we have even now. But I do believe it is possible, someday. Probably within our lifetime.
- orthoganol 10y ago
  I certainly think AGI is possible. I just don't think Word2vec, RNN's, i.e. stuff from the NLP textbooks, is in the same ballpark as what it will take to achieve. Edit - To be more clear I also agree AGI could be possible with text as the only input. I just think we need a new paradigm. Ostensibly AGI is meant to mimic human intelligence (minus the pitfalls), so IMO, the best approach will be that which mimics the underlying processes of human intelligence - not just the results. Traditional statistical NLP methods will probably have some role in this final system, but not the heart of it, as far as mimicking intelligence by mimicking intelligence's underlying processes goes.

Joof 10y ago

The problem seems to be in understanding the overall meaning over long periods of time. We can understand general structure quite easily and a single sentence might make sense, but overall it never ends up cohesive or meaningful. Maybe we can do that with just words, but humans certainly don't. Words are related to concepts first, then we figure out the meaning of the rest.

Cybiote 10y ago

I agree that AI is possible from text alone but only with the added stipulation that full understanding of text requires a very sophisticated learner. In order to predict what a human will say next, you need the ability to maintain extended context, use world models plus deduction to narrow and maintain multiple possibilities; all while being able to infer the state of the thing you are trying to predict (which means the AI itself has complex internal state far beyond anything an RNN or Neural Turing machine could manage today). If I said "That is one huge fan", could you predict what my next word will be? It would depend a lot on context and the ability to reason within a complex world model. Depending on whether I had gone to a concert or to a wind tunnel, your distribution over guesses would alter. If I had gone whale watching you might even suspect I made a typo. Changing huge to large would lead to major to no adjustments, depending on each guess. So while I agree an AI could emerge from text alone, it would have to be very sophisticated to do this.

wlievens 10y ago

So a related question becomes, can you learn to understand and thus predict physics (the way a child does - I'm not talking about quantum mechanics) from literature only, without interacting in space?

knodi 10y ago
Yes, the child doesn't learn physics he or she learns motor control of its body. Baby learning to talk is more about the brain learning to control its body through throwing its neurological system. Look at animal kingdom some animals are walking about 5 minutes of being born and running in hours.
xapata 10y ago
> from literature only How about when you include multimedia recordings? Or give the machine a camera and wheels?
- wlievens 10y ago
  Giving the machine a camera and wheels is basically embedding it in the real world. It goes against the spirit of my question, and since I was asking it skeptically it's actually the thing I intuitively (without any relevant expertise!) expect to be required.

hellameta 10y ago

> It's learning the meaning of words, and the relationships between them. It's learning a meaning, not the meaning. It's just a probabilistic model for the occurrence of a word based on the words that surround it. This should not serve as a base for the rest of your claims. Anyway -- the improvements gained by multi-modal systems essentially disprove your thesis. Which is a good news! We're making progress.

ACow_Adonis 10y ago

> All of the information of our world is contained in text Communicate the concept of "green" to me in text. The sound of a dog barking, a motor turning over, a sonic boom, or the experience of a Doppler shift. Beethoven's symphony. Sour. Sweet. What does "mint" taste like? Shame. Merit. Learn facial and object recognition via text. Vertigo. Tell a boxer how to box by reading? Hand eye coordination, bodies in 3 dimensional space. Look, I love text, maybe even more than yourself. But all these things imbibe, structure and influence or text, but are not contained in them. To make substantial inroads to something that looks like human esque AI, text is not enough. The division of these fields are artificial and based on our current limited tech and the specialisation of our researchers, faculties and limitations. When we read, we play back memories, visions, sounds, feelings, etc, and inherent ideas gained through experience of ourselves as physical bodies in space. Strong AI, at least to be vaguely recognised as such, must work with algorithms and machinery that understand these things, but which then works at that next level of abstraction to combine them into proper human type concepts. Of course, there is the question about why we would want to create a human like AI, it's my contention that human like AI isn't actually what many of us would want, but that's another topic...

rictic 10y ago
I think you're over-romanticizing the problem. I won't touch the qualia aspect, but everything necessary to flawlessly pretend to understand the color green, the sound of a dog's bark, or the experience of hearing a sonic boom can be represented with text. As an existence proof, you could encode an adult human as a serial stream of characters.
Houshalter 10y ago
Are blind or deaf people not intelligent? But if you must pretend to be sighted and hearing, there are many descriptions of green, of dogs barking, of motors, etc, scattered through the many books written in English (and other languages.) Are these descriptions perfect? Maybe not. But they are sufficient to mimic or communicate with humans through text. It's sufficient to beat a Turing test, to answer questions intelligently, to write books and novels, and political arguments, etc. If that's not AGI, I don't know what is.

eli_gottlieb 10y ago

Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers. It will need actual "sense-data" to do that. For instance, to get a model that generates image captions, you need to train it with actual images. If the much-vaunted "general intelligence" consists in both vague and precise causal reasoning and optimal control with respect to objects in the real world, then no, it obviously cannot be done with mere language. At least one sensor and one effector will be needed to train an ML/AI model to perform those tasks.

Houshalter 10y ago
There is nothing magical about "sense data". A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. The internet is so large and so comprehensive (especially if you include digitized books and papers, e.g. libgen or google books) that I doubt any important information that can be learned through video data, can't be obtained through text data.
- eli_gottlieb 10y ago
  >A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different. Uhhhh... there's this thing called entropy. A corpus of video data is vastly higher in entropy content than text data, and probably a good deal more tightly correlated too, making it much easier to learn from. Remember, for a human, the whole point of speech and text is to function as a robust, efficient code (in the information-theoretic sense) for activating generative models we already have in our heads when we acquire the words. This is why we have such a hard time with very abstract concepts like mathematical ones: the causal-role concepts (easy to theorize how to encode those as generative models) are difficult to acquire from nothing but the usage statistics of the words and symbols, in contrast to "concrete", sense-grounded concepts, which have large amounts of high-dimensional data to fuel the Blessing of Abstraction. Nevermind, I should probably just get someone to let me into a PhD program so I can publish this stuff. If only they'd consider the first paper novel enough already!

MichaelMoser123 10y ago

> Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words doesn't that depend on the syntax of a language? my guess is that it would not work for a language that has a more flexible word order than English; so it wouldn't quite work for such languages as Russian, Turkish and Finnish (and other languages).