5 ms·

If the machine or human is sufficiently intelligent and there is enough data and time, it is possible to do so as long as the non-random language is useful for

If the machine or human is sufficiently intelligent and there is enough data and time, it is possible to do so as long as the non-random language is useful for human communication and the language sample describes some experiences or things we understand. There are principles underlying all human languages. For example, they are used to describe the world or communicate intention and they must be understandable by a human. Given that the copious data of the non-random language should describe something we understand about the world or social dynamics, we can try to match them with world/domain models and find patterns of plausible interpretations that fit. The patterns should also satisfy the principle of communication economy and the limits of human cognition, such as the size of working memory. These constraints help narrow down the possibilities. The human/AI system can test and figure out the most likely patterns first and use them to decipher more patterns. All above is possible in principle but beyond the current state-of-the-art.

rvense 5y ago
This is true, but you're forgetting that the sign (ie. pairing of an expression and its meaning) is arbitrary. There's no particular reason that the sound 'cow' (or the squiggles c-o-w when we write) refers to cows, and you can see that quite clearly by the fact that it only does so in one out of the Earth's 6-7000 current languages. Note that this doesn't mean that relationships between words are arbitrary, they are of course highly systematic, else language wouldn't work. It also doesn't mean that language doesn't have history, the changes are also systematic. But if you take a random surface form (sound or writing), there is absolutely no way to relate it with any certainty to any content without a context that allows you to infer it.
- eru 5y ago
  You are being too pessimistic. Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly. And we do have context! Just knowing that a language is used by humans gives lots and lots of context. Human cultures and languages have more in common than you might realize. For example, have a look at Deixis (https://en.wikipedia.org/wiki/Deixis https://en.wikipedia.org/wiki/Deixis): > In linguistics, deixis (/ˈdaɪksɪs/, /ˈdeɪksɪs/)[1] is the use of general words and phrases to refer to a specific time, place, or person in context, e.g., the words tomorrow, there, and they. Words are deictic if their semantic meaning is fixed but their denoted meaning varies depending on time and/or place. Words or phrases that require contextual information to be fully understood—for example, English pronouns—are deictic. Deixis is closely related to anaphora. Although this article deals primarily with deixis in spoken language, the concept is sometimes applied to written language, gestures, and communication media as well. In linguistic anthropology, deixis is treated as a particular subclass of the more general semiotic phenomenon of indexicality, a sign "pointing to" some aspect of its context of occurrence. > Although this article draws examples primarily from English, deixis is believed to be a feature (to some degree) of all natural languages. I suspect with a large enough corpus you can most likely figure out the deixis of the given language. See https://linguistics.stackexchange.com/a/4046 https://linguistics.stackexchange.com/a/4046 for some more structure shared with all human languages: > However, all languages have some sort of a clause-type thing allowing them to express predication, attribution, etc. See Dixon's Basic Linguistic Theory: Basic Linguistic Theory Volume 1: Methodology . All languages also must have means of expressing cohesion and coherence (texture) although this is much less studied in cross linguistic perspective. Punctuated sentences are a kind of cohesive device. (A comment on this answer points out that the reality is probabilistic, of course:) > While I agree with you, it seems to me that linguists who study languages with a strong written tradition often talk about 'sentences', even when their examples are not from writing. Those of us who work on previously unwritten languages (I think) tend to talk about 'texts' or 'utterances'. I think this is an important issue as it relates to how some linguists have ignored the true messiness that is often found in examples of human language. Of course, all of this refers to natural human languages. Not totally arbitrary constructs.
  - rvense 5y ago
    > Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly. As I've said in a sibling comment, GPT-3 is able to construct new utterances in the same language that it learns from. It does not learn its language while learning a model of the world because it does not exist in the world, so it does not learn semantics like we do, which is requirement for making a translation. I am familiar with all these things you talk about, but you are not starting from a sound understanding of semiotics and the difference between content and expression. Say you learn a lot of syntax from text, you are able to write out a list of paradigms. How do you know which verb form is past and which form is present? How do you know which word means day and which word means night? The information required to make that judgement is not present in the text artifact.
    - eru 5y ago
      > How do you know which verb form is past and which form is present? There's lots and lots of context. One interpretation will fit the data much better than another. Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_. I agree that all these connections are only probabilistic and you need enormous amounts of data. Btw, GTP-3 can translate between English and French to certain very limited extent; but that's probably mostly because it has seen example translations. To give you a testable prediction of my theory: if you trained something like GPT-3 on a corpus that includes examples of natural languages A and B, but never any piece of text that contains both languages, yet alone example translations, I would expect that nevertheless, the resulting network would share the same internal representation for the concept of 'cow' in both languages.
      - rvense 5y ago
        > Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_. This still rests on a lot of assumptions about the language. Pronouns are used much more in some languages than others, some languages have more pronouns than others, etc.. Tense is not at all universal in the world's languages as you probably know. There are linguistic theories that even question the universality of nouns and verbs as categories, and even if they are to an extent universal, they are certainly not fixed in their boundaries across languages, e.g. most things that in English adjectives are expressed as verbs in Classical Arabic. Even if you could do a perfect job of extracting a set of syntactic categories and morphological paradigms from a language, I don't think you would be able to do anything with it. Word 3849 from category C often combines with words 201, 635, and 9913 from category F in conjugations 1-4, but never 5. And of course, even for nouns that refer to basic, physical things, often some languages will have several words where another will make do with one, or have none. It quickly becomes a question of culture.

rvense 5y ago

This is true, but you're forgetting that the sign (ie. pairing of an expression and its meaning) is arbitrary. There's no particular reason that the sound 'cow' (or the squiggles c-o-w when we write) refers to cows, and you can see that quite clearly by the fact that it only does so in one out of the Earth's 6-7000 current languages. Note that this doesn't mean that relationships between words are arbitrary, they are of course highly systematic, else language wouldn't work. It also doesn't mean that language doesn't have history, the changes are also systematic. But if you take a random surface form (sound or writing), there is absolutely no way to relate it with any certainty to any content without a context that allows you to infer it.

eru 5y ago
You are being too pessimistic. Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly. And we do have context! Just knowing that a language is used by humans gives lots and lots of context. Human cultures and languages have more in common than you might realize. For example, have a look at Deixis (https://en.wikipedia.org/wiki/Deixis https://en.wikipedia.org/wiki/Deixis): > In linguistics, deixis (/ˈdaɪksɪs/, /ˈdeɪksɪs/)[1] is the use of general words and phrases to refer to a specific time, place, or person in context, e.g., the words tomorrow, there, and they. Words are deictic if their semantic meaning is fixed but their denoted meaning varies depending on time and/or place. Words or phrases that require contextual information to be fully understood—for example, English pronouns—are deictic. Deixis is closely related to anaphora. Although this article deals primarily with deixis in spoken language, the concept is sometimes applied to written language, gestures, and communication media as well. In linguistic anthropology, deixis is treated as a particular subclass of the more general semiotic phenomenon of indexicality, a sign "pointing to" some aspect of its context of occurrence. > Although this article draws examples primarily from English, deixis is believed to be a feature (to some degree) of all natural languages. I suspect with a large enough corpus you can most likely figure out the deixis of the given language. See https://linguistics.stackexchange.com/a/4046 https://linguistics.stackexchange.com/a/4046 for some more structure shared with all human languages: > However, all languages have some sort of a clause-type thing allowing them to express predication, attribution, etc. See Dixon's Basic Linguistic Theory: Basic Linguistic Theory Volume 1: Methodology . All languages also must have means of expressing cohesion and coherence (texture) although this is much less studied in cross linguistic perspective. Punctuated sentences are a kind of cohesive device. (A comment on this answer points out that the reality is probabilistic, of course:) > While I agree with you, it seems to me that linguists who study languages with a strong written tradition often talk about 'sentences', even when their examples are not from writing. Those of us who work on previously unwritten languages (I think) tend to talk about 'texts' or 'utterances'. I think this is an important issue as it relates to how some linguists have ignored the true messiness that is often found in examples of human language. Of course, all of this refers to natural human languages. Not totally arbitrary constructs.
- rvense 5y ago
  > Yes, it's hard to figure out cow just from a big corpus; though you might learn enough about 'cow' to form meaningful sentences and answer questions. That's what GPT-3 is doing after all, and language models are still improving quickly. As I've said in a sibling comment, GPT-3 is able to construct new utterances in the same language that it learns from. It does not learn its language while learning a model of the world because it does not exist in the world, so it does not learn semantics like we do, which is requirement for making a translation. I am familiar with all these things you talk about, but you are not starting from a sound understanding of semiotics and the difference between content and expression. Say you learn a lot of syntax from text, you are able to write out a list of paradigms. How do you know which verb form is past and which form is present? How do you know which word means day and which word means night? The information required to make that judgement is not present in the text artifact.
  - eru 5y ago
    > How do you know which verb form is past and which form is present? There's lots and lots of context. One interpretation will fit the data much better than another. Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_. I agree that all these connections are only probabilistic and you need enormous amounts of data. Btw, GTP-3 can translate between English and French to certain very limited extent; but that's probably mostly because it has seen example translations. To give you a testable prediction of my theory: if you trained something like GPT-3 on a corpus that includes examples of natural languages A and B, but never any piece of text that contains both languages, yet alone example translations, I would expect that nevertheless, the resulting network would share the same internal representation for the concept of 'cow' in both languages.
    - rvense 5y ago
      > Eg in spoken languages the equivalent of 'I' is used much more often than any other pronoun. Similarly simpler sentences are perhaps more likely to refer to the present than the past; eg past tense is much more likely to come with an additional specification of _when_ stuff happened. That information is less kind of a given when we are talking about _now_. This still rests on a lot of assumptions about the language. Pronouns are used much more in some languages than others, some languages have more pronouns than others, etc.. Tense is not at all universal in the world's languages as you probably know. There are linguistic theories that even question the universality of nouns and verbs as categories, and even if they are to an extent universal, they are certainly not fixed in their boundaries across languages, e.g. most things that in English adjectives are expressed as verbs in Classical Arabic. Even if you could do a perfect job of extracting a set of syntactic categories and morphological paradigms from a language, I don't think you would be able to do anything with it. Word 3849 from category C often combines with words 201, 635, and 9913 from category F in conjugations 1-4, but never 5. And of course, even for nouns that refer to basic, physical things, often some languages will have several words where another will make do with one, or have none. It quickly becomes a question of culture.