5 ms·

Not a single mention if this is only applicable to english or to other natural languages. Afaict this mostly lists advancements in ELP (english language process

by mqus 6y ago

Not a single mention if this is only applicable to english or to other natural languages. Afaict this mostly lists advancements in ELP (english language processing), Especially the Winograd schema (or ar least the given example) seems to be heavily focused on english. Relevant article for this problem: https://news.ycombinator.com/item?id=24026511 https://news.ycombinator.com/item?id=24026511

MiroF 6y ago
But there's no reason the models are english specific...
- woodson 6y ago
  In a certain way, they could be, simply because their structure may work better on the type of language that English is. It may not work as well for languages that exhibit other grammatical patterns (e.g. morphologically rich languages, or those that exhibit superficially more flexible word order which nonetheless conveys information about topic/focus).
  - dvduval 6y ago
    Some languages like Chinese have a large corpus of information that can be studied, and there has been a continuous effort to standardize the language. it wouldn't surprise me if it turns out to be easier than English. I would expect it would be more difficult when you're talking about languages that have regional differences in usage of words, perhaps northern Vietnamese vs. Southern Vietnamese, or the Spanish of Spain versus the Spanish of Mexico.
    - east2west 6y ago
      Chinese has large regional differences, too. Cantonese has different grammar, not just word usage. Modern Chinese grammar is somewhat similar to English while Cantonese follows ancient Chinese, which places verb to the end of sentence.
      - dvduval 6y ago
        True, and Mandarin would be the better word to use here. Mandarin and Cantonese are for all intents and purposes separate languages. They are more different than English and Spanish.
        yorwba 6y ago
        If we're going down that path, then it should more specifically be Standard Mandarin, since Standard Mandarin and e.g. Southwestern Mandarin are only partially mutually intelligible like French and Spanish are partially mutually intelligible.
        DonaldFisk 6y ago
        Chinese is the written language, based on Mandarin. Mandarin and Cantonese are separate, though closely related, spoken languages. They're a lot closer to each other than English and Spanish (more like French and Spanish), which is why the same written language works for both sets of speakers: most of the vocabulary is shared.
        spacehunt 6y ago
        This is only because almost all native Cantonese speakers are trained to read and write in Standard Chinese from the very beginning. Standard Chinese can be spoken in Cantonese, but while understandable, it is never done, in contrast to Mandarin. In recent years written online conversations (like blogs, Facebook posts, instant messages) by Cantonese speakers are increasingly written in Cantonese instead, and this is slowly spreading to more "formal" media such as books. Source: I'm a native Cantonese speaker.
        DonaldFisk 6y ago
        Yes, it seems I underestimated the difference: "In spoken Cantonese the frequency of use of Cantonese dialect words is close to 50 percent (Li, et. al., 1995, p236)." (from https://www.aclweb.org/anthology/P98-2238.pdf https://www.aclweb.org/anthology/P98-2238.pdf). They're still a lot closer than English and Spanish or even French and Spanish (https://www.ezglot.com/most-similar-languages.php https://www.ezglot.com/most-similar-languages.php).
    - kingkawn 6y ago
      All language has significant regional difference if it covers enough regions
- akerro 6y ago
  I'm 100% sure NLP will do the same mistake with non-English language as face recognition did with faces of black people. I'm taking bets.
  - MiroF 6y ago
    ie. a dataset/corpus problem?
- m0zg 6y ago
  For some of them, there is. Russian (my mother tongue), for example, has the kind of morphology that makes linguists' hair stand on end. Also, verbs and nouns are gendered (and therefore they must agree, and you must know what everything refers to at all times, often pretty far away from where the noun was first mentioned), and there are declensions on various parts of speech, and they must agree as well. And words can be freely formed out of several roots, prefixes, and suffixes. For any of this to be understood or generated, your model has to be able to model all of that, which is much harder than modeling English.
  - MiroF 6y ago
    I think that modern BPE and techniques are more than sophisticated enough to handle things like this.
    - m0zg 6y ago
      Could be, but I've yet to see an example of any NLP task that performs anywhere near as well on Russian as it does on English. Trouble with NLP in general, is that humans, armed with their imperceptible error correction and general intelligence, expect unreasonable things from systems which fundamentally have neither. Error rate in single digit percentages is commonly perceived as terrible, and because of the cardinality of the input space "plausible" errors don't really happen and failure modes are mostly catastrophic. This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).
      - MiroF 6y ago
        > any NLP task that performs anywhere near as well on Russian as it does on English This has a lot more to do with smaller Russian corpora than anything to do with peculiarities of the Russian language. > This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking). I've trained NLP models used on over 15 different languages used in production
- YeGoblynQueenne 6y ago
  Yes, there is. A "model" is a set of parameters optimisted by some algorithm or system trained on the data in a specific dataset. Thus, a language model trained on a dataset of English language examples is only capable of representing English language utterances, not, e.g. French, or Greek, or Gujarati utterances. Diagrammatically: data --> system --> model What is not necessarily English-specific are the systems used to train different language models, at least in theory. In practice, systems are typically hand-crafted and fine-tuned to specific datasets, to such a degree that most of the work has to be done anew to train on a different dataset.
  - MiroF 6y ago
    I was using model to mean the architecture, ie. the way you're modeling the data flows - not the specific params, which are ofc language specific.
  - mannykannot 6y ago
    This is very interesting, and I would be interested in learning more about the language-specific accommodations that are made. My minimal understanding of the GPT systems is that they are initially trained on a large corpus of English-language text, and then often, though not necessarily, given few-shot training on example tasks before being tested with questions that are scored. It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. One guess is that the training data is annotated with grammatical information, in which case I would wonder if this is just a shortcut, or whether it solves a fundamental problem for such systems. Another guess is that the training set includes disambiguation, but that would seem to render meaningless the results on Winograd schema. (Update: I withdraw this last point, as presumably the Winograd-schema questions themselves are not disambiguated. Disambiguation of the training corpus would be a quite significant language-specific accommodation, if that is what is happening.)
    - MiroF 6y ago
      > It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. It really is that unremarkable, despite the GPs insinuations to the contrary.

by mqus 6y ago

MiroF 6y ago

But there's no reason the models are english specific...

woodson 6y ago
In a certain way, they could be, simply because their structure may work better on the type of language that English is. It may not work as well for languages that exhibit other grammatical patterns (e.g. morphologically rich languages, or those that exhibit superficially more flexible word order which nonetheless conveys information about topic/focus).
- dvduval 6y ago
  Some languages like Chinese have a large corpus of information that can be studied, and there has been a continuous effort to standardize the language. it wouldn't surprise me if it turns out to be easier than English. I would expect it would be more difficult when you're talking about languages that have regional differences in usage of words, perhaps northern Vietnamese vs. Southern Vietnamese, or the Spanish of Spain versus the Spanish of Mexico.
  - east2west 6y ago
    Chinese has large regional differences, too. Cantonese has different grammar, not just word usage. Modern Chinese grammar is somewhat similar to English while Cantonese follows ancient Chinese, which places verb to the end of sentence.
    - dvduval 6y ago
      True, and Mandarin would be the better word to use here. Mandarin and Cantonese are for all intents and purposes separate languages. They are more different than English and Spanish.
      - yorwba 6y ago
        If we're going down that path, then it should more specifically be Standard Mandarin, since Standard Mandarin and e.g. Southwestern Mandarin are only partially mutually intelligible like French and Spanish are partially mutually intelligible.
      - DonaldFisk 6y ago
        Chinese is the written language, based on Mandarin. Mandarin and Cantonese are separate, though closely related, spoken languages. They're a lot closer to each other than English and Spanish (more like French and Spanish), which is why the same written language works for both sets of speakers: most of the vocabulary is shared.
        spacehunt 6y ago
        This is only because almost all native Cantonese speakers are trained to read and write in Standard Chinese from the very beginning. Standard Chinese can be spoken in Cantonese, but while understandable, it is never done, in contrast to Mandarin. In recent years written online conversations (like blogs, Facebook posts, instant messages) by Cantonese speakers are increasingly written in Cantonese instead, and this is slowly spreading to more "formal" media such as books. Source: I'm a native Cantonese speaker.
        DonaldFisk 6y ago
        Yes, it seems I underestimated the difference: "In spoken Cantonese the frequency of use of Cantonese dialect words is close to 50 percent (Li, et. al., 1995, p236)." (from https://www.aclweb.org/anthology/P98-2238.pdf https://www.aclweb.org/anthology/P98-2238.pdf). They're still a lot closer than English and Spanish or even French and Spanish (https://www.ezglot.com/most-similar-languages.php https://www.ezglot.com/most-similar-languages.php).
  - kingkawn 6y ago
    All language has significant regional difference if it covers enough regions
akerro 6y ago
I'm 100% sure NLP will do the same mistake with non-English language as face recognition did with faces of black people. I'm taking bets.
- MiroF 6y ago
  ie. a dataset/corpus problem?
m0zg 6y ago
For some of them, there is. Russian (my mother tongue), for example, has the kind of morphology that makes linguists' hair stand on end. Also, verbs and nouns are gendered (and therefore they must agree, and you must know what everything refers to at all times, often pretty far away from where the noun was first mentioned), and there are declensions on various parts of speech, and they must agree as well. And words can be freely formed out of several roots, prefixes, and suffixes. For any of this to be understood or generated, your model has to be able to model all of that, which is much harder than modeling English.
- MiroF 6y ago
  I think that modern BPE and techniques are more than sophisticated enough to handle things like this.
  - m0zg 6y ago
    Could be, but I've yet to see an example of any NLP task that performs anywhere near as well on Russian as it does on English. Trouble with NLP in general, is that humans, armed with their imperceptible error correction and general intelligence, expect unreasonable things from systems which fundamentally have neither. Error rate in single digit percentages is commonly perceived as terrible, and because of the cardinality of the input space "plausible" errors don't really happen and failure modes are mostly catastrophic. This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).
    - MiroF 6y ago
      > any NLP task that performs anywhere near as well on Russian as it does on English This has a lot more to do with smaller Russian corpora than anything to do with peculiarities of the Russian language. > This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking). I've trained NLP models used on over 15 different languages used in production
YeGoblynQueenne 6y ago
Yes, there is. A "model" is a set of parameters optimisted by some algorithm or system trained on the data in a specific dataset. Thus, a language model trained on a dataset of English language examples is only capable of representing English language utterances, not, e.g. French, or Greek, or Gujarati utterances. Diagrammatically: data --> system --> model What is not necessarily English-specific are the systems used to train different language models, at least in theory. In practice, systems are typically hand-crafted and fine-tuned to specific datasets, to such a degree that most of the work has to be done anew to train on a different dataset.
- MiroF 6y ago
  I was using model to mean the architecture, ie. the way you're modeling the data flows - not the specific params, which are ofc language specific.
- mannykannot 6y ago
  This is very interesting, and I would be interested in learning more about the language-specific accommodations that are made. My minimal understanding of the GPT systems is that they are initially trained on a large corpus of English-language text, and then often, though not necessarily, given few-shot training on example tasks before being tested with questions that are scored. It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. One guess is that the training data is annotated with grammatical information, in which case I would wonder if this is just a shortcut, or whether it solves a fundamental problem for such systems. Another guess is that the training set includes disambiguation, but that would seem to render meaningless the results on Winograd schema. (Update: I withdraw this last point, as presumably the Winograd-schema questions themselves are not disambiguated. Disambiguation of the training corpus would be a quite significant language-specific accommodation, if that is what is happening.)
  - MiroF 6y ago
    > It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. It really is that unremarkable, despite the GPs insinuations to the contrary.