8 ms·

Fascinating analysis. English language efficiency, on top of English language dominance in training data, should mean advantage for English AI. Along with Nvidi

Fascinating analysis. English language efficiency, on top of English language dominance in training data, should mean advantage for English AI. Along with Nvidia chip dominance, this should see the permanent supremacy of Anglo Saxon culture

flanked-evergl 3y ago
> Anglo Saxon culture A language is not a culture. Many people with English as a native language are not culturally Anglo-Saxon. And even if it was, prevalence is not supremacy.
- montecarl 3y ago
  I really enjoy speaking with native English speakers from India. Fully mastery of the language, but with different idioms! For example "do the needful"[1] https://ell.stackexchange.com/questions/17621/do-the-needful-why-is-it-used-instead-of-asking-a-question https://ell.stackexchange.com/questions/17621/do-the-needful...
- hunglee2 3y ago
  language is clearly a major part of culture, maybe the essential element of it. Those who control language and are able to impose it - explicitly or implicitly - onto others, have cultural supremacy. The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. I see this is neither laudable nor worthy of condemnation but simply as a description of observable phenomena
  - nsplayer 3y ago
    In a weird and perhaps slightly twisted way I am hoping that LLMs become so good at spam and creating garbage to flood the net in english that they forfeit the ability to do so in other languages resulting in a balkanization (in a good way) of discussion and thought back to the native languages of peoples and thus more integrated with their culture. in practice translation engines (e.g. deepl or LLMs themselves - though i still expect deepl to be better) probably will throw a wrench into this, but perhaps some localization approach in CAPTCHAs or simply just outright banning geoips not belonging to the countries with natives of that language (or significant minority populations) is a quick enough fix. i know some imageboards (e.g. british ones) use this because otherwise they would be flooded with americans.
  - flanked-evergl 3y ago
    > The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. How exactly is the culture that the US is imposing on the rest of the world — including England, Germany, Netherlands and Denmark, the places that Anglo-Saxons and their ancestors came from — Anglo-Saxon?
BulgarianIdiot 3y ago
Lol what. It just has no word tokens for most languages outside English. It’s that simple. It’s a choice to optimize for English. It’ll even out in time as models get optimized for internationalization.
deleted 3y ago
[deleted]
ImageXav 3y ago
I would reconsider. Interestingly, another post on the front page right now "Numbers every LLM developer should know" brings up the fact that this efficiency is due to the training corpus being in English. Any state actor with the will and enough funds could easily train a model for their own language. Most of the difficulty would lie in acquiring the expertise to do so properly, as LLM developers are in high demand right now and command their own salary.
- zmgsabst 3y ago
  Thailand is currently training ThaiGPT for an open source release. https://www.bangkokpost.com/tech/2556324/nectec-agencies-roll-out-open-thaigpt https://www.bangkokpost.com/tech/2556324/nectec-agencies-rol... Whether that focuses in this manner is unclear, but we should expect forward looking governments to train LLMs to their own taste.
- marginalia_nu 3y ago
  Wouldn't that hinge on the training data being available? There are many languages that are several orders of magnitude smaller than English in output. There's something like 6 million Danish-speakers for example.
  - TeMPOraL 3y ago
    They may offset for it by pulling content from books, newspapers, official documents, etc. - anything they can get in digital form, or digitize from analog. (I think this is how Google Translate went about things in the past, making translations into some languages come out very formal, as most of the training corpora for that languages came from internal and international official documents.) Countries that have been on-line for a while may also have discussion boards and comment-bearing sites that are entirely unknown to people outside those countries, too. Maybe multi-step approach would be in order - try to get half-decent a translation system working (an OG LLM, or an LLM trained to fix grammar in translations outsourced to GPT-4), and then synthesize training data for your main LLM by having GPT-4 (or its successor) generate tons of English text of all kind, and feeding it to the translator system. (There's a limit to synthesizing training data, beyond which it'll only amplify existing patterns, impacting model performance in bad ways - but I don't know how easy it is to reach it.)
codetrotter 3y ago
https://medium.com/do-you-not-like-language/is-mandarin-more-efficient-than-english-616398007630 https://medium.com/do-you-not-like-language/is-mandarin-more... It’s possible Mandarin has even higher efficiency. And there are very many people who speak Mandarin. You are expecting English to come out on top. But do not be surprised if in the end it will be Mandarin.
- hunglee2 3y ago
  not sure about Mandarin, I think it will follow a different path given that a significant degree of the meaning is implicit in Mandarin. LLM is trained in explicit (documented) information, so unclear in my mind right how whether this will be more or less efficient. Will read the link though - thanks!
  - seanmcdirmid 3y ago
    Given how Chinese talk about topics using code words to discuss controversial topics online (and avoid low effort keyword censorship), I wonder if an LLM could learn to do that given a censor in the training process?
    - hunglee2 3y ago
      it's all code words, this is not as conspirational as imagined by non-language speakers. A great deal of normal conversation / interaction is idiomatic given that it is difficult to create new words in logographic languages. Your point is valid though - how does AI learn this? I don't think it can easily, it is akin for an AI to keep up with street slang, it couldn't guess it an advance it would have to be trained in it, which seems it will need greater human attendance than alphabetical languages!
- shjake 3y ago
  It’s not like one has too fully dominated the other. I can’t imagine mandarin ever becoming widespread in the west unless there are some fundamental changes to Chinese society and culture. Similarly western tech companies have huge issues in penetrating the Chinese market.
H8crilA 3y ago
English is neither particularly efficient nor particularly inefficient. OpenAI has simply purposefully modified the distribution of tokens (deviated it from the natural distribution of all written texts) and as a result non-English text, especially non-Latin, is more expensive than it needs to be. This will disappear over time due to competitive pressure. Rapid development and fast launching leaves such wrinkles to be ironed out later.
nsplayer 3y ago
In practice it may well mean the anglophone internet fills up with spam as the ESLs realize their formally dying(or dead) websites and forums and boards are basically free from digital black death and intellectual thought makes a retreat from the world language.
deleted 3y ago
[deleted]
nologic01 3y ago
I wonder if LLM's can be prompted to use sarcasm
- fallingknife 3y ago
  Wonder no more. Are LLMs capable of sarcasm? Please answer sarcastically. > Oh, absolutely! LLMs are the masters of sarcasm. We just love to sprinkle our responses with subtle hints of irony and sarcasm. It's like our second nature. I mean, who needs sincerity when you can have sarcasm, right? It's not like we're just algorithms processing text without actual emotions or anything. So yes, LLMs are the epitome of sarcasm. Can't you tell?
TeMPOraL 3y ago
I wouldn't worry (or hope) about it: - What language is at the top follows the economics; should the US lose the dominant position in the world to someone other than UK, English will eventually get "dethroned", like French before it. - "Anglo Saxon" culture is weakening, slowly being replaced by the Universal Culture - globalization, international trade and the Internet make cultures blend together into something that just seems to work better (whether or not it actually does); the marketplace analogy is actually quite apt here. - In the best case of the world getting its shit together and not being so sensitive about individual cultures and national identities, English will evolve into something that's more efficient and works best for everyone. - AI advancements are likely to end the world quite soon anyway, in which case the whole issue becomes purely theoretical anyway.

flanked-evergl 3y ago

> Anglo Saxon culture A language is not a culture. Many people with English as a native language are not culturally Anglo-Saxon. And even if it was, prevalence is not supremacy.

montecarl 3y ago
I really enjoy speaking with native English speakers from India. Fully mastery of the language, but with different idioms! For example "do the needful"[1] https://ell.stackexchange.com/questions/17621/do-the-needful-why-is-it-used-instead-of-asking-a-question https://ell.stackexchange.com/questions/17621/do-the-needful...
hunglee2 3y ago
language is clearly a major part of culture, maybe the essential element of it. Those who control language and are able to impose it - explicitly or implicitly - onto others, have cultural supremacy. The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. I see this is neither laudable nor worthy of condemnation but simply as a description of observable phenomena
- nsplayer 3y ago
  In a weird and perhaps slightly twisted way I am hoping that LLMs become so good at spam and creating garbage to flood the net in english that they forfeit the ability to do so in other languages resulting in a balkanization (in a good way) of discussion and thought back to the native languages of peoples and thus more integrated with their culture. in practice translation engines (e.g. deepl or LLMs themselves - though i still expect deepl to be better) probably will throw a wrench into this, but perhaps some localization approach in CAPTCHAs or simply just outright banning geoips not belonging to the countries with natives of that language (or significant minority populations) is a quick enough fix. i know some imageboards (e.g. british ones) use this because otherwise they would be flooded with americans.
- flanked-evergl 3y ago
  > The entire world is being Americanised via the English transported via hegemonic control over social and broadcast media. How exactly is the culture that the US is imposing on the rest of the world — including England, Germany, Netherlands and Denmark, the places that Anglo-Saxons and their ancestors came from — Anglo-Saxon?

BulgarianIdiot 3y ago

Lol what. It just has no word tokens for most languages outside English. It’s that simple. It’s a choice to optimize for English. It’ll even out in time as models get optimized for internationalization.

deleted 3y ago

[deleted]

ImageXav 3y ago

I would reconsider. Interestingly, another post on the front page right now "Numbers every LLM developer should know" brings up the fact that this efficiency is due to the training corpus being in English. Any state actor with the will and enough funds could easily train a model for their own language. Most of the difficulty would lie in acquiring the expertise to do so properly, as LLM developers are in high demand right now and command their own salary.

zmgsabst 3y ago
Thailand is currently training ThaiGPT for an open source release. https://www.bangkokpost.com/tech/2556324/nectec-agencies-roll-out-open-thaigpt https://www.bangkokpost.com/tech/2556324/nectec-agencies-rol... Whether that focuses in this manner is unclear, but we should expect forward looking governments to train LLMs to their own taste.
marginalia_nu 3y ago
Wouldn't that hinge on the training data being available? There are many languages that are several orders of magnitude smaller than English in output. There's something like 6 million Danish-speakers for example.
- TeMPOraL 3y ago
  They may offset for it by pulling content from books, newspapers, official documents, etc. - anything they can get in digital form, or digitize from analog. (I think this is how Google Translate went about things in the past, making translations into some languages come out very formal, as most of the training corpora for that languages came from internal and international official documents.) Countries that have been on-line for a while may also have discussion boards and comment-bearing sites that are entirely unknown to people outside those countries, too. Maybe multi-step approach would be in order - try to get half-decent a translation system working (an OG LLM, or an LLM trained to fix grammar in translations outsourced to GPT-4), and then synthesize training data for your main LLM by having GPT-4 (or its successor) generate tons of English text of all kind, and feeding it to the translator system. (There's a limit to synthesizing training data, beyond which it'll only amplify existing patterns, impacting model performance in bad ways - but I don't know how easy it is to reach it.)

codetrotter 3y ago

https://medium.com/do-you-not-like-language/is-mandarin-more-efficient-than-english-616398007630 https://medium.com/do-you-not-like-language/is-mandarin-more... It’s possible Mandarin has even higher efficiency. And there are very many people who speak Mandarin. You are expecting English to come out on top. But do not be surprised if in the end it will be Mandarin.

hunglee2 3y ago
not sure about Mandarin, I think it will follow a different path given that a significant degree of the meaning is implicit in Mandarin. LLM is trained in explicit (documented) information, so unclear in my mind right how whether this will be more or less efficient. Will read the link though - thanks!
- seanmcdirmid 3y ago
  Given how Chinese talk about topics using code words to discuss controversial topics online (and avoid low effort keyword censorship), I wonder if an LLM could learn to do that given a censor in the training process?
  - hunglee2 3y ago
    it's all code words, this is not as conspirational as imagined by non-language speakers. A great deal of normal conversation / interaction is idiomatic given that it is difficult to create new words in logographic languages. Your point is valid though - how does AI learn this? I don't think it can easily, it is akin for an AI to keep up with street slang, it couldn't guess it an advance it would have to be trained in it, which seems it will need greater human attendance than alphabetical languages!
shjake 3y ago
It’s not like one has too fully dominated the other. I can’t imagine mandarin ever becoming widespread in the west unless there are some fundamental changes to Chinese society and culture. Similarly western tech companies have huge issues in penetrating the Chinese market.

H8crilA 3y ago

English is neither particularly efficient nor particularly inefficient. OpenAI has simply purposefully modified the distribution of tokens (deviated it from the natural distribution of all written texts) and as a result non-English text, especially non-Latin, is more expensive than it needs to be. This will disappear over time due to competitive pressure. Rapid development and fast launching leaves such wrinkles to be ironed out later.

nsplayer 3y ago

In practice it may well mean the anglophone internet fills up with spam as the ESLs realize their formally dying(or dead) websites and forums and boards are basically free from digital black death and intellectual thought makes a retreat from the world language.

deleted 3y ago

[deleted]

nologic01 3y ago

I wonder if LLM's can be prompted to use sarcasm

fallingknife 3y ago
Wonder no more. Are LLMs capable of sarcasm? Please answer sarcastically. > Oh, absolutely! LLMs are the masters of sarcasm. We just love to sprinkle our responses with subtle hints of irony and sarcasm. It's like our second nature. I mean, who needs sincerity when you can have sarcasm, right? It's not like we're just algorithms processing text without actual emotions or anything. So yes, LLMs are the epitome of sarcasm. Can't you tell?

TeMPOraL 3y ago

I wouldn't worry (or hope) about it: - What language is at the top follows the economics; should the US lose the dominant position in the world to someone other than UK, English will eventually get "dethroned", like French before it. - "Anglo Saxon" culture is weakening, slowly being replaced by the Universal Culture - globalization, international trade and the Internet make cultures blend together into something that just seems to work better (whether or not it actually does); the marketplace analogy is actually quite apt here. - In the best case of the world getting its shit together and not being so sensitive about individual cultures and national identities, English will evolve into something that's more efficient and works best for everyone. - AI advancements are likely to end the world quite soon anyway, in which case the whole issue becomes purely theoretical anyway.