9 ms·

Wikipedia Has a Language Problem

34 points by _Microft 5y ago

h4wzdbf 5y ago
London Breed is a nazi. I hope she gets her ass fucked up on the street. Fucking cunt.
Chico75 5y ago
My impression was that wikidata was the main answer to this problem by centralizing all the facts in a central database to avoid each page having its own data and sources, but I can see how there is much more than the data at stake.
- tablespoon 5y ago
  > My impression was that wikidata was the main answer to this problem by centralizing all the facts in a central database to avoid each page having its own data and sources, but I can see how there is much more than the data at stake. IIRC, Wikidata doesn't centralize "all the facts," it just centralizes the basic infobox data, which is only a small part of any article
  - zaik 5y ago
    Wikidata can express more than just basic infobox data. That being said it's also a lot less expressive than natural language. I think "Abstract Wikipedia" tries to overcome this shortcoming, I wonder how it will turn out. It feels like an impossible feat.
rootsudo 5y ago
Disagree entirely on Japanese translating the English wiki - that is too many steps and most people do not do that. Also within Japan, SEO and such always put Japanese content first (especially on google.) vs en.wikipedia.org. The article is touching heavy on Japanese and if you do not have a basic fluency of the langauge, it's easy to jump to conclusion... Alongside, same with the renaming of Nanjing - what if the government renames the incident and it becomes the status quo? What is the source of truth? Machine translating one "wikipedia" is also wrong, because context is removed, and having English be the language of truth and all other languages translate from it - you'll miss out on many things, especially words that do not have a specific English translation. The article is poorly written and documented.
- mschuster91 5y ago
  > Alongside, same with the renaming of Nanjing - what if the government renames the incident and it becomes the status quo? What is the source of truth? The "source of truth" should be scientific consensus, not government decrees and policies. The problem is obvious: it takes a lot of resources to determine what the consensus is on hotly debated topics.
  - quanticle 5y ago
    What is the "scientific consensus" on the name of a city? When the government of India decides to rename Calcutta to Kolkata, then has the "scientific consensus" on the name of the city shifted?
    - 908B64B197 5y ago
      It's what the inhabitants calls it. Calcutta, Pondicherry and Pekin were "best attempts" by Europeans to transcribe the sounds from the locals that were relayed to them by illiterate sailors.
      - alisonkisk 5y ago
        Which inhabitants?
      - quanticle 5y ago
        The inhabitants of Calcutta called it "Calcutta" prior to the Indian government renaming it. Now they call it Kolkata. The people of Madras called it Madras before the government renamed it to Chennai. Now they call it Chennai. I think it's incorrect to pretend that there's some kind of "objective" scientific reality at play here. Naming geographic locations is an inherently political act, and choosing which of several names to call a place inherently means favoring some interpretations over others.
  - lwouis 5y ago
    See my sibling comment. You don't need a consensus. You can list multiple perspectives and have them coexist. That's how we have to do with religion for example. There can't be a consensus there clearly, so we share all perspectives. Facts of life such as what's on Wikipedia should be edited as well as narrated by a variety of sources so that it represents the diversity of humankind
deleted 5y ago
[deleted]
kensai 5y ago
The beauty of Wikipedia is exactly in its plurality. It's perfect in its (contributors') imperfection. As long as the standards of quoting and referencing are kept high, let these minor interpretation differences spark attention so we can promote discussion and interaction. One Wikipedia to rule them all will only promote the Anglo- narrative (or whichever lingua franca rules at a certain point). Be very careful what you are wishing for, it might very well be Chinese next.
- quanticle 5y ago
  Exactly. I take issue with the article's implicit assertion that English Wikipedia is somehow less subject to manipulation than Wikipedias in other languages. As Ryan Holliday pointed out all the way back in 2012, in his book Trust Me I'm Lying, the PR and reputation management industries love Wikipedia. They know that reporters are lazy, so in advance of an interview they'll go to Wikipedia for the background information, and that information sets the framing that they use for their narrative. If Wikipedia describes someone as a successful director, they'll get questions about movies they've directed, which actors they enjoyed working with most, etc. That same person described as a failed actor, will get questions about their acting career and why it never seemed to take off. Same person, same accomplishments, two completely different narratives, driven entirely by which facts are more prominent in the Wikipedia article.
  - TchoBeer 5y ago
    Clearly they should be described as a "failed actor turned successful director" and anything else is leaving out information.
sleavey 5y ago
This article talks about non-English Wikipedia sites, but it's worth noting that the co-founder of Wikipedia (the other one, not Jimmy Wales) no longer trusts the English site either. [1] https://www.youtube.com/watch?v=l0P4Cf0UCwU https://www.youtube.com/watch?v=l0P4Cf0UCwU
Peritract 5y ago
The article doesn't describe a language problem, but an abuse one - Wikipedia does not have effective tools to deal with misinformation for topics/pages with a small readership/number of editors. Machine translation doesn't fix that; it changes the nature of the bias and which targets count as niche.
tablespoon 5y ago
> But what if we could ensure that everyone across the globe saw the same page? What if we could create one universal Wikipedia — one shared, authoritative volume of pages that users from around the world could all read and edit in the language of their choice? > This would be a technological feat, but probably not an impossible one. Machine translation has been quickly improving in recent years and has become a part of everyday life in many non-English-speaking countries. Some Japanese users, rather than read the Japanese version of Wikipedia, choose to translate the English Wikipedia using Google Translate because they know the English version will be more comprehensive and less biased. As translation technology continues to improve, it’s possible to imagine people from all over the world will want to do the same. IIRC, Wikipedia used to have a machine translation utility, but they canned it because the results were so bad, and actually had a project to clean up the mess. I think the issue is that machine translation is usually pretty good, but also frequently introduces significant errors (e.g. missing or misapplying a negation, which totally inverts the meaning of a sentence).
deleted 5y ago
[deleted]
crazygringo 5y ago
I've often wondered if it would be not just possible, but user-friendly, to write in a way that guarantees machine translation is 100% accurate. Specifically, imagine you write a few sentences in English. Then when you hit "publish", it would look for the potentially amibiguous parts of your sentences, and then give you "nearby synonyms" for you to select which one you meant. It wouldn't change the text in your source language, but it would remove the ambiguity so all the output translations would be correct, no guessing. It would do things like ask which noun "it" referred to since some languages have male/female versions, ask for clarification of verb tenses as necessary, etc. After all, the hardest part of machine translation is inferring meaning. If that part could be guaranteed to work accurately, it seems like machine translation could become a ton more useful.
- yorwba 5y ago
  Abstract Wikipedia is kind of trying to do this, but using structured data instead of English. https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia
- a1369209993 5y ago
  > not just possible, but user-friendly, to write in a way that guarantees machine translation is 100% accurate. Not in (human-readable source) English, no, since it's too heavily[0] optimised for general-purpose colloquial communication, but it's probably possible to engineer some cross between text localization systems, procedural content generation for text games, and early-AI-style natural language processing data structures that could serve as a backing representation subject to linting/sanity-checking/etc. Edit: 0: To be clear, "heavily" is not necessarily the same as "well-".
Decabytes 5y ago
I know Esperanto isn't the greatest language in the world. It has its problems like other languages. But I feel like this is a good use case for it. I've heard of Esperanto being a good intermediate language to translate things into due to it being much more regular of a language. There have even been investigations in using it in Machine learning. It would be easier for someone to pick up, and translate to from their native language compared to other languages, even in cases where your native language had not cognates with Esperanto. Then since its grammar is regular it could be translated into another language with less effort. A benefit of using Esperanto as opposed to inventing a new language is that it does have some traction. There are websites and apps that teach it. It's on google translate, mozilla common voice etc. It is not completely unfamiliar, and it is not tied to any one nation. This simultaneously avoids the social issues that would inevitably arise if English were used, while also building off of an already established (and over 100 year old language). It also has support from countries who have had difficult relation ships with the U.S.A for example Radio China has a 1 hour broadcast in Esperanto, Cuba has a broadcast, and other countries as well! I've been learning Esperanto for a couple months and it has been fun. While the community is small it is definitely enthusiastic. It's #35 on wikipedia with over 300k articles, has both a normal, and meme(memeo) subreddit, and a large Telegram group (over 1k members). 1. https://www.proquest.com/openview/dfaf840b0267d1bb8a38d9d7195d9a88/1?pq-origsite=gscholar&cbl=2026366&diss=y https://www.proquest.com/openview/dfaf840b0267d1bb8a38d9d719... 2. https://en.wikipedia.org/wiki/Distributed_Language_Translation https://en.wikipedia.org/wiki/Distributed_Language_Translati... 3. http://esperanto.cri.cn/2521/more/2622/more2622.htm http://esperanto.cri.cn/2521/more/2622/more2622.htm 4. https://esperanto.net/en/esperanto-broadcasts/ https://esperanto.net/en/esperanto-broadcasts/ 5. https://en.wikipedia.org/wiki/List_of_Wikipedias https://en.wikipedia.org/wiki/List_of_Wikipedias 6. https://www.reddit.com/r/Esperanto/ https://www.reddit.com/r/Esperanto/
aaron695 5y ago
> English Wikipedia is considered by many researchers to be almost, but still not quite as accurate as traditional encyclopedias. This is garbage. No one's claimed this for years. The citations are all over 10 years old. What is fun and productive is reading Wikipedia in the language relevant to the article. So read about German beers on the German Wiki as well as English.
eynsham 5y ago
I am not sure how this would be implemented except by machine translation (probably from English). There are all sorts of problems with this—and since machine translation is already possible, switching to an English and translated Wiki would only really be justifiable if the disadvantages were very few. 1. Machine translation is highly uneven. The official status of certain languages means we have a large corpus of material from international organisations and multilingual governments. But there are still lots of languages where the corpus is much smaller—even if there are lots of users who contribute (e.g., Vietnamese.) The quality of these Wikipedias would decline substantially. 2. A few Wikipedia articles are a joy to read—they are idiomatic, easy to understand, and well-written. Machine translation simply cannot do this properly. This would be lost in every language but English. 3. What you want in an article probably differs by language. Frames of reference differ: one can probably expect e.g. certain pop culture analogies not to require explanation (or perhaps just a link) in some languages but to require fuller explanation in others. 4. The use of non-English sources would require much more work. At the moment, a French-language source may appear on French Wikipedia without any translation work. Now, it will only appear if a bilingual editor happens to find it and can be bothered to translate it. The pattern of course is general, and would be most undesirable. 5. There exist many perfectly good Wikipedia editors who do not know how to write in English, or would lack the confidence to do so. Are we to lose all their contributions? If not, we’d require legions of bilingual editors to verify machine translations from non-English languages. Experience suggests that there do not exist many such editors.
yongjik 5y ago
It's not feasible to create a multi-language Wikipedia without introducing a lot of inaccuracies and/or significant complication, because different languages don't necessarily divide words in the same way. A famous example is the word "blue" - the corresponding word can also include the green color in East Asian languages, while in Russian there are two different words for "light blue" and "dark blue" (or so I've heard). What would a single entry for "blue" look like? As another example, the English page for "goose" obviously explains both domestic and wild geese, because they're the same animal... in English. In Korean, domestic (geowi) and wild (gireogi) geese are different animals, and I don't think people even know they're related. (When I learned that they were the same in English, I was like "Huh, why do English speakers have one word for these different birds... oh wait.") Even in physics: Korean has words jajeon(자전)/gongjeon(공전) which mean "a stellar body's rotation about itself" and "a stellar body orbiting around another stellar body" - these are elementary-school-level terms. There are no corresponding entries in the English Wikipedia, because they're simply called "rotation" (or maybe "orbit" which is related but not exactly the same meaning). As for exposition, the English page for "poetry" is a tome with examples from Ancient Greece, China, and Shakespeare, but only a single passing mention of any Korean - and none from Sweden. You could meticulously translate the whole article into Korean or Swedish, and its readers will still find it wanting. And this is before we consider all the edit wars, POVs, and politics ...
- deleted 5y ago
  [deleted]
- zozbot234 5y ago
  > It's not feasible to create a multi-language Wikipedia without introducing a lot of inaccuracies and/or significant complication, because different languages don't necessarily divide words in the same way. This is obviously a concern, but there are ways to address it as described in https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia . Of course disambiguating all of these things would involve a non-trivial amount of work, but the expectation is that the result would at least be accurate and maintainable.
- jhanschoo 5y ago

34 points by _Microft 5y ago

London Breed is a nazi. I hope she gets her ass fucked up on the street. Fucking cunt.

My impression was that wikidata was the main answer to this problem by centralizing all the facts in a central database to avoid each page having its own data and sources, but I can see how there is much more than the data at stake.

tablespoon 5y ago
> My impression was that wikidata was the main answer to this problem by centralizing all the facts in a central database to avoid each page having its own data and sources, but I can see how there is much more than the data at stake. IIRC, Wikidata doesn't centralize "all the facts," it just centralizes the basic infobox data, which is only a small part of any article
- zaik 5y ago
  Wikidata can express more than just basic infobox data. That being said it's also a lot less expressive than natural language. I think "Abstract Wikipedia" tries to overcome this shortcoming, I wonder how it will turn out. It feels like an impossible feat.

rootsudo 5y ago

Disagree entirely on Japanese translating the English wiki - that is too many steps and most people do not do that. Also within Japan, SEO and such always put Japanese content first (especially on google.) vs en.wikipedia.org. The article is touching heavy on Japanese and if you do not have a basic fluency of the langauge, it's easy to jump to conclusion... Alongside, same with the renaming of Nanjing - what if the government renames the incident and it becomes the status quo? What is the source of truth? Machine translating one "wikipedia" is also wrong, because context is removed, and having English be the language of truth and all other languages translate from it - you'll miss out on many things, especially words that do not have a specific English translation. The article is poorly written and documented.

mschuster91 5y ago
> Alongside, same with the renaming of Nanjing - what if the government renames the incident and it becomes the status quo? What is the source of truth? The "source of truth" should be scientific consensus, not government decrees and policies. The problem is obvious: it takes a lot of resources to determine what the consensus is on hotly debated topics.
- quanticle 5y ago
  What is the "scientific consensus" on the name of a city? When the government of India decides to rename Calcutta to Kolkata, then has the "scientific consensus" on the name of the city shifted?
  - 908B64B197 5y ago
    It's what the inhabitants calls it. Calcutta, Pondicherry and Pekin were "best attempts" by Europeans to transcribe the sounds from the locals that were relayed to them by illiterate sailors.
    - alisonkisk 5y ago
      Which inhabitants?
    - quanticle 5y ago
      The inhabitants of Calcutta called it "Calcutta" prior to the Indian government renaming it. Now they call it Kolkata. The people of Madras called it Madras before the government renamed it to Chennai. Now they call it Chennai. I think it's incorrect to pretend that there's some kind of "objective" scientific reality at play here. Naming geographic locations is an inherently political act, and choosing which of several names to call a place inherently means favoring some interpretations over others.
- lwouis 5y ago
  See my sibling comment. You don't need a consensus. You can list multiple perspectives and have them coexist. That's how we have to do with religion for example. There can't be a consensus there clearly, so we share all perspectives. Facts of life such as what's on Wikipedia should be edited as well as narrated by a variety of sources so that it represents the diversity of humankind

deleted 5y ago

[deleted]

kensai 5y ago

The beauty of Wikipedia is exactly in its plurality. It's perfect in its (contributors') imperfection. As long as the standards of quoting and referencing are kept high, let these minor interpretation differences spark attention so we can promote discussion and interaction. One Wikipedia to rule them all will only promote the Anglo- narrative (or whichever lingua franca rules at a certain point). Be very careful what you are wishing for, it might very well be Chinese next.

quanticle 5y ago
Exactly. I take issue with the article's implicit assertion that English Wikipedia is somehow less subject to manipulation than Wikipedias in other languages. As Ryan Holliday pointed out all the way back in 2012, in his book Trust Me I'm Lying, the PR and reputation management industries love Wikipedia. They know that reporters are lazy, so in advance of an interview they'll go to Wikipedia for the background information, and that information sets the framing that they use for their narrative. If Wikipedia describes someone as a successful director, they'll get questions about movies they've directed, which actors they enjoyed working with most, etc. That same person described as a failed actor, will get questions about their acting career and why it never seemed to take off. Same person, same accomplishments, two completely different narratives, driven entirely by which facts are more prominent in the Wikipedia article.
- TchoBeer 5y ago
  Clearly they should be described as a "failed actor turned successful director" and anything else is leaving out information.

sleavey 5y ago

This article talks about non-English Wikipedia sites, but it's worth noting that the co-founder of Wikipedia (the other one, not Jimmy Wales) no longer trusts the English site either. [1] https://www.youtube.com/watch?v=l0P4Cf0UCwU https://www.youtube.com/watch?v=l0P4Cf0UCwU

Peritract 5y ago

The article doesn't describe a language problem, but an abuse one - Wikipedia does not have effective tools to deal with misinformation for topics/pages with a small readership/number of editors. Machine translation doesn't fix that; it changes the nature of the bias and which targets count as niche.

tablespoon 5y ago

> But what if we could ensure that everyone across the globe saw the same page? What if we could create one universal Wikipedia — one shared, authoritative volume of pages that users from around the world could all read and edit in the language of their choice? > This would be a technological feat, but probably not an impossible one. Machine translation has been quickly improving in recent years and has become a part of everyday life in many non-English-speaking countries. Some Japanese users, rather than read the Japanese version of Wikipedia, choose to translate the English Wikipedia using Google Translate because they know the English version will be more comprehensive and less biased. As translation technology continues to improve, it’s possible to imagine people from all over the world will want to do the same. IIRC, Wikipedia used to have a machine translation utility, but they canned it because the results were so bad, and actually had a project to clean up the mess. I think the issue is that machine translation is usually pretty good, but also frequently introduces significant errors (e.g. missing or misapplying a negation, which totally inverts the meaning of a sentence).

deleted 5y ago

[deleted]

crazygringo 5y ago

I've often wondered if it would be not just possible, but user-friendly, to write in a way that guarantees machine translation is 100% accurate. Specifically, imagine you write a few sentences in English. Then when you hit "publish", it would look for the potentially amibiguous parts of your sentences, and then give you "nearby synonyms" for you to select which one you meant. It wouldn't change the text in your source language, but it would remove the ambiguity so all the output translations would be correct, no guessing. It would do things like ask which noun "it" referred to since some languages have male/female versions, ask for clarification of verb tenses as necessary, etc. After all, the hardest part of machine translation is inferring meaning. If that part could be guaranteed to work accurately, it seems like machine translation could become a ton more useful.

yorwba 5y ago
Abstract Wikipedia is kind of trying to do this, but using structured data instead of English. https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia
a1369209993 5y ago
> not just possible, but user-friendly, to write in a way that guarantees machine translation is 100% accurate. Not in (human-readable source) English, no, since it's too heavily[0] optimised for general-purpose colloquial communication, but it's probably possible to engineer some cross between text localization systems, procedural content generation for text games, and early-AI-style natural language processing data structures that could serve as a backing representation subject to linting/sanity-checking/etc. Edit: 0: To be clear, "heavily" is not necessarily the same as "well-".

Decabytes 5y ago

I know Esperanto isn't the greatest language in the world. It has its problems like other languages. But I feel like this is a good use case for it. I've heard of Esperanto being a good intermediate language to translate things into due to it being much more regular of a language. There have even been investigations in using it in Machine learning. It would be easier for someone to pick up, and translate to from their native language compared to other languages, even in cases where your native language had not cognates with Esperanto. Then since its grammar is regular it could be translated into another language with less effort. A benefit of using Esperanto as opposed to inventing a new language is that it does have some traction. There are websites and apps that teach it. It's on google translate, mozilla common voice etc. It is not completely unfamiliar, and it is not tied to any one nation. This simultaneously avoids the social issues that would inevitably arise if English were used, while also building off of an already established (and over 100 year old language). It also has support from countries who have had difficult relation ships with the U.S.A for example Radio China has a 1 hour broadcast in Esperanto, Cuba has a broadcast, and other countries as well! I've been learning Esperanto for a couple months and it has been fun. While the community is small it is definitely enthusiastic. It's #35 on wikipedia with over 300k articles, has both a normal, and meme(memeo) subreddit, and a large Telegram group (over 1k members). 1. https://www.proquest.com/openview/dfaf840b0267d1bb8a38d9d7195d9a88/1?pq-origsite=gscholar&cbl=2026366&diss=y https://www.proquest.com/openview/dfaf840b0267d1bb8a38d9d719... 2. https://en.wikipedia.org/wiki/Distributed_Language_Translation https://en.wikipedia.org/wiki/Distributed_Language_Translati... 3. http://esperanto.cri.cn/2521/more/2622/more2622.htm http://esperanto.cri.cn/2521/more/2622/more2622.htm 4. https://esperanto.net/en/esperanto-broadcasts/ https://esperanto.net/en/esperanto-broadcasts/ 5. https://en.wikipedia.org/wiki/List_of_Wikipedias https://en.wikipedia.org/wiki/List_of_Wikipedias 6. https://www.reddit.com/r/Esperanto/ https://www.reddit.com/r/Esperanto/

aaron695 5y ago

> English Wikipedia is considered by many researchers to be almost, but still not quite as accurate as traditional encyclopedias. This is garbage. No one's claimed this for years. The citations are all over 10 years old. What is fun and productive is reading Wikipedia in the language relevant to the article. So read about German beers on the German Wiki as well as English.

eynsham 5y ago

I am not sure how this would be implemented except by machine translation (probably from English). There are all sorts of problems with this—and since machine translation is already possible, switching to an English and translated Wiki would only really be justifiable if the disadvantages were very few. 1. Machine translation is highly uneven. The official status of certain languages means we have a large corpus of material from international organisations and multilingual governments. But there are still lots of languages where the corpus is much smaller—even if there are lots of users who contribute (e.g., Vietnamese.) The quality of these Wikipedias would decline substantially. 2. A few Wikipedia articles are a joy to read—they are idiomatic, easy to understand, and well-written. Machine translation simply cannot do this properly. This would be lost in every language but English. 3. What you want in an article probably differs by language. Frames of reference differ: one can probably expect e.g. certain pop culture analogies not to require explanation (or perhaps just a link) in some languages but to require fuller explanation in others. 4. The use of non-English sources would require much more work. At the moment, a French-language source may appear on French Wikipedia without any translation work. Now, it will only appear if a bilingual editor happens to find it and can be bothered to translate it. The pattern of course is general, and would be most undesirable. 5. There exist many perfectly good Wikipedia editors who do not know how to write in English, or would lack the confidence to do so. Are we to lose all their contributions? If not, we’d require legions of bilingual editors to verify machine translations from non-English languages. Experience suggests that there do not exist many such editors.

yongjik 5y ago

It's not feasible to create a multi-language Wikipedia without introducing a lot of inaccuracies and/or significant complication, because different languages don't necessarily divide words in the same way. A famous example is the word "blue" - the corresponding word can also include the green color in East Asian languages, while in Russian there are two different words for "light blue" and "dark blue" (or so I've heard). What would a single entry for "blue" look like? As another example, the English page for "goose" obviously explains both domestic and wild geese, because they're the same animal... in English. In Korean, domestic (geowi) and wild (gireogi) geese are different animals, and I don't think people even know they're related. (When I learned that they were the same in English, I was like "Huh, why do English speakers have one word for these different birds... oh wait.") Even in physics: Korean has words jajeon(자전)/gongjeon(공전) which mean "a stellar body's rotation about itself" and "a stellar body orbiting around another stellar body" - these are elementary-school-level terms. There are no corresponding entries in the English Wikipedia, because they're simply called "rotation" (or maybe "orbit" which is related but not exactly the same meaning). As for exposition, the English page for "poetry" is a tome with examples from Ancient Greece, China, and Shakespeare, but only a single passing mention of any Korean - and none from Sweden. You could meticulously translate the whole article into Korean or Swedish, and its readers will still find it wanting. And this is before we consider all the edit wars, POVs, and politics ...

deleted 5y ago
[deleted]
zozbot234 5y ago
> It's not feasible to create a multi-language Wikipedia without introducing a lot of inaccuracies and/or significant complication, because different languages don't necessarily divide words in the same way. This is obviously a concern, but there are ways to address it as described in https://meta.wikimedia.org/wiki/Abstract_Wikipedia https://meta.wikimedia.org/wiki/Abstract_Wikipedia . Of course disambiguating all of these things would involve a non-trivial amount of work, but the expectation is that the result would at least be accurate and maintainable.
jhanschoo 5y ago