4 ms·

Alternate take, it can also help people learn niche languages if native speakers contribute to data sets. For example, I've been using Clozemaster [1] for the p

Alternate take, it can also help people learn niche languages if native speakers contribute to data sets. For example, I've been using Clozemaster [1] for the past few months as a way to work on vocabulary in some languages, and they pull their dataset from Tatoeba [2]. I was very surprised to see that my father's native language, Kabylie, which is admittedly a somewhat niche language, is one of the top languages by sentence contribution in the dataset (over 700k entries, more than French or Spanish or German). I showed him the sentences once and he confirmed that yes, they all seem like what a native speaker would say. Not all of them have translations into other languages of course, and a lot of them are slight variations on each other, but some native speakers are there contributing. It's not currently an option to use in Clozemaster -- I'm guessing the TTS isn't really there -- but I totally could see these as gaps that are easily filled. Same with my wife's native language (Bengali). There are surprisingly few language learning resources for Bangla, even though it's the 7th most spoken language in the world. But there it is in the data set with TTS and the ability for Clozemaster to have ChatGPT "explain" what's going on in the sentence (a very useful feature for new speakers). Anyway, I don't view AI as good or bad, just another tool that we should be intentional about when we cultivate the data sets underlying the tool. [1] https://www.clozemaster.com/ https://www.clozemaster.com/ [2] https://tatoeba.org https://tatoeba.org

yorwba 2y ago
You wouldn't even need Kabyle TTS for something like Clozemaster to support the language, since there are more than 50 000 recordings https://tatoeba.org/en/audio/index/kab https://tatoeba.org/en/audio/index/kab (The reason for these large numbers is that there's a political movement demanding independence or at least autonomy from Algeria, which has latched onto language promotion as a way to further their cause. Similar to Catalan topping the number of recorded hours on Mozilla Common Voice.)
- mtalantikite 2y ago
  Oh yeah, absolutely -- I was still surprised they had gotten the data set to that level. My dad likes to point out an old French map from 1846 that has "Kabilie independante" written on the region [1]. A few years back I was in Bejaia when a lot of the protests were going on, which were both against Bouteflika and also pro-independence. Regardless of where anyone stands on the issue, it's great that people are making that data set available! [1] https://www.loc.gov/resource/g8241e.ct003442/ https://www.loc.gov/resource/g8241e.ct003442/
scoot 2y ago
Unfortunately the Tatoeba data set is fundamentally flawed, as they have assumed that there is only one instance of any particular language, when that is clearly not the case for all languages. For example, Brazilian Portuguese is very different to European Portuguese in pronunciation, idioms, and to varying degree grammar and vocabulary, to the extent that spoken European Portuguese is largely unintelligible to Brazilian Portuguese speakers. Even Brazilian Portuguese has significant regional difference, and there is no "standard" Brazilian, but at least differentiating at the country level would make the dataset somewhat more useful. To top it off, they use national flags to represent languages, for example the UK flag to represent English, despite the spoken examples being overwhelmingly American English, or some mashup for example the Brazilian and Portuguese flags slapped together.
- mtalantikite 2y ago
  Yeah, I don't disagree, there are problems with it for sure. But at least for language learning in most languages, as a beginner you're probably just trying to get yourself to a B1-ish level in your target language before switching over to (easy) native content. The quirks of regional pronunciation and idioms can come later. And having some resources is better than having no resources, particularly when we're talking about languages without much in regards to learning materials. That's where a dataset like Tatoeba could be helpful for training ML models or being used as a base for learning apps like Clozemaster. For a language like Portugese, though, you already have Assimil courses and many levels of Pimsleur, etc in both Brazilian and European dialects to get you going. There looks to even be some FSI courses, if you're coming from English. Again, I do agree that it's weird they have both European and Brazilian Portugese mixed on Tatoeba, whereas for Arabic, where many dialects are also not mutually intelligible, there are different categories. I wonder why that happened.

yorwba 2y ago

You wouldn't even need Kabyle TTS for something like Clozemaster to support the language, since there are more than 50 000 recordings https://tatoeba.org/en/audio/index/kab https://tatoeba.org/en/audio/index/kab (The reason for these large numbers is that there's a political movement demanding independence or at least autonomy from Algeria, which has latched onto language promotion as a way to further their cause. Similar to Catalan topping the number of recorded hours on Mozilla Common Voice.)

mtalantikite 2y ago
Oh yeah, absolutely -- I was still surprised they had gotten the data set to that level. My dad likes to point out an old French map from 1846 that has "Kabilie independante" written on the region [1]. A few years back I was in Bejaia when a lot of the protests were going on, which were both against Bouteflika and also pro-independence. Regardless of where anyone stands on the issue, it's great that people are making that data set available! [1] https://www.loc.gov/resource/g8241e.ct003442/ https://www.loc.gov/resource/g8241e.ct003442/

scoot 2y ago

Unfortunately the Tatoeba data set is fundamentally flawed, as they have assumed that there is only one instance of any particular language, when that is clearly not the case for all languages. For example, Brazilian Portuguese is very different to European Portuguese in pronunciation, idioms, and to varying degree grammar and vocabulary, to the extent that spoken European Portuguese is largely unintelligible to Brazilian Portuguese speakers. Even Brazilian Portuguese has significant regional difference, and there is no "standard" Brazilian, but at least differentiating at the country level would make the dataset somewhat more useful. To top it off, they use national flags to represent languages, for example the UK flag to represent English, despite the spoken examples being overwhelmingly American English, or some mashup for example the Brazilian and Portuguese flags slapped together.

mtalantikite 2y ago
Yeah, I don't disagree, there are problems with it for sure. But at least for language learning in most languages, as a beginner you're probably just trying to get yourself to a B1-ish level in your target language before switching over to (easy) native content. The quirks of regional pronunciation and idioms can come later. And having some resources is better than having no resources, particularly when we're talking about languages without much in regards to learning materials. That's where a dataset like Tatoeba could be helpful for training ML models or being used as a base for learning apps like Clozemaster. For a language like Portugese, though, you already have Assimil courses and many levels of Pimsleur, etc in both Brazilian and European dialects to get you going. There looks to even be some FSI courses, if you're coming from English. Again, I do agree that it's weird they have both European and Brazilian Portugese mixed on Tatoeba, whereas for Arabic, where many dialects are also not mutually intelligible, there are different categories. I wonder why that happened.