4 ms·

I'll be looking at this in detail. I've started a company to do similar things, https://6k.ai https://6k.ai I'm currently concentrating on better data gatherin

by ks2048 7mo ago

I'll be looking at this in detail. I've started a company to do similar things, https://6k.ai https://6k.ai I'm currently concentrating on better data gathering for low-resource languages. When you look in detail at data like Common Crawl, finepdfs, and fineweb, (1) they are really lacking quality data sources if you know where to look, and (2) the sources they have are not processed "finely" enough (e.g. finepdfs classify each page of PDF as having a specific language, where-as many language learning sources have language pairs, etc.

ccgreg 7mo ago
Common Crawl has been running a low-resource language project for 1.5 years now -- it's a hard problem.
intended 7mo ago
There’s many nation states working on this, have you looked into availability of those data sets? What languages are you prioritizing?
- ks2048 7mo ago
  Yes, there are government datasets, languge "acadamies" (or "regulators") - organizations focused on preserving / teaching the language, and often smaller, local publishers that publish material in their local language. I'm living in Guatemala, so have been focusing on the Mayan languages here (22 languages, millions of speakers).
  - dhosek 7mo ago
    As an aside, I remember visiting Guatemala (in the border area near Chiapas) in the early 90s and discovering that “Mayan” was not the monolith that I had been led to believe by my culturally narrow American education, but was a diverse collection of related cultures with multiple languages. In one of the villages we visited, there was a language school where foreigners were learning Jacalteco. One student was from Israel and where most of the students had vocabulary lists in three columns (Jacalteco - Spanish - English), his had four columns where he did one more step of translation to Hebrew.
omneity 7mo ago
Hey, this is super cool! I’ve been working on a similar problem, focusing on low-resource and underserved languages including the Mayan family, and have published some research and open resources around that [0, 1]. On the data side, I’ve found that the biggest bottleneck isn’t collecting text (it’s out there!) but reliable language identification. It’s often difficult or ambiguous to separate languages cleanly in datasets like Common Crawl, Fineweb, or others. I worked on improving this a bit for Fineweb 2 for my native language, that might inspire you [3]. Many of the challenges you mention seem to recur across regions and language families, so I’d love to connect and compare notes sometime. Feel free to reach me at omar [at] the labs site below. 0: https://wikilangs.org https://wikilangs.org 1: https://omneitylabs.com https://omneitylabs.com 2: https://huggingface.co/blog/omarkamali/gherbal-multilingual-fineweb-moroccan-arabic https://huggingface.co/blog/omarkamali/gherbal-multilingual-...
- mandeepj 7mo ago
  You both might find it useful - https://news.ycombinator.com/item?id=44950661 https://news.ycombinator.com/item?id=44950661 I’ve also recently started in this space: building an agent, for a client, who can communicate in multiple languages.
  - omneity 7mo ago
    Excellent, thank you mandeepj! Curious about the language coverage of your agent and if / how you plan to eval your agent, if you're willing to share more.
    - mandeepj 7mo ago
      Regarding language coverage, we will start with the most frequently spoken languages first. evaluating your agent: we are documenting the details, but it should give you some idea about an approach https://news.ycombinator.com/item?id=47232903 https://news.ycombinator.com/item?id=47232903 Also, you might find this useful - https://open.substack.com/pub/bytebytego/p/how-roblox-uses-ai-to-translate-16?utm_campaign=post&utm_medium=email https://open.substack.com/pub/bytebytego/p/how-roblox-uses-a...
quantumwoke 7mo ago
It's sad that I didn't see any languages on your website from Australia, where there are hundreds of languages that need translating.
- ks2048 7mo ago
  It’s a small sample and not specifically ones we’re working on. It’s biased towards alternative scripts for visual interest. Australian languages are definitely interesting! and I will say, from what I’ve seen, Australian government (and other orgs) have done better than most to help document them (in recent years, at least)

by ks2048 7mo ago

ccgreg 7mo ago

Common Crawl has been running a low-resource language project for 1.5 years now -- it's a hard problem.

intended 7mo ago

There’s many nation states working on this, have you looked into availability of those data sets? What languages are you prioritizing?

ks2048 7mo ago
Yes, there are government datasets, languge "acadamies" (or "regulators") - organizations focused on preserving / teaching the language, and often smaller, local publishers that publish material in their local language. I'm living in Guatemala, so have been focusing on the Mayan languages here (22 languages, millions of speakers).
- dhosek 7mo ago
  As an aside, I remember visiting Guatemala (in the border area near Chiapas) in the early 90s and discovering that “Mayan” was not the monolith that I had been led to believe by my culturally narrow American education, but was a diverse collection of related cultures with multiple languages. In one of the villages we visited, there was a language school where foreigners were learning Jacalteco. One student was from Israel and where most of the students had vocabulary lists in three columns (Jacalteco - Spanish - English), his had four columns where he did one more step of translation to Hebrew.

omneity 7mo ago

Hey, this is super cool! I’ve been working on a similar problem, focusing on low-resource and underserved languages including the Mayan family, and have published some research and open resources around that [0, 1]. On the data side, I’ve found that the biggest bottleneck isn’t collecting text (it’s out there!) but reliable language identification. It’s often difficult or ambiguous to separate languages cleanly in datasets like Common Crawl, Fineweb, or others. I worked on improving this a bit for Fineweb 2 for my native language, that might inspire you [3]. Many of the challenges you mention seem to recur across regions and language families, so I’d love to connect and compare notes sometime. Feel free to reach me at omar [at] the labs site below. 0: https://wikilangs.org https://wikilangs.org 1: https://omneitylabs.com https://omneitylabs.com 2: https://huggingface.co/blog/omarkamali/gherbal-multilingual-fineweb-moroccan-arabic https://huggingface.co/blog/omarkamali/gherbal-multilingual-...

mandeepj 7mo ago
You both might find it useful - https://news.ycombinator.com/item?id=44950661 https://news.ycombinator.com/item?id=44950661 I’ve also recently started in this space: building an agent, for a client, who can communicate in multiple languages.
- omneity 7mo ago
  Excellent, thank you mandeepj! Curious about the language coverage of your agent and if / how you plan to eval your agent, if you're willing to share more.
  - mandeepj 7mo ago
    Regarding language coverage, we will start with the most frequently spoken languages first. evaluating your agent: we are documenting the details, but it should give you some idea about an approach https://news.ycombinator.com/item?id=47232903 https://news.ycombinator.com/item?id=47232903 Also, you might find this useful - https://open.substack.com/pub/bytebytego/p/how-roblox-uses-ai-to-translate-16?utm_campaign=post&utm_medium=email https://open.substack.com/pub/bytebytego/p/how-roblox-uses-a...

quantumwoke 7mo ago

It's sad that I didn't see any languages on your website from Australia, where there are hundreds of languages that need translating.

ks2048 7mo ago
It’s a small sample and not specifically ones we’re working on. It’s biased towards alternative scripts for visual interest. Australian languages are definitely interesting! and I will say, from what I’ve seen, Australian government (and other orgs) have done better than most to help document them (in recent years, at least)