4 ms·

Slightly off-topic, but figured this might be the best place to connect with like minded people: I’ve been working on NLP tooling for low resource languages, an

Slightly off-topic, but figured this might be the best place to connect with like minded people: I’ve been working on NLP tooling for low resource languages, and had quite the success developing an LLM that speaks Moroccan natively [0], started from a super tiny dataset I manually annotated and scaling it to find anything written in Moroccan on the whole web. Currently helping out a research group to extend the same approaches from my project to Indic languages and writing scripts. If anyone here is working on NLP for low resource languages, hit me up! I’d love to exchange ideas, techniques and possibly to collaborate. [0]: https://sawalni.com https://sawalni.com

vintermann 3y ago
That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandinavian languages and in particular the two written forms of Norwegian. I had the same problem of lacking training data, but I tried an approach of bootstrapping from rule-based translators. There exist decent rule-based translators for the Scandinavian languages. Of course they make mistakes, but I figured that as long as I used the rule-translated text only as source, never as target, it wouldn't matter. Effectively, instead of learning to translate from real Danish to correct Norwegian, it learned to translate from slightly broken machine-Danish to correct Norwegian. And my hope was that it would learn to translate from real Danish to correct Norwegian as a side effect. I then figured that I might as well let it take input in slightly broken machine-Swedish as well, hopefully helping its generalization. Then I could use this translator for the next step, again using the real text as the target, but using output from this translator as input. It worked reasonably well. It certainly beat Google Translate at the time, since that at the time went via English even for closely related languages like the Scandinavian ones. I seriously considered making a start-up and commercializing it, some people had expressed interest (among them newspapers - they used a rule-based translator/grammar checker at the time, I think maybe they still do.) But in the end, it's probably good I didn't. These days I understand they don't even train sequence to sequence models, they just train a general-purpose language model and instruction-tune it to do translation.
- omneity 3y ago
  Thank you for sharing your experience! It's awesome to hear what worked, and I see many parallels in our work as well (a manual rule system was key for me to unlock seq2seq and then the rest of the ML chain). By the way, Seq2seq models are still super useful, I've trained and I'm using many in the data pipeline of this project. LLMs are a nuclear hammer and sometimes you just have a teeny tiny nail. There's also limited commercial value to what I'm working on, but it's a work of passion and I want to be able to speak in my native language to voice assistants (among other objectives). How did you communicate about it at the time and gather newspaper interest, if you don't mind sharing your experience?
  - vintermann 3y ago
    I went to a machine learning conference in Tromsø and talked with many people there, and I met the newspaper people at a local business fair in my hometown. I had a little android app to demo it, but the backend was too expensive to have up... optimizing the use of cloud resources was a can of worms I didn't even get started on!
    - omneity 3y ago
      Thanks again for sharing, that was super helpful. And yeah I agree it is indeed a massive endeavor, I'm sorry it didn't pan out the way you wanted it to. Are you still interested in this space, and do you write about your experiences anywhere? I'd love to connect if you want to.
      - vintermann 3y ago
        Sorry it took a little time to get back to you. Yes, I wrote a few articles, on linkedin of all places, but they didn't say much more than what I've said here, as I recall. You can contact me on vintermann at gmail if you want.
OfSanguineFire 3y ago
Why are you focusing on “Moroccan” (IMO a problematic term, considering the country’s language diversity), instead of on Maghrebi Arabic in general?
- omneity 3y ago
  > Why are you focusing on “Moroccan”, instead of on Maghrebi Arabic in general? There are quite a few reasons. - I am a Moroccan myself, I want an AI I can speak to the way I speak with friends. And I also want other people who can only speak my language to be able to make use of AI technology. - Constraints wise, I'm working solo on this with no funding. Can't do it all. - Maghrebi Arabic is not a language, rather a linguistic construct to try to simplify the complex web of languages spoken in North Africa that have arabic influences. On the ground, there is no such thing as Maghrebi Arabic. - My personal expertise does not lie in these other languages. I do not have the necessary background to ensure the quality of the outcome, nor the relevance of the language used. - While there might be some level of mutual intelligibility between Moroccan and Algerian for example, mostly when _speaking_, different languages in this group have very different rules, vocabularies, and most importantly different writing habits. Some of the simpler examples: * Code switching between arabic and latin scripts, which follows different conventions in each country (region even). It's already tough to cover Moroccan. * Another difference is reminiscent of the relationship between English and French. Many common words in French are rarely used in English or could mean something different entirely, and that's the same case here. * Moroccan is significantly influenced by Tamazight and Spanish, which does not hold the same way for other north African countries. Ultimately, keep in mind that NA is not a single cultural block. Not being exhaustive, but Morocco has some Andalusian components to its culture, Algeria has some Ottoman, Tunisia and Lybia were very close to Italy at some point, and it gets crazier when you zoom in closer. > (IMO a problematic term, considering the country’s language diversity) Is there a debate I am not familiar with? I am Moroccan. To whom is this a problematic term?

vintermann 3y ago

That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandinavian languages and in particular the two written forms of Norwegian. I had the same problem of lacking training data, but I tried an approach of bootstrapping from rule-based translators. There exist decent rule-based translators for the Scandinavian languages. Of course they make mistakes, but I figured that as long as I used the rule-translated text only as source, never as target, it wouldn't matter. Effectively, instead of learning to translate from real Danish to correct Norwegian, it learned to translate from slightly broken machine-Danish to correct Norwegian. And my hope was that it would learn to translate from real Danish to correct Norwegian as a side effect. I then figured that I might as well let it take input in slightly broken machine-Swedish as well, hopefully helping its generalization. Then I could use this translator for the next step, again using the real text as the target, but using output from this translator as input. It worked reasonably well. It certainly beat Google Translate at the time, since that at the time went via English even for closely related languages like the Scandinavian ones. I seriously considered making a start-up and commercializing it, some people had expressed interest (among them newspapers - they used a rule-based translator/grammar checker at the time, I think maybe they still do.) But in the end, it's probably good I didn't. These days I understand they don't even train sequence to sequence models, they just train a general-purpose language model and instruction-tune it to do translation.

omneity 3y ago
Thank you for sharing your experience! It's awesome to hear what worked, and I see many parallels in our work as well (a manual rule system was key for me to unlock seq2seq and then the rest of the ML chain). By the way, Seq2seq models are still super useful, I've trained and I'm using many in the data pipeline of this project. LLMs are a nuclear hammer and sometimes you just have a teeny tiny nail. There's also limited commercial value to what I'm working on, but it's a work of passion and I want to be able to speak in my native language to voice assistants (among other objectives). How did you communicate about it at the time and gather newspaper interest, if you don't mind sharing your experience?
- vintermann 3y ago
  I went to a machine learning conference in Tromsø and talked with many people there, and I met the newspaper people at a local business fair in my hometown. I had a little android app to demo it, but the backend was too expensive to have up... optimizing the use of cloud resources was a can of worms I didn't even get started on!
  - omneity 3y ago
    Thanks again for sharing, that was super helpful. And yeah I agree it is indeed a massive endeavor, I'm sorry it didn't pan out the way you wanted it to. Are you still interested in this space, and do you write about your experiences anywhere? I'd love to connect if you want to.
    - vintermann 3y ago
      Sorry it took a little time to get back to you. Yes, I wrote a few articles, on linkedin of all places, but they didn't say much more than what I've said here, as I recall. You can contact me on vintermann at gmail if you want.

OfSanguineFire 3y ago

Why are you focusing on “Moroccan” (IMO a problematic term, considering the country’s language diversity), instead of on Maghrebi Arabic in general?

omneity 3y ago
> Why are you focusing on “Moroccan”, instead of on Maghrebi Arabic in general? There are quite a few reasons. - I am a Moroccan myself, I want an AI I can speak to the way I speak with friends. And I also want other people who can only speak my language to be able to make use of AI technology. - Constraints wise, I'm working solo on this with no funding. Can't do it all. - Maghrebi Arabic is not a language, rather a linguistic construct to try to simplify the complex web of languages spoken in North Africa that have arabic influences. On the ground, there is no such thing as Maghrebi Arabic. - My personal expertise does not lie in these other languages. I do not have the necessary background to ensure the quality of the outcome, nor the relevance of the language used. - While there might be some level of mutual intelligibility between Moroccan and Algerian for example, mostly when _speaking_, different languages in this group have very different rules, vocabularies, and most importantly different writing habits. Some of the simpler examples: * Code switching between arabic and latin scripts, which follows different conventions in each country (region even). It's already tough to cover Moroccan. * Another difference is reminiscent of the relationship between English and French. Many common words in French are rarely used in English or could mean something different entirely, and that's the same case here. * Moroccan is significantly influenced by Tamazight and Spanish, which does not hold the same way for other north African countries. Ultimately, keep in mind that NA is not a single cultural block. Not being exhaustive, but Morocco has some Andalusian components to its culture, Algeria has some Ottoman, Tunisia and Lybia were very close to Italy at some point, and it gets crazier when you zoom in closer. > (IMO a problematic term, considering the country’s language diversity) Is there a debate I am not familiar with? I am Moroccan. To whom is this a problematic term?