4 ms·

Related: I built a translation app[0]* for language pairs that are not traditionally supported by Google Translate or DeepL (Moroccan Arabic with a dozen of oth

Related: I built a translation app[0]* for language pairs that are not traditionally supported by Google Translate or DeepL (Moroccan Arabic with a dozen of other major languages), and also trained a custom translation model for it - a BART encoder/decoder derivative, using data I collected, curated and corrected from scratch, and then I built a continuous training pipeline for it taking people's corrections into account. Happy to answer questions if anyone is interested in building translation models for low-resource languages, without being a GPT wrapper. A great resource for this is Marian-NMT[1] and the Opus & Tatoeba projects (beware of data quality). 0: https://tarjamli.ma https://tarjamli.ma * Unfortunately not functioning right now due to inference costs for the model, but I plan to launch it sometime soon. 1: https://marian-nmt.github.io https://marian-nmt.github.io

philomath868 1y ago
How does the "continuous training pipeline" work? You rebuild the model after every N corrections, with the corrections included in the data?
- omneity 1y ago
  Yes. There's a scoring and filtering pipeline first, whereby I try to automatically check for the quality of the correction using a custom multilingual embedding model, madmon[0] and language identification model, gherbal[1]. Above a certain similarity threshold it goes into the training dataset, below it it's flagged for human review. This is mostly to stave off the trolls or blatant mistakes. For the continuous training itself, yes I simply continue training the model from the last checkpoint (cosine lr scheduler). I am considering doing a full retraining at some point when I collect enough data to compare with this progressive training. Apologies for the poor links, it takes a lot of time to work on this let alone fully document everything. 0: https://api.sawalni.com/docs#tag/Embeddings https://api.sawalni.com/docs#tag/Embeddings 1: https://api.sawalni.com/docs#tag/Language-Identification https://api.sawalni.com/docs#tag/Language-Identification
yorwba 1y ago
I'm curious how large your training corpus is and your process for dealing with data quality issues. Did you proofread everything manually or were you able to automate some parts?
- omneity 1y ago
  I started seeing results as early as 5-10k pairs, but you want something closer to 100k, especially if the language has a lot of variations (aka morphologically rich, agglutinative, or written in a non-standardized way). Manual proof-reading (and data generation) was a big part of it, it's definitely not a glamorous magic process. But as I went through it I could notice patterns and wrote some tools to help. There's a way to leverage LLMs to help with this if your language is supported (my target wasn't at the time), but I still strongly recommend a manual review part. That's really the secret sauce and no way around it if you're serious about the translation quality of your model.
ks2048 1y ago
Any major challenges beyond gathering high-quality sentence pairs? Did the Marian training recipes basically work as-is? Any special processing needed for Arabic compared to Latin-script-based languages?
- omneity 1y ago
  Marian was a good starting point and allowed me to iterate faster when I first started, but I quickly found it a bit limiting as it performs better for single pairs. My goal was a google translate style multilingual translation model, and for that the BART architecture proved ultimately to be better because you benefit from cross-language transfer learning. If your model learns the meaning of "car" in language pair (A, B), and it knows it in language (B, C), then it will perform decently when you ask it to translate between A and C. It compounds very quickly the more you add language pairs. One big limitation of BART (where LLMs become more attractive) is that it becomes extremely slow for longer sentences, and is less good at understanding and translating complex sentences. > Any special processing needed for Arabic compared to Latin-script-based Yes indeed, quite a lot. Especially for Moroccan Arabic which is written in both Arabic and Latin scripts (I made sure to support both and they're aligned in the model's latent space). For this I developed semantic and phonetic embedding models along the way that helped a lot. I am in the process of publishing a paper on the phonetic processing aspect, if you're interested let's stay in touch and I'll let you know when it's out. But beyond the pre-processing and data pipeline, the model itself didn't need any special treatment besides the tokenizer.
deivid 1y ago
How big are the models that you use/built? Can't you run them on the browser? Asking because I built a translator app[0] for Android, using marian-nmt (via bergamot), with Mozilla's models, and the performance for on-device inference is very good. [0]: https://github.com/DavidVentura/firefox-translator https://github.com/DavidVentura/firefox-translator
- omneity 1y ago
  Thanks for the tip and cool project! The model I trained is relatively large, as it's a single model that supports all language pairs (to leverage transfer learning). With that said while running it client-side is indeed an option, openly distributing the model is not something I would like to do, at least at this stage. Unlike the bigger projects in the NMT space, including Marian and Bergamot, I don't have any funding, and my monetization plan is to offer inference via API[0]. 0: https://api.sawalni.com/docs https://api.sawalni.com/docs
  - klipt 1y ago
    > I trained is relatively large, as it's a single model that supports all language pairs (to leverage transfer learning). Note that you have the larger model, if you wanted a smaller model for just one language pair, I guess you could use distillation?
WalterBright 1y ago
> for language pairs that are not traditionally supported Maybe translate X to English, and then to Y?
- omneity 1y ago
  Many languages (with a sizable speaker population) do not have machine translation to or from any other language. The technique makes sense though, but in the training data stage mostly. BART-style translation models already represent concepts in latent space regardless of the input-output language sidestepping English entirely so you have something like: `source lang —encoded into-> latent space —decoded into—> target lang` Works great to get translation support for arbitrary language combinations.
- djvdq 1y ago
  It's a bad idea. It makes a lot of mistakes and might totally change the meaning of some sentences.
woodson 1y ago
Not sure if you tried that already, but ctranslate2 can run BART and MarianNMT models quite efficiently, also without GPUs.
- omneity 1y ago
  It does! I do use CT2! On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user. But it’s definitely worth revisiting that. Thanks!
  - woodson 1y ago
    Oh, that’s pretty slow. Have you tried using quantization (int8 or int8_float32)? In my experience that can help speed up CT2 execution. Personally, I haven’t had much luck with small-ish decoder-only models (i.e., typical LLMs) for translation. Sure, GPT4 etc. work extremely well, but not so much local models capable of running on small form-factor devices. Perhaps I should revisit that.

philomath868 1y ago

How does the "continuous training pipeline" work? You rebuild the model after every N corrections, with the corrections included in the data?

omneity 1y ago
Yes. There's a scoring and filtering pipeline first, whereby I try to automatically check for the quality of the correction using a custom multilingual embedding model, madmon[0] and language identification model, gherbal[1]. Above a certain similarity threshold it goes into the training dataset, below it it's flagged for human review. This is mostly to stave off the trolls or blatant mistakes. For the continuous training itself, yes I simply continue training the model from the last checkpoint (cosine lr scheduler). I am considering doing a full retraining at some point when I collect enough data to compare with this progressive training. Apologies for the poor links, it takes a lot of time to work on this let alone fully document everything. 0: https://api.sawalni.com/docs#tag/Embeddings https://api.sawalni.com/docs#tag/Embeddings 1: https://api.sawalni.com/docs#tag/Language-Identification https://api.sawalni.com/docs#tag/Language-Identification

yorwba 1y ago

I'm curious how large your training corpus is and your process for dealing with data quality issues. Did you proofread everything manually or were you able to automate some parts?

omneity 1y ago
I started seeing results as early as 5-10k pairs, but you want something closer to 100k, especially if the language has a lot of variations (aka morphologically rich, agglutinative, or written in a non-standardized way). Manual proof-reading (and data generation) was a big part of it, it's definitely not a glamorous magic process. But as I went through it I could notice patterns and wrote some tools to help. There's a way to leverage LLMs to help with this if your language is supported (my target wasn't at the time), but I still strongly recommend a manual review part. That's really the secret sauce and no way around it if you're serious about the translation quality of your model.

ks2048 1y ago

Any major challenges beyond gathering high-quality sentence pairs? Did the Marian training recipes basically work as-is? Any special processing needed for Arabic compared to Latin-script-based languages?

omneity 1y ago
Marian was a good starting point and allowed me to iterate faster when I first started, but I quickly found it a bit limiting as it performs better for single pairs. My goal was a google translate style multilingual translation model, and for that the BART architecture proved ultimately to be better because you benefit from cross-language transfer learning. If your model learns the meaning of "car" in language pair (A, B), and it knows it in language (B, C), then it will perform decently when you ask it to translate between A and C. It compounds very quickly the more you add language pairs. One big limitation of BART (where LLMs become more attractive) is that it becomes extremely slow for longer sentences, and is less good at understanding and translating complex sentences. > Any special processing needed for Arabic compared to Latin-script-based Yes indeed, quite a lot. Especially for Moroccan Arabic which is written in both Arabic and Latin scripts (I made sure to support both and they're aligned in the model's latent space). For this I developed semantic and phonetic embedding models along the way that helped a lot. I am in the process of publishing a paper on the phonetic processing aspect, if you're interested let's stay in touch and I'll let you know when it's out. But beyond the pre-processing and data pipeline, the model itself didn't need any special treatment besides the tokenizer.

deivid 1y ago

How big are the models that you use/built? Can't you run them on the browser? Asking because I built a translator app[0] for Android, using marian-nmt (via bergamot), with Mozilla's models, and the performance for on-device inference is very good. [0]: https://github.com/DavidVentura/firefox-translator https://github.com/DavidVentura/firefox-translator

omneity 1y ago
Thanks for the tip and cool project! The model I trained is relatively large, as it's a single model that supports all language pairs (to leverage transfer learning). With that said while running it client-side is indeed an option, openly distributing the model is not something I would like to do, at least at this stage. Unlike the bigger projects in the NMT space, including Marian and Bergamot, I don't have any funding, and my monetization plan is to offer inference via API[0]. 0: https://api.sawalni.com/docs https://api.sawalni.com/docs
- klipt 1y ago
  > I trained is relatively large, as it's a single model that supports all language pairs (to leverage transfer learning). Note that you have the larger model, if you wanted a smaller model for just one language pair, I guess you could use distillation?

WalterBright 1y ago

> for language pairs that are not traditionally supported Maybe translate X to English, and then to Y?

omneity 1y ago
Many languages (with a sizable speaker population) do not have machine translation to or from any other language. The technique makes sense though, but in the training data stage mostly. BART-style translation models already represent concepts in latent space regardless of the input-output language sidestepping English entirely so you have something like: `source lang —encoded into-> latent space —decoded into—> target lang` Works great to get translation support for arbitrary language combinations.
djvdq 1y ago
It's a bad idea. It makes a lot of mistakes and might totally change the meaning of some sentences.

woodson 1y ago

Not sure if you tried that already, but ctranslate2 can run BART and MarianNMT models quite efficiently, also without GPUs.

omneity 1y ago
It does! I do use CT2! On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user. But it’s definitely worth revisiting that. Thanks!
- woodson 1y ago
  Oh, that’s pretty slow. Have you tried using quantization (int8 or int8_float32)? In my experience that can help speed up CT2 execution. Personally, I haven’t had much luck with small-ish decoder-only models (i.e., typical LLMs) for translation. Sure, GPT4 etc. work extremely well, but not so much local models capable of running on small form-factor devices. Perhaps I should revisit that.