5 ms·

Can you help, please? I want to use AI to translate (localize) messages for free software, in my case, Ukrainian language. My plan to improve quality of automa

by drran 6y ago

Can you help, please? I want to use AI to translate (localize) messages for free software, in my case, Ukrainian language. My plan to improve quality of automated translation is to translate from similar languages in parallel, i.e. give a same message in English, Russian, Polish, and expect message in Ukrainian as output. Where I should start? Which libraries to use? How to connect them? How to train them?

chartpath 6y ago
I've been using LASER from Facebook Research via https://github.com/yannvgn/laserembeddings https://github.com/yannvgn/laserembeddings to accept multi-lingual input in front of the the domain-specific models for recommendations and stuff (that are trained on English annotated examples).
- zburatorul 6y ago
  This sounds interesting. Can you share more please? It sounds like there is some multilingual input text on the basis of which you make recommendations, but I think you would have called that a search engine rather than recommender.
  - chartpath 6y ago
    That's true, I'm making recommendations based on Multinomial Naive Bayes (and SGDClassifier) over custom TF-IDF bags of words, so it is like search plus text classification. And some endpoints do just check the cosine or Jaccard distance between things. There is a lot of overlap between search and NLP. My approach to AI is somewhat conservative because of working in a law-adjacent field where explainability is paramount. When it comes to getting "smart" I prefer forward-chaining logic over facts, and facts include predictions from models too. But at least there is a "judge"/engine to coordinate how the predictions from the ensemble of models maps to actions. I love me some pertained neural nets, but use them more as black box appliances.
    - krasi0 6y ago
      Which forward-chaining engine do you use? Something based on prolog?
      - chartpath 6y ago
        Currently https://github.com/nilp0inter/experta https://github.com/nilp0inter/experta but https://github.com/noxdafox/clipspy https://github.com/noxdafox/clipspy seems nice, I just shied away from using it due to uneasiness about FFI and debugging, even though the original CLIPS is still awesome and has a very interesting manual. There's also https://github.com/jruizgit/rules https://github.com/jruizgit/rules but haven't tried it yet.
        krasi0 6y ago
        Interesting. I guess your driving factor was that you can use those directly from Python. How's the performance (with many many rules)?
        chartpath 6y ago
        Yes exactly, I want to be able to do rich auditing of the predictions. Not sure about performance yet, still prototyping!
kthejoker2 6y ago
MarianMT has a lot of pre-trained models from one language to another, here is one for Polish to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk and English to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-en-uk https://huggingface.co/Helsinki-NLP/opus-mt-en-uk You can test the models with the form input on the right-hand side And here's a site with docs and code examples for basic usage in your app https://huggingface.co/transformers/model_doc/marian.html https://huggingface.co/transformers/model_doc/marian.html You can also use Marian's training code on PyTorch with custom source and target texts.
- drran 6y ago
  Translation from just English language creates a lot of ambiguity. The idea is to use two or more input languages to reduce ambiguity. Currently, volunteers manually translate from English to all other languages. It's possible to use AI to help, but quality of translation is too low. Dumb Translation Memory database is much more helpful than AI translator. In the new system, one volunteer will translate messages from English to his native language, while all others will be able to use this translation as the additional constraint, to improve quality of translation to their native language. More input languages -> more constraints -> better quality of translation. My best idea, so far, is to use a multilingual tokenizer (SpaCy looks good) and a linear transformer, because linear transformers are able to accept large inputs, with thousands of tokens. IMHO, I can input multiple translations of messages (they are short), and define loss as expected Ukrainian translation. However, I'm completely new to this field. I completed just one AI project so far: recognition of animals at video. I have no idea how to start. For example, I have no idea how to hint transformer that input messages are the same, just in 4 different languages. Should I interleave messages, like "Cannot Не можу open відкрити file файл : : ", or put them aside, via separator: "\0Cannot open file: \0Не можу відкрити файл: \0", or create few independent inputs? Or use a memory and improve quality of output message incrementally? Any hints where to start?

by drran 6y ago

chartpath 6y ago

I've been using LASER from Facebook Research via https://github.com/yannvgn/laserembeddings https://github.com/yannvgn/laserembeddings to accept multi-lingual input in front of the the domain-specific models for recommendations and stuff (that are trained on English annotated examples).

zburatorul 6y ago
This sounds interesting. Can you share more please? It sounds like there is some multilingual input text on the basis of which you make recommendations, but I think you would have called that a search engine rather than recommender.
- chartpath 6y ago
  That's true, I'm making recommendations based on Multinomial Naive Bayes (and SGDClassifier) over custom TF-IDF bags of words, so it is like search plus text classification. And some endpoints do just check the cosine or Jaccard distance between things. There is a lot of overlap between search and NLP. My approach to AI is somewhat conservative because of working in a law-adjacent field where explainability is paramount. When it comes to getting "smart" I prefer forward-chaining logic over facts, and facts include predictions from models too. But at least there is a "judge"/engine to coordinate how the predictions from the ensemble of models maps to actions. I love me some pertained neural nets, but use them more as black box appliances.
  - krasi0 6y ago
    Which forward-chaining engine do you use? Something based on prolog?
    - chartpath 6y ago
      Currently https://github.com/nilp0inter/experta https://github.com/nilp0inter/experta but https://github.com/noxdafox/clipspy https://github.com/noxdafox/clipspy seems nice, I just shied away from using it due to uneasiness about FFI and debugging, even though the original CLIPS is still awesome and has a very interesting manual. There's also https://github.com/jruizgit/rules https://github.com/jruizgit/rules but haven't tried it yet.
      - krasi0 6y ago
        Interesting. I guess your driving factor was that you can use those directly from Python. How's the performance (with many many rules)?
        chartpath 6y ago
        Yes exactly, I want to be able to do rich auditing of the predictions. Not sure about performance yet, still prototyping!

kthejoker2 6y ago

MarianMT has a lot of pre-trained models from one language to another, here is one for Polish to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk and English to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-en-uk https://huggingface.co/Helsinki-NLP/opus-mt-en-uk You can test the models with the form input on the right-hand side And here's a site with docs and code examples for basic usage in your app https://huggingface.co/transformers/model_doc/marian.html https://huggingface.co/transformers/model_doc/marian.html You can also use Marian's training code on PyTorch with custom source and target texts.

drran 6y ago
Translation from just English language creates a lot of ambiguity. The idea is to use two or more input languages to reduce ambiguity. Currently, volunteers manually translate from English to all other languages. It's possible to use AI to help, but quality of translation is too low. Dumb Translation Memory database is much more helpful than AI translator. In the new system, one volunteer will translate messages from English to his native language, while all others will be able to use this translation as the additional constraint, to improve quality of translation to their native language. More input languages -> more constraints -> better quality of translation. My best idea, so far, is to use a multilingual tokenizer (SpaCy looks good) and a linear transformer, because linear transformers are able to accept large inputs, with thousands of tokens. IMHO, I can input multiple translations of messages (they are short), and define loss as expected Ukrainian translation. However, I'm completely new to this field. I completed just one AI project so far: recognition of animals at video. I have no idea how to start. For example, I have no idea how to hint transformer that input messages are the same, just in 4 different languages. Should I interleave messages, like "Cannot Не можу open відкрити file файл : : ", or put them aside, via separator: "\0Cannot open file: \0Не можу відкрити файл: \0", or create few independent inputs? Or use a memory and improve quality of output message incrementally? Any hints where to start?