6 ms·

Does anyone know of any good primers on building your own machine translation? My wife is from a small European country, and all machine translation I've used u

Does anyone know of any good primers on building your own machine translation? My wife is from a small European country, and all machine translation I've used usually end up with incompressible results. My theory is that it's not that there is anything hard about the language, it's just they don't put as much effort into training this language as the number of users is small - so it ends up with poor results. As an experiment I'd like to try building my own, but I don't know where to begin.

nl 10y ago
https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/index.html#neural-translation-model https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/... If the European country she is from is within the EU, you can probably get parallel corpus from the EU parliamentary translations. I believe there are other EU research projects which release useful corpus.
- xbmcuser 10y ago
  Sadly one of the largest tranlation corpus of multiple languages can't be used for copyright reasons i.e the user made subtitle files that are shared all over the internet. Every week a few hundred episodes of different shows are released in different countries that are translated by users into many languages. Now that I think about it an app that could transcribe the words spoken in a TV show and then would translate it using google translate would be a huge with the manga crowd. Though Japanese hasn't been added yet.
  - nl 10y ago
    It's actually not at all clear if they couldn't be used. Things like Google's word2vec model are derived from copyrighted data and seem to be fine. Perhaps distribution of the subtitles is illegal under copyright law, but I suspect the trained model would be fine. It would be an interesting case anyway.
- urubu 10y ago
  This is a directory of available multilingual corpora: http://opus.lingfil.uu.se/ http://opus.lingfil.uu.se/
londons_explore 10y ago
Poor results are probably lack of enough data rather than lack of enough researcher attention. NMT requires massive amounts of data to give even okayish results.
PeterisP 10y ago
If it's an EU language, then it shouldn't be caused by "they don't put as much effort" as there are a bunch of datasets and research projects that, as a policy, work on all 24 EU languages. These systems aren't always publicly available for anyone (e.g. http://ec.europa.eu/dgs/translation/translationresources/machine_translation/index_en.htm http://ec.europa.eu/dgs/translation/translationresources/mac... is freely available for gov't institutions only, https://www.letsmt.eu/Login.aspx https://www.letsmt.eu/Login.aspx has commercial pricing), but the freely available resources (e.g. EU corpora) tend to be well integrated in the main free MT systems (google/bing/etc). What you see there is just a bit worse than the maximum currently achievable - you can get a better system for a particular narrow domain (which is often done in practical applications) by integrating domain-specific data and terminology, but for general purpose translation don't expect to get anything significantly better than e.g. google translate even for a small language unless you can invest many man-years of work - e.g. a major funded research project for that particular language. Regarding "My theory is that it's not that there is anything hard about the language", there are two factors that you must keep in mind. First, the current (pre-neuron) mainstream approach of phrase based SMT works much worse if the target language is highly inflectional, as many European languages tend to be. Second, the main available large training data sets for European languages are mostly legal/government documents, so the systems work better for that domain but much worse on "common" topics since they're poorly covered in training data.

nl 10y ago

https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/index.html#neural-translation-model https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/... If the European country she is from is within the EU, you can probably get parallel corpus from the EU parliamentary translations. I believe there are other EU research projects which release useful corpus.

xbmcuser 10y ago
Sadly one of the largest tranlation corpus of multiple languages can't be used for copyright reasons i.e the user made subtitle files that are shared all over the internet. Every week a few hundred episodes of different shows are released in different countries that are translated by users into many languages. Now that I think about it an app that could transcribe the words spoken in a TV show and then would translate it using google translate would be a huge with the manga crowd. Though Japanese hasn't been added yet.
- nl 10y ago
  It's actually not at all clear if they couldn't be used. Things like Google's word2vec model are derived from copyrighted data and seem to be fine. Perhaps distribution of the subtitles is illegal under copyright law, but I suspect the trained model would be fine. It would be an interesting case anyway.
urubu 10y ago
This is a directory of available multilingual corpora: http://opus.lingfil.uu.se/ http://opus.lingfil.uu.se/

londons_explore 10y ago

Poor results are probably lack of enough data rather than lack of enough researcher attention. NMT requires massive amounts of data to give even okayish results.

PeterisP 10y ago

If it's an EU language, then it shouldn't be caused by "they don't put as much effort" as there are a bunch of datasets and research projects that, as a policy, work on all 24 EU languages. These systems aren't always publicly available for anyone (e.g. http://ec.europa.eu/dgs/translation/translationresources/machine_translation/index_en.htm http://ec.europa.eu/dgs/translation/translationresources/mac... is freely available for gov't institutions only, https://www.letsmt.eu/Login.aspx https://www.letsmt.eu/Login.aspx has commercial pricing), but the freely available resources (e.g. EU corpora) tend to be well integrated in the main free MT systems (google/bing/etc). What you see there is just a bit worse than the maximum currently achievable - you can get a better system for a particular narrow domain (which is often done in practical applications) by integrating domain-specific data and terminology, but for general purpose translation don't expect to get anything significantly better than e.g. google translate even for a small language unless you can invest many man-years of work - e.g. a major funded research project for that particular language. Regarding "My theory is that it's not that there is anything hard about the language", there are two factors that you must keep in mind. First, the current (pre-neuron) mainstream approach of phrase based SMT works much worse if the target language is highly inflectional, as many European languages tend to be. Second, the main available large training data sets for European languages are mostly legal/government documents, so the systems work better for that domain but much worse on "common" topics since they're poorly covered in training data.