4 ms·
That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandi
by vintermann 3y ago
That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandinavian languages and in particular the two written forms of Norwegian.
I had the same problem of lacking training data, but I tried an approach of bootstrapping from rule-based translators. There exist decent rule-based translators for the Scandinavian languages. Of course they make mistakes, but I figured that as long as I used the rule-translated text only as source, never as target, it wouldn't matter.
Effectively, instead of learning to translate from real Danish to correct Norwegian, it learned to translate from slightly broken machine-Danish to correct Norwegian. And my hope was that it would learn to translate from real Danish to correct Norwegian as a side effect.
I then figured that I might as well let it take input in slightly broken machine-Swedish as well, hopefully helping its generalization.
Then I could use this translator for the next step, again using the real text as the target, but using output from this translator as input.
It worked reasonably well. It certainly beat Google Translate at the time, since that at the time went via English even for closely related languages like the Scandinavian ones. I seriously considered making a start-up and commercializing it, some people had expressed interest (among them newspapers - they used a rule-based translator/grammar checker at the time, I think maybe they still do.)
But in the end, it's probably good I didn't. These days I understand they don't even train sequence to sequence models, they just train a general-purpose language model and instruction-tune it to do translation.
- omneity 3y agoThank you for sharing your experience! It's awesome to hear what worked, and I see many parallels in our work as well (a manual rule system was key for me to unlock seq2seq and then the rest of the ML chain). By the way, Seq2seq models are still super useful, I've trained and I'm using many in the data pipeline of this project. LLMs are a nuclear hammer and sometimes you just have a teeny tiny nail. There's also limited commercial value to what I'm working on, but it's a work of passion and I want to be able to speak in my native language to voice assistants (among other objectives). How did you communicate about it at the time and gather newspaper interest, if you don't mind sharing your experience?
- vintermann 3y agoI went to a machine learning conference in Tromsø and talked with many people there, and I met the newspaper people at a local business fair in my hometown. I had a little android app to demo it, but the backend was too expensive to have up... optimizing the use of cloud resources was a can of worms I didn't even get started on!
- omneity 3y agoThanks again for sharing, that was super helpful. And yeah I agree it is indeed a massive endeavor, I'm sorry it didn't pan out the way you wanted it to. Are you still interested in this space, and do you write about your experiences anywhere? I'd love to connect if you want to.
- vintermann 3y agoSorry it took a little time to get back to you. Yes, I wrote a few articles, on linkedin of all places, but they didn't say much more than what I've said here, as I recall. You can contact me on vintermann at gmail if you want.