4 ms·

Hi everyone, I'm a creator of OpenNMT and run the NLP group at Harvard (http://nlp.seas.harvard.edu http://nlp.seas.harvard.edu). A lot has changed in both tran

by srush 8y ago

Hi everyone, I'm a creator of OpenNMT and run the NLP group at Harvard (http://nlp.seas.harvard.edu http://nlp.seas.harvard.edu). A lot has changed in both translation and deep learning over the last couple years. Happy to answer any questions about the area. (BTW, the title of this article is crazy hyperbole, almost all this work is just research implementations of ideas originally from Google/MILA)

zawerf 8y ago
Do you rearchitect the code upon new research breakthroughs? Why use this over the Transformer model in tensor2tensor?
- srush 8y ago
  > Do you rearchitect the code upon new research breakthroughs? Great question. So we now have implementations of Transformer in both the PyTorch and TensorFlow version of the library. It did require some rearchitecting, particularly for inference, but at heart the model is basically a sequence-to-sequence model. I wrote a blog post describing the process of implementing it here: http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html . > Why use this over the Transformer model in tensor2tensor? OpenNMT is a bit more accessible to tensor2tensor which is a very powerful library but requires buying into a heavyweight framework. For instance OpenNMT uses plain text files, whereas tensor2tensor manages the entire data pipeline. I also personally find TensorFlow to be difficult for developing new research code.
luxpir 8y ago
Good to see you here! I have a few questions if you're still about. Have you worked with professional translators much along the way? Most of us use and keep our own translation memories stretching into the millions of segments. What would be the ideal training material for these models to work from? Do you think NMT could ever recognise the type of text to be translated and apply style and context accordingly? Displaying an element of creativity, for example, drawing from relevant contextual themes. I suspect it might do this automatically if trained widely enough?
- srush 8y ago
  Sure, happy to respond. > Have you worked with professional translators much along the way? Most of us use and keep our own translation memories stretching into the millions of segments. I personally haven't worked too much with translators in my research, but we built OpenNMT with Systran who employ a group of translators and linguists. We also host a yearly workshop (http://workshop-paris-2018.opennmt.net/ http://workshop-paris-2018.opennmt.net/) where many professional translator came to talk. There is a lot of interesting work on integrating translation memories and automatic systems. > What would be the ideal training material for these models to work from? So the curt answer is "the best type of training is more training". In practice though the best type of training is in-domain data for whatever type of problem is currently of interest. > Do you think NMT could ever recognise the type of text to be translated and apply style and context accordingly? Displaying an element of creativity, for example, drawing from relevant contextual themes. I suspect it might do this automatically if trained widely enough? Creativity is a slippery term, so I will avoid that (personally the models don't seem too "clever" to me). There is a lot of interest though in models that can mimic a certain style, whether that be politeness, tone, genre, or technical material. Often that means learning "knobs" to tune for these properties.

by srush 8y ago

zawerf 8y ago

Do you rearchitect the code upon new research breakthroughs? Why use this over the Transformer model in tensor2tensor?

srush 8y ago
> Do you rearchitect the code upon new research breakthroughs? Great question. So we now have implementations of Transformer in both the PyTorch and TensorFlow version of the library. It did require some rearchitecting, particularly for inference, but at heart the model is basically a sequence-to-sequence model. I wrote a blog post describing the process of implementing it here: http://nlp.seas.harvard.edu/2018/04/03/attention.html http://nlp.seas.harvard.edu/2018/04/03/attention.html . > Why use this over the Transformer model in tensor2tensor? OpenNMT is a bit more accessible to tensor2tensor which is a very powerful library but requires buying into a heavyweight framework. For instance OpenNMT uses plain text files, whereas tensor2tensor manages the entire data pipeline. I also personally find TensorFlow to be difficult for developing new research code.

luxpir 8y ago

Good to see you here! I have a few questions if you're still about. Have you worked with professional translators much along the way? Most of us use and keep our own translation memories stretching into the millions of segments. What would be the ideal training material for these models to work from? Do you think NMT could ever recognise the type of text to be translated and apply style and context accordingly? Displaying an element of creativity, for example, drawing from relevant contextual themes. I suspect it might do this automatically if trained widely enough?

srush 8y ago
Sure, happy to respond. > Have you worked with professional translators much along the way? Most of us use and keep our own translation memories stretching into the millions of segments. I personally haven't worked too much with translators in my research, but we built OpenNMT with Systran who employ a group of translators and linguists. We also host a yearly workshop (http://workshop-paris-2018.opennmt.net/ http://workshop-paris-2018.opennmt.net/) where many professional translator came to talk. There is a lot of interesting work on integrating translation memories and automatic systems. > What would be the ideal training material for these models to work from? So the curt answer is "the best type of training is more training". In practice though the best type of training is in-domain data for whatever type of problem is currently of interest. > Do you think NMT could ever recognise the type of text to be translated and apply style and context accordingly? Displaying an element of creativity, for example, drawing from relevant contextual themes. I suspect it might do this automatically if trained widely enough? Creativity is a slippery term, so I will avoid that (personally the models don't seem too "clever" to me). There is a lot of interest though in models that can mimic a certain style, whether that be politeness, tone, genre, or technical material. Often that means learning "knobs" to tune for these properties.