4 ms·

Yup it's multi-layer bidir seq2seq with attention, and a few tricks like teacher forcing. Same as Google Translate. Their version takes a long time to train on

by jph00 8y ago

Yup it's multi-layer bidir seq2seq with attention, and a few tricks like teacher forcing. Same as Google Translate. Their version takes a long time to train on a lot of GPUs, so we simplify it by using less layers, and a smaller, simplified corpus (it only contains questions, and limits them to 30 words long). By "solve end-to-end problems" I only mean that we show how to do the whole process from beginning to end - I didn't mean to imply that the final model would be human-equivalent or perfect or anything like that.

pixelHD 8y ago
Oh, nice! Yeah, I understood the intention behind that statement. Great work with the course! While you're here, what do you think about using temporal convolution for sequence tasks? I've read a few articles, this particular one by my professor comes to mind now [0], which say CNNs could work extremely well for the tasks traditionally done with RNNs. A recent paper by the people at Google Brain [1] mentioned that their CNN with attention network beats traditional RNN approaches. More surprising is that the network is 130+ layers deep, and yet trains faster than RNNs. Do you think we can potentially switch most machine translation tasks to CNNs? [0]: https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c74ce0 https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c7... [1]: https://twitter.com/lmthang/status/989261575482560513 https://twitter.com/lmthang/status/989261575482560513
- jph00 8y ago
  Yup potentially. The jury is still out on which will win, or whether both have their place!
YeGoblynQueenne 8y ago
>> By "solve end-to-end problems" I only mean that we show how to do the whole process from beginning to end (...) Then why not write just that? What is the point of using language that implies you can teach people how to solve a very hard problem that nobody knows how to solve yet? I find it extremely disreputable to claim to be able to accomplish feats that go far beyond the limits of current technology. That is the tactic of charlatans and snake oil salesmen, not of scientists and technologists.

by jph00 8y ago

pixelHD 8y ago

Oh, nice! Yeah, I understood the intention behind that statement. Great work with the course! While you're here, what do you think about using temporal convolution for sequence tasks? I've read a few articles, this particular one by my professor comes to mind now [0], which say CNNs could work extremely well for the tasks traditionally done with RNNs. A recent paper by the people at Google Brain [1] mentioned that their CNN with attention network beats traditional RNN approaches. More surprising is that the network is 130+ layers deep, and yet trains faster than RNNs. Do you think we can potentially switch most machine translation tasks to CNNs? [0]: https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c74ce0 https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c7... [1]: https://twitter.com/lmthang/status/989261575482560513 https://twitter.com/lmthang/status/989261575482560513

jph00 8y ago
Yup potentially. The jury is still out on which will win, or whether both have their place!

YeGoblynQueenne 8y ago

>> By "solve end-to-end problems" I only mean that we show how to do the whole process from beginning to end (...) Then why not write just that? What is the point of using language that implies you can teach people how to solve a very hard problem that nobody knows how to solve yet? I find it extremely disreputable to claim to be able to accomplish feats that go far beyond the limits of current technology. That is the tactic of charlatans and snake oil salesmen, not of scientists and technologists.