3 ms·

I haven’t done any significant projects using neural networks for sequence modeling or analysis, but when I do, I plan to start with a temporal convolutional ne

I haven’t done any significant projects using neural networks for sequence modeling or analysis, but when I do, I plan to start with a temporal convolutional network, based on [0]. They argue that RNNs being standard is likely an artefact of the history of the field, while they get superior performance from TCNs. [0] https://github.com/locuslab/TCN https://github.com/locuslab/TCN

yablak 8y ago
The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem. Disclaimer, I wrote the TF RNN api.
- alexcnwy 8y ago
  Totally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size. Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.
- Zephyr314 8y ago
  For graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects. Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent" [0]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia
netheril96 8y ago
From my cursory understanding of TCN, it always outputs sequence of the same length as the input. If that is true, then its usage is severely limited compared to recurrent networks. Please point to more substantive sources if I am wrong. I am very interested to make TCN work, as it is much faster.
nl 8y ago
I do a lot of sequence work (text). While CNNs are tempting and fast to train I've never been able to get the accuracy I can from RNNs. In NLP, accuracy is important because for lots of tasks NLP is right at that inflection point of being good enough to be useful.. if it's good enough. It's worth noting that this TCN paper gets a perplexity of 45.19 on WikiText-103. That was competitive in 2015. The current state of the art is 29.2[1] - not their claim of 48.4 (unclear what that came from). Still, CNNs are nice in an ensemble model if that's your thing. They do tend to pick up different things to RNNs, which can be useful. Edit: I now understand why their reported metrics are so wrong. They use generic models to compare against. They list SOTA performance in their supplementary material (they still get them wrong though). [1] http://nlpprogress.com/english/language_modeling.html http://nlpprogress.com/english/language_modeling.html
- stochastic_monk 8y ago
  Thanks for digging into how their claims are misleading. As someone in an adjacent subfield, I was wondering if I was missing something. It does at least sound like an interesting potential component.

yablak 8y ago

The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem. Disclaimer, I wrote the TF RNN api.

alexcnwy 8y ago
Totally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size. Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.
Zephyr314 8y ago
For graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects. Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent" [0]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia

netheril96 8y ago

From my cursory understanding of TCN, it always outputs sequence of the same length as the input. If that is true, then its usage is severely limited compared to recurrent networks. Please point to more substantive sources if I am wrong. I am very interested to make TCN work, as it is much faster.

nl 8y ago

I do a lot of sequence work (text). While CNNs are tempting and fast to train I've never been able to get the accuracy I can from RNNs. In NLP, accuracy is important because for lots of tasks NLP is right at that inflection point of being good enough to be useful.. if it's good enough. It's worth noting that this TCN paper gets a perplexity of 45.19 on WikiText-103. That was competitive in 2015. The current state of the art is 29.2[1] - not their claim of 48.4 (unclear what that came from). Still, CNNs are nice in an ensemble model if that's your thing. They do tend to pick up different things to RNNs, which can be useful. Edit: I now understand why their reported metrics are so wrong. They use generic models to compare against. They list SOTA performance in their supplementary material (they still get them wrong though). [1] http://nlpprogress.com/english/language_modeling.html http://nlpprogress.com/english/language_modeling.html

stochastic_monk 8y ago
Thanks for digging into how their claims are misleading. As someone in an adjacent subfield, I was wondering if I was missing something. It does at least sound like an interesting potential component.