3 ms·
The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fi
by yablak 8y ago
The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem.
Disclaimer, I wrote the TF RNN api.
- alexcnwy 8y agoTotally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size. Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.
- Zephyr314 8y agoFor graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects. Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent" [0]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia