3 ms·

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard sai

Really great work on visualizing neurons! Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges? Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

denzil_correa 9y ago
> simpler GRUs What does GRU stand for?
- davik 9y ago
  Gated recurrent unit, it has fewer gates than LSTMs usually.
olavgg 9y ago
Our chatbot uses LSTM(GRU) for prediction, we see about the same performance as LSTM, but training is more efficient.
agibsonccc 9y ago
Yes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance) Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can learn some fairly useful classification and forecasting problems. Beyond that standard neural net tuning applies. Eg: normalize your data, pay attention to your weight initialization, understand what loss function you're using,..
- griffinkelly 9y ago
  So if LSTMs are purposely forgetting, do you need less training data than a CNN?
  - agibsonccc 9y ago
    LSTMs don't "forget" more than "remember the things that matter". They don't necessarily need less data. They do have a limit on the "length" of time steps they can handle though. Eg: You can't do thousands in to the future (maybe a few hundred or so) The long part of "LSTM" means remember good long ranging dependencies.
msamwald 9y ago
A recent paper exploring various variatons of neural machine translation architectures found that LSTMs consistently outperformed GRUs in that task. https://arxiv.org/abs/1703.03906 https://arxiv.org/abs/1703.03906
jph00 9y ago
I hope that wasn't quite what I said :) I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.
- jph00 9y ago
  Oh also I did say that CNNs often beat RNNs, even for time series, language, and audio. So I generally try them first. You can always stick an RNN on top of the CNN and get the best of both worlds.
  - rjtavares 9y ago
    I just wanted to take the chance to say thank you for the course. As someone who is interested in the topic but isn't a practicioner, just watching it all work in practice has been fascinating and highly enlightening.

denzil_correa 9y ago

> simpler GRUs What does GRU stand for?

davik 9y ago
Gated recurrent unit, it has fewer gates than LSTMs usually.

olavgg 9y ago

Our chatbot uses LSTM(GRU) for prediction, we see about the same performance as LSTM, but training is more efficient.

agibsonccc 9y ago

Yes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance) Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can learn some fairly useful classification and forecasting problems. Beyond that standard neural net tuning applies. Eg: normalize your data, pay attention to your weight initialization, understand what loss function you're using,..

griffinkelly 9y ago
So if LSTMs are purposely forgetting, do you need less training data than a CNN?
- agibsonccc 9y ago
  LSTMs don't "forget" more than "remember the things that matter". They don't necessarily need less data. They do have a limit on the "length" of time steps they can handle though. Eg: You can't do thousands in to the future (maybe a few hundred or so) The long part of "LSTM" means remember good long ranging dependencies.

msamwald 9y ago

A recent paper exploring various variatons of neural machine translation architectures found that LSTMs consistently outperformed GRUs in that task. https://arxiv.org/abs/1703.03906 https://arxiv.org/abs/1703.03906

jph00 9y ago

I hope that wasn't quite what I said :) I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.

jph00 9y ago
Oh also I did say that CNNs often beat RNNs, even for time series, language, and audio. So I generally try them first. You can always stick an RNN on top of the CNN and get the best of both worlds.
- rjtavares 9y ago
  I just wanted to take the chance to say thank you for the course. As someone who is interested in the topic but isn't a practicioner, just watching it all work in practice has been fascinating and highly enlightening.