4 ms·
Hey guys, I'm the writer. As you can see from the post, I'm still very much learning. What I want the most from this site is for more experienced people to hel
by strikingloo 7y ago
Hey guys, I'm the writer. As you can see from the post, I'm still very much learning.
What I want the most from this site is for more experienced people to help me out with some of my questions.
Here they come:
- Can you use Batch Normalization (the one from tf.keras) on an LSTM layer? Or will it break the model?
- How do you deal with extremely infrequent words if you do a word-based LSTM (with a one-hot encode of each word in the corpus?)? Do you remove them? Replace them? Cluster them?
- Do you think there's any other architecture that would've had better results -while still not taking too long to train-?
- srean 7y ago> Do you think there's any other architecture that would've had better results -while still not taking too long to train-? Yes! A 5th or 6th order word level Markov chain with smoothing.
- strikingloo 7y agoThanks! I hadn't thought of doing Markov Chains since I was focusing on RNN/LSTMs, but I think I'll try that for the next article, see how that goes.
- nestorD 7y ago> Do you think there's any other architecture that would've had better results -while still not taking too long to train? You could easily finetune a pretrained GPT2 model on your lovecraft dataset : https://github.com/minimaxir/gpt-2-simple https://github.com/minimaxir/gpt-2-simple