5 ms·
I thought I understood transformers well, even though I had never implemented them. Then one day I implemented them, and they didn't work/train nearly as well a
by thomasahle 3y ago
I thought I understood transformers well, even though I had never implemented them. Then one day I implemented them, and they didn't work/train nearly as well as the standard pytorch transformer.
I eventually realized that I had ignored the dropout, because I thought my data could never overfit. (I trained the transformer to add numbers, and I never showed it the same pair twice.) Turns out dropout has a much bigger role than I had realized.
TLDR, just go and implement a transformer.
The more from scratch the better.
Everyone I know who tried it, ended up learning something they hadn't expected.
From how training is parallelized over tokens down to how backprop really works.
It's different for every person.
- fransje26 3y agoWould you have any literature to recommend, to help get some insight in how to approach the task?
- juliangoldsmith 3y agoAndrej Karpathy has a series on YouTube where he builds everything up from scratch. He implements a transformer when he builds a GPT-style model: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
- fransje26 3y agoThank you for the tip!
- thomasahle 3y agoThere are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_mixformer_sequential.py https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_... It's well written, and very modern, including rotary embeddings and a kv-cache for inference.
- fransje26 3y agoThank you!