4 ms·
Would you have any literature to recommend, to help get some insight in how to approach the task?
by fransje26 3y ago
Would you have any literature to recommend, to help get some insight in how to approach the task?
- juliangoldsmith 3y agoAndrej Karpathy has a series on YouTube where he builds everything up from scratch. He implements a transformer when he builds a GPT-style model: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
- fransje26 3y agoThank you for the tip!
- thomasahle 3y agoThere are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_mixformer_sequential.py https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_... It's well written, and very modern, including rotary embeddings and a kv-cache for inference.
- fransje26 3y agoThank you!