There are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main

There are a lot of "simple transformer" implementations on github. But I'll recommend Microsofts new Phi 1.5: https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_mixformer_sequential.py https://huggingface.co/microsoft/phi-1_5/blob/main/modeling_... It's well written, and very modern, including rotary embeddings and a kv-cache for inference.