You can manually program transformers: https://srush.github.io/raspy/ https://srush.github.io/raspy/ I don't know if you can integrate them into a model. I th

You can manually program transformers: https://srush.github.io/raspy/ https://srush.github.io/raspy/ I don't know if you can integrate them into a model. I think you might run out of space, since these aren't polysemantic and so would take up a lot more "room" than learned neurons.