This requires retraining from scratch, so no, you can't use Llama 2 pretrained weight. As far as I can tell you can take Llama 2 modelling code, training infra

This requires retraining from scratch, so no, you can't use Llama 2 pretrained weight. As far as I can tell you can take Llama 2 modelling code, training infrastructure, training data and apply proposed modification (they provide PyTorch nn.Module which should be drop in replacement of nn.Linear) and run the training if you have enough compute and it should work. Doesn't mean it would work, there are always lots of practical problems, but it should work in principle.