Thanks for the detailed feedback! Yes, supporting LoRA fine-tuning is one of the things we are already working on. btw, we have LoRA supported with Llama3 PyTo

Thanks for the detailed feedback! Yes, supporting LoRA fine-tuning is one of the things we are already working on. btw, we have LoRA supported with Llama3 PyTorch-XLA model. Check that out in meanwhile.