2 ms·
There is no reason to leak the Alpaca weights, finetuning the model is not that expensive, eventually someone will finetune one and release it. (Also LoRA shoul
by GaggiX 4y ago
There is no reason to leak the Alpaca weights, finetuning the model is not that expensive, eventually someone will finetune one and release it. (Also LoRA should performs on-par or better than fine-tuning in model quality)
- ComplexSystems 4y agoThere are different fine tuning methods. Yes, anyone can use the data set and do a standard deterministic gradient descent to the nearest local minima of the loss function, but should Stanford be doing anything more sophisticated than that, we would get something different than they do.
- GaggiX 4y agoIt is enough to use a different seed to get a different result; I do not think it is important to recreate a copy of the Stanford model weights, as long as the performance is equivalent.
- danielbln 4y agoStanford released the exact training data as well as the training script with all parameters. Boot up a p4.2xlarge (8 A100 GPUs) which costs about $40/hour and let it run for a 2-3 hours and voila. See the Readme in their repo where it mentions the fine-tuning script[0] [0] https://github.com/tatsu-lab/stanford_alpaca https://github.com/tatsu-lab/stanford_alpaca