5 ms·
Can anyone share their thoughts on how does this compare to fine-tuning Llama2 13B or 70B which usually requires renting GPUs from one of the cloud providers?
by thumbsup-_- 3y ago
Can anyone share their thoughts on how does this compare to fine-tuning Llama2 13B or 70B which usually requires renting GPUs from one of the cloud providers?
In case of openAI, the infra doesn't needs to be rented but the cost is covered in usage pricing but want to hear from someone who has experience fine-tuning Llama2 models that how it compares.
- SparkyMcUnicorn 3y agoI haven't fine-tuned GPT-3.5-turbo yet, but overall llama2 is probably going to be cheaper. Especially if 13B is good enough for you. You can get roughly $0.003/1K tokens using modal.com for inference on a fine-tuned llama2 13B model, and there's definitely cheaper options available. Unless you have a ton of data, the cost for training llama2 is usually a few dollars.
- swyx 3y ago> the cost for training llama2 is usually a few dollars. nit: thats the cost for finetuning, not pretraining
- SparkyMcUnicorn 3y agoYeah, definitely not pretraining. I've been referring to fine-tuning as training, since that's what's happening when you "fine-tune" a llama model.
- easygenes 3y agoI'm curious about this. Can you point me to, e.g. some example code for setting up an inference endpoint with a base llama2 model on modal.com?
- SparkyMcUnicorn 3y agoHere's one if their tutorials using vLLM, and they have a few other guides and example repos as well. https://modal.com/docs/guide/ex/vllm_inference https://modal.com/docs/guide/ex/vllm_inference https://github.com/modal-labs https://github.com/modal-labs Alternatively, Runpod is fairly cheap and easy to get stuff running in a few minutes and can be point/click only using their templates. https://www.runpod.io/console/gpu-secure-cloud?template=f1pf20op0z https://www.runpod.io/console/gpu-secure-cloud?template=f1pf... ("serverless" example) https://github.com/ashleykleynhans/runpod-worker-oobabooga https://github.com/ashleykleynhans/runpod-worker-oobabooga
- easygenes 3y agoThanks for that. I've used RunPod GPU cloud to setup vLLM as an Open-AI API compatible endpoint before, but haven't tried any of the serverless options yet.
- drcode 3y agoI'm not an expert on this subject, but I have done a bit of gpt3 finetuning through their api: I think it's clear that "fine tuning" with GPT is different from fine tuning something like Llama2, in that it probably isn't adjusting all the weights of the network, only a tiny subfragment of the network- Exactly how OpenAI accomplishes this is properietary. The tradeoff is that OpenAI fine tuning is less expensive, but it is also less powerful than "real" fine tuning.
- swyx 3y ago> it probably isn't adjusting all the weights of the network, only a tiny subfragment of the network source please? this actually isnt all that clear to me
- drcode 3y agoIt was what I read on forums when I learned about the process. It's possible that I am mistaken.
- deleted 3y ago[deleted]
- lgvld 3y agoI've been taught in many cases you can indeed fine-tune the last (i.e. closest from the output) layer(s) of a network. Of course, it does not give as good results as fine-tuning the whole model, but it is obviously way less expensive in compute. i.e. you actually don't want your model to re-learn _everything_.