4 ms·
> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https
by throwme_123 3y ago
> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used.
$300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925 https://news.ycombinator.com/item?id=35662925
- sp332 3y agoA100's cost about $4/hour/GPU to rent. So the total cost depends on the amount of time, but 8x for 24 hours would cost $768.
- akiselev 3y ago8x A100 80GB are $12/hour or so on Lambda Cloud. If you get lucky enough to snag capacity.
- lsaferite 3y agoThey called out the costs incurred: $30 - dataset generation (OpenAI GPT 3.5-turbo) $24 - llama 7b fine-tuning (8x A100 80GB instance costs) $84 - llama 13b fine-tuning (8x A100 80GB instance costs)
- brucethemoose2 3y agoYou can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model https://github.com/Lightning-AI/lit-llama#finetune-the-model What OP did is more intense, but Lora/Adapter still can give excellent results. Pure AI cards for mere mortals aren't really a thing yet.
- theolivenbaum 3y agoAnyone tried fine tunning on CPU already? I expect it to be much slower, but is it even practical?
- brucethemoose2 3y agoOh no not a chance, finetuning is really compute intense.
- brucethemoose2 3y agoAlso I bet the pytorch training code is written with CUDA semantics. Maybe a JAX version would work without messing with the code.