3 ms·
I think anyone can train one of these models. “anyone” being someone who can throw $1-5M in cloud resources at compute. Or a bored sysadmin with access to a few
by oceanplexian 4y ago
I think anyone can train one of these models. “anyone” being someone who can throw $1-5M in cloud resources at compute. Or a bored sysadmin with access to a few thousand GPUs and a cool boss.
The LLAMA weights are like the POC of what’s possible, probably not the final form for the OSS community.
- bitL 4y agoThere seems to be a big difference between efficiently training a "large-ish" model on 4-8 GPUs and a gigantic model on 1000+ GPUs. The same techniques might not work due to different warm up steps, gradient overlaps etc. All you can see running in the wild are quantized LLaMA variants (4 or 8-bit) whereas the original model is 32-bit.
- oezi 4y agoWhats possible now (March 2023). The field is moving so quick.
- wkat4242 4y agoSounds like a great Kickstarter option for a truly free model. I'd back it.
- kir-gadjello 4y agoPlease do it, people shouldn't put up with the apathetic siloed status quo. I'm sure people will find all sorts of beneficial uses for these models they are going to run on their own hardware!
- pdc56 4y agoThese models are as valuable as code now. And we know how open source eventually took over pretty much all code. I expect this to be the same, but instead of four decades, it will be four months