5 ms·
"We have also trained 34B variants, which we report on in this paper but are not releasing." "We are delaying the release of the 34B model due to a lack of tim
by msp26 3y ago
"We have also trained 34B variants, which we report on in this paper but are not releasing."
"We are delaying the release of the 34B model due to a lack of time to sufficiently red team."
From the Llama 2 paper
- swyx 3y agoif you red team the 13b and the 70b and they pass, what is the danger of 34B being significantly more dangerous? edit: turns out I should RTFP. there was a ~2x spike in safety violations for 34B https://twitter.com/yacineMTB/status/1681358362057883680?s=20 https://twitter.com/yacineMTB/status/1681358362057883680?s=2...
- DebtDeflation 3y agoA 34B model is probably about the largest you can run on a consumer GPU with 24GB VRAM. 70B will require A100's or a cloud host. 13B models are everywhere already. I'm sure this was a very deliberate choice - let people play with the 13B model locally to whet their appetite and then they can pay to run the 70B model on Azure.
- bloaf 3y agoI'm running a 30B model on an amd 5600x cpu at 2-3 tokens/s, which is just under a "read-aloud" pace. I'd wager that you can run a 70B model at about the same speed with a 7900x and a bit more RAM.
- fmajid 3y agoOr a $5000 128GB Mac Studio, that you can get for 1/2 the price of a 40GB A100 or 1/7 the price of a 80GB H100.
- brucethemoose2 3y ago70B should work reasonably well on 64GB CPU RAM + any decent GPU, or maybe a 24GB GPU + 32GB CPU RAM.
- swyx 3y agodo you mind teaching how to do CPU/GPU RAM math? all i know is 34B 16bit = 68GB total RAM needed (because 1B of 8bytes = 1GB definitionally), but i dont know how it splits between CPU/GPU and whether the tradeoff in tok/s is acceptable
- sbierwagen 3y agoPeople running LLMs on CPU are generally running them integer quantized, so they use fewer bits per parameter.
- brucethemoose2 3y agoIf you are doing ~4 bit quantization, a good rule of thumb is just under 1 Gigabyte per 1B parameters, plus a little room for the operating system. Longer contexts require a bit more VRAM. For reference, 4 bit LlamaV1 33B fits snugly on a 24GB GPU with 2K context with the exLLaMA backend. But it won't do really long inputs. Llama.cpp is pretty much the only backend that can offload to CPU efficiently. Its still quite fast and offers very flexible 3-5 bit quantization, with the leanest 3 bit quant just barely fitting LlamaV1 33b on my 6GB + 16GB laptop.