7 ms·
Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
- trollied 4y agoNeeds merging with https://news.ycombinator.com/item?id=35100086 https://news.ycombinator.com/item?id=35100086
- Retr0id 4y agoWhy?
- Retr0id 4y agoIt's not a new technique at this point, but it still blows my mind that you can quantize a model trained at fp16 down to 4 bits, and still have coherent inference results. > While running, the model uses about 4GB of RAM and Activity Monitor shows it using 748% CPU - which makes sense since I told it to use 8 CPU cores. > I imagine it's possible to run a larger model such as 13B on this hardware, but I've not figured out how to do that yet. Naively it seems like you could repeat the same process outlined in the article but with "13B" in place of "7B". What's the catch?
- summarity 4y agoNo catch, just works. 30B works fine on an M1 Max with 64GB of RAM, had to go for the M1 Ultra at 128GB for 65B.
- detrites 4y agoWhat's the tokens/s on those?
- minxomat 4y agoWith 16 threads, about 140ms per token for 30B, 300ms per token for 65B I should also mention that 65B should be able to run on 64GB systems. Total system memory consumption on M1 Ultra is about 67GB when running nothing else.
- bojangleslover 4y agoI'm not able to run 13B and from his wiki: > Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models.
- minxomat 4y agoThis has been fixed almost 2 days ago now. It’s literally mentioned at the top of the repo.
- simonw 4y agoThis commit landed 7 hours ago (since I wrote my TIL): https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e...
- xiphias2 4y agoYou have both at home / work?
- minxomat 4y agoA laptop and a desktop (Mac Studio)
- cjbprime 4y agoI was wondering if Apple Silicon would be uniquely suited for high-GPU-RAM tasks because it shares memory across the system. But I guess in this case it's a CPU model, so that's unrelated. Is that right? Do you think you could run these models on GPU instead?
- raihansaputra 4y agoI think it's about the quantization process and loading the model. I needed just above 64GB of RAM + Swap to quantize the 35B model to int4. Not to mention much slower inference time.
- montebicyclelo 4y agoHmm, wonder if the whole model is being loaded into memory for the quantisation, and whether that's necessary. (Also, shame the models can't be distributed in 4bit, (due to the license).)
- raihansaputra 4y agoYeah it's basically wholly loaded to memory for quantisation. Some optimization should be possible, and yes the quantized models can't be distributed sadly.
- eternalban 4y agoMy intuition for this is that a high dimensional vector with 16bits per dimension reduced to 4bits would be ok if these are effectively low cardinality dimensions and even with 4 bits things can be distinctly mapped into the vector space. It should break down on high cardinality dimensions (say things which are only distinguished in a few significant dimensions & rest being effectively zerovalue). In otherwords, given a vector of say 12288 dimensions (GPT), a 4-bit dimension, if vectors were uniformly distributed in the embedding space, is a choice space of 16^12288. That's in 4 bits. The 16-bit space is huge. I think serious errors will crop up only if we're looking at items that cluster in a small subset 'd' of those 12288 dimensions. So at some small d, 16^d will result in vector collisions for certain type or category of inputs.
- aledalgrande 4y agoWhat's the best read on quantization of a model? Thx
- akira2501 4y ago> and still have coherent inference results. With language being as expressive as it is, I'm not sure why people consider highly subjective measure of "coherence" to be a selling point? You can generate random text and get semi-coherent sentences a surprising amount of the time. Why not accuracy?
- zamadatix 4y agoYou probably want more than one measure for a meaningful conclusion. I'm not sure how you can generate random text and get semi-coherent sentences though. Even bending that to allow markov chains and you get clearly incoherent thoughts. On the flip side it's possible to program in perfect clarity and accuracy but it's hard to do that without making the output feel like you're talking with a spreadsheet instead of having a conversation.
- zoba 4y agoI was able to get this running on an 8gb M2. It’s a lot faster than I expected. It’s not lighting fast, but, it is moderately usable.
- tmalsburg2 4y agoYeah, 63ms/token (number from the post) is totally usable. Didn't the CPU version that was recently posted take 8min/token? But I think that was a larger model.
- aledalgrande 4y agoUnless I'm misunderstanding, it would be usable for tests/dev but not for production at that speed. If you feed in a complex prompt of 500 tokens it would take way too long.
- shad0wca7 4y agoSince the quantization requires more ram than running it, why can’t the quantized models be uploaded for use by those with 16-32gb?
- sp332 4y agoIt's against the license terms, but seeing how quickly the weights were leaked in the first place, I wouldn't be surprised if a torrent shows up with quantized ones soon.
- MacsHeadroom 4y agoThey can and they have been look on HuggingFace or search torrent sites. You only actually need about 30GB of VRAM (or unified memory) and no ram to run the largest 65B model.
- seydor 4y agoIs nobody running this on a PC?
- pletnes 4y agoThe mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs are capable of running LLMs.
- zamadatix 4y agoThe article and linked GitHub project both say this runs on the CPU. The GitHub project notes the quantization doesn’t work right with other systems for some reason but if you have enough RAM to quantize it you have enough RAM to run it without quantizing it anyways.
- pletnes 4y agoAha. Then I guess performance is the only issue. I guess it should run fine on Intel.
- washadjeffmad 4y agoThis is a Mac thread. Virtually everyone is using a PC. I test between a HEDT Linux system and a recent Intel build on Windows 11. Took me around 8 hours to build and deploy 4bit. 7B and 13B worked great, still working on the quantized 30B weights.
- lerela 4y agoThe FP16 7B version runs on my Ubuntu XPS with 32GB memory, ~300ms/token. 13B also works but results aren't really good (the model will loop after a few sentences) so parameters probably need tuning. So far I'm unable to reliably generate outputs in a different language than English, the model will very quickly start to translate (even if it's not asked) or just switch to English.
- sourcecodeplz 4y agoOMG, IT IS happening! Wish I had the machinery to play with this sooo much...
- code51 4y agoWhy is nobody commenting about the quality of these models? I totally understand that quantization is decreasing quality and capabilities a bit but I haven't seen anybody verifying the claim: LLaMA 13B > GPT-3. I was expecting LLaMA 65B to be as coherent as GPT-3 but LLaMA 65B (when run quantized) seems to think 2012 is in the future.
- greggh 4y agoLLaMA doesn't have any RHLF, human filtering / reinforcement training or any of the extra stuff that ChatGPT does. So the people saying that the LLaMA models are as good as GPT-3 are correct, but anyone saying it's as good as ChatGPT might be wrong. The real issue here is that LLaMA has to have a lot more input and prompt engineering to get good answers. If you want it to know the correct year while answering you, you have to tell it that. "The current year is 2023, some prompt here..."
- pixl97 4y agoHmmm, rather than teach it the current year, on some questions like the date in particular implementing tooling to call 'date' and "intelligently" parse the year out of it seems better, as the current year is not a static value.
- sp332 4y agoWhat prompt were you using to turn GPT-3 into a chat bot?
- deleted 4y ago[deleted]
- gwern 4y agoAt least one issue seems to be that the hyperparameters may be quite different from what you'd assume from the OA Playground: https://twitter.com/theshawwn/status/1632569215348531201 https://twitter.com/theshawwn/status/1632569215348531201
- code51 4y ago
- deleted 4y ago[deleted]
- winstonprivacy 4y agoI happen to have a couple of 16 core Xeons available with 128GB RAM and a decent high end Radeon GPU. Could I run a decent text transform model on this?
- sp332 4y agoIt's not as starightforward, because the M1 has a unified memory model while the PC will have to copy data between system RAM and VRAM. But it should be possible, see FlexGen for example.
- cjbprime 4y agoIt is straightforward for this code, which is CPU-only.
- sp332 4y agoWithin the last couple of hours, AVX support was added to the repo. Should be a lot faster on your Intel CPUs.
- netruk44 4y agoI wonder if someone is working on LoRA training for LLaMA? Maybe someone could fine tune an Instruct-LLaMA.
- underlines 4y agoI would suggest helping open-assistant.io intsead of wasting resources for LLaMA, which has very restrictive ToS and thus cripples it's full potential for the open source community. open-assistant is in the process of creating a crowd based fine tuning data set for instructions/assistant, and then select a truly open source model, not LLaMA with it's restrictive ToS.
- mxby7e 4y agohttps://github.com/tloen/alpaca-lora https://github.com/tloen/alpaca-lora
- 19h 4y agoCurrently running 65B on my 96GB M2 Max.. it's pretty good.
- sheepscreek 4y agoNice. I didn’t know Pros could be bumped above 64GB. What did that setup set you back?
- 19h 4y agoGermany -- 5.529,00 € excl. Apple Care which is 149,99 €/Year
- wincy 4y agoFor the super high provisioned laptops it seems like Applecare is a steal.
- satvikpendem 4y agoMeanwhile I just bought 64 GB RAM to try the 65B model on my desktop for like 150 bucks, lol
- snek_case 4y agoWhat seems unclear to me is, does this use the onboard GPU or neural accelerator at all, or is it all CPU-based? Cool if it's able to run 100% CPU-based because that makes portability and deployment a lot easier. Makes this code a lot more accessible.
- ggerganov 4y agoIt currently uses only the CPU via ARM NEON intrinsics - no GPU, no ANE and no Apple Accelerate. Plan is in the future to utilize respective SIMD intrinsics for other architectures (AVX, WASM SIMD, etc) and also add other more accurate quantization approaches. It's actually not a lot of work and I have most of the stuff ready, so hopefully soon! Edit: AVX2 support has just been added
- mft_ 4y agoTotal tangent, but this is something that inrigued me in recent days, after struggling with a similar problem: > You also need Python 3 - I used Python 3.10, after finding that 3.11 didn't work because there was no torch wheel for it yet. Is there a solid reason for denying forwards compatability by default in this manner? I'd faaaar from an expert on the inner workings of Python, but how often does a incrementally newer version (e.g. 3.11 over 3.10) introduce changes which would break module installation or function? I can see that in some cases, a module taking advantage of newly-introduced features would require limiting backwards compatability, but the opposite?
- SnowflakeOnIce 4y agoIt's more an issue of torch not yet providing prebuilt binary wheels for Python 3.11. You could probably get it working, but it would involve building torch from source, which can be rather more involved than 'pip install'. It's a packaging/release engineering limitation more than a language incompatibility thing.
- IanCal 4y agoAlso using an environment with a different version of python is very easy so it'd be my first solution too.
- mft_ 4y agoAgain (with little understanding) I'd ask a similar question: is there such a huge difference under the skin between Python 3.10 and 3.11 to render a prebuilt binary (torch or otherwise) for 3.10 incompatible with 3.11, on an otherwise identical platform?
- SnowflakeOnIce 4y agoThe C API for Python changes between major Python versions. This means that native-code extensions (like torch) need to be build specifically for each version of Python they want to support.
- boesboes 4y ago7B & 13B can be ran on a M1 Air with 16G memory: 7B uses about 4.5G max & runs at 203.38 ms per token, 13B about 8G and does 396.58 ms per token. 30B needs about 20G and basically hangs due to swapping i guess with 16G.
- arvinsim 4y agoJust the comment that I was looking for. Guess I can run it on my M1 Pro 32GB model
- habibur 4y agoHow does a 7b model compare with 170b parameter GPT3? Worth it?
- taf2 4y agoFunny I'm doing this this morning too and was just thinking I should have bought that 96 GB memory option instead of thinking who will ever need more than 64GB on a laptop... darn
- simonw 4y agoThanks to this commit from 8 hours ago: https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6eb56678fdee4c09219ddb85b640 https://github.com/ggerganov/llama.cpp/commit/007a8f6f459c6e... I have now successfully run the 13B model too! I updated my TIL with details: https://til.simonwillison.net/llms/llama-7b-m2#user-content-running-13b https://til.simonwillison.net/llms/llama-7b-m2#user-content-... 13B is the model that Facebook claim is competitive with original GPT3: > LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B
- mrtksn 4y ago13B runs fine on my M1 Air with 16GB Ram and 8C8G. It's not even stressing out the memory and the swap as much as Stable Diffusion. I'm getting like 350-450ms per token and it feels as fast as ChatGPT on a busy day. This obviously isn't using the Neural Engine. With Apple's Stable Diffusion implementation, when the neural engine is used I can see how my CPU and GPU stays mostly idle and the temp on the Neural Engine cores is rising but it is rising significantly less than when run on the GPU or the CPU. I wonder of it's not possible to have this run on the Neural Engine? Given that it's mostly idle, running this locally will only impact the RAM use and on a machine with a large RAM it might feel like doesn't have a performance hit and run continuously for various task.
- yc-kraln 4y agoany chance to get this to run on Radeon hardware? starting to feel a bit miffed
- MacsHeadroom 4y agohttps://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ has been running 4bit LLaMA on Radeon hardware with faster speeds and lower memory requirements than this LLaMA.cpp for the past week!