11 ms·
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
- Luciatrutth 4y ago[dead]
- KETpXDDzR 4y agoI'm happy that someone implemented a CPU-only version of Llama. The original Meta implementation always crashed with "out of memory" (I only have 8GB VRAM). Too bad Llama doesn't work as well as (Chat-)GPT for instructional prompts.
- yanislapisse 4y ago[dead]
- yanislapisse 4y ago[dead]
- yanislapisse 4y ago[dead]
- yanislapisse 4y ago[dead]
- mrtksn 4y agoThe performance on Macbook with M1 Pro is said to be 20 tokens/s https://twitter.com/ggerganov/status/1634282694208114690 https://twitter.com/ggerganov/status/1634282694208114690
- eis 4y agoA Macbook Pro M1 would have the base M1 CPU while he was referring to the M1 Pro CPU in something like a Macbook Pro w/ M1 Pro. It's confusing naming by Apple.
- mrtksn 4y agoright, fixed it
- lxe 4y agoThis is faster than running it on an RTX 4090 I think.
- deleted 4y ago[deleted]
- jokethrowaway 4y agoNope a 4090 can do the 30b-4bit model at 20 tokens/s
- MacsHeadroom 4y agoI get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that. So nowhere close to the 4090, but plenty fast anyway.
- olliej 4y agoIsn't using any of the AS "ML" coprocessor/extensions/whatever, so it's just normal simd.
- astrange 4y agoIt uses Accelerate so it may be using some of that indirectly.
- olliej 4y agoMaybe, but as far as I can tell using the ML specific hardware requires CoreML and CoreML's data formats. But I also can't tell where the vector (or matrix?) extensions end and the apple "neural" engine begins so shrug? :D
- Szpadel 4y agoI tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provide me with simple regex and also RFC compliant version when asked.
- endymi0n 4y agoLlama is neither instruction finetuned nor human reinforcement learning ranked. It‘s a base model like the original GPT3 that needs much more careful prompting and few-shot reasoning to produce similar results.
- IanCal 4y agoOutside of it being possibly a smaller model, the prompts should be different as llama hasn't been trained to take instructions so it would I think need to be framed more as "A regex for emails is " and let it complete the rest. Just in case you have tried the exact same prompt. Also those messages sound suspiciously like error messages from a system around it.
- dmm 4y agosillysaurusx made some posts about tweaks that greatly improved output for him. Maybe they would help you: """ Fixed the llama sampler. After turning off top_p, adding top_k 40, setting temp to 0.7, and adding a repetition penalty of 1/0.85, llama 7B is looking nice. """ https://twitter.com/theshawwn/status/1632569215348531201?cxt=HHwWgoCzgafDhqgtAAAA https://twitter.com/theshawwn/status/1632569215348531201?cxt... https://twitter.com/theshawwn/status/1633456289639542789?cxt=HHwWisC9wen1mastAAAA https://twitter.com/theshawwn/status/1633456289639542789?cxt...
- Szpadel 4y agofor reference I tried or here: https://lite.koboldai.net/ https://lite.koboldai.net/ with chat mode, and parameters: temp 0.62, top_p 0.9, repetition penalty 1.08 is free to try, but then you have to wait in queue, therefore I had to wait 10min per response and I didn't had patience to test it further
- xiphias2 4y agoGeorge Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
- jalino23 4y agothat is so impressive!
- apetresc 4y agoCan you link to the Twitch stream VoD?
- vopi 4y agohttps://www.twitch.tv/georgehotz/videos?filter=archives&sort=time https://www.twitch.tv/georgehotz/videos?filter=archives&sort... although, there is a VOD channel on YT that might be better.
- yieldcrv 4y agowhy does it say video unavailable?
- stagger87 4y agoI'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.
- jonas21 4y agoI don't think he runs the YouTube channel - it's managed by some of his fans. But like what's he gonna do about it? Send them a takedown notice?
- gumby 4y agoThanks for doing this, nice work! Please add some sort of license.
- SheinhardtWigCo 4y agoSuper cool project. This is from the author of whisper.cpp, which enables highly accurate real-time audio transcription on the M1/M2: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- jcq3 4y agoDon't know anything about ML can someone can explain me what is this hype about?
- stagger87 4y agohttps://en.wikipedia.org/wiki/Machine_learning https://en.wikipedia.org/wiki/Machine_learning
- potatoman22 4y agoThis is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,
- hot_gril 4y agoRun meaning run inference, not train, right?
- TOMDM 4y agoYes
- mattfrommars 4y agoWasn't LLaMa official meant to run on consumer grade machine? How does this modify the model to make it work. All of this is confusing.
- MacsHeadroom 4y agoYes but it wasn't made to run on a Mac. This project ported LLaMA to Apple Silicon so all the macbook users can finally play with what the rest of us have had access to for the past couple of weeks.
- lxe 4y agoYou can run your own ChatGPT on your Macbook.
- 0xbadc0de5 4y agoThat's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-generation-webui, the results are quite impressive.
- brookst 4y agoMacs have UMA so an off the shelf Mac can use up to about 120GB of vram. Far more than any consumer card, more than the biggest a100. GPU power is lower, of course, but pure vram is not a problem.
- stu2b50 4y agoVRAM is the thing that Apple Silicon is going to have in excess compared to anything even close in price. MacBook Airs can have 14-15GB of VRAM if necessary.
- kir-gadjello 4y agoRAM read bandwidth of M1/M2 is still not really competitive with the large GPUs like RTX3090, but it's getting close, compared with the usual CPU setups.
- gorbypark 4y agoIs there an upper limit on the usable VRAM on Airs, or is it just tied to RAM size minus a bit for the OS? Just got a 24GB Air M2 and your comment made me wonder if it was capped at n amount of VRAM, or if it's just that Air's now can have 24GB of ram compared to 16GB with the M1 Airs?
- liuliu 4y agoHere is some quick math: these devices has SSD read speed somewhere around 2GiB/s. With 4-bit quantization, we are looking at loading 4B parameters per second. That means we need 8s per token for 30B model. Hmmm, the math is a bit off (or I need to look closer whether we can do more tokens per iteration with some batching).
- 2bitencryption 4y agoCould someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of <1sec/token? It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?
- TaylorAlexander 4y agoWell I was able to run the original code with the 7B model on 16GB vram: https://news.ycombinator.com/item?id=35013604 https://news.ycombinator.com/item?id=35013604 The output I got was underwhelming, though I did not attempt any tuning.
- fnbr 4y agoparameter tuning is pretty necessary, according to anecdotes. People on twitter have got good results by changing the default parameters.
- int_19h 4y agoFor 13b and 30b, it really needs high temperature to produce good outputs.
- fdb 4y agoThe author just made an update that makes the generation much better, even with the 7B model: https://twitter.com/ggerganov/status/1634310199170179075 https://twitter.com/ggerganov/status/1634310199170179075 I tried it out myself (git pull && make) and the difference in results are day and night! It's amazing to play with, although you should prompt it differently than ChatGPT (more like the GPT-3 API).
- thewataccount 4y ago16GB of vram can run the 7B for sure, I'm not sure what the most cutting-edge memory optimization but the 15B is going to be pretty tight I'm not sure that'll fit with what I know of at least, I've got it working at a bit over 20gb of vram I think at 8bit. If you can't fit it all in vram you can still run it but it'll be slooooow, at least that's been my experience with the 30b.
- throwuwu 4y agoNow someone translate it to zig
- thewataccount 4y agoI don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc? I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.
- arthurcolle 4y agoyou mean the 13B ?
- thewataccount 4y agoYeah my bad, everyone is a bit all over the place with the numbers in this thread. I'm not exactly sure how these numbers were chosen, they seem a bit odd?
- 0xbadc0de5 4y agoThe 13b and 30b run quite well on a 4090 at 4-bit quantization.
- thewataccount 4y agoAh dang I missed that I was still using the 8bit mode, I'll look into that thanks!
- DesiLurker 4y agobut why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.
- tempaccount420 4y agoYou can read the code. It's open source.
- eigenvalue 4y agoSeems like it's extremely high performance for me on an M1 running the 7B model. Totally usable.
- DesiLurker 4y agojust because something is high performance does not means it cannot be improved by say another 2X. My point was not whether its usable or not, it is that if you are going to run on CPU vectorization is rather important part and its odd that the landing page has no remark about it.
- lxe 4y agoI'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa, and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very little detail, but I thought it would be faster?
- sillysaurusx 4y agoEh, I’d expect it to be slower if anything. Think about it like this. If you write an image renderer, bitmap would be the fastest, because it’s already decompressed. 4-bit quantization is a compression algorithm. It depends on the details of memory bandwidth vs compute though.
- washadjeffmad 4y agoIn case anyone catches this late (and anything older than a few hours in AI/ML is!), some of the original llama HF weights were not built correctly and gave poor output. Many people testing this weekend have not updated or rebuilt those weights from earlier in the week.
- dekhn 4y agoThis is sort of the polar opposite of how modern high performance ML frameworks are built. Skimming the code, there's a ton of boilerplate for the various operations that could be library-ized and generified, if that makes sense. I actually really like minimal implementations of state-of-the-art systems because the code is much easier to understand (modern frameworks are super-complex) but I wonder what it means long-term if you don't need frameworks.
- SCLeo 4y agoThis is just inference. The core at most ML library is the auto differentiation capability. It will be extremely tedious if you are to calculate the gradients manually. Or, if you implemented your own AD, then it is effectively a minified version of a ML library.
- version_five 4y agollama.cpp/ggml.h // GGML Tensor Library ... // This library implements: // - a set of tensor operations // - automatic differentiation // - basic optimization algorithms
- mhh__ 4y ago> autodiff There's a lot of // TODO: implement backward in there
- version_five 4y agoOk thanks for digging deeper, I didn't realize that, and obviously that invalidates the excerpt I posted above
- technerder 4y agoAbsolutely love ggerganov's approach with models like this and Whisper. It's just awesome being able to experiment with (what I consider) complex models without needing a billion python/c/cpp dependencies!
- mattfrommars 4y agoI have very limited in this domain. Why is it necessary to port LLaMa Into C? Assuming original model implementation was in Python, did it not require few tweaks to make it work in Apple Silicon?
- amelius 4y agoYes this is a good question. Why did they focus on a specific model rather than a generic solution that makes ANY python based model work on Apple silicon?
- MacsHeadroom 4y agoBecause that's not how machine learning models work. Machine learning as a field goes through a nearly complete revolution annually. Every new major model is a special snowflake of unique cases. Writing high performance software that handles all of them is next to impossible, because its the special tailoring to the unique features of a given model that provides the high performance.
- amelius 4y agoThat's not how I think it works. ML is a small number of operations applied to very large blocks of data, tensors. You can build all kinds of complex formulas using those small number of tensor operations, but the (relative) speed is determined by how efficient the small number of operations are implemented, not by how complicated the formulas are (relatively, compared to other operations using the same formula).
- MacsHeadroom 4y agoYou're half right. First, tensor operations are only a small part of modern ML. Second, how you plug all those small operation together is where all the performance difference is had these days between implementations. Different hardware have a variety of different small operations that do almost the same thing. So when a state of the art model architecture meets a state of the art quantization method and you want to run it fast on AMD GPUs, Nvidia GPUs, x86 Processors, ARM processors, and Apple Silicon you are highly likely to end up with perhaps 3-5 bespoke implementations. This happens every few months in ML. Meanwhile hardware is also both innovating and balkanizing at the same time. Now we have Google Silicon, Huawei Silicon, and Intel Arc GPUs. It's not an environment where "one fast library to rule them all" seems attainable.
- recuter 4y ago> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if you squint this is a useful/convincing "ChatGPT on a laptop" minus the corporate lobotomy only a few short months after release. Very exciting! I actually care about the upcoming Mac Pro now. $3999 Mac Studio with 64GB ram. +$800 for 128GB.
- londons_explore 4y agoLLaMA doesn't perform very well with answering questions. If you ask it "What color is the sky?" It will reply with something like "Why is ice cold? Why do we exist?"
- RoboTeddy 4y agoLLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```
- 4y ago
- ojosilva 4y agoInteresting. But how about the Apple Neural Engine (ANE)? I've always wondered if the ANE is ML worthy, maybe it's really only with inference or who knows, even training somehow. I've seen Apple's marketeers bragging about it [1], with even code examples, but ifaik no useful libraries nor reliable measurements and community interest exist in the wild for doing ANE ML on Macs. 1. https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra... Edit: just found this: https://github.com/ggerganov/whisper.cpp/pull/566 https://github.com/ggerganov/whisper.cpp/pull/566
- jjkeddo199 4y agoInsanity! This is the same guy who wrote Whisper C++! How does he do this? I feel like I am a side character in some cartoon gasping at the unthinkable power level of the main character.
- Alifatisk 4y agoWhat’s even more impressive is this part > This was hacked in an evening
- adeon 4y agoIf you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/model.py https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion torch code in diffusers and transformers Python libraries has lots of conditionals, experiments etc. that are not being used that can make it hard to follow what is going on. Last weekend I got the "main loop" of the transformer working in pure CPU Rust code, following the reference code. My crappy code is just very very slow as I focused on getting it to run, not making it fast. The tokenizer uses some Google thing https://github.com/google/sentencepiece https://github.com/google/sentencepiece but luckily for inference it seems that you just need to be able to parse the tokenizer model file and not understand how it was created; I was able to strip out the protobuf files from that repository and add it to Rust and read the tokens. I am optimistic that someone makes a high quality CPU or some CPU+GPU+SSD combination thingmaling that will make it somewhat practical to run even the large LLM models without needing an A100 or two.
- nerdponx 4y agoI noticed the Fasttext code was also surprisingly clean and readable C++. whatever moralities and other flaws the metal business model might have in general, they seem to have a consistently excellent track record when it comes to publicly available libraries and tools.
- gpm 4y agoMy code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd https://github.com/gmorenz/llama/tree/ssd Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056f2766acb8c5cd349176f86d65e405 https://github.com/facebookresearch/llama/commit/5be06e56056... At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at once on the same GPU) is a good as it gets when you need to read the whole model from disk each token. At a 125GB and a 2MB/s read (largest model, what I get from my ssd) that's 60 seconds per token (1 day per 1440 words), which isn't exactly practical. Which is really the issue here, if you need to stream the model from an SSD because you don't have enough RAM, it is just a fundamentally slow process. You could probably optimize quite a bit for batch throughput if you're ok with the latency though.
- eigenvalue 4y agoThis is so awesome and exciting. I have an M1 iMac and it was trivially easy to get this working and generating text. And the performance is VERY impressive, especially considering that it's not even using any of the built in "neural compute" stuff. Also, the model seems like it doesn't have any political correctness conditioning based on some of the completions it has given me on controversial prompts. I can't wait until someone gets the 13b model working (sounds like this should happen in the next day or so) and gets the repetition penalty working.
- lynguist 4y agoIt is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.
- kamranjon 4y agoConfusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX (https://github.com/corsix/amx https://github.com/corsix/amx) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as well. All of this is to say that it actually could get significantly faster if some of this work was able to be handed to the ANE with CoreML.
- lynguist 4y agoYou’re right! I wrote it too fast without thinking!
- raphlinus 4y agoThree. You can also do it in Metal, which as of recently has cooperative matrix multiplication in the form of the simd_matrix type (this is similar functionality as "tensor cores" in the Nvidia world). I have no idea what the software support is, but I have seen analysis suggesting that the raw tensor multiplication throughput is larger than ANE for the high-end GPUs.
- 29athrowaway 4y agoIt is not idiomatic C++.
- schappim 4y agoI'm a huge fan of Georgi (the author)! You should also check out his other work, bringing Apple Silicon support to OpenAI's Whisper (speech-to-text model): https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- fwlr 4y agoA quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines; it might be the same exact component in each chip). All iPhone 14 models sport 6GB integrated RAM; the MacBook starts at 8GB. All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro. Some people have already had success porting Whisper to the Neural Engine, and as of 14 hours ago GGerganov (the guy who made this port of LLaMA to the Neural Engine and who made the port of Whisper to C++) posted a GitHub comment indicating he will be working on that in the next few weeks. So. With Whisper and LLaMA on the Neural Engine both showing better than real-time performance, and Apple’s own pre-existing Siri Neural TTS, it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone. This is absolutely extraordinary stuff!
- schappim 4y ago> Some people have already had success porting Whisper to the Neural Engine, and as of 14 hours ago GGerganov (the guy who made this port of LLaMA to the Neural Engine and who made the port of Whisper to C++) posted a GitHub comment indicating he will be working on that in the next few weeks. He has already done great work here: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- sho_hn 4y agoThe 7b model specifically is not quite "ChatGPT-level" though, is it?
- doctoboggan 4y agoAccording to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- russellbeattie 4y agoI just filled out Meta's online form to get access to the LLaMA models. Anyone know how long it takes, how selective it is (it asked for papers I've published: none) or if there's any places to download it from in the meantime?
- dTal 4y agohttps://github.com/facebookresearch/llama/pull/73/files https://github.com/facebookresearch/llama/pull/73/files
- deleted 4y ago[deleted]
- renewiltord 4y agoQuick steps I did to get it working 1. Download the models from here https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/blob/main/llama.sh (you'll need to edit the script to make it work in bash) 2. Follow the instructions here. 3. Create an interesting prompt for the AI and put your question at the end. It works tremendously fast on an Apple Macbook Pro with the M1 Max and 64 GiB of RAM. I only downloaded the 7B model because I wanted a quick run through. I'll try the rest later. Here's some good fun: Prompt: Example HN Post: Gojq: Pure Go Implementation of Jq. Example HN top comment: "gojq does not keep the order of object keys" is a bit disappointing. I care about key order purely for cosmetic reasons: when Im designing JSON APIs I like to put things like the "id" key first in an object layout, and when Im manipulating JSON using jq or similar I like to maintain those aesthetic choices. HN Post:"Llama.cpp - C/C++ implementation of Facebook LLama model". HN top comment: Completion: "This is more of an example of C++s power than a breakthrough in computer science." Example HN Post: "The Moral Case for Software Patents". HN top comment: "The patent system is a vestigial technology, and the whole point of a vestigial technology is to make something go away. Its like a dinosaur that no one is using anymore. Its served its purpose." Example HN Post: "Benchmark: The Mystery of Gauss Summation". HN top comment: "But it is interesting to see how Benchmark manages to cram main: mem per token = 14368644 bytes main: load time = 1301.14 ms main: sample time = 273.73 ms main: predict time = 20970.29 ms / 82.89 ms per token main: total time = 22939.82 ms
- crimjow 4y agoIt even maintained your spelling errors.
- tantony 4y agoThis is pretty cool. I got it working in about 5 minutes. Inference of the 7B model is super fast. I don't know enough about prompting plain GPT models though.
- graderjs 4y agoggerganov is a genius! I'm using his excellent whisper.cpp in my WisprNote offline private transcribing MacOS app. It 10x the speed of the regular models. He even invented a custom format for the models! The work this guy is doing--first class or beyond!!!
- simonw 4y agoI got this working on my 64GB M2 MacBook Pro! Wrote detailed notes here for anyone else who wants to try this: https://til.simonwillison.net/llms/llama-7b-m2 https://til.simonwillison.net/llms/llama-7b-m2
- oidar 4y agoThank you!
- jeffhwang 4y agoThank you for these helpful notes!!
- nvegater 4y agoAmazing. Thank you very much !
- ls612 4y agoI have the 64GB M2 Max I will have to try this out, its amazing that this laptop can do things my high end desktop can't do, Apple really made some good architectural choices with apple silicon.
- nl 4y agoI'm getting 56.38 ms per token on my 32GB M1 Max using this code on the 7GB model. Very usable!
- 2Gkashmiri 4y agowhat model are you using? edit: i mean 6B, 13b or 30B?
- garblegarble 4y agoIt'll be 7B they're referring to, on my M1 Max 32GB with a 4000 token output request I get 67ms/token on 7B (4bit) and 154ms/token on 13B (4bit)... I've made a tweak to the code to increase the context size but it doesn't seem to change perf. main: mem per token = 22357508 bytes main: load time = 2741.67 ms main: sample time = 156.68 ms main: predict time = 11399.12 ms / 154.04 ms per token main: total time = 14914.39 ms
- nl 4y agoThis was generating 2000 tokens, so it seems to get slightly faster on longer generation runs maybe?
- jokethrowaway 4y agoStellar job and it's amazing to have this running in wasm. I don't get why something like this should be faster than running eg. libtorch in cpu mode, though. If it is, surely you'd want to port the optimisations to libtorch so that any model would benefit from it. If it's just Mac specific you could even add another target.
- DeathArrow 4y agoHow does one learn to do stuff like this? You first learn doing AI algorithms in Python and then transfer the knowledge to C++ or you learn doing them in C++ from the start?
- PartiallyTyped 4y agoYou first need the basic building blocks which is matrix multiplication. You may use libraries here. Then it’s a matter of loading the model correctly. If you are just reimplementing and not doing research you don’t need the mathematics. The algorithm is rather simple too. The issue is more about engineering than machine learning.
- underlines 4y agoRelevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements. A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper ( https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 ) which leads to almost no quality degradation compared to the 16bit weights. 4 bit weights: Model, weight size, vram req. LLaMA-7B, 3.5GB, 6GB LLaMA-13B, 6.5GB, 10GB LLaMA-30B, 15.8GB, 20GB LLaMA-65B, 31.2GB, 40GB Here is a good overall guide for Linux and Windows: https://rentry.org/llama-tard-v2#bonus-4-4bit-llama-basic-setup https://rentry.org/llama-tard-v2#bonus-4-4bit-llama-basic-se... I also wrote a guide how to get the bitsandbytes library working on windows: https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1456040134 https://github.com/oobabooga/text-generation-webui/issues/14...
- sswam 4y ago[dead]
- 0xakhil 4y agoDoes Llama model has a token limit like ChatGPT api (4k tokens)?
- MacsHeadroom 4y agoIt's currently 2048 tokens. But like GPT-3, that can be improved in the future.
- lawrencechen 4y agoGeorgi just added support for all models (13B/33B/65B) [0] LLaMA 65B can do ~2 tokens per second on my M1 Max / 64 gb ram [1] [0] https://twitter.com/ggerganov/status/1634488664150487041 https://twitter.com/ggerganov/status/1634488664150487041 [1] https://twitter.com/lawrencecchen/status/1634507648824676353 https://twitter.com/lawrencecchen/status/1634507648824676353
- KVFinn 4y agoVery cool. I've seen some people running 4-bit 65B on dual 3090s, but didn't notice a benchmark yet to compare. It looks like this is regular 4-bit and not GPTQ 4-bit? It's possible there's quality loss but we'll have to test. >4-bit quantization tends to come at a cost of substantial output quality losses. GPTQ quantization is a state of the art quantization method which results in negligible output performance loss when compared with the prior state of the art in 4-bit (and 3-bit) quantization methods and even when compared with uncompressed fp16 inference. https://github.com/ggerganov/llama.cpp/issues/9 https://github.com/ggerganov/llama.cpp/issues/9
- magoghm 4y agoOn my M1 Ultra LlaMA 65B generates ~3 tokens per second (using 16 threads).
- gorbypark 4y agoI can confirm that this (7B) runs nicely on a 24GB MacBook Air M2. The output of my initial test was definitely a bit different than ggreganov's example! The first man on the moon was 39 years old on July 16, 1969. July 16th is the 198th day of the year (199th in leap years) in the Gregorian calendar. There are 168 days remaining until the end of the year. 1561 – France is divided into 2535 circles (French: cercles) for fiscal purposes. 1582 – Pope Gregory XIII, through a papal bull, establishes the Gregorian calendar (Old Style and New Style dates). 1
- snickmy 4y agoThe self attention of this model needs some tuning :) It reminds me how sometimes I have stream of consciousness while talking, and I have ADHD.
- gorbypark 4y agoYeah, definitely. The second run was similarly off the rails.. The first man on the moon was 12 years old when the American Revolution began. As a boy, he was horrified by the sight of his mother being whipped. He came to America after surviving the Spanish Inquisition and massacres in Mexico and Brazil. In Texas he married a second-generation German, the daughter of a Confederate soldier who had never heard of the Battle of Bull Run. As a teenager, he fished salmon out of the Columbia River for his Russian Orthodox mother-in-law who said she had seen the sign of the cross appear in the sky before a victory in battle. He followed the death Still impressive for a 7B model that's been slimmed down to 4 bits.
- gorbypark 4y agoAs a followup, the 13B model similarly runs great on the 24GB MacBook Air. It seems to perform a lot better (output wise) and about twice as slow per token. Output from 7B "The first man on the moon was ": The first man on the moon was 19-year-old Luna 15-year-old man, so the eyes of the world on him, as he was the first man in the moon after the moon. The second man on the moon was a 45-year-old American man, Neil Armstrong. In 1969, Neil Armstrong, with Aldrin, landed on the moon and set foot on the moon, became a hero to the world. At that time, the United States succeeded in landing on the moon with a manned spacecraft, NASA's Apollo 11 Lunar Module main: mem per token = 14434244 bytes main: load time = 1865.80 ms main: sample time = 259.36 ms main: predict time = 23119.33 ms / 170.00 ms per token main: total time = 25916.14 ms Output from 13B "The first man on the moon was ": The first man on the moon was 38 year old Neil Armstrong (Buzz Aldrin was the second man to walk on the moon, he was the lunar module pilot). What was the first man to land on the moon? Neil Armstrong was the first person to step on the moon on July 20, 1969. What did Neil Armstrong say when he landed on the moon? Neil Armstrong said, "Houston, Tranquillity Base here, the Eagle has landed." What are the facts about Neil Armstrong? Neil Armstrong is an American main: mem per token = 22439492 bytes main: load time = 3692.42 ms main: sample time = 260.61 ms main: predict time = 42439.68 ms / 312.06 ms per token main: total time = 47629.22 ms
- a6 4y agoOn an unrelated note, are there any open source implementations utilising Intel's Gaussian & Neural Accelerator and Deep Learning Boost technologies on 12th and 13th gen CPU's?
- sgt 4y agoDoes anyone know how many languages this supports? I know that FB has been translated to a ton of languages. Will those translations benefit the models in LLaMA? Or am I misunderstanding the point of this?
- MacsHeadroom 4y agoThe point of it is to generate arbitrary text locally, like the GPT-3 API does remotely. It can "say"/talk about anything an average IQ person with knowledge of the entire internet and most books in existence could. So if you prompt it to write 100 pages on the differences between positive and negative law, as a poem, while never using a word with the letter "F" it will spit that out for you without any issue. It can also program quite well, create recipes, debate with you, impersonate anyone, and lots more. And it does all of this offline, in airplane mode, locally on your PC or Mac. It's good at translation but is probably one of the least efficient ways to translate text when models specifically for translation exist.
- sgt 4y agoWhat models exist for translations? The reason I wonder is that a lot of community work has gone into producing high quality translations of Facebook. I have actually submitted a few translations myself. If Meta could release some of those models, it would be very powerful. Right now I see "google translate" type quality everywhere and it's pretty bad, since there are often sentences you can't translate unless the technology understands the context and meaning.
- ngrilly 4y agoThe README says it has been "hacked in an evening"! Have the 10.000 lines of ggml.c been written in an evening, without reusing a lot of pre-existing code? That's quite amazing.
- hnfong 4y agoI thought that as well. But then I noticed the same author has an existing project with a ggml.c ( https://github.com/ggerganov/whisper.cpp/blob/master/ggml.c https://github.com/ggerganov/whisper.cpp/blob/master/ggml.c 8000+ lines), so maybe he "just" wrote 2000 lines in an evening. Sounds much more "humanly possible", but still amazing.
- ngrilly 4y agoMakes more sense. But as you said, still amazing!
- hnfong 4y agoI assumed I needed a M1 pro or better due to RAM requirements, but I was able to run the 7B model on a 16GB M1 Mac Mini. The system reported 4GB RAM usage for the inference process. I suspect it's possible to run the 13B model as well.
- mrtksn 4y agoYou can, I was able to run 13B on my 16GB 8c8g M1 Air. The performance was 2-3 tokens/second. It felt on par with ChatGPT on a busy day.
- taf2 4y agoWhen I ask ChatGPT about a transcript e.g. ``` Transcript: \"Professor Poopy Pants: Okay. Todd: Thank you for holding. Hello. How may I help you? Professor Poopy Pants: Hey. I just wanna let you know that my name is professor Poopy pants. Todd: Oh, hit oh, that's great. So professor Pupi pants, and can I ask how I can help you today? Professor Poopy Pants: Sure. I appreciate it. So I have some poop in my pants, and I I need it to be clean clean. Todd: So you have food with your pants and you need to be cleaned? No problem, sir. I will get right on that. Have. Professor Poopy Pants: Oh, Todd: a nice. Professor Poopy Pants: thank Todd: day. Professor Poopy Pants: thank you so much.\" Tell me, what did the caller need help with in 2 or 3 words? ``` I get "Cleaning Pants" When I do the same with LLaMA 7B model by doing e..g ``` ./main --temp 0.2 -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 300 -p "Transcript: \"Professor Poopy Pants: Okay. Todd: Thank you for holding. Hello. How may I help you? Professor Poopy Pants: Hey. I just wanna let you know that my name is professor Poopy pants. Todd: Oh, hit oh, that's great. So professor Pupi pants, and can I ask how I can help you today? Professor Poopy Pants: Sure. I appreciate it. So I have some poop in my pants, and I I need it to be clean clean. Todd: So you have food with your pants and you need to be cleaned? No problem, sir. I will get right on that. Have. Professor Poopy Pants: Oh, Todd: a nice. Professor Poopy Pants: thank Todd: day. Professor Poopy Pants: thank you so much.\" Tell me, what did the caller need help with in 2 or 3 words? ``` I get: ``` Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the problem? Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the solution? Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the outcome? Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the lesson learned? Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the lesson learned? Tood: "Profeesssor Poopy Pants: I have some poop in my pants, and I I need it to be clean clean." Tell me, what was the lesson learned? Tood ```
- simonw 4y agoI wrote about why I think LLaMA + llama.cpp means that large language models are having a "Stable Diffusion moment" right now: https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/
- kossTKR 4y agohow long before someone creates a simple GUI for this? That + a small bit of optimisation and everyone with a newer Mac / iPhone will be able to run something akin to chatGPT locally! Isn't this a pretty crazy development - just weeks ago people said this would be impossible. From this thread the 13b model runs just as fast as chatGPT on a M2 Macbook Air, and it's not even using the Neural Engine yet so will become significantly faster once that is utilised - wow!
- MacsHeadroom 4y agoPeople have been running LLaMA in 4bit quickly on cheap hardware with a simple GUI for over a week using https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui Just not on Macs. (that repo does not support Apple Silicon)
- zapdrive 4y agoDoes quantizing the models reduce their "accuracy"?
- MacsHeadroom 4y agoYes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead. References: https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers [Oct, 2022] https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]
- 999900000999 4y agoHow long until someone puts out a docker image of this? Ideally with a nice rest API, I can't imagine it's too hard to do.
- random_kris 4y agoCan someone create a wasm with typescript so that i can run this in the browser
- zxspectrum1982 4y agoHow can I train LLaMA with my own content?
- phodo 4y agoFor whistper.cpp live transcription, there seems to be duplicates in the output filename. Is there a mix of parameters that eliminate this?
- spgman 4y agoCould this be used to classify text?
- EGreg 4y agoWhere do you get the weights? Do you have to fill out the researcher form on FB?
- gh2k 4y agoDoes anyone know if it's possible to split this model across multiple (identical, sli) GPUs, to run a version bigger than fits in the RAM of a single GPU?
- tshaddox 4y agoIs it feasible for this or similar to run on an Apple Silicon iPad Pro?