17 ms·
How Is LLaMa.cpp Possible?
- Havoc 3y agoWhat I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.
- gct 3y agoBasically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.
- KerrAvon 3y agoFirst, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.
- Waterluvian 3y agoHumans just do spicy autocomplete too.
- goatlover 3y agoNo they don't. You're "just" doing what everyone else in the past has done with the brain/human intelligence and using the latest technology as a metaphor without realizing it.
- Waterluvian 3y agoWe want to think we’re exceptional but all we can do is say “human consciousness is special” without having any way of measuring it or disproving the assertion that we’re just really fancy pattern matchers. Take any metaphor you want, it’s the same outcome: we may all be philosophical zombies.
- roarcher 3y agoWe may "just" be neural networks that run on meat instead of silicon, but that does not mean that we're LLMs.
- Waterluvian 3y agoWhy doesn’t it?
- marginalia_nu 3y agoIt's a formal logical error. One does not follow from the other without affirming the consequent.
- roarcher 3y agoBecause not all neural networks are LLMs. A GAN is a neural network, does that make it an LLM?
- hkt 3y agoWe have inputs other than words, for a start
- goatlover 3y agoI’m conscious, maybe you’re not? Not that I really believe that. I think you probably see colors and hear sounds, even in your dreams! But engineering types tend to be persuaded by a particular view of the world, failing to understand that it is a view and not nature itself.
- not2b 3y agoNo, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.
- Waterluvian 3y agoSo a human is different because it keeps training its neural network?
- eurekin 3y agoWhoa, imagine you get a good base LLM model and save all conversations with it. Run a batch process every night to fine tune a LORA on convo dataset. If I ever came across such a chat bot I'd probably freak out as to why it remembers things outside of the context window, without summarisation
- MichaelZuo 3y agoThat's a pretty neat idea, I would be surprised if no one is already working on that.
- catchnear4321 3y agomonkeys make monkeys accidentally. monkeys make meseeks on purpose. there is a difference, but will it be fun?
- Waterluvian 3y agoI am 100% invested in how much ridiculous fun this era is going to be. Right up until the moment when it becomes a horror.
- gerdesj 3y agoReally? I can guess at what spicy autocomplete might actually mean but I doubt a LLM ... OK ChatGPT did a pretty good job of it (I've just asked it), whilst sidestepping the definition of spicy in this context. It is after all a very good next word guesser, given a context, and its not ... me! I am capable of hallucinating but it was 30 odd years ago since I hunted out certain fungi on Dartmoor, or smoked hemp. To be fair, we humans do often interrupt each other to second guess a sentence completion. Done correctly it is a brief satisfying collaboration. Done wrong ... I've been married for 18 years and know when to bite my tongue, but I still get it wrong from time to time - sometimes deliberately. Despite that, me and the wiff can autocomplete each other's sentences with uncanny accuracy and end up with perfect harmony or a cough slight disagreement as a result. We are getting some phenomenal slide rules these days but the darleks are not going to be flying up the stairwell just yet, nor will SkyNet be taking over tomorrow. That said, you just know that some noddy is trying to sell a nuclear "deterrent" LLM AI thingie somewhere. Thankfully, production military equipment takes quite a while to get to deployment. There is a good chance that we will get to grips with all this stuff before SkyNet is let loose for real 8)
- enterprise_cog 3y agoMaybe you do. Luckily, not everyone is quite so simple.
- ggm 3y agoThe irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence.
- coolspot 3y ago> the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence. Can say the same about a half of population, tbh
- binary132 3y agoSo what you’re saying is.... we’re going to have human-level AI, and it’s going to be incredibly stupid
- bugglebeetle 3y agoBAAGI (Below Average Artificial General Intelligence).
- djmips 3y agoOr more simply not AGI but AGS
- glimshe 3y agoWhy do you say they do nothing to advance AGI? Do you know what it takes to advance AGI? It's hard to state that without knowing how AGI would work yourself. LLMs would be considered magic just a couple years ago. Sure, not AGI but behaves just like one for certain workloads. I find hard to believe we're not a bit closer now - or maybe even a lot closer.
- csjh 3y agoIMO the direction we're going seems more like having a few small models in a MoE that are equivalent to a current 200bn model
- Aerroon 3y agoA 200b 4-bit quantized model could potentially fit into 128 GB of RAM. The inference would just be really slow. Ie you could technically run something like that today. I think more VRAM on GPUs isn't necessarily a technical limitation either. I think GPU manufacturers could add a lot more VRAM to their cards if they wanted to. The question is whether it would be worth the price increase.
- rootusrootus 3y ago> Ie you could technically run something like that today. Yep, on higher end machines it should already be feasible. I can do 2.5-3 tok/sec on a 70B model quantized at 4 bit today with my MacBook Pro M2 MAX w/96GB. It's a little slower than a 30B, but the difference is less than I had guessed it would be. That's not super fast, but it's usable. And that's on a machine that isn't designed for this workload. Over the next few years things should improve quite a bit. 200B does not seem like a reach.
- eurekin 3y agoAbout the RAM. I doubt they wanted to do that, since basic gpu function is to render a frame in as little ms as possible. Currently VRAM is latency optimized on consumer gpus and all memory chips are an inch away from the gpu. Light only travels as far in the gigahertz realm. Thats why they started mounting vram chips on both sides of the board, cause there was no more place left on the first side. Just checked: light travels 30cm in one nanosecond. So if the gpu is running at 4GHz it goes only 7.5 cm.
- Mertax 3y agoAnd then things like neural implants and BCIs -- seems like your dog could have language capabilities sooner than you'd think ;)
- TMWNN 3y agoBah. We still haven't equaled the rude and hateful AI achieved in a microcomputer in 1981. <https://scp-wiki.wikidot.com/scp-079 https://scp-wiki.wikidot.com/scp-079>
- RosanaAnaDana 3y agoWe can keep reaching for that rainbow.
- ggm 3y agoA good analogy: as you approach the rainbow moves off. Others see you in it, but you can confirm it's somewhere else. It's an effect, a side effect, it's pretty and we value it. There's no pot of gold in literal sense, it's ephemeral value in other products.
- __loam 3y agoWill be interesting to see what people can do with local models, particularly for open source programming tools and PCG models for video games.
- cratermoon 3y agoThey'll probably write fanfic and some Harry Potter ships.
- _kava 3y agoIn case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of inference only. Training is a whole other beast and requires much more power. Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter. Bandwidth and memory size are probably, in my ignorance mind, easier to increase than raw compute so maybe we will soon actually have "smart" devices.
- ripvanwinkle 3y agoThank you! Is there a sweet spot with quantization. how much can you quantize for given model type and size and still be useful.
- pseudonom- 3y agoTim Dettmers recently (https://www.manifold1.com/episodes/ai-on-your-phone-tim-dettmers-on-quantization-of-neural-networks-41/transcript https://www.manifold1.com/episodes/ai-on-your-phone-tim-dett...): "But what we found with these neural networks is, if you use 32 bits, they're just fine. And then you use 16 bits, and they're just fine. And then with eight bits, you need to use a couple of tricks and then it's just fine. And now we find if you can go to four bits, and for some networks, that's much easier. For some networks, it's much more difficult, but then you need a couple more tricks. And so it seems they're much more robust."
- KirillPanov 3y ago> And now we find if you can go to four bits That will be really interesting for FPGAs, because the current ones are basically oceans of 4-bit computers. Yes, you can gang together a pair of 4LUTs to make a 5LUT, and a pair of 5LUTs to make a 6LUT, but you halve your parallelism each time you do that. OTOH you can't turn a 4LUT into a pair of 3LUTs on any currently-manufactured FPGA. It's simply the "quantum unit" of currently-available hardware -- and it's been that way for at least 15 years (Altera had 3LUTs back in the 2000s). There's no fundamental reason for the number 4 -- but it is a very, very deep local minimum for the current (non-AI) customers of FPGA vendors.
- jmorgan 3y agoThis project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has a pretty extensive "server" tool (in its examples directory in the repo) with a web ui and support for grammar (e.g. forcing the output to be JSON)
- jay-barronville 3y agoThank you for the link to your project! I'll be playing around with it.
- jcims 3y agoJust wanted to say thanks homie for making something so hairy so accessible. Love it!!!
- wfurney 3y agoI've also been experimenting with the C# implementation https://github.com/trrahul/llama2.cs https://github.com/trrahul/llama2.cs
- pjmlp 3y agoWhy Parallel.For() instead of SIMD types? I would expect a much better performance.
- lovelyviking 3y ago[flagged]
- oblio 3y agoBecause. On a more serious node, your questions seem very... aggressive. Especially the one about the name, is the name offensive or what? To me it sounds quite benign.
- cameron_b 3y agoI’ve been working through that repo and managed the 13B dataset on a single Pi4 8gig I’ve also replicated the work in OpenMPI ( from a thread on the llama.cpp GitHub repo ) and today I managed to get the 65B dataset operational on three pi4 nodes. I’m not saying this as any achievement of mine, but as a comment on the current reality of reproducible LLM At home on anything you’ve got. It really feels like this technique has arrived. https://github.com/cameronbunce/ClusterConfig https://github.com/cameronbunce/ClusterConfig
- eurekin 3y agoHow many tokens a second?
- cameron_b 3y agoThe other way around is whole number math. I added the 3-node output from the 13B model to github, the timings are below. The 3-node 65B job hasn't finished yet. llama_print_timings: load time = 17766.29 ms llama_print_timings: sample time = 264.42 ms / 128 runs ( 2.07 ms per token, 484.07 tokens per second) llama_print_timings: prompt eval time = 10146.71 ms / 8 tokens ( 1268.34 ms per token, 0.79 tokens per second) llama_print_timings: eval time = 287157.12 ms / 127 runs ( 2261.08 ms per token, 0.44 tokens per second) llama_print_timings: total time = 297598.22 ms
- loxias 3y ago> I’ve also replicated the work in OpenMPI... Oh cool! How did it perform? I wonder if this would be an exciting test for Amazon's SRD protocol which appears to be built for HPC. I'm looking for an excuse to play with it...
- Animats 3y agoIs this single-thread? Or are they putting all available CPUs on the problem?
- sharms 3y agoThe problem is memory bandwidth rather than CPU cores: "Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. Anything that reduces the memory requirements for these models makes them much easier to serve"
- brucethemoose2 3y agoIts complicated. If you dont have a GPU, prompt ingestion is totally threaded. The more cores the better. For generating tokens, more cores helps to a point, but then: - You start saturating the memory bus, and performance plateaus. - There is some overhead from the threading implementation, and too many threads hurts performance. The ideal number of threads varys per CPU. For instance, using hyperthreaded cores or Apple/Intel e cores typically hurts performance... But not always. You just have to test and see.
- regularfry 3y agoIt's multithreaded.
- tysam_and 3y agoIn my best estimation, Finbarr makes pretty great content, he and I have had a number of positive interactions on Twitter. I tend to have a pretty grumpy disposition towards a lot of modern ML and such as I feel it's shovelware, but whenever Finbarr puts out work, I tend to set aside some time to give it a good gander, as I feel like it's generally pretty "meaty" (which I honestly find pretty hard to do past a certain pace). Well worth the subscribe if you have not done so already (I'm not affiliated with him, I just really like his work!).
- gautamcgoel 3y agoI enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units. Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a modern GPU can perform by a factor of 2 and not notice any appreciable difference in ML performance.
- zamadatix 3y agoThe primary use case for GPUs in devices, graphics, is not as often memory bound. Even for other general data compute purposes that may not always be the case. It's specifically neural nets that have extremely wide batches of extremely simple operations occurring across extremely large chunks of memory where being memory bound is the default case.
- reissbaker 3y ago1. I think nanosecond-scale latency numbers on operations taking dozens to hundreds of ms are probably overkill? 2. Inference is only one aspect of what GPUs are used for. Many other workloads are compute-bound. That being said, given the recent rise of these kinds of open-source, pre-trained large language models, I wouldn't be surprised if future Nvidia product launches offered variants with significantly more VRAM. There would probably be a lot of interest in "3080-equivalent compute, but 48GB VRAM" these days — certainly I would take one over a 4090 with 24GB VRAM. (Then again, that's basically an A6000, and those go for nearly $7k...)
- sbierwagen 3y agoYeah, Nvidia won't do big VRAM consumer cards until AMD forces them to. They're running flat out just trying to keep up with demand for H100s at forty thousand USD each.
- ColonelPhantom 3y agoOr Intel! They're not making any high-end cards currently, but the A770 16GB is pretty decent if you're on a budget. The software support isn't really amazing yet, but their GPUs have some quite decent matmul acceleration. Unfortunately support for their GPUs is not upstream in Tensorflow or Pytorch, and I don't think they're well-supported by Llama.cpp either, but Intel does look quite promising. I also believe that Intel oneMKL works on Arc GPUs these days, which in CPU land is amongst the fastest BLAS libraries out there. Also their matrix hardware is accessible using OpenCL extensions, which means that rolling a custom kernel for things related to quantization should be quite possible. (For Tensorflow and PyTorch, you need to install a custom package called Intel Extension for $FRAMEWORK. The PyTorch one got updated to PyTorch 2.0 recently, which is promising.) Currently rumors seem to indicate that their second generation GPUs will go from 32 to 64 Xe cores for the top model, and keep the 256 bit bus. If Intel were to double the VRAM to 32 GB as well (at least as an option, just like the A770 comes in both 8GB and 16GB variants), it'd immediately make them the crown of consumer VRAM size, which I'd wager would drive a lot of interest from the ML community.
- thomasfromcdnjs 3y agoThank you very much for writing this out!
- fnbr 3y agoYou’re welcome :)
- oars 3y agoGreat article. Don't see content like this anywhere else outside of HN.
- redox99 3y agoYou can find content like this on Twitter if you follow the right people. In fact I read this article before it was even posted here because @karpathy tweeted about it.
- notuseful 3y ago[flagged]
- Joe_Boogz 3y agoShameful? Just because you dislike it doesn’t mean everyone has to.
- PKop 3y agoSays you. I'd prefer to take advantage of the resources available there. Your silly moralism is easy to ignore
- DiggyJohnson 3y agoI’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”
- class4behavior 3y agoIt's betraying the word shameful in that it's an utter understatement. If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem. To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.
- m3kw9 3y agoIs there a Llama2 65b quantized version for Mac M2?
- eurekin 3y agoI'd look for llama2 thebloke 70b GGML on hugginface. Update, this might work: https://huggingface.co/TheBloke/Llama-2-70B-GGML https://huggingface.co/TheBloke/Llama-2-70B-GGML Example command mentions the 4bit variant: ./main -m llama-2-70b.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "Llamas are"
- ionwake 3y agoI also found this link ( https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-apple-silicon-mac-c3f46143f327 https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-a...) but havent verified if it helps, but thanks for your reply I am taking a look now.
- eurekin 3y agoJust checked on a m1 w 32GB ram. 7b and 13b work as a charm. 70b doesn't fit, even the 2 bit version
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- pedrovhb 3y agoGiven the massive imbalance in the memory bandwidth bottleneck, I wonder why specialized hardware is the way it is. Is there some use case in which processing is the bottleneck, or at least it's more even? Are we expecting some software paradigm shift which will change the balance? Why couldn't they just make a cheaper, more rounded card which isn't heavily underutilized because of a large bottleneck?
- brucethemoose2 3y agoSee: https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_wall https://en.m.wikipedia.org/wiki/Random-access_memory#Memory_... But llama.cpp, and llms in general, are very atypically memory bound, even for AI workloads. Other models/backends will typically make more use of compute and cache. For llms specifically, cloud hosts will batch requests to make better use of GPUs. But you are not far off. There is a proposition to pipe chips with very fast memory (aka no external memory at all) together: https://www.nextplatform.com/2023/07/12/microsofts-chiplet-cloud-to-bring-the-cost-of-llms-way-down/ https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c...
- PeterisP 3y agoIMHO the current bottleneck is driven by the partial overlap between the gaming GPU market and machine learning needs. The expensive things that are needed by both (i.e. parallel calculations) are included in all cards, but expensive things that aren't needed by games (i.e. memory bandwidth) become a bottleneck unless you use expensive ML-specific niche hardware instead of consumer (gamer) GPUs.
- andrewstuart 3y ago>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....
- BobbyJo 3y agoMost CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.
- winwang 3y agoFor reference, DDR4-3200 in quad channel is ~100 GB/s while a 3090's VRAM is 960 GB/s. Of course, most consumers only have dual channel. M1 Pro is 200 and M1 Max is 400. Which is slow for GPU memory, but incredible for main memory -- although I'm not sure how much of that a single core can actually pull.
- KolenCh 3y agoThe Anand tech article has profiled this. IIRC the CPU cores has access to half that band width only, which is still quite a lot.
- andrewstuart 3y agoThe AMD Ryzen™ 9 7940HS uses DDR5-5600, which I understand to be about 89.6GB/s in a dual channel setup.
- mhh__ 3y agoThis is basically the appeal of the apple chips in this domain. Apple have fuck-you money so they have a bunch of high-bandwidth decent-latency soldered onto the chip.
- fulafel 3y ago
- ggm 3y ago80/20 rule. Approximations have smaller, less accurate approximates which do in many contexts. If your goal is to reach America, a crude compass and speed reasoning works. If you want to target an ICBM you need better positional accuracy.
- kitanata 3y ago[flagged]
- brucethemoose2 3y agoThis leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but is still massively smaller. - Batching is a big thing to ignore outside of personal deployments. - The real magic of llama.cpp is model splitting. A small discrete GPU can completely offload prompt ingestion and part of the model inference. And it doesn't have to be an Nvidia GPU! There is no other backend that will do that so efficiently in the generative AI space. - Hence the GPU backends (OpenCL, Metal, CUDA, soon ROCm and Vulkan) are the defacto way to run llama.cpp these days. Without them, I couldn't even run 70B on my desktop, or 33B on my (16GB RAM) laptop.
- PoignardAzur 3y agoWhat's prompt ingestion?
- brucethemoose2 3y agoIts the phase where llama.cpp processeses the input, as opposed to the phase where its generating the response. Speed is particularly important. The response can be streamed word by word, so it doesn't have to be particularly fast, but slow input processing leads to very noticable latency.
- deleted 3y ago[deleted]
- ant6n 3y agoWhat’s perplexity?
- regularfry 3y agoPerplexity is a measure of how certain the model is of the next token. It's calculated by looking at the probabilities that the model calculates for the next token in a stream. If there are several choices for the next token with similar probabilities, that's telling you that the model is having a hard time telling what the right answer should be: the model is more perplexed, perplexity is higher. If there's a single option with a much higher implied probability than any other, that means the model is more certain, and perplexity is lower. Note that this has nothing to do with whether the answer is objectively correct. It's just measuring how confident it is.
- kitanata 3y ago[flagged]
- kitanata 3y ago[flagged]
- gdiamos 3y agoMemory bound token generation is a limitation of transformer decoders. In the past, hardware has motivation algorithm innovations. I’m curious how long it will take until we see more hardware friendly models.
- marmaduke 3y agoThe Rwkv family of models qualifies, since it computes like a recurrent network at inference time.
- fouc 3y agoDid anyone notice that apparently a M2 Macbook Pro is only 16x faster than a Pixel 5? Not sure that makes sense.
- sharkjacobs 3y agoI’m not an expert in this but my “does that feel right wrong” sense isn’t going off Pixel 5 is 2-3 years old. CPUs aren’t doubling in speed every 2 years, but let’s very generously say we expect current designs to be 4x faster than 2-3 year old equivalent. Apple silicon is faster than other ARM chips, so if we imagine that’s another 2x we’re up to 8x Common wisdom is that “real computers” are faster than phones, but the difference between the A16 and M2 is less than 2x for multi core, and much much less for single core benchmarks. Rounding up, another 2x is 16x Maybe there are characteristics of the two devices which make this more surprising, I’d be interested to learn more.
- astrange 3y agoThe phone slows down as it heats up, and then slows down infinitely when the battery dies. (With some workloads and chargers, it's possible running on a phone would drain the battery faster than it can recharge even if you plugged it in.)
- Roark66 3y agoIt is useful to mention running inference on modern cpus that have AVX2 is not that bad. Sure it is slower than on the gpu, but you get the benefit of having a single long continuous region of ram. But there is one huge problem why this is not that popular on x86_64. Having to run in fp32. As far as I know our most common ml libraries (pytorch, tf, onnx etc) do not have an option to quantize to 4 bits and they don't have an option to run inference at anything other than fp32 on the x86_64 cpus. It is a huge shame. There is openvino which supports int8, but if you can't easily quantize large models without a gpu, what use is it? (For small models I suppose). So if anyone figured out a way to quantize a transformer model to 4/8 bit and run it on the x86_64 cpu platform I'm very interested in hearing about it.
- _nalply 3y agoSubjective experience: AVX512 helps a lot. I would have liked to read more about this. It seems that AVX512 supports fp16 in hardware and allows 32 fused multiplication-add operations per core. So I imagine on a Ryzen 9 with 12 cores you can have 384 simultaneous fused multiplication-add operations. I am not sure whether my estimation is off. Anyone know more than me?
- gliptic 3y agoSorry, but the topic of this post, llama.cpp, runs quantized 4/8 bit models just fine on x86_64 with AVX2, or am I missing some requirement you have?
- Roark66 3y agoWait, I wasn't aware llama.cpp even runs on x86_64.i thought it is arm hw only. If what you say is correct that indeed is very interesting. Especially if I can extend it to other models like falcon.
- gliptic 3y agoIt doesn't support Falcon right now, but there's a fork that does (https://github.com/cmp-nct/ggllm.cpp/ https://github.com/cmp-nct/ggllm.cpp/).
- aargh_aargh 3y agoOriginally posted here: https://news.ycombinator.com/item?id=35192102 https://news.ycombinator.com/item?id=35192102
- ant6n 3y agoThis article would probably be useful for a lot more people if it spent just a couple of sentences introducing the various parameters, rather than just throwing variable names at the reader. Interestingly, a whole paragraph is spent on explaining n_bytes.
- djmips 3y agoAnd BTW this analysis holds true for many workloads. Memory bandwidth is often at the core of performance.
- ionwake 3y agoDoes anyone know what the next breakthroughs will be and their rough timelines regarding locally run models? For instance will anything like chat gpt 4 be runnable on an M1 Mac within the next year?
- loudmax 3y ago"Breakthroughs" are inherently hard to predict. This field is advancing at a very rapid pace. A lot of the improvements are incremental, but even these are coming very fast, and moving in different directions at once. I don't think there's any likelihood of replicating GPT-4 on your M1 in the next twelve months, especially not if you're expecting responsive performance. What we could see are a plethora of models dedicated to doing particular tasks. Say, specialists in aspects of programming or accounting or law or medicine. Or general knowledge models with access to a local cache of Wikipedia. Individually, none of these models would have to come close to GPT-4's overall level of power and flexibility. But collectively, they could reach that level of utility.
- SleekEagle 3y agoFantastic article - I would love to see an analysis comparing larger quantized models to smaller unquantized models. e.g. is a 14b quantized model better than a 7b unquantized model?
- lhl 3y agoBasically, the most aggressive quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 Tim Dettmers did a bunch of research on this last year as well: https://arxiv.org/abs/2212.09720 https://arxiv.org/abs/2212.09720
- fulafel 3y agoAre there benchmark figures for CPU based setups somewhere? Eg a 192-core / 24-channel 2P Zen 4 box. Or is the CPU side too unturned to be interesting?
- simne 3y agoI'm shocked - before I seen it working myself, I was not believe, so large (smart) LLM could run on desktop cpu. - core i7, 4 cores, 3.3GHz, 64G ddr4-2400, 70b model, ~0.5 tokens/s; 30b model ~1 tokens/s.