25 ms·
Llama.cpp 30B runs with only 6GB of RAM now
- brucethemoose2 4y agoDoes that also mean 6GB VRAM? And does that include Alpaca models like this? https://huggingface.co/elinas/alpaca-30b-lora-int4 https://huggingface.co/elinas/alpaca-30b-lora-int4
- terafo 4y agoNo(llama.cpp is cpu-only) and no(you need to requantize the model).
- sp332 4y agoAccording to https://mobile.twitter.com/JustineTunney/status/1641902010198036480 https://mobile.twitter.com/JustineTunney/status/164190201019... you can probably use the conversion tools from the repo on Alpaca and get the same result. If you want to run larger Alpaca models on a low VRAM GPU, try FlexGen. I think https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ is one of the easier ways to get that going.
- brucethemoose2 4y agoYeah, or deepspeed presumably. Maybe torch.compile too. I dunno why I thought llama.cpp would support gpus. shrug
- sp332 4y agoLots of C++ programs use the GPU. It's irrelevant.
- KingMachiavelli 4y agoI think those specific Alpaca models are all in safetensor now and there isn't simple converter to ggml.
- sp332 4y agoLate edit: Deep Speed, not FlexGen. I don't know if FG could work, but that repo only supports it for Opt models.
- deleted 4y ago[deleted]
- ducktective 4y agoI wonder if Georgi or jart use GPT in their programming and design. I guess the training data was lacking for the sort of stuff they do due to their field of work especially jart.
- jart 4y agoNot yet. GPT-4 helped answer some questions I had about the WIN32 API but that's the most use I've gotten out of it so far. I'd love for it to be able to help me more, and GPT-4 is absolutely 10x better than GPT 3.5. But it's just not strong enough at the kinds of coding I do that it can give me something that I won't want to change completely. They should just train a ChatJustine on my code.
- detrites 4y agoThe pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.
- datadeft 4y agoI have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
- terafo 4y agoYou mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.
- datadeft 4y agoI was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.
- rickrollin 4y agoPeople have actually ran it on phones.
- datadeft 4y agoThis is what my comment is about. It happened much sooner than I thought.
- politician 4y agoRoughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.
- jart 4y agoAuthor here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
- 0xakhil 4y agoSome OS’s zram compress the unpinned pages instead of swapping to disk. It might be faster than fetching the pages again from disk. I wonder if this is a reason why folks see different results.
- nynx 4y agoWhy is it behaving sparsely? There are only dense operations, right?
- w1nk 4y agoI also have this question, yes it should be. The forward pass should require accessing all the weights AFAIK.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- HarHarVeryFunny 4y agoFrom what I've read there's no evidence it's "behaving sparsely".. That was just offered as a suggestion why it might not be loading all the weights, but makes no sense in terms of the model. It's going to be using all the weights. Another suggestion is that not all of the word/token embedding table might be used, which would be a function of the input used to test, but that would be easy enough to disprove as there would then be different memory usage for different inputs. It seems possible the reported memory usage is lower than reality if that's how mmap/top work. In any case, a good use of mmap it seems, especially since for a multi-layer model layer weights will be used sequentially so paged load-on-demand will work relatively well even in a low memory situation.
- lukev 4y agoHas anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs. I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.
- bakkoting 4y agoSome results here: https://github.com/ggerganov/llama.cpp/discussions/406 https://github.com/ggerganov/llama.cpp/discussions/406 tl;dr quantizing the 13B model gives up about 30% of the improvement you get from moving from 7B to 13B - so quantized 13B is still much better than unquantized 7B. Similar results for the larger models.
- terafo 4y agoI wonder where such difference between llama.cpp and [1] repo comes from. F16 difference in perplexity is .3 on 7B model, which is not insignificant. ggml quirks are definitely need to be fixed. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa https://github.com/qwopqwop200/GPTQ-for-LLaMa
- bakkoting 4y agoI'd guess the GPTQ-for-LLaMa repo is using a larger context size. Poking around it looks like GPTQ-for-llama is specifying 2048 [1] vs the default 512 for llama.cpp [2]. You can just specify a longer size on the CLI for llama.cpp if you are OK with the extra memory. [1] https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8ef4024e52a3c0c5893d86db84f1b52b6/llama.py#L20 https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/934034c8e... [2] https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bdc8ec3f0e6e40c47251608700#latest-measurements https://github.com/ggerganov/llama.cpp/tree/3525899277d2e2bd...
- gliptic 4y agoGPTQ-for-LLaMa recently implemented some quantization tricks suggested by the GPTQ authors that improved 7B especially. Maybe llama.cpp hasn't been evaluated with those in place?
- arka2147483647 4y agoWhat is lama? What can it do?
- terafo 4y agoRead readme in repo.
- arka2147483647 4y agoFrom readme: - Can it run doom - Inference of LLaMA model in pure C/C++ - Plain C/C++ implementation without dependencies It really does not explain itsef to the uniniated. I infer it is some kind of language model. Why/how it differs any other impl/model, i do not know.
- sillysaurusx 4y agoOn the legal front, I’ve been working with counsel to draft a counterclaim to Meta’s DMCA against llama-dl. (GPT-4 is surprisingly capable, but I’m talking to a few attorneys: https://twitter.com/theshawwn/status/1641841064800600070?s=61&t=jQbmCk1JqL7depzFWJNuPA https://twitter.com/theshawwn/status/1641841064800600070?s=6...) An anonymous HN user named L pledged $200k for llama-dl’s legal defense: https://twitter.com/theshawwn/status/1641804013791215619?s=61&t=jQbmCk1JqL7depzFWJNuPA https://twitter.com/theshawwn/status/1641804013791215619?s=6... This may not seem like much vs Meta, but it’s enough to get the issue into the court system where it can be settled. The tweet chain has the details. The takeaway for you is that you’ll soon be able to use LLaMA without worrying that Facebook will knock you offline for it. (I wouldn’t push your luck by trying to use it for commercial purposes though.) Past discussion: https://news.ycombinator.com/item?id=35288415 https://news.ycombinator.com/item?id=35288415 I’d also like to take this opportunity to thank all of the researchers at MetaAI for their tremendous work. It’s because of them that we have access to such a wonderful model in the first place. They have no say over the legal side of things. One day we’ll all come together again, and this will just be a small speedbump in the rear view mirror. EDIT: Please do me a favor and skip ahead to this comment: https://news.ycombinator.com/item?id=35393615 https://news.ycombinator.com/item?id=35393615 It's from jart, the author of the PR the submission points to. I really had no idea that this was a de facto Show HN, and it's terribly rude to post my comment in that context. I only meant to reassure everyone that they can freely hack on llama, not make a huge splash and detract from their moment on HN. (I feel awful about that; it's wonderful to be featured on HN, and no one should have to share their spotlight when it's a Show HN. Apologies.)
- terafo 4y agoWish you all luck in the world. We need much more clarity in legal status of these models.
- sillysaurusx 4y agoThanks! HN is pretty magical. I think they saw https://news.ycombinator.com/item?id=35288534 https://news.ycombinator.com/item?id=35288534 and decided to fund it. I’m grateful for the opportunity to help protect open source projects such as this one. It will at least give Huggingface a basis to resist DMCAs in the short term.
- bsaul 4y agohow is llama performance relative to chatgpt ? is it as good as chatgpt3 or even 4 ?
- terafo 4y agoIt is as good as GPT-3 at most sizes. Instruct layer needs to be put on top in order for it to compete with GPT 3.5(which powers ChatGPT). It can be done with comparatively little amount of compute(couple hundred bucks worth of compute for small models, I'd assume low thousands for 65B).
- stavros 4y agoThat's surprising to read, given that ChatGPT (at least the first version) was much worse than text-davinci-003 at following instructions. The new version seems to be much better, though.
- weird-eye-issue 4y agoNo, it wasn't
- UncleOxidant 4y agoWhat's the difference between llama.cpp and alpaca.cpp?
- cubefox 4y agoI assume the former is just the foundation model (which only predicts text) while the latter is instruction tuned.
- deleted 4y ago[deleted]
- cubefox 4y agoI don't understand. I thought each parameter was 16 bit (two bytes) which would predict minimally 60GB of RAM for a 30 billion parameter model. Not 6GB.
- heap_perms 4y agoI was thinking something similar. Turns out that you don't need all the weights for any given prompt. > LLaMA 30B appears to be a sparse model. While there's 20GB of weights, depending on your prompt I suppose only a small portion of that needs to be used at evaluation time [...] Found the answer from the author of this amazing pull request: https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu...
- xsmasher 4y agoDoes this mean LLaMA only uses 10% of it's brain? An urban legend come to life!
- w1nk 4y agoNo, your OP is mistaken. The model weights have to all be accessed for the forward pass. What has happened is that using mmap changes where the memory is consumed (kernel vs process) and so it was being incorrectly interpreted. There are still 30B parameters, and you'll need that times however big your floating point representation is to use the model still.
- xsmasher 4y agoBut do they all need to be accessed at the same time? If not, pages that are not being actively used can be dropped from memory until needed again.
- gamegoblin 4y agoParameters have been quantized down to 4 bits per parameter, and not all parameters are needed at the same time.
- qwertox 4y agoIs the 30B model clearly better than the 7B? I played with Pi3141/alpaca-lora-7B-ggml two days ago and it was super disappointing. In percentage between 0% = alpaca-lora-7B-ggml and 100% GPT-3.5, where would LLaMA 30B be positioned?
- Rzor 4y agoI haven't been able to run it myself yet, but according to what I read so far from people who did, the 30B model is where the "magic" starts to happen.
- sp332 4y agoCheck out the graph on page 3 of this PDF: https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 The 33B model started beating the 7B when it had been trained on only 1/3 as much data. And then they kept training it to 40% more than the total that 7B was trained on. It's better.
- victor96 4y agoLess memory than most Electron apps!
- terafo 4y agoWith all my dislike to Electron, I struggle to remember even one Electron app that managed to use 6 gigs.
- mrtksn 4y agoI've seen WhatsApp doing it. It start with 1.5G anyway, so after some images and stuff it inflates quite a lot.
- baobabKoodaa 4y agoI assume it was a joke
- xiphias2 4y agoMaybe they should do more mmap as well.
- joshbochu 4y agoLMAO
- TaylorAlexander 4y agoGreat to see this advancing! I’m curious if anyone knows what the best repo is for running this stuff on an Nvidia GPU with 16GB vram. I ran the official repo with the leaked weights and the best I could run was the 7B parameter model. I’m curious if people have found ways to fit the larger models on such a system.
- terafo 4y agoI'd assume that 33B model should fit with this(only repo that I know of that implements SparseGPT and GPTQ for LLaMa), I, personally, haven't tried though. But you can try your luck https://github.com/lachlansneff/sparsellama https://github.com/lachlansneff/sparsellama
- enlyth 4y agohttps://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- TaylorAlexander 4y agolooks great thank you!
- w1nk 4y agoDoes anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)? From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data. Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
- matsemann 4y agoMaybe lots of the data is embedding values or tokenizer stuff, where a single prompt uses a fraction of those values. And then the rest of the model is quite small.
- w1nk 4y agoThat shouldn't be the case. 30B is a number that directly represents the size of the model, not the size of the other components.
- jhatemyjob 4y agoIf you read a file with malloc and memcpy, it copies the data from the kernel to userspace. With mmap there is no copying.
- losteric 4y agoyeah, I believe some readers are misinterpreting the report. The OS manages mmap, it won't show up as "regular" memory utilization because it's lazy-loaded and automatically managed. If the OS can keep the whole file in memory, it will, and it will also magically swap to disk prioritizing explicit memory allocation (malloc). Sounds like the big win is load time from the optimizations. Also, maybe llama.cpp now supports low-memory systems through mmap swapping? ... at the end of the day, 30B quantized is still 19GB...
- l33tman 4y agoIt's not a bug, but it's misreading the htop output as mmap doesn't show up as a resident set size there. The pages are RO and not dirty so it's "on the OS" to count it and the OP had lots of RAM on the computer so the model just resides in his page cache instead.
- danShumway 4y agoI messed around with 7B and 13B and they gave interesting results, although not quite consistent enough results for me to figure out what to do with them. I'm curious to try out the 30B model. Start time was also a huge issue with building anything usable, so I'm glad to see that being worked on. There's potential here, but I'm still waiting on more direct API/calling access. Context size is also a little bit of a problem. I think categorization is a potentially great use, but without additional alignment training and with the context size fairly low, I had trouble figuring out where I could make use of tagging/summarizing. So in general, as it stands I had a lot of trouble figuring out what I could personally build with this that would be genuinely useful to run locally and where it wouldn't be preferable to build a separate tool that didn't use AI at all. But I'm very excited to see it continue to get optimized; I think locally running models are very important right now.
- kossTKR 4y agoDoes this mean that we can also run the 60B model on a 16GB ram computer now? I have the M2 air and can't wait until further optimisation with the Neural Engine / multicore gpu + shared ram etc. I find it absolutely mind boggling that GPT-3.5(4?) level quality may be within reach locally on my $1500 laptop / $800 m2 mini.
- thomastjeffery 4y agoI doubt it: text size and text pattern size don't scale linearly.
- kossTKR 4y agoInteresting, i wonder how it scales.
- singularity2001 4y agoDoes that only happen with the quantized model or also with the float16 / float32 model? Is there any reason to use float models at all?
- abujazar 4y agoI love how LLMs have got the attention of proper programmers such that the Python mess is getting cleaned up.
- faitswulff 4y agoHow so?
- kccqzy 4y agoThere are two distinct kinds of jobs: ML researchers and software engineers. A lot of ML researchers write pretty bad code by software engineering standards but that's okay; it's not their job to produce clean code. They string together libraries in Python and do a lot of experimentation and analysis. When they produced something ready to be productionized, software engineers then come in and optimize things. This is totally the right way. Make it work, then make it right, then make it fast.
- Hortinstein 4y agoNot sure if you came up with "Make it work, then make it right, then make it fast." but I just screenshotted it and made it the mantra for my current project...which is by far more complicated than anything I have done as a side project. I am struggling with the desire to go "make it right" as I work on shipping the deployable prototype (right now running on cloud services I control)...thanks for this
- faitswulff 4y agohttps://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast > This formulation of this statement has been attributed to [KentBeck][0]; it has existed as part of the [UnixWay][1] for a long time. [0]: https://wiki.c2.com/?KentBeck https://wiki.c2.com/?KentBeck [1]: https://wiki.c2.com/?UnixWay https://wiki.c2.com/?UnixWay
- buzzert 4y ago
- Dwedit 4y ago> 6GB of RAM > Someone mentioning "32-bit systems" Um no, you're not mapping 6GB on RAM on a 32-bit system. The address space simply doesn't exist.
- jiggawatts 4y agoWindows Server could use up to 64 GB for a 32-bit operating system. Individual processes couldn't map more than 4 GB, but the total could be larger: https://en.wikipedia.org/wiki/Physical_Address_Extension https://en.wikipedia.org/wiki/Physical_Address_Extension
- muyuu 4y agoIt actually happened. With paging and bigger word sizes, which were common in 32 bit systems even with hardware acceleration. A 32bit address space only means you have 4GibiAddresses, which do not need to be pointing to single bytes. In fact the natural thing to do in a 32 bit system for a structure like this is moving 32bit words, which actually means you're addressing a 16GB space, flat. And then there's segmentation. For instance the 286 had a 24bit address showing for 16MB in direct addressing mode and 1GB via segmentation (what back then was usually referred to by virtual memory) The 386 had a 32 bit address width and its MMU allowed access to 64TB in virtual mode and 4GB in protected mode.This was indeed one of the reasons Linux was not made 286-compatible. Its protected mode was only 1GB and segmented rather than 4GB flat, so Linus didn't have to deal with XMS or EMS for a chip that was becoming obsolete soon anyway. But the 1GB space was there, and at the time that was plenty.
- yodsanklai 4y agoTotal noob questions. 1. How does this compare with ChatGPT3 2. Does it mean we could eventually run a system such as ChatGPT3 on a computer 3. Could LLM eventually replace Google (in the sense that answers could be correct 99.9% of the time) or is the tech inherently flawed
- addisonl 4y agoMinor correction, chatGPT uses GPT-3.5 and (most recently, if you pay $20/month) GPT-4. Their branding definitely needs some work haha. We are in track for you to be able to run something like chatGPT locally!
- nmca 4y agoworse, yes, yes, no
- retrac 4y agoThe largest LLaMA model at ~60 billion parameters is not quite as large as ChatGPT 3 in size, and probably not quite as well trained, but it's basically in the same class. Even the complete, not quantitized model, can be run with llama.cpp on ARM64 and x86_64 CPUs already, assuming you have enough RAM (128 GB?).
- simonw 4y ago"Could LLM eventually replace Google" If you try to use LLMs as a Google replacement you're going to run into problems pretty quick. LLMs are better thought of as "calculators for words" - retrieval of facts is a by-product of how they are trained, but it's not their core competence at all. LLaMA at 4bit on my laptop is around 3.9GB. There's no way you could compress all of human knowledge into less than 4GB of space. Even ChatGPT / GPT-4, though much bigger, couldn't possible contain all of the information that you might want them to contain. https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web https://www.newyorker.com/tech/annals-of-technology/chatgpt-... "ChatGPT Is a Blurry JPEG of the Web" is a neat way of thinking about that. But... it turns out you don't actually need a single LLM that contains all knowledge. What's much more interesting is a smaller LLM that has the ability to run tools - such as executing searches against larger indexes of data. That's what Bing and Google Bard do already, and it's a pattern we can implement ourselves pretty easily: https://til.simonwillison.net/llms/python-react-pattern https://til.simonwillison.net/llms/python-react-pattern The thing that excites me is the idea of having a 4GB (or 8GB or 16GB even) model on my own computer that has enough capabilities that it can operate as a personal agent, running searches, executing calculations and generally doing really useful stuff despite not containing a great deal of detailed knowledge about the world at all.
- wkat4242 4y agoWow I continue being amazed by the progress being made on language models in the scope of weeks. I didn't expect optimisations to move this quickly. Only a few weeks ago we were amazed with ChatGPT knowing it would never be something to run at home, requiring $100.000 in hardware (8xA100 card).
- smoldesu 4y agoBefore ChatGPT was in beta, there were already models that fit into 2gb and smaller. They were complete shit, but they did exist.
- wkat4242 4y agoI know but what's changing is that they aren't shit now. Not on par with GPT but getting much closer. Especially with a little massaging like Stanford has done.
- m3kw9 4y agoLooks like we can run OpenAI gpt3.5 with 32g ram?
- sp332 4y agoWhere are you going to download that from?
- operatingthetan 4y agoHey if Meta can see a leak, then a former non-profit like OpenAI definitely can.
- llui85 4y agoDifference is that Meta was "publishing" the model to researchers, and one of them leaked it. I suspect OpenAI is a little less likely to share the GPT models around unfortunately...
- dislikedtom2 4y ago1. turn the swap off or monitor it closely 2. try to load a big model, like 65b-q4 or 30b-f16 3. observe the OOM - It's not so hard to test this.
- gg82 4y agoUsing a memory mapped file doesn't use swap. The memory is backed by the file that is memory mapped!
- vrglvrglvrgl 4y ago[dead]
- macromaniac 4y agoMy guess is that there was an error during quantization that resulted in a large amount of the weights not properly being used. A potential test would be to compare the number of page faults between quantized an unquantized model and confirm they are roughly the same proportionally. This could also explain how e.g. gpt4all seem to notice better performance on unquantized weights when there really shouldn't be.
- dvt 4y agoThis seems suspiciously like a bug (either in inference or in mmap reporting), as these models are not sparse enough for the savings to come from anywhere viable.
- jacooper 4y agoStill can't run it, thanks AMD ROCm.
- arthurcolle 4y ago"How much RAM did you shave off last week?" "Oh, you know, like 12-18GB" "Haha shut the fuck up, how much RAM did you shave off last week" "12-18GB" "Let me tell you what - you show me your commits right now, if you shaved off 12-18GB of RAM last week I quit my job right now and come work for you" https://www.youtube.com/watch?v=TxHITqC5rxE https://www.youtube.com/watch?v=TxHITqC5rxE
- PragmaticPulp 4y agoMaybe not so fast. Other users are reporting that it’s not actually running properly in environments with limited RAM. The reduced memory usage might be more of a reporting misunderstanding, not an actual reduction in memory usage.
- muyuu 4y agothe README.md file of the repository is not updated to reflect these changes yet, right?
- kccqzy 4y agoI might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.
- losteric 4y agoAre the weights on NVME? Old SSD? HDD?
- alchemist1e9 4y agoIt’s interesting how NVMe will be even more critically important if this lazy weights loading approach works out. PCIe 5 has arrived just in time for LLM interference it seems.
- freehorse 4y agoWell in this case it does not have to do with SSDs, quite the opposite here a performance gain seems to happen by caching the file in RAM in the beginning.
- alchemist1e9 4y agoThat’s not my understanding. The entire point is the model can’t fit in RAM. mmap allows lazy loading from storage.
- freehorse 4y agoYes but to compute a token it has to eventually read the data, either cached in RAM or from storage. There is no way that a fast SSD can compete with RAM in terms of I/O speed. To achieve any speed benefit the whole file has to be cached in RAM. This has different benefits eg threads can share memory, and the file does not have to be reread next time it is called because it is already cached in RAM, but, in final analysis, you need to have the RAM, or then you are reading from the disk, and reading 20gb for each token means you need to read 1T for a paragraph of 50 tokens. My m1, which by no means has a slow SSD, reads the file at 500-600mb/s, while a thunderbolt pci-4 enclosure reads at 700-800mb/s, even if you double that it will still take 10-20 seconds per token. To get less than 1 second per token for the 30B model one has to read there at 20gb/s. At the time we have done that, there will be even huger (v)RAMs and even larger models.
- deleted 4y ago[deleted]
- kthakore 4y agoWhere do you download the tokenizer.model that is needed to convert the GPT4ALL model to the appropriate format?
- Rzor 4y agolesmo provides a magnet uri to all models here. it's inside (499kb): https://github.com/facebookresearch/llama/pull/73/files/016a53608c5eae1021e171b9c4f06a9783fc14c0 https://github.com/facebookresearch/llama/pull/73/files/016a...
- aledalgrande 4y agoThe real question is when can I run Jarvis
- alduin32 4y agoWell, only if you don't count the filesystem cache as "RAM". You still need enough memory so that the kernel can hold all these pages, even if Llama.cpp is not using this memory itself.
- solarpunk 4y agoI'm curious if someone will have to port these enhancements elsewhere, ie: https://github.com/rustformers/llama-rs https://github.com/rustformers/llama-rs
- ar9av 4y ago[dead]
- abhimskywalker 4y ago"The recent change also means you can run multiple LLaMA ./main processes at the same time, and they'll all share the same memory resources." So this could have a main and multiple sub-worker llm processes possibly collaborating while sharing same memory footprint?
- l33tman 4y agoYes, if the model is mmap'ed read-only (as I'm sure it is). There are other bottlenecks than CPU cores though, it might not be very useful to run multiple in parallel..
- acamboy 4y ago[flagged]
- acamboy 4y ago[flagged]
- Aardwolf 4y agoWould it in theory be possible to back up only 1 file (of several TB in size), your disk as a block device file, here? Or would this require reuploading the entire disk every time any single file inside changes rather than only the blocks that changed?
- nothrowaways 4y agoSomebody please teach ggreganov python. I am sure he will create same magic with python.
- mazbox 4y agommap saves the day again
- mazbox 4y agommap saves the day again.
- rootw0rm 4y agoi think i found a decent 30b alpaca model if anyone else is just now diving in: https://huggingface.co/Pi3141/alpaca-lora-30B-ggml/tree/main https://huggingface.co/Pi3141/alpaca-lora-30B-ggml/tree/main
- muyuu 4y agothe ggml-model-f16.bin file for 30B is taking me a bunch of hours to process are these made available somewhere?
- zikohh 4y agoHas this been backported to alpaca CPP?
- k__ 4y agoHalf-OT: How do all these models compare to each other? Are there any metrics that can tell me how much better or worse LLaMA is compared to GPT3? What does it even mean to be better?
- DoctorOetker 4y agoIs there a reason Llama is getting so much attention compared to say T5 11B? Not sure how neutral or what benchmarks are used on the following link, but T5 seems to sit a lot higher on this leaderboard? https://accubits.com/large-language-models-leaderboard/ https://accubits.com/large-language-models-leaderboard/
- itake 4y agollama can run on an m1. T5 still needs a specialized gpu
- DoctorOetker 4y agoWhat is the reason T5 needs a specialized GPU and Llama doesn't? In the end they are mathematical models, so what would prevent someone from loading T5 into a machine with plenty of RAM (like a server)? Would the codebase truly require that much refactoring? How difficult would it be to rewrite the model arhitecture as a set of mathematical equations (Einstein summation) and reimplement inference for CPU?
- itake 4y agoI'm far from an expert in this area. But llama has been updated so anyone can hack with it on their m1 macbook (which many developers have). If someone updated T5 to be as easy to dev against, then I am sure they would see similar community interest. Most people don't have the hardware or budget to access these specialized high vram GPUs.
- barbariangrunge 4y agoIs llama open source? I heard it was pirated from Facebook
- DoctorOetker 4y agoI did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics). Anyway, T5 being available for download from Huggingface only makes my question more pertinent...
- lucabs 4y ago[dead]
- Ambix 4y agoHow cool is that! There's also exciting news about early access for GPT5 13T model [1] features from Noosphere AI [1] https://noosphere.chat https://noosphere.chat
- logicallee 4y agoIt's possible to run llama.cpp on windows, e.g. see this tutorial: https://www.youtube.com/watch?v=coIj2CU5LMU https://www.youtube.com/watch?v=coIj2CU5LMU Would this version (ggerganov) work with one of those methods?
- breput 4y agoYes, the 30B model is working for me on Windows 10 / AMD 5600G CPU / 32GB RAM, with llama.cpp release master-3525899 (already one release out of date!), in PowerShell, using the Python 3.10 version that automatically installs when you type "python3". I did the following: 1. Create a new working directory. 2. git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp 3. Download the latest release from https://github.com/ggerganov/llama.cpp/releases https://github.com/ggerganov/llama.cpp/releases (note the CPU requirements in the filename) and unzip directly into the working directory's llama.cpp/ - you'll have the .exe files and .py scripts in the same directory. 4. Open PowerShell, cd to the working directory/llama.cpp, and create a new Python virtual environment: python3 -m venv env and activate the environment: .\env\Scripts\Activate.ps1 5. Obtain the LLaMA model(s) via the magnet torrent link and place them in the models directory. I used 30B and it is slow, but usable, on my system. Not even ChatGPT 3 level especially for programming questions, but impressive. 6. python3 -m pip install torch numpy sentencepiece 7. python3 convert-pth-to-ggml.py models/30B/ 1 (you may delete the original .pth model files after this step to save disk space) 8. .\quantize.exe ./models/30B/ggml-model-f16.bin ./models/30B/ggml-model-q4_0.bin 2 9. I copied the examples/chat-13B.bat to a new chat-30B.bat file, updated the model directory, and changed the last line of the script to: .\main.exe 10. Run using: .\examples\chat-30B.bat https://github.com/ggerganov/llama.cpp#usage https://github.com/ggerganov/llama.cpp#usage has details, although it assumes 7B and skips a few of the above steps.
- boppo1 4y agoWhat about Slaren? https://rentry.org/Jarted https://rentry.org/Jarted
- koolamusic 4y agoThis is super amazing stuff and I am still in awe.
- mew12623 4y ago[flagged]