12 ms·
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/l
by jart 4y ago
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome, but like all things in science a small amount of healthy skepticism is warranted.
- 0xakhil 4y agoSome OS’s zram compress the unpinned pages instead of swapping to disk. It might be faster than fetching the pages again from disk. I wonder if this is a reason why folks see different results.
- nynx 4y agoWhy is it behaving sparsely? There are only dense operations, right?
- w1nk 4y agoI also have this question, yes it should be. The forward pass should require accessing all the weights AFAIK.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- HarHarVeryFunny 4y agoFrom what I've read there's no evidence it's "behaving sparsely".. That was just offered as a suggestion why it might not be loading all the weights, but makes no sense in terms of the model. It's going to be using all the weights. Another suggestion is that not all of the word/token embedding table might be used, which would be a function of the input used to test, but that would be easy enough to disprove as there would then be different memory usage for different inputs. It seems possible the reported memory usage is lower than reality if that's how mmap/top work. In any case, a good use of mmap it seems, especially since for a multi-layer model layer weights will be used sequentially so paged load-on-demand will work relatively well even in a low memory situation.
- eternalban 4y agoGreat work. Is the new file format described anywhere? Skimming the issue comments I have a vague sense that r/o matter was colocated somewhere for zero copy mmap or is there more to it?
- jart 4y agoThat's something I'm working on presently.
- conradev 4y ago> But we don't have a compelling enough theory yet to explain the RAM usage miracle. My guess would be that the model is faulted into memory lazily page by page (4K or 16K chunks) as the model is used, so only the actual parts that are needed are loaded. The kernel also removes old pages from the page cache to make room for new ones, and especially so if the computer is using a lot of its RAM. As with all performance things, this approach trades off inference speed for memory usage, but likely faster overall because you don't have to read the entire thing from disk at the start. Each input will take a different path through the model, and will require loading more of it. The cool part is that this memory architecture should work just fine with hardware acceleration, too, as long as the computer has unified memory (anything with an integrated GPU). This approach likely won't be possible with dedicated GPUs/VRAM. This approach _does_ still work to run a dense model with limited memory, but the time/memory savings would just be less. The GPU doesn't multiply every matrix in the file literally simultaneously, so the page cache doesn't need to contain the entire model at once.
- liuliu 4y agoMetal only recent versions (macOS 13 / iOS 16) supports mmap and use that in GPU directly. CUDA does have unified memory mode even it is dedicated GPU, would be interesting to try that out. Probably going to slow down quite a bit, but still interesting to have that possibility.
- jart 4y agoI don't think it's actually trading away inference speed. You can pass an --mlock flag, which calls mlock() on the entire 20GB model (you need root to do it), then htop still reports only like 4GB of RAM is in use. My change helps inference go faster. For instance, I've been getting inference speeds of 30ms per token after my recent change on the 7B model, and I normally get 200ms per eval on the 30B model.
- conradev 4y agoVery cool! Are you testing after a reboot / with an empty page cache?
- zone411 4y agoIt really shouldn't act as a sparse model. I would bet on something being off.
- world2vec 4y ago>I'm glad you're happy with the fact that LLaMA 30B (a 20gb file) can be evaluated with only 4gb of memory usage! Isn't LLaMA 30B a set of 4 files (60,59Gb)? -edit- nvm, It's quantized. My bad
- sillysaurusx 4y agoHey, I saw your thoughtful comment before you deleted it. I just wanted to apologize — I had no idea this was a de facto Show HN, and certainly didn’t mean to make it about something other than this project. The only reason I posted it is because Facebook had been DMCAing a few repos, and I wanted to reassure everyone that they can hack freely without worry. That’s all. I’m really sorry if I overshadowed your moment on HN, and I feel terrible about that. I’ll try to read the room a little better before posting from now on. Please have a wonderful weekend, and thanks so much for your hard work on LLaMA! EDIT: The mods have mercifully downweighted my comment, which is a relief. Thank you for speaking up about that, and sorry again. If you'd like to discuss any of the topics you originally posted about, you had some great points.
- hackernewds 4y agogreat apology :)
- Kiro 4y agoI don't think you need to apologize for your comment. Even if I post a Show HN I expect no special treatment on my comments. That would be ridiculous and this is not even a Show HN.
- d3nj4l 4y agoMaybe off topic, but I just wanted to say that you're an inspiration!
- htrp 4y agoJust shows how inefficient some of the ML research code can be
- robrenaud 4y agoTraining tends to require a lot more precision and hence memory than inference. I bet many of the tricks here won't work well for training.
- actually_a_dog 4y agoAs a former grad student, I can tell you, that's all research code, not just ML, or even "performance-oriented" research code.
- rvz 4y agoExactly. It also shows the number of impostors in this thread and inflated titles of self-proclaimed 'seniors' who can't optimize ML code to even be on the same league as Tunney (jart), and Gerganov (ggerganov). Not even ChatGPT or Copilot could even submit a change or in-fact completely rewrite and optimize this code like they have done.
- visarga 4y agoRemember this moment when you're about to criticise LLMs. People can act suboptimal too, even experts.
- alduin32 4y agoFor now we've just shown how measuring memory consumption can be tricky at times.
- sr-latch 4y agoHave you tried running it against a quantized model on HuggingFace with identical inputs and deterministic sampling to check if the outputs you're getting are identical? I think that should confirm/eliminate any concern of the model being evaluated incorrectly.
- intelVISA 4y agoDidn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
- ok123456 4y agoYou can mmap from python.
- westurner 4y agoThe CPython mmap module docs: https://docs.python.org/3/library/mmap.html https://docs.python.org/3/library/mmap.html zero_buffer (CFFI, 2013) https://github.com/alex/zero_buffer/blob/master/zero_buffer.py https://github.com/alex/zero_buffer/blob/master/zero_buffer.... "Buffers on the edge: Python and Rust" (2022) https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ : > If you have a Python object and want to obtain its buffer, you can do so with memoryview in Python or PyObject_GetBuffer in C. If you’re defining a class and want to expose a buffer, you can do so in Python by… actually you can’t, only classes implemented in C can implement the buffer protocol. To implement the buffer protocol in C, you provide the bf_getbuffer and bf_releasebuffer functions which are called to obtain a buffer from an object and when that buffer is being released, respectively. iocursor (CPython C API, ~Rust std::io::Cursor) https://github.com/althonos/iocursor https://github.com/althonos/iocursor Arrow Python (C++) > On disk and MemoryMappedFile s: https://arrow.apache.org/docs/python/memory.html#on-disk-and-memory-mapped-files https://arrow.apache.org/docs/python/memory.html#on-disk-and... "Apache Arrow: Read DataFrame With Zero Memory" (2020) https://towardsdatascience.com/apache-arrow-read-dataframe-with-zero-memory-69634092b1a https://towardsdatascience.com/apache-arrow-read-dataframe-w... pyarrow.Tensor: https://arrow.apache.org/docs/python/generated/pyarrow.Tensor.html#pyarrow.Tensor https://arrow.apache.org/docs/python/generated/pyarrow.Tenso... ONNX is built on protocolbuffers/protobufs (google/protobufs), while Arrow is built on google/flatbuffers. FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers https://en.wikipedia.org/wiki/FlatBuffers : > It supports “zero-copy” deserialization, so that accessing the serialized data does not require first copying it into a separate part of memory. This makes accessing data in these formats much faster than data in formats requiring more extensive processing, such as JSON, CSV, and in many cases Protocol Buffers. Compared to other serialization formats however, the handling of FlatBuffers requires usually more code, and some operations are not possible (like some mutation operations).
- thomastjeffery 4y agoHow diverse is the training corpus?
- dchest 4y agohttps://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971
- thomastjeffery 4y agoIs there any measure, not of size or token amount, but of diversity in the content of the text? Did that metric meaningfully change when the amount of required memory dropped? If the amount of diversity is lowered, I would expect that to lower the amount of patterns to be modeled from the text. If that is the case, then the resulting model size itself would be lowered, during and after training.
- actually_a_dog 4y agoBy "diversity," do you mean something like "entropy?" Like maybe H_s(x) := -\sum_{x \in X_s} p(x) log(p(x)) where X_s := all s-grams from the training set? That seems like it would eventually become hard to impossible to actually compute. Even if you could what would it tell you? Or, wait... are you referring to running such an analysis on the output of the model? Yeah, that might prove interesting....
- thomastjeffery 4y agoI'm really just speculating here. Because the text we write is not evenly distributed random noise, what we encode into it (by writing) is entropy. Because LLMs model text with inference, they model all of the entropy that is present. That would mean that the resulting size would be a measure of entropy (sum of patterns) divided by repetition (recurring patterns). In this count, I would consider each unique token alone an instance of the identity pattern. So to answer both questions: yes.
- smaddox 4y agoBased on that discussion, it definitely sounds like some sort of bug is hiding. Perhaps run some evaluations to compare perplexity to the standard implementation? Edit: looks like there's now confirmation that running it on a 10GB VM slows inference down massively, so looks like the only thing strange is the memory usage reading on some systems.
- diimdeep 4y agoIs the title misleading here ? 30B quantized requires 19.5 GB, not 6GB; Otherwise severe swapping to disk model original size quantized size (4-bit) 7B 13 GB 3.9 GB 13B 24 GB 7.8 GB 30B 60 GB 19.5 GB 65B 120 GB 38.5 GB
- renewiltord 4y agoThat's the size on disk, my man. When you quantize it to a smaller float size you lose precision on the weights and so the model is smaller. Then here they `mmap` the file and it only needs 6 GiB of RAM!
- deleted 4y ago[deleted]
- gliptic 4y agoThe size mentioned is already quantized (and to integers, not floats). mmap obviously doesn't do any quantization.
- xiphias2 4y agoNow it's clear that there was a bug in the measurement. The author used a machine with lots of RAM, so I guess most of us are still stuck with quantized 13B. Still, the improvement hopefully translates, and I hope that 30B will run with 3 bit quantization in a few days.
- diimdeep 4y agoAlso, current SSD's achieve 7.5 GB/s+ read speed, opposed to older SSD from 2013 with 500 MB/s, so performance will drastically differ depending on your system specs in case of pulling weights from disk to RAM on demand. Also, there is $ vmmap <pid> where we can see various statistics about process memory and used swap, that are not available in top or htop.
- freehorse 4y ago
- alchemist1e9 4y agoI’m hopeful that when especially skilled developers like you have banged on minimizing inference resources, the others and you will start looking at distributed training ideas. Probably there is a way to decentralize the training so we can all throw in our GPUs together on building the most useful models for code generation than can be free to use and relatively cheap to run inference on. If you have any thoughts on that side of the LLM space I’m sure we would all be super curious to hear them. Thank you for the amazing work. It’s so appreciated by so many on HN like me I’m sure.
- parano1d 4y agoCould mmap also be used to improve the memory usage of whisper.cpp?
- bestcoder69 4y agoThanks for this! I was able to integrate alpaca-30B into a slack bot & a quick tkinter GUI (coded by GPT-4 tbh) by just shelling out to `./main` in both cases, since model loading is so quick now. (I didn't even have to ask GPT-4 to code me up Python bindings to llama's c-style api!)
- bugglebeetle 4y agoWhat’s your setup for running these? I’m not seeing performance improvements on off the shelf hardware that would allow for this.
- MacsHeadroom 4y agoI host a llama-13B IRC chatbot on a spare old android phone.
- bugglebeetle 4y agoHave a repo anywhere?
- MacsHeadroom 4y agoIt's just the same llama.cpp repo everyone else is using. You just git clone it to your android phone in termux and then run make and you're done. https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Assuming you have the model file downloaded (you can use wget to download it) these are the instructions to install and run: pkg install git pkg install cmake pkg install build-essential git clone https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp cd llama.cpp make -j ./main
- bugglebeetle 4y agoYeah, I’ve already been running llama.cpp locally, but not found it to perform at the level attested in the comment (30B model as a chat bot on commodity hardware). 13B runs okay, but inference appears generally too slow on to do anything useful on my MacBook. I wondered what you might be doing to get usable performance in that context.
- ec109685 4y agoBefore, it had to read the 16GB into a buffer, which itself could be written to disk if the system needs to page out. That is a big reason startup time is fast.
- StillBored 4y agoTook a look at it, did you try MAP_HUGETLB? This looks like the kind of application that can gain very large runtime advantages from avoiding TLB pressure. It might take a bit longer (or fail entirely) on machines where you can't get enough huge pages, but attempting it (or probing for free pages via /proc/meminfo) and then falling back to mapping it without might take slightly longer for the mmap() but the advantages of taking an order of magnitude (assuming you can get 1G pages) fewer TLB misses might be worth it.
- rcarmo 4y agoThis is nothing short of legendary. Was following the thread on Twitter and LOLed at the replies of “Praise be Jart”, but there’s something of the sublime here. Great weight wrangling judo :)
- shock-value 4y agoIt appears that this was just a misreading of how memory usage was being reported and there was actually no improvement here. At least nothing so sensational as being able to run a larger-than-RAM model without swapping from disk on every iteration.
- jart 4y agoPlease read the original link to the pull request, where I stated my change offered a 2x improvement in memory usage. You actually are able to load models 2x larger without compromising system stability, because pages are no longer being copied. That's because you previously needed 40gb of RAM to load a 20GB model, in order to ensure your file cache wasn't destroyed and need to reread from disk the next time. Now you only need 20GB to load a 20GB model. The peculiarity here is that tools like htop were reporting the improvement as being an 8x improvement, which is interesting, because RAM use is only 2x better due to my change. The rusage.com page fault reporting was also interesting too. This is not due to sparseness. It's because htop was subtracting MAP_SHARED memory. The htop docs say on my computer that the color purple is used to display shared memory, and yellow is used to display kernel file caches. But it turned out it just uses yellow for both, even though it shouldn't, because mincore() reported that the shared memory had been loaded into the resident set size.
- shock-value 4y agoIt's obviously a productive change and kudos for taking it on, but much of the enthusiasm being generated here was driven by the entirely unanticipated prospect of running a model at full speed using less memory than the model's own footprint, and by the notion that inference with a dense model somehow behaved in a sparse manner at runtime. Best to be a bit more grounded here, particularly with regard to claims that defy common understanding.
- Manjuuu 4y agoThanks for all the nice projects you are releasing for free, btw.
- AnthonyMouse 4y agoGosh, thank you for getting to this before I did. The first thing I said when I saw it loading tens of GB from the disk on each run is, is there some reason they're not using mmap? This isn't just a matter of making the 30B model run in 6GB or whatever. You can now run the largest model, without heavy quantization, and let the OS figure it out. It won't be as fast as having "enough" memory, but it will run. In theory you could always have done this with swap, but swap is even slower because evictions have to be written back to swap (and wear out your SSD if your swap isn't on glacially slow spinning rust) instead of just discarded because the OS knows where to read it back from the filesystem. This should also make it much more efficient to run multiple instances at once because they can share the block cache. (I wonder if anybody has done this with Stable Diffusion etc.)
- deleted 4y ago[deleted]
- irthomasthomas 4y agoAuthorship of this change is contested. https://news.ycombinator.com/item?id=35431865 https://news.ycombinator.com/item?id=35431865