12 ms·

Using mmap to make LLaMA load faster

269 points by yawboakye 4y ago

ftxbro 4y ago
related hacker news discussion https://news.ycombinator.com/item?id=35411909 https://news.ycombinator.com/item?id=35411909
- ingenieroariel 4y ago
  Should we be linking flagged items? That discussion was very ugly and unproductive.
  - Technetium 4y ago
    Usually you'd be very right, but it's contextually relevant here due to slaren and jart replying there.
valine 4y ago
It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?
- tantony 4y ago
  Vicune
- menzoic 4y ago
  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/ https://vicuna.lmsys.org/
  - isoprophlex 4y ago
    Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...
    - danielbln 4y ago
      The weight deltas are available: https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights
    - TehCorwiz 4y ago
      The github page (https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights) is better: > We release Vicuna weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Vicuna weights.
    - detrites 4y ago
      Vicuna's fine-tune of the LLaMA weights are available for download, called "deltas". So you get the LLaMA weights (somewhere), then apply the Vicuna deltas to them to end up with the Vicuna model.
    - patrakov 4y ago
      Officially, only as deltas against LLaMa weights, and needing a complicated and resource-intensive conversion procedure. Unofficially, yes, a pre-converted llama.cpp compatible ggml file is available, but obviously I won't publish the link here to avoid violating the Y Combinator's terms of use.
  - adeon 4y ago
    I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUDs for a text adventure or analyze data or recognize languages or write stories. AFAIK it's been trained on ChatGPT discussions so makes sense. This AI still gets uppity sometimes about offensive content but unlike ChatGPT, you can edit the prompts to put words in its mouth to encourage it to answer properly.
    - VadimPR 4y ago
      Mind sharing how you got it to work in your setup?
      - adeon 4y ago
        I work on an independent LLM implementation here: https://github.com/Noeda/rllama/ https://github.com/Noeda/rllama/ I only got it working at all yesterday and there's no nice UX at all. Not sure I recommend trying to use this as llama.cpp will probably have this in no time with a much better user experience, although I am also trying to make it more usable. If you follow the instructions on Vicuna page over how to apply the deltas, and you can compile the project, then you could run: cargo run --release --features opencl -- --model-path /models/vicuna13b --param-path /models/vicuna13b/config.json --tokenizer-path /models/vicuna13b/tokenizer.model --prompt-file prompt --top-p 1.0 --top-k 20 --repetition-penalty 1 --temperature 0.9 --max-seq-len 2048 --f16 --percentage-to-gpu 0.9 Where /models/vicuna13b is the HuggingFace-compatible model. This will put 90% of weights on GPU and remaining 10% non CPU which is just barely enough to not run out of GPU memory (on a 24 gig card) Create a text file 'prompt' with the prompt. I've been using this template: You are a helpful and precise assistant for checking the quality of the answer.###Human: Can you explain nuclear power to me?###Assistant: (the model seems to use ### as delimiters to distinguish Human and Assistant). The "system prompt" is whatever text is written at the beginning.
  - dang 4y ago
    Related: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality - https://news.ycombinator.com/item?id=35378683 https://news.ycombinator.com/item?id=35378683 - March 2023 (167 comments)
- a5huynh 4y ago
  I recently found this list of models that works with llama.cpp: https://rentry.org/nur779 https://rentry.org/nur779 (with dl links, albeit given llama's licensing gray area, use at your own risk) The latest so far would be Vicuna, whose weights were just recently release.
radicality 4y ago
Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?
- simion314 4y ago
  There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to try to fit the biggest model possible in our VRAM or RAM.
  - astrange 4y ago
    > There are no memory improvements, people were not measuring correct. Using filebacked pages instead of anonymous memory is a real improvement because it doesn't have to get swapped out if there's memory pressure. And this program probably isn't the only thing running on the machine.
    - simion314 4y ago
      I was referring that you would not gain any memory, there was no magic compression so you could use a bigger model on the same hardware. There were some wild claims made but it was some people meassuring memory usage wrong, but you are correct there might be some small memory improvements and soem speed improvements.
      - astrange 4y ago
        Well, you can use a bigger model now, it will "just" be really slow. This is different from GPUs, which would just fail to load larger models than VRAM because they don't support paging (unless you build that yourself.)
NickGerleman 4y ago
Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711 https://github.com/ggerganov/llama.cpp/pull/711
- dekhn 4y ago
  I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.
  - londons_explore 4y ago
    In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.
    - bigodanktime 4y ago
      What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage (https://github.com/postgres/postgres/blob/master/src/backend/access/transam/xlog.c#L8048 https://github.com/postgres/postgres/blob/master/src/backend...).
      - astrange 4y ago
        Copying the file backed pages to heap memory and possibly having to swap them out.
        bigodanktime 4y ago
        I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swapping can occur in the buffer cache or mmap, this is why so many databases implement their own buffer cache to ensure specific data isn't flushed, leaving them in an inconsistent state. An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re...)
rossgellerty 4y ago
[flagged]
candrewlee14 4y ago
jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.
- stephc_int13 4y ago
  She is very good indeed, but this mmap thing is not comparable to her previous work. Hopefully we should publicize this kind of achievement as a way to teach more devs about mmap... (this really should be common knowledge)
  - candrewlee14 4y ago
    Of course its simpler, I just wanted it to make sure the comment still related to her work on this :) In summation, what she’s done is awesome.
- astrange 4y ago
  Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.
  - csmpltn 4y ago
    Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.
    - saagarjha 4y ago
      …I don’t see how this is related?
    - baq 4y ago
      it's as much engineering as is gluing mmap, recv and send... there are people who can do that and can't piece together a dataframe pipeline, no need to be passive agressive here
crims0n 4y ago
Read that as nmap and got really confused.
secondcoming 4y ago
Isn't `madvise` useful if you want to use Huge Pages?
- scottlamb 4y ago
  If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.
  - the8472 4y ago
    Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/<firefox process>/smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_Dirty: 0 kB Referenced: 58420 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 57344 kB <==== huge pages Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 1 VmFlags: rd ex mr mw me sd
    - scottlamb 4y ago
      Exciting if true! I see https://docs.kernel.org/filesystems/proc.html https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel hasn't allocated it as contiguous physical memory (and consistently aligned with the userspace virtual addresses), so there's something I'm missing. What kernel version did that output come from? Do you happen to know if Firefox did anything special to set that up? What filesystem type is this?
gmork13 4y ago
This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.
- jerf 4y ago
  "Blatant lie" seems a bit strong. Running a large model for a second time in a row is a pretty common use case and that speedup strikes me as real in that common case. Attribution may have been wrong but the time saved is real.
  - gmork13 4y ago
    So it can load twice as large models somehow?
- saurik 4y ago
  Justine knows this and it is stated right there on the page: > The first time you load a model after rebooting your computer, it's still going to go slow, because it has to load the weights from disk. However each time it's loaded afterwards, it should be fast (at least until memory pressure causes your file cache to be evicted).
  - Zuiii 4y ago
    Is the initial load via mmap slower than the non mmap approach? If it is, why is it slower?
- blitzkrieg3 4y ago
  mmap() will keep things in memory after first loading, but the page cache will _also_ keep things in memory after first loading. The difference is in order to re-use that you still need to read the file and store yourself (requiring 2x memory), instead of just doing a memory access. This has two consequences: * 2x memory. A 20G data set requires 40G (20 for page cache and 20 for LLaMA) * Things would be _even slower_ if they weren't in page cache after first loading. mmap is fast because it does not require a copy and reduces the working set size
astrange 4y ago
> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.
- mhh__ 4y ago
  I'm not sure what really needs abstracting about it other than tuning stuff for a given kernel/system. It's already extremely high level from a certain perspective :)
  - astrange 4y ago
    Someone has to own the memory region and know it gets munmap()ed not free()d. (deleted?) If you send it over IPC it's nice to keep it mmapped instead of accidentally copying it too.
  - plorkyeran 4y ago
    Boost.interprocess is an example of an abstraction over mmap which solves some of the things the blog post mentions. It abstracts away the difference between mmap() and CreateViewOfFile(), and gives you smart pointers and container types which are close to being drop-in replacements for std::vector and std::map that can be stored in a memory-mapped file.
- dundarious 4y ago
  I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.
  - 10000truths 4y ago
    Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.
mhh__ 4y ago
I've seen this on my HN feed about 5 times now, twitter twice. Why? (Author's blog is acceptable as they did the change, but the other 4?) Is this AI people learning that mmap exists? People simping cos' there a Justine involved?
- MuffinFlavored 4y ago
  [flagged]
  - HopenHeyHi 4y ago
    No? I mean yes, probably will get downvotes from the usual crowd. There is as a matter of fact some of that explicitly going on in the GitHub issue, which derailed the conversation. Such are the woes of being an internet micro celebrity I guess. Truly we live in the stupidest timeline.
    - MuffinFlavored 4y ago
      > There is as a matter of fact some of that explicitly going on in the GitHub issue Could you link an example so I can learn/retract my comment? I didn't think there was actually going on.
      - HopenHeyHi 4y ago
        You can just read it (at least ctrl-f jart) before telling people what is/isn't appropriate. As I recall in short order people started arguing about jealousy/credit, the file format initials being changed in jarts honor, "I'm not technical and don't know what's going on but..", 'etc. Goofy stuff.
mb_18850 4y ago
[dead]
kuroguro 4y ago
Lol, there's a "broken" image in the footer that claims your IP for ipv4.games http://ipv4.games/user.html?name=jart http://ipv4.games/user.html?name=jart
- MrStonedOne 4y ago
  [dead]
antimatter15 4y ago
Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment-1493385890 https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
- version_five 4y ago
  I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.
  - ncann 4y ago
    I found this summary from that thread: https://rentry.org/Jarted https://rentry.org/Jarted
    - bestcoder69 4y ago
      Click the 4chan links and you’ll see in their own words what this was really about (trans maintainer, channers worried about their waifu bot getting cucked.) Most seem to not know what they’re talking about- some admit to being retarded. Highly suspicious of this being tech related. Also note the stats on GH subscribers and stuff. This is a lolcow dossier… Events like this make me glad I don’t contribute OSS. I’ll keep my coombots proprietary.
      - EGreg 4y ago
        I didn’t know Justine was trans. Her wikipedia article doesn’t mention it, and has another female middle name which suggests that her parents gave the names to her. I am not much familiar with her work except the impressive Cosmopolitan / Redbean mentioned on HN in the past. But she seems to be quite a controversial figure that is for some weird technocracy and against democracy and leftists, despite being a leader in the zucotti park protests… in short, someone who is no stranger to drama and controversy, and actively courts it: https://www.thedailybeast.com/articles/2014/08/01/occupying-the-throne-justine-tunney-neoreactionaries-and-the-new-1-percent https://www.thedailybeast.com/articles/2014/08/01/occupying-...
dasl 4y ago
> One of the downsides of the Linux cp command, is copying a file larger than RAM will destroy every existing entry in the file cache. Under normal circumstances this is a good thing, since a least recently used strategy usually works. However it can be problematic if you're just organizing your files on a production system where you don't want to disrupt performance. As far as I know, no standard command line utility offers a way to exploit this functionality. I think `dd` in conjunction with the `oflag=direct` has this functionality. See: https://stackoverflow.com/questions/33485108/why-is-dd-with-the-direct-o-direct-flag-so-dramatically-faster https://stackoverflow.com/questions/33485108/why-is-dd-with-...
- kccqzy 4y ago
  I choose to use a modern file system and use cp --reflink=always.
jancsika 4y ago
I still don't understand: why was the magic number changed in addition to the file version number? Edit: can someone running llama.cpp ask it whether it thinks it's a good idea to concatenate a running list of vanity initials of important developers into a magic filetype constant?
- vkoskiv 4y ago
  It was a significant change, and Greg - the original author of llama.cpp was fine with it: > Regarding the version comment - yes, the plan was to bump versions and no the magic. But I'm ok to change the magic to commemorate the significance of this update. In fact, maybe we can make this a thing and everybody who makes a significant contribution to the project will get their initials appended to the version. What do you think? smile
baffo32 4y ago
justine worked hard in these changes in good faith and was attacked and experienced character assassination by orchestrated trolling, it’s so sad
manv1 4y ago
There was a post a few months ago about a developer who's job it was to rework ML stuff into actual efficient code. This reminds me of that post because it seems that lots of ML stuff is just plain inefficient...in that they use way too many resources given the problem. And they're arguing about mmap, something that's been around forever. It reminds me of that speedup in a package manager because they were reading uncached byte-at-a-time off of disk. You need to explicitly turn buffered reads off...but why would you do that in the first place? Unbuffered reads are almost never a good idea, ever. It makes me wonder what other weird sub-optimal stuff is lying underneath the resource behemoth that is ML.
- bob1029 4y ago
  > it seems that lots of ML stuff is just plain inefficient This is just like any other technology. Use it wrong, you will get burned. Doesn't matter how shiny the container is. "ML stuff" also (mostly?) includes purely statistical methods from the 90s that are deterministic and arguably the best way to solve a large variety of non-generative problems. In fact, unless generation of arbitrary output is a major objective, it's likely you can solve whatever ML task on a workstation from 2010 that uses intel integrated graphics.
stuaxo 4y ago
The original change made intuitive sense, some of the arguments against seem a bit weird - asserting that MMAPing the file could mean the memory sticks around after the program stops.. no. Suggesting that MMAP limits things to the size of the ram, well - no, as well - paging may happen, but then we are just back out to the file. Honestly, some of the weird assertions wouldn't take long for people to double check an verify (or falsify).
ingenieroariel 4y ago
The post is a technical article that describes a very cool systems engineering approach to a problem that is usually coded away in proprietary code from nvidia. No wonder a lot of people see it as "mmap, nothing new". But this is not the case in a lot of libraries where the norm is to just budget for a lot of time moving things to/from gpu and just relying on someone else's code. Instead of accumulating technical debt the owner of the repo decided to merge this, have a breaking change and move on. When there was some community backlash to the breaking changes there was a pull request trying to revert all changes instead working through the issues (it was a net win for several users but not all, some configurations with slower drives were better served by the older approach). There was an ugly back and forth and the repo owner decided to ban both the person who did the pull request and the author of this post. This article brings the conversation back to the technical merits, the roadmap, credit the the original authors and tones down the ownership tone that may have pissed off some community members. That pull request has now been closed by the owner of the repo. They are trying to move on and be productive, let's do the same.
up2isomorphism 4y ago
Smart people contribute to open source and fight with internet, wise people provide a service and profit from it. Maybe sarcastic, but it is how things look like today.
dekhn 4y ago
If I've learned two lessons in computing over 35+ years, it's: you can never have enough RAM, and IO is the hardest bottleneck to solve.

269 points by yawboakye 4y ago

related hacker news discussion https://news.ycombinator.com/item?id=35411909 https://news.ycombinator.com/item?id=35411909

ingenieroariel 4y ago
Should we be linking flagged items? That discussion was very ugly and unproductive.
- Technetium 4y ago
  Usually you'd be very right, but it's contextually relevant here due to slaren and jart replying there.

valine 4y ago

It’s hard to keep up with all developments around LLaMA. What’s the best RLHF alpaca like model you can download right now?

tantony 4y ago
Vicune
menzoic 4y ago
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality by the Team with members from UC Berkeley, CMU, Stanford, and UC San Diego https://vicuna.lmsys.org/ https://vicuna.lmsys.org/
- isoprophlex 4y ago
  Are the Vicuna weights available for download, and are they llama.cpp compatible? I can't grok that by skimming the page...
  - danielbln 4y ago
    The weight deltas are available: https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights
  - TehCorwiz 4y ago
    The github page (https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights) is better: > We release Vicuna weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Vicuna weights.
  - detrites 4y ago
    Vicuna's fine-tune of the LLaMA weights are available for download, called "deltas". So you get the LLaMA weights (somewhere), then apply the Vicuna deltas to them to end up with the Vicuna model.
  - patrakov 4y ago
    Officially, only as deltas against LLaMa weights, and needing a complicated and resource-intensive conversion procedure. Unofficially, yes, a pre-converted llama.cpp compatible ggml file is available, but obviously I won't publish the link here to avoid violating the Y Combinator's terms of use.
- adeon 4y ago
  I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU with 90% of weights on the GPU (otherwise running out of memory) at around 500ms per token. This model is really good for a (semi-)open source model. I think this may be the first locally runnable model that I will actually use for real stuff rather than just play around for fun. It's not ChatGPT level but it's not that far behind. It will draw ASCII art HUDs for a text adventure or analyze data or recognize languages or write stories. AFAIK it's been trained on ChatGPT discussions so makes sense. This AI still gets uppity sometimes about offensive content but unlike ChatGPT, you can edit the prompts to put words in its mouth to encourage it to answer properly.
  - VadimPR 4y ago
    Mind sharing how you got it to work in your setup?
    - adeon 4y ago
      I work on an independent LLM implementation here: https://github.com/Noeda/rllama/ https://github.com/Noeda/rllama/ I only got it working at all yesterday and there's no nice UX at all. Not sure I recommend trying to use this as llama.cpp will probably have this in no time with a much better user experience, although I am also trying to make it more usable. If you follow the instructions on Vicuna page over how to apply the deltas, and you can compile the project, then you could run: cargo run --release --features opencl -- --model-path /models/vicuna13b --param-path /models/vicuna13b/config.json --tokenizer-path /models/vicuna13b/tokenizer.model --prompt-file prompt --top-p 1.0 --top-k 20 --repetition-penalty 1 --temperature 0.9 --max-seq-len 2048 --f16 --percentage-to-gpu 0.9 Where /models/vicuna13b is the HuggingFace-compatible model. This will put 90% of weights on GPU and remaining 10% non CPU which is just barely enough to not run out of GPU memory (on a 24 gig card) Create a text file 'prompt' with the prompt. I've been using this template: You are a helpful and precise assistant for checking the quality of the answer.###Human: Can you explain nuclear power to me?###Assistant: (the model seems to use ### as delimiters to distinguish Human and Assistant). The "system prompt" is whatever text is written at the beginning.
- dang 4y ago
  Related: Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality - https://news.ycombinator.com/item?id=35378683 https://news.ycombinator.com/item?id=35378683 - March 2023 (167 comments)
a5huynh 4y ago
I recently found this list of models that works with llama.cpp: https://rentry.org/nur779 https://rentry.org/nur779 (with dl links, albeit given llama's licensing gray area, use at your own risk) The latest so far would be Vicuna, whose weights were just recently release.

radicality 4y ago

Since the post is from day, so the improvements were all ‘real’? I didn’t follow closely but I remember multiple points people brought up earlier like: is the memory counting correct, why aren’t all the weights accessed for a query, whether quantisation is a problem etc. Were all these fixed?

simion314 4y ago
There are no memory improvements, people were not measuring correct. The giant improvement is the load times after the first run(if you do not invalidate your caches). Quantization to 4 bit is a big gain, the loss appears to be minimal from benchmarks. So with quantization you gain the ability to try a bigger model, if you have the hardware to fit the biggest model then you can skip it but for most people we need to try to fit the biggest model possible in our VRAM or RAM.
- astrange 4y ago
  > There are no memory improvements, people were not measuring correct. Using filebacked pages instead of anonymous memory is a real improvement because it doesn't have to get swapped out if there's memory pressure. And this program probably isn't the only thing running on the machine.
  - simion314 4y ago
    I was referring that you would not gain any memory, there was no magic compression so you could use a bigger model on the same hardware. There were some wild claims made but it was some people meassuring memory usage wrong, but you are correct there might be some small memory improvements and soem speed improvements.
    - astrange 4y ago
      Well, you can use a bigger model now, it will "just" be really slow. This is different from GPUs, which would just fail to load larger models than VRAM because they don't support paging (unless you build that yourself.)

NickGerleman 4y ago

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711 https://github.com/ggerganov/llama.cpp/pull/711

dekhn 4y ago
I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.
- londons_explore 4y ago
  In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.
  - bigodanktime 4y ago
    What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage (https://github.com/postgres/postgres/blob/master/src/backend/access/transam/xlog.c#L8048 https://github.com/postgres/postgres/blob/master/src/backend...).
    - astrange 4y ago
      Copying the file backed pages to heap memory and possibly having to swap them out.
      - bigodanktime 4y ago
        I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swapping can occur in the buffer cache or mmap, this is why so many databases implement their own buffer cache to ensure specific data isn't flushed, leaving them in an inconsistent state. An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re...)

rossgellerty 4y ago

[flagged]

candrewlee14 4y ago

jart is a genius. What they’ve done with Blink, Cosmopolitan C, Redbean, and now llama.cpp is incredible. It gives me hope for the future of systems/low-level programming.

stephc_int13 4y ago
She is very good indeed, but this mmap thing is not comparable to her previous work. Hopefully we should publicize this kind of achievement as a way to teach more devs about mmap... (this really should be common knowledge)
- candrewlee14 4y ago
  Of course its simpler, I just wanted it to make sure the comment still related to her work on this :) In summation, what she’s done is awesome.
astrange 4y ago
Using mmap is basic systems programming knowledge and not "incredible" even though it's a useful contribution.
- csmpltn 4y ago
  Glad to see we're slowly coming to the realization that wrangling NumPy and Pandas doesn't actually count as Software Engineering.
  - saagarjha 4y ago
    …I don’t see how this is related?
  - baq 4y ago
    it's as much engineering as is gluing mmap, recv and send... there are people who can do that and can't piece together a dataframe pipeline, no need to be passive agressive here

crims0n 4y ago

Read that as nmap and got really confused.

secondcoming 4y ago

Isn't `madvise` useful if you want to use Huge Pages?

scottlamb 4y ago
If you're hoping file-backed mmaps from any filesystem other than tmpfs/shmem will use huge pages, you will be disappointed. :-( See the following note at https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html https://www.kernel.org/doc/html/latest/admin-guide/mm/transh... > Currently THP only works for anonymous memory mappings and tmpfs/shmem.
- the8472 4y ago
  Not sure if that's true any longer on recent kernels. There have been some changes with folios that I think enable huge pages in the page cache. $ cat /proc/<firefox process>/smaps [...] 7efca62e3000-7efcaa13d000 r-xp 00ae3000 00:18 75354786 /usr/lib/libLLVM-15.so Size: 63848 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Rss: 58420 kB Pss: 19213 kB Pss_Dirty: 0 kB Shared_Clean: 56372 kB Shared_Dirty: 0 kB Private_Clean: 2048 kB Private_Dirty: 0 kB Referenced: 58420 kB Anonymous: 0 kB LazyFree: 0 kB AnonHugePages: 0 kB ShmemPmdMapped: 0 kB FilePmdMapped: 57344 kB <==== huge pages Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB Swap: 0 kB SwapPss: 0 kB Locked: 0 kB THPeligible: 1 VmFlags: rd ex mr mw me sd
  - scottlamb 4y ago
    Exciting if true! I see https://docs.kernel.org/filesystems/proc.html https://docs.kernel.org/filesystems/proc.html describes FilePmdMapped as "Page cache mapped into userspace with huge pages", consistent with what you are saying. I don't fully understand the distinction between that and FileHugePages: "Memory used for filesystem data (page cache) allocated with huge pages". I wouldn't think it'd be possible to map it into userspace as huge pages if the kernel hasn't allocated it as contiguous physical memory (and consistently aligned with the userspace virtual addresses), so there's something I'm missing. What kernel version did that output come from? Do you happen to know if Firefox did anything special to set that up? What filesystem type is this?

gmork13 4y ago

This is from today apr 5 saying the mmap change loads twice as big models with x100 speed up - is this not a blatant lie? Wasn’t it discovered last week that loading larger models was an error in measurement and the speed up was from keeping things in memory after the first loading? Please do correct me if I’m wrong.

jerf 4y ago
"Blatant lie" seems a bit strong. Running a large model for a second time in a row is a pretty common use case and that speedup strikes me as real in that common case. Attribution may have been wrong but the time saved is real.
- gmork13 4y ago
  So it can load twice as large models somehow?
saurik 4y ago
Justine knows this and it is stated right there on the page: > The first time you load a model after rebooting your computer, it's still going to go slow, because it has to load the weights from disk. However each time it's loaded afterwards, it should be fast (at least until memory pressure causes your file cache to be evicted).
- Zuiii 4y ago
  Is the initial load via mmap slower than the non mmap approach? If it is, why is it slower?
blitzkrieg3 4y ago
mmap() will keep things in memory after first loading, but the page cache will _also_ keep things in memory after first loading. The difference is in order to re-use that you still need to read the file and store yourself (requiring 2x memory), instead of just doing a memory access. This has two consequences: * 2x memory. A 20G data set requires 40G (20 for page cache and 20 for LLaMA) * Things would be _even slower_ if they weren't in page cache after first loading. mmap is fast because it does not require a copy and reduces the working set size

astrange 4y ago

> I don't think I've ever seen a high-level library that's able to do what mmap() does, because it defies attempts at abstraction. I'm not sure what this means but I'm pretty sure I can name several "high level libraries" that mmap things. None of those are the STL, but it's not exactly perfect design.

mhh__ 4y ago
I'm not sure what really needs abstracting about it other than tuning stuff for a given kernel/system. It's already extremely high level from a certain perspective :)
- astrange 4y ago
  Someone has to own the memory region and know it gets munmap()ed not free()d. (deleted?) If you send it over IPC it's nice to keep it mmapped instead of accidentally copying it too.
- plorkyeran 4y ago
  Boost.interprocess is an example of an abstraction over mmap which solves some of the things the blog post mentions. It abstracts away the difference between mmap() and CreateViewOfFile(), and gives you smart pointers and container types which are close to being drop-in replacements for std::vector and std::map that can be stored in a memory-mapped file.
dundarious 4y ago
I read it to mean mmap is irreplaceable. There is no other sophisticated dance of system calls or userspace trickery that can achieve what mmap can achieve. She's saying that everything up and down the stack, including high level libraries, do just call mmap, because there would be no DIY alternative with similar cost-benefit.
- 10000truths 4y ago
  Except it’s not irreplaceable, at least on Linux. userfaultfd allows you to define custom page fault handling. With it, you can even do crazy things like “mmap” a remote resource by making HTTP range requests on a read fault.

mhh__ 4y ago

I've seen this on my HN feed about 5 times now, twitter twice. Why? (Author's blog is acceptable as they did the change, but the other 4?) Is this AI people learning that mmap exists? People simping cos' there a Justine involved?

MuffinFlavored 4y ago
[flagged]
- HopenHeyHi 4y ago
  No? I mean yes, probably will get downvotes from the usual crowd. There is as a matter of fact some of that explicitly going on in the GitHub issue, which derailed the conversation. Such are the woes of being an internet micro celebrity I guess. Truly we live in the stupidest timeline.
  - MuffinFlavored 4y ago
    > There is as a matter of fact some of that explicitly going on in the GitHub issue Could you link an example so I can learn/retract my comment? I didn't think there was actually going on.
    - HopenHeyHi 4y ago
      You can just read it (at least ctrl-f jart) before telling people what is/isn't appropriate. As I recall in short order people started arguing about jealousy/credit, the file format initials being changed in jarts honor, "I'm not technical and don't know what's going on but..", 'etc. Goofy stuff.

mb_18850 4y ago

[dead]

kuroguro 4y ago

Lol, there's a "broken" image in the footer that claims your IP for ipv4.games http://ipv4.games/user.html?name=jart http://ipv4.games/user.html?name=jart

MrStonedOne 4y ago
[dead]

antimatter15 4y ago

Note that as a result of some llama drama associated with this change, @jart (author of this post) and @anzz1 are apparently no longer welcome as collaborators on llama.cpp: https://github.com/ggerganov/llama.cpp/pull/711#issuecomment-1493385890 https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...

version_five 4y ago
I can't really understand what's going on after a few minutes poking around (best I can tell an argument over attribution?), but I certainly sympathize with the project author having to deal with some stupid internet drama that takes away from a cool project. This seems to be one of the less-discussed challenges with popular open-source.
- ncann 4y ago
  I found this summary from that thread: https://rentry.org/Jarted https://rentry.org/Jarted
  - bestcoder69 4y ago
    Click the 4chan links and you’ll see in their own words what this was really about (trans maintainer, channers worried about their waifu bot getting cucked.) Most seem to not know what they’re talking about- some admit to being retarded. Highly suspicious of this being tech related. Also note the stats on GH subscribers and stuff. This is a lolcow dossier… Events like this make me glad I don’t contribute OSS. I’ll keep my coombots proprietary.
    - EGreg 4y ago
      I didn’t know Justine was trans. Her wikipedia article doesn’t mention it, and has another female middle name which suggests that her parents gave the names to her. I am not much familiar with her work except the impressive Cosmopolitan / Redbean mentioned on HN in the past. But she seems to be quite a controversial figure that is for some weird technocracy and against democracy and leftists, despite being a leader in the zucotti park protests… in short, someone who is no stranger to drama and controversy, and actively courts it: https://www.thedailybeast.com/articles/2014/08/01/occupying-the-throne-justine-tunney-neoreactionaries-and-the-new-1-percent https://www.thedailybeast.com/articles/2014/08/01/occupying-...

dasl 4y ago

> One of the downsides of the Linux cp command, is copying a file larger than RAM will destroy every existing entry in the file cache. Under normal circumstances this is a good thing, since a least recently used strategy usually works. However it can be problematic if you're just organizing your files on a production system where you don't want to disrupt performance. As far as I know, no standard command line utility offers a way to exploit this functionality. I think `dd` in conjunction with the `oflag=direct` has this functionality. See: https://stackoverflow.com/questions/33485108/why-is-dd-with-the-direct-o-direct-flag-so-dramatically-faster https://stackoverflow.com/questions/33485108/why-is-dd-with-...

kccqzy 4y ago
I choose to use a modern file system and use cp --reflink=always.

jancsika 4y ago

I still don't understand: why was the magic number changed in addition to the file version number? Edit: can someone running llama.cpp ask it whether it thinks it's a good idea to concatenate a running list of vanity initials of important developers into a magic filetype constant?

vkoskiv 4y ago
It was a significant change, and Greg - the original author of llama.cpp was fine with it: > Regarding the version comment - yes, the plan was to bump versions and no the magic. But I'm ok to change the magic to commemorate the significance of this update. In fact, maybe we can make this a thing and everybody who makes a significant contribution to the project will get their initials appended to the version. What do you think? smile

baffo32 4y ago

justine worked hard in these changes in good faith and was attacked and experienced character assassination by orchestrated trolling, it’s so sad

manv1 4y ago

There was a post a few months ago about a developer who's job it was to rework ML stuff into actual efficient code. This reminds me of that post because it seems that lots of ML stuff is just plain inefficient...in that they use way too many resources given the problem. And they're arguing about mmap, something that's been around forever. It reminds me of that speedup in a package manager because they were reading uncached byte-at-a-time off of disk. You need to explicitly turn buffered reads off...but why would you do that in the first place? Unbuffered reads are almost never a good idea, ever. It makes me wonder what other weird sub-optimal stuff is lying underneath the resource behemoth that is ML.

bob1029 4y ago
> it seems that lots of ML stuff is just plain inefficient This is just like any other technology. Use it wrong, you will get burned. Doesn't matter how shiny the container is. "ML stuff" also (mostly?) includes purely statistical methods from the 90s that are deterministic and arguably the best way to solve a large variety of non-generative problems. In fact, unless generation of arbitrary output is a major objective, it's likely you can solve whatever ML task on a workstation from 2010 that uses intel integrated graphics.

stuaxo 4y ago

The original change made intuitive sense, some of the arguments against seem a bit weird - asserting that MMAPing the file could mean the memory sticks around after the program stops.. no. Suggesting that MMAP limits things to the size of the ram, well - no, as well - paging may happen, but then we are just back out to the file. Honestly, some of the weird assertions wouldn't take long for people to double check an verify (or falsify).

ingenieroariel 4y ago

The post is a technical article that describes a very cool systems engineering approach to a problem that is usually coded away in proprietary code from nvidia. No wonder a lot of people see it as "mmap, nothing new". But this is not the case in a lot of libraries where the norm is to just budget for a lot of time moving things to/from gpu and just relying on someone else's code. Instead of accumulating technical debt the owner of the repo decided to merge this, have a breaking change and move on. When there was some community backlash to the breaking changes there was a pull request trying to revert all changes instead working through the issues (it was a net win for several users but not all, some configurations with slower drives were better served by the older approach). There was an ugly back and forth and the repo owner decided to ban both the person who did the pull request and the author of this post. This article brings the conversation back to the technical merits, the roadmap, credit the the original authors and tones down the ownership tone that may have pissed off some community members. That pull request has now been closed by the owner of the repo. They are trying to move on and be productive, let's do the same.

up2isomorphism 4y ago

Smart people contribute to open source and fight with internet, wise people provide a service and profit from it. Maybe sarcastic, but it is how things look like today.

dekhn 4y ago

If I've learned two lessons in computing over 35+ years, it's: you can never have enough RAM, and IO is the hardest bottleneck to solve.