8 ms·

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://g

Worth pointing out, there has been quite a bit of contention around this change, both technical, and some accusations of plagiarism/miscrediting here. https://github.com/ggerganov/llama.cpp/pull/711 https://github.com/ggerganov/llama.cpp/pull/711

dekhn 3y ago
I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.
- londons_explore 3y ago
  In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.
  - bigodanktime 3y ago
    What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage (https://github.com/postgres/postgres/blob/master/src/backend/access/transam/xlog.c#L8048 https://github.com/postgres/postgres/blob/master/src/backend...).
    - astrange 3y ago
      Copying the file backed pages to heap memory and possibly having to swap them out.
      - bigodanktime 3y ago
        I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swapping can occur in the buffer cache or mmap, this is why so many databases implement their own buffer cache to ensure specific data isn't flushed, leaving them in an inconsistent state. An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re...)
        astrange 3y ago
        > With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.) > An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-re https://www.mongodb.com/blog/post/getting-storage-engines-re...) Darwin kernel does though. I believe Linux uses the builtin old memcpy instructions on Intel, just to force CPU vendors to keep them usable.
        bigodanktime 3y ago
        > There is no copy with mmap You are right, if you are directly modifying the mmaped region. I always internally model my data as staging my changes to be synchronized to the mmaped region, so thats my mistake there. > the page is either unwritable or CoW. This is not universally true, or maybe I'm confused on this statement. MAP_SHARED exists, but maybe you are referencing a specific kernels' implementation on how they achieve coherence between file backed shared memory regions in two processes? Im not sure. > Darwin kernel does though. Sure we can always point to a kernel that has has implemented some feature or another, which is why I said typically you don't see it.
        saagarjha 3y ago
        > Darwin kernel does though. It does not. Compare the implementation of _bcopyout against _platform_memmove, you'll see the difference :)
        astrange 3y ago
        Huh, maybe I was thinking of "you can use floating point in the kernel". That doesn't work in every kernel because they don't want to bother saving/restoring the extra registers.
  - dekhn 3y ago
    Yes- I have 35 years experience with UNIX and used to use mmapping with BLAST, a sequence search tool, as well as my own codes. I'll repeat myself: mmap is subtle. If what you mmap is larger than your host RAM, only some of the pages will be loaded at any time, and depending on access patterns, can lead to significant paging.
- marginalia_nu 3y ago
  > only work when you have much more RAM than the files you're mapping in. Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache and the least accessed data is paged out. I don't disagree with the subtlety part though. It's very difficult to reason about I/O performance in general. Modern systems are like an onion of hidden performance optimization tricks and caching layers (both in software and hardware).
  - jjoonathan 3y ago
    > least accessed data is paged out Aren't all the weights touched in every pass?
    - marginalia_nu 3y ago
      Speaking in general.
  - Karrot_Kream 3y ago
    Yeah and on top of that, different systems (software and hardware combos) are different, so I can see the performance of this depending on the implementation of mmap on the system and the implementation of caches and virtual memory on the architecture. When I've debugged stuff like this, it's either been for myself in which case I know what combo I'm running on or it's been for work where we know which combinations we target and we run regression tests to observe perf implications.
yieldcrv 3y ago
only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?
- super256 3y ago
  > only thing this discussion has showed me is that more people need Computer Science degrees again You have too much faith in unis. Mine did not teach me about mmap at all.
  - crysin 3y ago
    I'm in a grad program for Software Engineering. At my university, the only difference between the Comp Sci and Software Engineering degree is that comp sci requires an advanced algorithm class whereas software engineering has a capstone class where you have to work with a team to build a MVP that is unit tested, uses CI/CD, and obviously works. I say this to highlight the parent comment. I'm essentially in a computer science program and we have learned absolutely 0 about paging or memory in any of my required courses. We practically don't touch OS anything in any of the classes. That's not to say the courses for that aren't offered but they aren't part of the core curriculum and over my time in my program, they've mostly not been offered due to lack of student interest. I did learn how to use linked lists like a champion though!
- deleted 3y ago
  [deleted]
- stephc_int13 3y ago
  I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
  - yieldcrv 3y ago
    I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs, while some of the most basic optimizations are seemingly a lost art. On the other hand, its only been a few weeks, so maybe I should ignore this absurdity and just wait.
HopenHeyHi 3y ago
Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.
- popip 3y ago
  [flagged]
- djur 3y ago
  Is this related to her advocacy for neoreactionary politics or is it just a transphobia thing?
  - HopenHeyHi 3y ago
    [flagged]
  - saagarjha 3y ago
    I mean, there's also the part where she's wrong a lot.
- oceanplexian 3y ago
  I'm not a fan or a hater, I didn't even know who this person was until this thread. Does the change deserve a blog post or wild claims like "llama.cpp is 100x faster and uses half the memory!"? No. The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic (i.e. lots of language like "We spent several weeks volunteering" and "our project") uh whatever. It also breaks a backwards compatibility when there's no technical reason it couldn't have been optional or put behind a feature flag, plus a ton of condescending language in the PR. Not really the kind of work I'd be proud of or would be advertising in a blog post.
  - HopenHeyHi 3y ago
    Yes, exactly. The claim that it uses half the memory was probably a honest mistake. The ensuing disappointment that it did not in fact halve memory usage and drama attracted trolls and white knights and is icky. The discussion around nmap I suppose is subtle and when emotion abounds can no longer be had. :/
bigodanktime 3y ago
I can understand these folks struggling with what mmap is actually doing. But this isn't a new discussion about the qualities of MMAP versus file based IO etc. Although, many of the comments stated are quite wrong. Related Work on this problem: 1. https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re... - talks about developments on MongoDB's backend to use mmap. 2. https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf - Talks about some of the cons of mmap, some I think are not as prevalent due to the existence of low latency, high throughput storage devices. 3. https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_uploads/vmcache.pdf https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up... - less relevant but related.
bestcoder69 3y ago
Also worth pointing out that you can follow the thread’s link to Rentry, which links to a 4chan (?) archived thread, where you can see anons getting worked up over jart being a trans internet celebrity. And unless you’re playing dumb, you have to admit they were looking for an excuse to troll jart. Unless you seriously want me to believe they were all that mad about… mmap
pantalaimon 3y ago
Is mmap really that broken on Windows? Or is the poster just confused that the data stays in the page cache? But that’s what the page cache does - that memory will be used for other things if needed, but if the memory is not needed it might as well keep the old data in cache.
- Analemma_ 3y ago
  No, mmap on Windows is fine. A generous, charitable statement would be that the OP on that thread is very confused, but based on some comments elsewhere on this thread about jart attracting a chorus of haters, it seems more likely that they're just trolling.
- tedunangst 3y ago
  There's a weird breed of programmer who only wants to see the free memory column in top be maximized. I bought all this RAM and I want to make sure none of it is used in case I want to use it later.
chad1n 3y ago
[flagged]
- vore 3y ago
  jart is a woman, for what it's worth.
  - chad1n 3y ago
    I just presented the drama and I referred to him as `he` as a developer, my teammates are mostly male, so I usually say `he`.
    - whimsicalism 3y ago
      Right, so maybe you shouldn't do that?
      - timpani 3y ago
        [flagged]
    - tedunangst 3y ago
      Do you also refer to your female teammates as he?
      - timpani 3y ago
        [flagged]
  - timpani 3y ago
    [flagged]
    - dekhn 3y ago
      This sort of statement really don't contribute anythign to the discussion and in fact greatly distracts from the technical content. We don't need to hear your opinions about trans people in this thread.
514723553 3y ago
I don't get the "plagiarism/miscrediting" accusations. This was in the original PR (https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. Given all this, it feels like the drama--shouldn't exist? Like, if there's an issue with attribution, it's not because of bad-faith, and I feel like a good-faith conversation could have just resolved this, instead of bringing in trolls.
- IAmNotACellist 3y ago
  That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 https://github.com/ggerganov/llama.cpp/pull/586. Jart's archived comments: "my changes" "Here's how folks in the community have been reacting to my work." "I just wrote a change that's going to let your LLaMA models load instantly..." https://archive.ph/PyPFZ https://archive.ph/PyPFZ "I'm the author" https://archive.ph/qFrcY https://archive.ph/qFrcY "Author here..." "Tragedy of the commons...We're talking to a group of people who live inside scientific papers and jupyer notebooks." "My change helps inference go faster." "The point of my change..." "I stated my change offered a 2x improvement in memory usage." https://archive.ph/k34V2 https://archive.ph/k34V2 "I can only take credit for a 2x recrease in RAM usage." https://archive.ph/MBPN0 https://archive.ph/MBPN0 "I just wrote a change that's going to let your LLaMA models load instantly, thanks to custom malloc() and the power of mmap()" https://archive.ph/yrMwh https://archive.ph/yrMwh slaren replied to jart on HN asking her why she was doing and saying those things, and she didn't bother to reply to him, despite replying to others in that subthread within minutes. https://archive.ph/zCfiJ https://archive.ph/zCfiJ
  - empathy_m 3y ago

dekhn 3y ago

I feel significantly dumber for reading that merge request. The one thing to understand is that the performance implications of mmap are subtle and only work when you have much more RAM than the files you're mapping in.

londons_explore 3y ago
In this case, the main benefit is from multiple invocations of the same program. Using mmap, you avoid doing any work at all the 2nd time you load the file.
- bigodanktime 3y ago
  What do you mean by work. The underlying page cache will keep much of the data actual cached if it's recent. Even databases like PostGreSQL use this to their advantage (https://github.com/postgres/postgres/blob/master/src/backend/access/transam/xlog.c#L8048 https://github.com/postgres/postgres/blob/master/src/backend...).
  - astrange 3y ago
    Copying the file backed pages to heap memory and possibly having to swap them out.
    - bigodanktime 3y ago
      I may have parsed your statement incorrectly, but I'm assuming you are talking about the copy of data when using either mmap or File IO (memcpy versus write) Whether you do File IO versus mmap, there's going to be copy. With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. Swapping can occur in the buffer cache or mmap, this is why so many databases implement their own buffer cache to ensure specific data isn't flushed, leaving them in an inconsistent state. An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re...)
      - astrange 3y ago
        > With files, the copy occurs within kernel space with data being copied into the pages in the buffer cache, with mmap the copy occurs in userspace with data being copied into the address space. There is no copy with mmap, the page is either unwritable or CoW. There's always a copy with read(). (But read() can still be faster and more memory efficient nevertheless.) > An advantage of copying in userspace is the ability to use more performant instructions to perform the memcopy, which the kernel does not typically have access to (https://www.mongodb.com/blog/post/getting-storage-engines-re https://www.mongodb.com/blog/post/getting-storage-engines-re...) Darwin kernel does though. I believe Linux uses the builtin old memcpy instructions on Intel, just to force CPU vendors to keep them usable.
        bigodanktime 3y ago
        > There is no copy with mmap You are right, if you are directly modifying the mmaped region. I always internally model my data as staging my changes to be synchronized to the mmaped region, so thats my mistake there. > the page is either unwritable or CoW. This is not universally true, or maybe I'm confused on this statement. MAP_SHARED exists, but maybe you are referencing a specific kernels' implementation on how they achieve coherence between file backed shared memory regions in two processes? Im not sure. > Darwin kernel does though. Sure we can always point to a kernel that has has implemented some feature or another, which is why I said typically you don't see it.
        saagarjha 3y ago
        > Darwin kernel does though. It does not. Compare the implementation of _bcopyout against _platform_memmove, you'll see the difference :)
        astrange 3y ago
        Huh, maybe I was thinking of "you can use floating point in the kernel". That doesn't work in every kernel because they don't want to bother saving/restoring the extra registers.
- dekhn 3y ago
  Yes- I have 35 years experience with UNIX and used to use mmapping with BLAST, a sequence search tool, as well as my own codes. I'll repeat myself: mmap is subtle. If what you mmap is larger than your host RAM, only some of the pages will be loaded at any time, and depending on access patterns, can lead to significant paging.
marginalia_nu 3y ago
> only work when you have much more RAM than the files you're mapping in. Really depends on what you're doing, like memory access patterns. I've definitely seen scenarios when mapping hundreds of gigabytes of data on dozens of gigabytes of ram where mmap has been an almost absurd performance boost over traditional I/O, both immediately but also asymptotically as all the most frequently accessed data ends up in cache and the least accessed data is paged out. I don't disagree with the subtlety part though. It's very difficult to reason about I/O performance in general. Modern systems are like an onion of hidden performance optimization tricks and caching layers (both in software and hardware).
- jjoonathan 3y ago
  > least accessed data is paged out Aren't all the weights touched in every pass?
  - marginalia_nu 3y ago
    Speaking in general.
- Karrot_Kream 3y ago
  Yeah and on top of that, different systems (software and hardware combos) are different, so I can see the performance of this depending on the implementation of mmap on the system and the implementation of caches and virtual memory on the architecture. When I've debugged stuff like this, it's either been for myself in which case I know what combo I'm running on or it's been for work where we know which combinations we target and we run regression tests to observe perf implications.

yieldcrv 3y ago

only thing this discussion has showed me is that more people need Computer Science degrees again like, wow, mmap and paging. really guys?

super256 3y ago
> only thing this discussion has showed me is that more people need Computer Science degrees again You have too much faith in unis. Mine did not teach me about mmap at all.
- crysin 3y ago
  I'm in a grad program for Software Engineering. At my university, the only difference between the Comp Sci and Software Engineering degree is that comp sci requires an advanced algorithm class whereas software engineering has a capstone class where you have to work with a team to build a MVP that is unit tested, uses CI/CD, and obviously works. I say this to highlight the parent comment. I'm essentially in a computer science program and we have learned absolutely 0 about paging or memory in any of my required courses. We practically don't touch OS anything in any of the classes. That's not to say the courses for that aren't offered but they aren't part of the core curriculum and over my time in my program, they've mostly not been offered due to lack of student interest. I did learn how to use linked lists like a champion though!
deleted 3y ago
[deleted]
stephc_int13 3y ago
I feel the same. I maybe should not be surprised, given that we live in the era of Unity and Electron, but using mmap() to load large files should be not be seen as rocket science. And this is basically available on almost any platform with a MMU and a kernel.
- yieldcrv 3y ago
  I think there just hasn't been a consumer application that is really resource constrained, for a long time now. Only things for enthusiasts have been. LLMs have product market fit, but running a useful one client side is resource constrained, but instead of it truly being a consumer hardware limitation, it just turns out they were never optimized to begin with - coming from the perceived "top AI/ML minds" at FAANGs, while some of the most basic optimizations are seemingly a lost art. On the other hand, its only been a few weeks, so maybe I should ignore this absurdity and just wait.

HopenHeyHi 3y ago

Unfortunately Justine has attracted a peculiar fanbase+haterbase. As their numbers swell the collective intelligence and technical understanding diminishes. So the discussions end up gravitating towards weird drama. I wish you wouldn't have linked this thread. Theres going to be a bunch of stupid comments here as well about how great/awful jart is.

popip 3y ago
[flagged]
djur 3y ago
Is this related to her advocacy for neoreactionary politics or is it just a transphobia thing?
- HopenHeyHi 3y ago
  [flagged]
- saagarjha 3y ago
  I mean, there's also the part where she's wrong a lot.
oceanplexian 3y ago
I'm not a fan or a hater, I didn't even know who this person was until this thread. Does the change deserve a blog post or wild claims like "llama.cpp is 100x faster and uses half the memory!"? No. The original PR looks like a decent addition but the blog posts reads as incredibly narcissistic (i.e. lots of language like "We spent several weeks volunteering" and "our project") uh whatever. It also breaks a backwards compatibility when there's no technical reason it couldn't have been optional or put behind a feature flag, plus a ton of condescending language in the PR. Not really the kind of work I'd be proud of or would be advertising in a blog post.
- HopenHeyHi 3y ago
  Yes, exactly. The claim that it uses half the memory was probably a honest mistake. The ensuing disappointment that it did not in fact halve memory usage and drama attracted trolls and white knights and is icky. The discussion around nmap I suppose is subtle and when emotion abounds can no longer be had. :/

bigodanktime 3y ago

I can understand these folks struggling with what mmap is actually doing. But this isn't a new discussion about the qualities of MMAP versus file based IO etc. Although, many of the comments stated are quite wrong. Related Work on this problem: 1. https://www.mongodb.com/blog/post/getting-storage-engines-ready-fast-storage-devices https://www.mongodb.com/blog/post/getting-storage-engines-re... - talks about developments on MongoDB's backend to use mmap. 2. https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf - Talks about some of the cons of mmap, some I think are not as prevalent due to the existence of low latency, high throughput storage devices. 3. https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_uploads/vmcache.pdf https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up... - less relevant but related.

bestcoder69 3y ago

Also worth pointing out that you can follow the thread’s link to Rentry, which links to a 4chan (?) archived thread, where you can see anons getting worked up over jart being a trans internet celebrity. And unless you’re playing dumb, you have to admit they were looking for an excuse to troll jart. Unless you seriously want me to believe they were all that mad about… mmap

pantalaimon 3y ago

Is mmap really that broken on Windows? Or is the poster just confused that the data stays in the page cache? But that’s what the page cache does - that memory will be used for other things if needed, but if the memory is not needed it might as well keep the old data in cache.

Analemma_ 3y ago
No, mmap on Windows is fine. A generous, charitable statement would be that the OP on that thread is very confused, but based on some comments elsewhere on this thread about jart attracting a chorus of haters, it seems more likely that they're just trolling.
tedunangst 3y ago
There's a weird breed of programmer who only wants to see the free memory column in top be maximized. I bought all this RAM and I want to make sure none of it is used in case I want to use it later.

chad1n 3y ago

[flagged]

vore 3y ago
jart is a woman, for what it's worth.
- chad1n 3y ago
  I just presented the drama and I referred to him as `he` as a developer, my teammates are mostly male, so I usually say `he`.
  - whimsicalism 3y ago
    Right, so maybe you shouldn't do that?
    - timpani 3y ago
      [flagged]
  - tedunangst 3y ago
    Do you also refer to your female teammates as he?
    - timpani 3y ago
      [flagged]
- timpani 3y ago
  [flagged]
  - dekhn 3y ago
    This sort of statement really don't contribute anythign to the discussion and in fact greatly distracts from the technical content. We don't need to hear your opinions about trans people in this thread.

514723553 3y ago

I don't get the "plagiarism/miscrediting" accusations. This was in the original PR (https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613): > This PR was written in collaboration with @slaren. This PR is also rebased on PR #586 so please do not squash merge! Use either merge or rebase. jart made sure to that the other user got credit, in addition to making sure that their name was properly attributed in the commit log. Given all this, it feels like the drama--shouldn't exist? Like, if there's an issue with attribution, it's not because of bad-faith, and I feel like a good-faith conversation could have just resolved this, instead of bringing in trolls.

IAmNotACellist 3y ago
That's not the original PR. jart was working on a malloc() approach that didn't work and slaren wrote all the code actually doing mmap, which jart then rebased in a random new PR, changed to support an unnecessary version change, magic numbers, a conversion tool, and WIN32 support when that was already working in the draft PR. https://archive.ph/Uva8c https://archive.ph/Uva8c This is the original PR: https://github.com/ggerganov/llama.cpp/pull/586 https://github.com/ggerganov/llama.cpp/pull/586. Jart's archived comments: "my changes" "Here's how folks in the community have been reacting to my work." "I just wrote a change that's going to let your LLaMA models load instantly..." https://archive.ph/PyPFZ https://archive.ph/PyPFZ "I'm the author" https://archive.ph/qFrcY https://archive.ph/qFrcY "Author here..." "Tragedy of the commons...We're talking to a group of people who live inside scientific papers and jupyer notebooks." "My change helps inference go faster." "The point of my change..." "I stated my change offered a 2x improvement in memory usage." https://archive.ph/k34V2 https://archive.ph/k34V2 "I can only take credit for a 2x recrease in RAM usage." https://archive.ph/MBPN0 https://archive.ph/MBPN0 "I just wrote a change that's going to let your LLaMA models load instantly, thanks to custom malloc() and the power of mmap()" https://archive.ph/yrMwh https://archive.ph/yrMwh slaren replied to jart on HN asking her why she was doing and saying those things, and she didn't bother to reply to him, despite replying to others in that subthread within minutes. https://archive.ph/zCfiJ https://archive.ph/zCfiJ
- empathy_m 3y ago