4 ms·

The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actu

The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actually being able to shrink the model size from 20GB > 6GB were just completely false, and while there was a model loading time improvement, actual memory required and used did not change. A number of people saw this and said that making a breaking change to the repo that a lot of people are using and have forked for other models was a bad idea, thus this new PR.

blibble 4y ago
here mmap is being used for essentially lazy loading that's it
- chpatrick 4y ago
  But then it's pretty much the same as using the old version and turning on swap, so I don't really see the point. As far as I understand the whole model needs to be read constantly so there's no benefit from the random access mmap provides.
  - sitkack 4y ago
    Swap is a system level property not a program level property. They are similar, use similar mechanisms, but the experience that a user would see are very different.
    - chpatrick 4y ago
      I'm not sure it would be that different but mmap has the benefit that it can swap directly to the model file on the disk instead of making a copy in swap space.
      - sitkack 4y ago
        mmap is like tightly scoped, targeted swap. If the PoR is on disk, the OS is free to reclaim that memory for something else at anytime. It really is a beautiful hack, but if you turn on swap for the system in that way, it has to balance memory usage across all the running programs. In this use of mmap, there is nothing to swap out as the source of truth is always on disk. During general swap usage, memory has to travel in both directions.
simonw 4y ago
The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613
- trsohmers 4y ago
  Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting 1200+ upvotes resulted in multiple conversations in my social circle that took that untrue statement as fact.
  - sitkack 4y ago
    > [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. <supposition ... I would look at IO rates during inference ...> You have an opportunity to teach rather than admonish.
    - trsohmers 4y ago
      I appreciate your feedback, and I agree I could have better worded and taken it in a more constructive direction. Thank you, and I hope myself and others will try to have more productive discourse.
  - fastball 4y ago
    The comment you're looking for is here[1], where @jart does indeed seem to be echoing the claim that memory usage is reduced to surprising levels given the LLM architecture. Justine's original guess was that this is because the model is sparse (?) but in actuality it seems like memory usage is just being reported strangely by the system and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu...

blibble 4y ago

here mmap is being used for essentially lazy loading that's it

chpatrick 4y ago
But then it's pretty much the same as using the old version and turning on swap, so I don't really see the point. As far as I understand the whole model needs to be read constantly so there's no benefit from the random access mmap provides.
- sitkack 4y ago
  Swap is a system level property not a program level property. They are similar, use similar mechanisms, but the experience that a user would see are very different.
  - chpatrick 4y ago
    I'm not sure it would be that different but mmap has the benefit that it can swap directly to the model file on the disk instead of making a copy in swap space.
    - sitkack 4y ago
      mmap is like tightly scoped, targeted swap. If the PoR is on disk, the OS is free to reclaim that memory for something else at anytime. It really is a beautiful hack, but if you turn on swap for the system in that way, it has to balance memory usage across all the running programs. In this use of mmap, there is nothing to swap out as the source of truth is always on disk. During general swap usage, memory has to travel in both directions.

simonw 4y ago

The 20GB to 6GB confusion appears to have come from the title of the Hacker News post the other day: https://news.ycombinator.com/item?id=35393284 https://news.ycombinator.com/item?id=35393284 The PR it linked to said nothing of the sort: https://github.com/ggerganov/llama.cpp/pull/613 https://github.com/ggerganov/llama.cpp/pull/613

trsohmers 4y ago
Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting 1200+ upvotes resulted in multiple conversations in my social circle that took that untrue statement as fact.
- sitkack 4y ago
  > [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. <supposition ... I would look at IO rates during inference ...> You have an opportunity to teach rather than admonish.
  - trsohmers 4y ago
    I appreciate your feedback, and I agree I could have better worded and taken it in a more constructive direction. Thank you, and I hope myself and others will try to have more productive discourse.
- fastball 4y ago
  The comment you're looking for is here[1], where @jart does indeed seem to be echoing the claim that memory usage is reduced to surprising levels given the LLM architecture. Justine's original guess was that this is because the model is sparse (?) but in actuality it seems like memory usage is just being reported strangely by the system and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu...