3 ms·

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actual

Sadly I’m on my phone at the moment and can’t find the specific post, but in that PR or related discussion there was talk of only a few GB of the weights actually being used during the computation, which anyone who understands how a multi headed attention transformer works would know is impossible… your QKV matmuls need to touch all of the weights once you go through all the layers. Since that post yesterday getting 1200+ upvotes resulted in multiple conversations in my social circle that took that untrue statement as fact.

sitkack 4y ago
> [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. <supposition ... I would look at IO rates during inference ...> You have an opportunity to teach rather than admonish.
- trsohmers 4y ago
  I appreciate your feedback, and I agree I could have better worded and taken it in a more constructive direction. Thank you, and I hope myself and others will try to have more productive discourse.
fastball 4y ago
The comment you're looking for is here[1], where @jart does indeed seem to be echoing the claim that memory usage is reduced to surprising levels given the LLM architecture. Justine's original guess was that this is because the model is sparse (?) but in actuality it seems like memory usage is just being reported strangely by the system and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu...

sitkack 4y ago

> [which anyone who understands] how a multi headed attention transformer works [would know is impossible] How is that productive? The author already claimed in their changes that they aren't an ML expert and asked for advice. It could be written like multi-headed attention transformers frobnicate all the weights in their QKV matmuls. <supposition ... I would look at IO rates during inference ...> You have an opportunity to teach rather than admonish.

trsohmers 4y ago
I appreciate your feedback, and I agree I could have better worded and taken it in a more constructive direction. Thank you, and I hope myself and others will try to have more productive discourse.

fastball 4y ago

The comment you're looking for is here[1], where @jart does indeed seem to be echoing the claim that memory usage is reduced to surprising levels given the LLM architecture. Justine's original guess was that this is because the model is sparse (?) but in actuality it seems like memory usage is just being reported strangely by the system and that a model that takes up 20GB on disk still effectively needs 20GB of RAM in the end. [1] https://github.com/ggerganov/llama.cpp/discussions/638#discussioncomment-5492916 https://github.com/ggerganov/llama.cpp/discussions/638#discu...