3 ms·

I wonder how much of the model you can avoid loading before you start to see a real performance difference. Let’s say you want to maintain 90% of everything-in

by MBCook 3y ago

I wonder how much of the model you can avoid loading before you start to see a real performance difference. Let’s say you want to maintain 90% of everything-in-RAM performance. Can you get away with only using half the memory? Do you need 90% of the memory? Maybe 95%? Basically how fast do you lose performance compared to the maximum by cutting RAM. The charts are comparing their algorithm vs a basic one for the less RAM case, which is a different (but good!) question. If you can get good performance by NOT loading an entire eight gig model into memory of a cell phone that’s obviously a very useful thing.

dwd 3y ago
Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models. The paper is worth a read in full as what they are doing is pretty cool: https://arxiv.org/pdf/2312.11514 https://arxiv.org/pdf/2312.11514 Highlight from the paper... "Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput: • Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights. • Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."
- majestic5762 3y ago
  I was thinking a couple of days ago about this concept of windowing for LLMs, but I lack the technical skills to implement it. Now Apple just published a paper on it. This is what I call synchronicity
- rbr91 3y ago
  Is this https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-20231219.pdf https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-... related? Could it be the origin?
  - keivanalizadeh 3y ago
    By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 https://arxiv.org/abs/2310.04564 and dejavu https://arxiv.org/abs/2310.17157 https://arxiv.org/abs/2310.17157. They are targeting limited gpu memory and limit cpu to gpu memory transfer. I don't know how it could be useful on Macs because MacBooks have a unified memory and you don't need to do that transfer necessarily.
- keivanalizadeh 3y ago
  Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash. p.s: The window size in the paper is showing how many token's feed forward layer is in the memory.
RecycledEle 3y ago
I'm just thinking out loud. Nothing in this post is authoritative. Theoretically, the time consumed to inference a single token with part of the model stored in flash should be equal to the time to inference that token if the whole model was in RAM, plus the time required to load the part of the model stored in flash memory. I assume we do not need to write back to flash, but I'm not an LLM expert so I could be wrong. I assume we have many (more than 10) layers so we can leave a fairly small amount of our RAM available to load one layer after another. Most nontrivial LLMs have many dozens of layers, so this seems plausible. If we are not bottlenecking on our RAM during inferencing then we might be able to load the next layer from flash into RAM through a DMA transfer while inferencing our current layer. I don't think that would work on single-processor systems due to us always bottlenecking on RAM. Maybe a dual-processor system could load one layer into RAM on one processor while inferencing on the previous layer on the other processor, and thus run really big LLMs in a small amount of RAM? I'm sitting next to a pile of parts to build a new LLM AI machine. (z840, dual processor) and I look forward to playing with this stuff.
- dwd 3y ago
  There was also a throwaway comment about experts: "Mixture of Experts (Yiet al., 2023) have a sparse structure in their feed forward layer. This property can be used to combine with our method for enabling larger MoEs on device." Assuming this implementation would allow for running Mixtral 8x7b on a 16Gb M1, I'm happy.
  - treffer 3y ago
    I would think so, I have it quantized at 8 bit (q8) and that ticks in at ~47GB. Q4 should be well below the 32GB (2x 16GB). I am wondering if the same implementation could be done for RAM to CPU, too. Those are said to be data transfer limited, so minimizing the RAM to CPU cache transfer should help there, too?
    - indexerror 3y ago
      Q4 comes out to be ~26GB but Apple doesn't let you load it on a 32GB Mac machine because they put a limit on the max usable unified memory at ~21GB (`device. recommendedMaxWorkingSetSize`) [1]. So for Q4 Mixtral MoE you'd need a 64GB Mac machine unfortunately. Unless you use this hack [2]. [1] https://developer.apple.com/forums/thread/732035 https://developer.apple.com/forums/thread/732035 [2] https://github.com/ggerganov/llama.cpp/discussions/2182#discussioncomment-7698315 https://github.com/ggerganov/llama.cpp/discussions/2182#disc...

by MBCook 3y ago

dwd 3y ago

Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models. The paper is worth a read in full as what they are doing is pretty cool: https://arxiv.org/pdf/2312.11514 https://arxiv.org/pdf/2312.11514 Highlight from the paper... "Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput: • Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights. • Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."

majestic5762 3y ago
I was thinking a couple of days ago about this concept of windowing for LLMs, but I lack the technical skills to implement it. Now Apple just published a paper on it. This is what I call synchronicity
rbr91 3y ago
Is this https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-20231219.pdf https://ipads.se.sjtu.edu.cn/_media/publications/powerinfer-... related? Could it be the origin?
- keivanalizadeh 3y ago
  By the time we put our paper on arxiv this paper was not out so we were not aware of, but it is similar in some ways. Both this paper and us are reliant on our previous paper https://arxiv.org/abs/2310.04564 https://arxiv.org/abs/2310.04564 and dejavu https://arxiv.org/abs/2310.17157 https://arxiv.org/abs/2310.17157. They are targeting limited gpu memory and limit cpu to gpu memory transfer. I don't know how it could be useful on Macs because MacBooks have a unified memory and you don't need to do that transfer necessarily.
keivanalizadeh 3y ago
Half the memory was just an example not a sweet spot. With smaller window sizes you can use less memory, but it will come with the cost of loading more from flash. p.s: The window size in the paper is showing how many token's feed forward layer is in the memory.

RecycledEle 3y ago

I'm just thinking out loud. Nothing in this post is authoritative. Theoretically, the time consumed to inference a single token with part of the model stored in flash should be equal to the time to inference that token if the whole model was in RAM, plus the time required to load the part of the model stored in flash memory. I assume we do not need to write back to flash, but I'm not an LLM expert so I could be wrong. I assume we have many (more than 10) layers so we can leave a fairly small amount of our RAM available to load one layer after another. Most nontrivial LLMs have many dozens of layers, so this seems plausible. If we are not bottlenecking on our RAM during inferencing then we might be able to load the next layer from flash into RAM through a DMA transfer while inferencing our current layer. I don't think that would work on single-processor systems due to us always bottlenecking on RAM. Maybe a dual-processor system could load one layer into RAM on one processor while inferencing on the previous layer on the other processor, and thus run really big LLMs in a small amount of RAM? I'm sitting next to a pile of parts to build a new LLM AI machine. (z840, dual processor) and I look forward to playing with this stuff.

dwd 3y ago
There was also a throwaway comment about experts: "Mixture of Experts (Yiet al., 2023) have a sparse structure in their feed forward layer. This property can be used to combine with our method for enabling larger MoEs on device." Assuming this implementation would allow for running Mixtral 8x7b on a 16Gb M1, I'm happy.
- treffer 3y ago
  I would think so, I have it quantized at 8 bit (q8) and that ticks in at ~47GB. Q4 should be well below the 32GB (2x 16GB). I am wondering if the same implementation could be done for RAM to CPU, too. Those are said to be data transfer limited, so minimizing the RAM to CPU cache transfer should help there, too?
  - indexerror 3y ago
    Q4 comes out to be ~26GB but Apple doesn't let you load it on a 32GB Mac machine because they put a limit on the max usable unified memory at ~21GB (`device. recommendedMaxWorkingSetSize`) [1]. So for Q4 Mixtral MoE you'd need a 64GB Mac machine unfortunately. Unless you use this hack [2]. [1] https://developer.apple.com/forums/thread/732035 https://developer.apple.com/forums/thread/732035 [2] https://github.com/ggerganov/llama.cpp/discussions/2182#discussioncomment-7698315 https://github.com/ggerganov/llama.cpp/discussions/2182#disc...