3 ms·
I'm just thinking out loud. Nothing in this post is authoritative. Theoretically, the time consumed to inference a single token with part of the model stored i
by RecycledEle 3y ago
I'm just thinking out loud. Nothing in this post is authoritative.
Theoretically, the time consumed to inference a single token with part of the model stored in flash should be equal to the time to inference that token if the whole model was in RAM, plus the time required to load the part of the model stored in flash memory.
I assume we do not need to write back to flash, but I'm not an LLM expert so I could be wrong.
I assume we have many (more than 10) layers so we can leave a fairly small amount of our RAM available to load one layer after another. Most nontrivial LLMs have many dozens of layers, so this seems plausible.
If we are not bottlenecking on our RAM during inferencing then we might be able to load the next layer from flash into RAM through a DMA transfer while inferencing our current layer. I don't think that would work on single-processor systems due to us always bottlenecking on RAM.
Maybe a dual-processor system could load one layer into RAM on one processor while inferencing on the previous layer on the other processor, and thus run really big LLMs in a small amount of RAM?
I'm sitting next to a pile of parts to build a new LLM AI machine. (z840, dual processor) and I look forward to playing with this stuff.
- dwd 3y agoThere was also a throwaway comment about experts: "Mixture of Experts (Yiet al., 2023) have a sparse structure in their feed forward layer. This property can be used to combine with our method for enabling larger MoEs on device." Assuming this implementation would allow for running Mixtral 8x7b on a 16Gb M1, I'm happy.
- treffer 3y agoI would think so, I have it quantized at 8 bit (q8) and that ticks in at ~47GB. Q4 should be well below the 32GB (2x 16GB). I am wondering if the same implementation could be done for RAM to CPU, too. Those are said to be data transfer limited, so minimizing the RAM to CPU cache transfer should help there, too?
- indexerror 3y agoQ4 comes out to be ~26GB but Apple doesn't let you load it on a 32GB Mac machine because they put a limit on the max usable unified memory at ~21GB (`device. recommendedMaxWorkingSetSize`) [1]. So for Q4 Mixtral MoE you'd need a 64GB Mac machine unfortunately. Unless you use this hack [2]. [1] https://developer.apple.com/forums/thread/732035 https://developer.apple.com/forums/thread/732035 [2] https://github.com/ggerganov/llama.cpp/discussions/2182#discussioncomment-7698315 https://github.com/ggerganov/llama.cpp/discussions/2182#disc...
- astrange 3y agosysctls aren't a hack exactly, it's there so you can change it. As for why it's not the default, it's mostly because wiring all your memory will crash the computer pretty fast.
- endymi0n 3y agoThere’s a brand new hybrid quantization for Mixtral out that uses 4b for shared neurons and 2b for experts, which does not bleed much perplexity, but fits it into a 32G machine. Haven’t had it in hand yet and no link here on mobile, but can’t wait to try.