4 ms·
Not 7B, 8x7B. It will run with the speed of a 7B model while being much smarter but requiring ~24GB of RAM instead of ~4GB (in 4bit).
by MacsHeadroom 3y ago
Not 7B, 8x7B.
It will run with the speed of a 7B model while being much smarter but requiring ~24GB of RAM instead of ~4GB (in 4bit).
- dragonwriter 3y agoGiven the config parametes posted, its 2 experts per token, so the conputation cost per token should be the cost of the conponent that selects experts + 2× cost of a 7B model.
- stavros 3y agoYes, but I also care about "can I load this onto my home GPU?" where, if I need all experts for this to run, the answer is "no".
- MacsHeadroom 3y agoThe answer is yes if you have a 24GB GPU. Just wait for 4bit quantization. Or watch Tim Dettmers, who is releasing code to run Mixtral 8x7b in just 4GB of RAM.
- MacsHeadroom 3y agoAh good catch. Upon even closer examination, the attention layer (~2B params) is shared across experts. So in theory you would need 2B for the attention head + 5B for each of two experts in RAM. That's a total of 12B, meaning this should be able to be run on the same hardware as 13B models with some loading time between generations.
- deleted 3y ago[deleted]