3 ms·
Inference should be closer to llama 13b, since it runs 2/8 experts for each token.
by Palmik 3y ago
Inference should be closer to llama 13b, since it runs 2/8 experts for each token.
- sgt101 3y agoDoes it have to run them sequentially? I guess the cost will be 12/13bn level but latency may be faster?