3 ms·
Is it maybe because DeepSeek is a MoE and doesn't require all parameters for a given token? That's not ideal from a token throughput perspective, but I can see
by monocasa 2y ago
Is it maybe because DeepSeek is a MoE and doesn't require all parameters for a given token?
That's not ideal from a token throughput perspective, but I can see min working set of weight memory gains if you can load pieces into vram for each token.
- throwaway314155 2y agoIt still wouldn't fit in 16 GB memory. Further there's too much swapping going on with MoE models to move expert layers to and from gpu without bottlenecks.