4 ms·
Running a 35B LLM at 128K Context, Full Speed, on €870 of Used Hardware
- cyanydeez 2mo ago35B MoE => 3B active parameters which basically means, the memory goes towards storing the various "expert" layers but the processing speed is only experiencing 3B.