4 ms·
Mixtral 8x7B going 378 tokens per second on CPU
- dangoodmanUT 3y agoOnly for prompt eval, it's only spitting out 10 tok/s if I am understanding that screenshot correctly
- deleted 3y ago[deleted]
- PeterStuer 3y agoI think it's processing the tokenized prompt at 378 tokens per second, but emitting predicted tokens by the model at 11 tokens per second.
- d-z-m 3y agoSeems misleading to phrase it this way if you aren't talking about inference tok/s.
- kkielhofner 3y agoVery misleading, likely deliberately so. Per usual it’s working - here we are talking about it. It’s evaluating input prompt at that speed. Important metric and impressive on its own but “going X tokens per second” in the mind of anyone looking at this is generated tokens. That number: 11 tokens per second 368 on input is impressive, 11 on output is too. The need to taint legitimate and real progress by stretching, misleading, etc is yet another classic issue greatly exacerbated by social media.
- T-A 3y ago... with 96 cores. Is that a Threadripper?
- pella 3y agowith 8 Memory Channels + AMD Ryzen Threadripper PRO 7995WX