2 ms·

How great is the performance? Tokens/s?

• yjftsjthsd-h 2y ago
  Random sample query ("What shape should a kumquat be?") against a 7B model quantised to 4b running on an i7-9750H (so a good CPU, but also a good laptop CPU from 2019) gives: 148 tokens predicted, 159 ms per token, 6.27 tokens per second
  • bagels 2y ago
    Thanks, that helps.