3 ms·

how long for one token to infer on an average cpu?

• kristianp 4y ago
  From the readme: On a Ryzen 7900X, the 7B model is able to infer several words per second, quite a lot better than you'd expect!
• markasoftware 4y ago
  I tested on a decidedly above average CPU, and got several words per second on the 7B model. I'd guess maybe one word per second on a more average one?
  • raverbashing 4y ago
    Cool so we're back to the days of 2400 baud modems
    • asplake 4y ago
      Simulating a slow typist
    • Tepix 4y ago
      More like 300 baud. At 300 baud (30 cps) you can still read it as it arrives.
• deleted 4y ago
  [deleted]