4 ms·

Do you know what kind of speed you can expect with inference on CPU?

• MacsHeadroom 4y ago
  Yes, around 2 tokens per second on a newer CPU with fast RAM.