2 ms·

And it's hilariously bad (in comparison to regular chatgpt).

• Der_Einzige 3y ago
  And slow. They never tell you that quantization of many LLMs slows down your inference, sometimes by orders of magnitude.
  • arugulum 3y ago
    It depends on the quantization method, but yes some of the most commonly used ones are extremely slow.