4 ms·
I'm running a chat LLM on my local pc. It spits out text just slightly faster than I can type, but it is using much of my CPU and redlining my GPU. Is it proc
by kbelder 2y ago
I'm running a chat LLM on my local pc. It spits out text just slightly faster than I can type, but it is using much of my CPU and redlining my GPU.
Is it processing at a dozen bits per second, or hundreds of millions?
If the text the LLM generates is "that is true", can I consider that one bit of information?
I agree, they're artificially simplifying the framing of the question to generate a lower number than is sensible.