3 ms·
I agree it’s probably not representative token speed. But I do believe the overall observation holds: The monetary value of local inference is bound by the wall
by millicentricism 1mo ago
I agree it’s probably not representative token speed. But I do believe the overall observation holds: The monetary value of local inference is bound by the wall clock.
I agree that there are many other reasons than cost alone.