2 ms·
Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this har
by montroser 4mo ago
Result is ~12 tokens per second, as reported by OP down in these comments here.
An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.
- andix 4mo agoEspecially if you consider those smaller models are really cheap and fast on platforms like openrouter. Often by the factor 100-500 cheaper than SOTA models, and 2-5x in TPS.
- causal 4mo agoYeah took way too long to find that result. Being able to run on slow RAM isn't surprising considering you can run a model off an SSD.
- greenavocado 4mo agoI was about to ask that
- gowld 4mo agoRight. You can also perform RSA encryption on pencil and paper with a scientific calculator. It works, but it's not useful throughput for serious work
- kingnothing 4mo agoIt's not terrible for interactive... https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text And it should be just fine for plenty of background use cases.