3 ms·
You can just use a CPU in that case, no? You can run most ML inference on vectorized operations on modern CPUs at a fraction of the price.
by m00x 2y ago
You can just use a CPU in that case, no? You can run most ML inference on vectorized operations on modern CPUs at a fraction of the price.
- marcyb5st 2y agoMy 7800x says not really. Compared to my 3070 it feels so incredibly slow that gets in the way of productivity. Specifically, waiting ~2 seconds vs ~20 for a code snippet is much more detrimental to my productivity than the time difference would suggest. In ~2 seconds I don't get distracted, in ~20 seconds my mind starts wandering and then I have to spend time refocusing. Make a GPU that is 50% slower than a 2 generations older mid-range GPU (in tokens/s) but on bigger models and I would gladly shell out 1000+$. So much so that I am considering getting a 5090 if nVdia actually fixes the connector mess they made with 4090s or even a used v100.
- refulgentis 2y agoI don't understand, make it slower so it's faster?
- magicalhippo 2y agoMy 2080Ti at half speed would still beat the crap out of my 5900X CPU for inference, as long as the model fits in VRAM. I think that's what GP was alluding to.
- m00x 2y agoI'm running codeseeker 13B model on my macbook with no perf issues and I get a response within a few seconds. Running a specialist model makes more sense on small devices.