3 ms·
Indeed, but it probably won‘t get any easier for us devs when clients also get easier access to open weight LLMs. I mean from personal experience 40 tok/s on an
by p2detar 2mo ago
Indeed, but it probably won‘t get any easier for us devs when clients also get easier access to open weight LLMs. I mean from personal experience 40 tok/s on an M3 pro with gpt-oss-20b holds up quite well for lots of tasks. Thinks are changing so fast.
- ShinyLeftPad 2mo agoProbably zero of your clients would run them locally (you might, but this is a tiny minority). Also they will never be as good as commercial models. And legality/ethics is still questionable if it is trained on GPL code but is offered under AL2.