3 ms·
Okay but does any one actually _want_ a reasoning model at such low tok/sec speeds?!
by behnamoh 2y ago
Okay but does any one actually _want_ a reasoning model at such low tok/sec speeds?!
- ripped_britches 2y agoLots of use cases don’t require low latency. Background work for agents. CI jobs. Other stuff I haven’t thought of.
- behnamoh 2y agoIf my "automated" CI job takes more than 5 minutes, I'll do it myself..
- bee_rider 2y agoI bet the raspberry pi takes a smaller salary though.
- chickenzzzzu 2y agoThere are tasks that I don't want to do whose delivery sensitivity is 24 hours, aka they can be run while I'm sleeping.
- baq 2y agoWhere I’ve been doing CI 5 minutes was barely enough to warm caches on a cold runner
- Xeoncross 2y agoNo, but the alternative in some places is no reasoning model. Just like people don't want old cell phones / new phones with old chips - but often that's all that is affordable in some places. If we can get something working, then improving it will come.
- rvnx 2y agoYou can have questions that are not urgent. It's like Cursor, I'm fine with the slow version until a certain point, I launch the request then I alt-tab to something else. Yes it's slower, but well, for free (or cheap) it is acceptable.
- deadbabe 2y agoOnly interactive uses cases need high tps, if you just want a process running somewhere ingesting and synthesizing data it’s fine. It’s done when it’s done.
- JKCalhoun 2y agoIt's local?
- baq 2y agoHaving one running in the background constantly looking at your home assistant instance might be an interesting use case for a smart home