3 ms·
IDK about "lazy", but it's certainly an extremely expensive solution.
by darby_eight 3y ago
IDK about "lazy", but it's certainly an extremely expensive solution.
- refulgentis 3y agoA 3B model runs on Android phones from 2 years ago at 6 tkns/s.
- darby_eight 3y agoI'm not sure what you're comparing this to or how you're making this comparison—can you enlighten us? (Somehow I doubt whatever caldav software the above poster references takes more than a second to process multiple emails.)
- throwaway11460 3y agoI don't think it's a comparison - they're just saying that it's fast enough even on old mobile hardware, so it can be used on new hardware even faster. I don't have a problem with a background task taking a minute or something...
- refulgentis 3y agoCost isnt a relevant factor for this
- dugite-code 3y agoIsn't the expensive part of LLM's the training? My understanding is once they are trained they can often be optimized to run quite cheaply. Not as cheaply as a well designed program but cheaply enough it shouldn't be too prohibitive to run.
- denton-scratch 3y agoI'd love to be shown I'm wrong; but I thought most 'runtime' LLMs required a shit-ton of memory. Just downloading one seems to require more storage than I have on this laptop.