4 ms·
They are still too large to run on a normal laptop. Furthermore there must be room left for doing our job. It's a long way until what we use online will be wit
by pmontra 1y ago
They are still too large to run on a normal laptop. Furthermore there must be room left for doing our job. It's a long way until what we use online will be within the reach of a $2000 laptop, better if a $1000 one. My laptop won't run any of them even at unreasonable speed (actually slowness).
- briansun 1y agoTotally fair. On a normal laptop you also need headroom to do your actual job, and KV cache + context length can eat that quickly.