3 ms·
> LLMs can’t run locally yet. "Yet" is a big word here when it comes to the field as a whole. I got Alpaca-LoRA up and running on my desktop machine with a 308
by crooked-v 4y ago
> LLMs can’t run locally yet.
"Yet" is a big word here when it comes to the field as a whole. I got Alpaca-LoRA up and running on my desktop machine with a 3080 the other day and I'd say it's about 50% as good as ChatGPT 3.5 and fast enough to already be usable for most minor things ("summarize this text", etc) if only the available UIs were better.
I feel like we're not far off from the point where it'll be possible to buy something of ChatGPT 3.5 quality as a home hardware appliance that can then hook into a bunch of things.
- pbronez 4y agoAgree that Alpaca is Important. I got the smallest one running on a pathetic notebook… 2 cores, 8GB RAM. It was slow. It was sloppy. But it worked. Getting these things running on GPU/NPU will be very compelling, especially if we don’t hit a wall on compression. I think a sweet spot exists where consumer clients are powerful enough and models are small enough to deliver value and privacy.