3 ms·

I am willing to bet that we will have useful LLMs running locally on high-end home desktops within the year. Certainly ones that exceed Alexa or Siri as they ar

I am willing to bet that we will have useful LLMs running locally on high-end home desktops within the year. Certainly ones that exceed Alexa or Siri as they are today.

jeroenhd 4y ago
I think the difficulty will be teaching tricks to the AI, or making them output actions in the right format. Pygmalion + Tavern/Kobold can run a very convincing chatbot with personality on about 10GiB of VRAM, even better with more. The less VRAM you dedicate, the easier the model runs. For about a thousand dollars you can get an NVIDIA A4000 with 16GiB of VRAM. The GPU isn't particularly fast, but VRAM seems much more important with these models anyway. Gaming GPUs (which perform worse on ML tasks but are still equipped with plenty of VRAM) cost more but are already in the hands of high-end home desktop users. The hardware is ready and available for those wanting to get started. All you need to do is find a way to connect the output of the AI into some kind of smart home system+interact with cron jobs. A few weekends with the Home Assistant API may be enough to get most things done already. The only thing I haven't seen open source AIs do is look for information online, like Bing Chat does.
ClapperHeid 4y ago
>Certainly ones that exceed Alexa or Siri as they are today... Exceeding Alexa wouldn't be difficult. ---- Alexa, play Radio 4 I'm sorry. I don't know any station called Radio 4 Alexa, play Radio 4 [plays Radio 4] ---- This following on from the 1000+ preceding days, where Alexa had also either played Radio 4 on request, or told me she didn't know any station called Radio 4... or done both in the same conversation.
- Baeocystin 4y ago
  I have had much the same experience as what you describe, which is one of the reasons I feel very confident in my prediction!
Baeocystin 4y ago
Less than 10 days after I post this, and someone's got the small version of LLaMa running (slowly! but running) on a Pi. https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209

I am willing to bet that we will have useful LLMs running locally on high-end home desktops within the year. Certainly ones that exceed Alexa or Siri as they are today.

jeroenhd 4y ago

I think the difficulty will be teaching tricks to the AI, or making them output actions in the right format. Pygmalion + Tavern/Kobold can run a very convincing chatbot with personality on about 10GiB of VRAM, even better with more. The less VRAM you dedicate, the easier the model runs. For about a thousand dollars you can get an NVIDIA A4000 with 16GiB of VRAM. The GPU isn't particularly fast, but VRAM seems much more important with these models anyway. Gaming GPUs (which perform worse on ML tasks but are still equipped with plenty of VRAM) cost more but are already in the hands of high-end home desktop users. The hardware is ready and available for those wanting to get started. All you need to do is find a way to connect the output of the AI into some kind of smart home system+interact with cron jobs. A few weekends with the Home Assistant API may be enough to get most things done already. The only thing I haven't seen open source AIs do is look for information online, like Bing Chat does.

ClapperHeid 4y ago

>Certainly ones that exceed Alexa or Siri as they are today... Exceeding Alexa wouldn't be difficult. ---- Alexa, play Radio 4 I'm sorry. I don't know any station called Radio 4 Alexa, play Radio 4 [plays Radio 4] ---- This following on from the 1000+ preceding days, where Alexa had also either played Radio 4 on request, or told me she didn't know any station called Radio 4... or done both in the same conversation.

Baeocystin 4y ago
I have had much the same experience as what you describe, which is one of the reasons I feel very confident in my prediction!

Baeocystin 4y ago

Less than 10 days after I post this, and someone's got the small version of LLaMa running (slowly! but running) on a Pi. https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209