4 ms·

How long before people can buy their own in-home personal chatbot off-the-grid system? Might be profitable at >$100,000 per unit at present, though the service

How long before people can buy their own in-home personal chatbot off-the-grid system? Might be profitable at >$100,000 per unit at present, though the service and maintenance fees would likely be pretty steep per month, including the electricity bill. See Joi, Blade Runner 2049.

Baeocystin 4y ago
I am willing to bet that we will have useful LLMs running locally on high-end home desktops within the year. Certainly ones that exceed Alexa or Siri as they are today.
- jeroenhd 4y ago
  I think the difficulty will be teaching tricks to the AI, or making them output actions in the right format. Pygmalion + Tavern/Kobold can run a very convincing chatbot with personality on about 10GiB of VRAM, even better with more. The less VRAM you dedicate, the easier the model runs. For about a thousand dollars you can get an NVIDIA A4000 with 16GiB of VRAM. The GPU isn't particularly fast, but VRAM seems much more important with these models anyway. Gaming GPUs (which perform worse on ML tasks but are still equipped with plenty of VRAM) cost more but are already in the hands of high-end home desktop users. The hardware is ready and available for those wanting to get started. All you need to do is find a way to connect the output of the AI into some kind of smart home system+interact with cron jobs. A few weekends with the Home Assistant API may be enough to get most things done already. The only thing I haven't seen open source AIs do is look for information online, like Bing Chat does.
- ClapperHeid 4y ago
  >Certainly ones that exceed Alexa or Siri as they are today... Exceeding Alexa wouldn't be difficult. ---- Alexa, play Radio 4 I'm sorry. I don't know any station called Radio 4 Alexa, play Radio 4 [plays Radio 4] ---- This following on from the 1000+ preceding days, where Alexa had also either played Radio 4 on request, or told me she didn't know any station called Radio 4... or done both in the same conversation.
  - Baeocystin 4y ago
    I have had much the same experience as what you describe, which is one of the reasons I feel very confident in my prediction!
- Baeocystin 4y ago
  Less than 10 days after I post this, and someone's got the small version of LLaMa running (slowly! but running) on a Pi. https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209
Gigachad 4y ago
Still cheaper than a real person + divorce
nullc 4y ago
It's really just a gpu vram limitation: affordable GPUs are rather memory starved. Fortunately people have started writing implementations for pipelining across multiple gpus. https://github.com/Ying1123/FlexGen https://github.com/Ying1123/FlexGen
ClapperHeid 4y ago
>How long before people can buy their own in-home personal chatbot off-the-grid system?... That's something else that I don't think has been discussed much; people worry about internet security and big tech [often Google] knowing all about them. How much more potentially damaging or blackmail-worthy would it be to have a security breach at one of these AI chatbot companies and have the fact come to light that you like to have your chatbot avatar wear a school uniform and jackboots and [textually, speaking] spank you on the bare bum with a copy of Mein Kampf? I think self-hosted is definitely the way to go here!

Baeocystin 4y ago

I am willing to bet that we will have useful LLMs running locally on high-end home desktops within the year. Certainly ones that exceed Alexa or Siri as they are today.

jeroenhd 4y ago
I think the difficulty will be teaching tricks to the AI, or making them output actions in the right format. Pygmalion + Tavern/Kobold can run a very convincing chatbot with personality on about 10GiB of VRAM, even better with more. The less VRAM you dedicate, the easier the model runs. For about a thousand dollars you can get an NVIDIA A4000 with 16GiB of VRAM. The GPU isn't particularly fast, but VRAM seems much more important with these models anyway. Gaming GPUs (which perform worse on ML tasks but are still equipped with plenty of VRAM) cost more but are already in the hands of high-end home desktop users. The hardware is ready and available for those wanting to get started. All you need to do is find a way to connect the output of the AI into some kind of smart home system+interact with cron jobs. A few weekends with the Home Assistant API may be enough to get most things done already. The only thing I haven't seen open source AIs do is look for information online, like Bing Chat does.
ClapperHeid 4y ago
>Certainly ones that exceed Alexa or Siri as they are today... Exceeding Alexa wouldn't be difficult. ---- Alexa, play Radio 4 I'm sorry. I don't know any station called Radio 4 Alexa, play Radio 4 [plays Radio 4] ---- This following on from the 1000+ preceding days, where Alexa had also either played Radio 4 on request, or told me she didn't know any station called Radio 4... or done both in the same conversation.
- Baeocystin 4y ago
  I have had much the same experience as what you describe, which is one of the reasons I feel very confident in my prediction!
Baeocystin 4y ago
Less than 10 days after I post this, and someone's got the small version of LLaMa running (slowly! but running) on a Pi. https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209

Gigachad 4y ago

Still cheaper than a real person + divorce

nullc 4y ago

It's really just a gpu vram limitation: affordable GPUs are rather memory starved. Fortunately people have started writing implementations for pipelining across multiple gpus. https://github.com/Ying1123/FlexGen https://github.com/Ying1123/FlexGen

ClapperHeid 4y ago

>How long before people can buy their own in-home personal chatbot off-the-grid system?... That's something else that I don't think has been discussed much; people worry about internet security and big tech [often Google] knowing all about them. How much more potentially damaging or blackmail-worthy would it be to have a security breach at one of these AI chatbot companies and have the fact come to light that you like to have your chatbot avatar wear a school uniform and jackboots and [textually, speaking] spank you on the bare bum with a copy of Mein Kampf? I think self-hosted is definitely the way to go here!