6 ms·

Founder Home Assistant here. Let me know if anyone has any questions. Edit: if you want to keep in the loop of the work we're doing, subscribe to our free mont

Founder Home Assistant here. Let me know if anyone has any questions. Edit: if you want to keep in the loop of the work we're doing, subscribe to our free monthly newsletter @ https://building.open-home.io/ https://building.open-home.io/

miki123211 3y ago
Have you considered integrating RHVoice in addition to the Piper models? It's nowhere near as exciting (it uses HMMs instead of cool new shiny neural network goodness), but the use-cases are sort of similar (it was originally developed for screen-readers, which require extremely low latencies and high responsiveness.) It doesn't support as many languages, but for some of the languages it does support (Polish for example), I find the quality superior to what Piper can offer.
moritonal 3y ago
Does the API of the Agent support the phrase "In ten minutes, turn off any light bulb beginning with the letter B"? I hope you saw the recent demo of a ChatGPT plug-in for HA that might add a lot of functionality to HA if there is the ability to use it to construct action plans and then execute them. https://www.reddit.com/r/homeassistant/comments/12l10se/introducing_a_home_assistant_plugin_for_chatgpt https://www.reddit.com/r/homeassistant/comments/12l10se/intr...
- balloob 3y ago
  In Home Assistant a voice assistant is a combination of a speech-to-text, conversation and text-to-speech engines. A user can create as many combinations as they want. For a conversation agent, it's text command in, text response out. We already have the option to use OpenAI GPT 3.5, so it's possible in the future to hook it up to a large language model and allow it to control your house. Fun fact: we supported Almond (later Genie), the IoT AI from Stanford which they released a couple of years ago. It would convert natural language into their ThingTalk language, which could do things like you mention. It was not popular at all and Stanford ended up shutting it down.
  - windexh8er 3y ago
    I remember the Almond days. Back then they were up against Vera which, IMO, was the clear leader. Then the guys from SmartThings happened to come out with their product which I saw at a MinneStar event. I asked one of the founders after the preso what their perspective on cloud connected services was and the guy couldn't believe I had an opinion on a home automation system that couldn't operate without cloud services. They're were clearly going after a much simpler consumer market and looking to sell out, of which they did. But it was a disaster for anyone that was used to the configurability and flexibility of Vera. Then Vera started to lose focus and HomeSeer seemed to come up as a viable replacement, along with the early days of HomeAssistant. It's interesting that HA is only getting better while all the rest are slowly dying off. HomeSeer v4 is still pretty rough transition from v3. I only hope HA stays the course and continues to hold their high standards. It's such a phenomenal project. From simple to complex automation, it is truly a great work of open source.
outworlder 3y ago
That's pretty good. Amazon Echo integration is one of the main Cloud dependencies I have. I also like that you are directly supporting ESPHome and I'll definitely take that route and build something nice. However, for the more casual users, are you thinking about releasing an off-the-shelf "voice assistant" device?
- balloob 3y ago
  Yes, that's the plan. We're still exploring different ML on the edge chips to help with wake word and audio processing. One hard requirement that we have is that the tooling is openly available, so that anyone can convert an AI model and run it on the chip. We don't accept tooling to be available as a website or under NDA. That challenge is harder than we expected!
  - photonbeam 3y ago
    What does an ML-on-the-edge system chip do differently from other chips? Does it have a gpu or other accelerator on it?
    - flangola7 3y ago
      Some labs are developing analogue chips instead of digital
    - baq 3y ago
      Domain specific fixed function hardware could consume 10-100x less power than a gpu for the same workload. Think video decode or ‘YouTube battery life’ benchmarks but for speech recognition.
xrd 3y ago
Can I repurpose my Mycroft device with this? I'm worried it is now a dead rock.
- synesthesiam 3y ago
  Mark I or Mark II?
  - xrd 3y ago
    Mark II.
    - synesthesiam 3y ago
      I used to work for Mycroft, so I'm hoping to eventually create an image that's compatible with Home Assistant pipelines. For now, though, you may want to check out OVOS: https://openvoiceos.com/ https://openvoiceos.com/
      - xrd 3y ago
        I checked it out but didn't see that it runs on my hardware. I'm worried this gets me closer not father to brick status with my device.
        synesthesiam 3y ago
        Neon AI and OVOS are taking over development of Mycroft Core and the Mark II: https://www.reddit.com/r/Mycroftai/comments/1212h87/neon_ai_and_openvoiceos_ovos_are_taking_the_lead/ https://www.reddit.com/r/Mycroftai/comments/1212h87/neon_ai_...
  - bluGill 3y ago
    I have a mark I that hasn't been plugged in for years. Iirc The SD card is bad.
chrismiceli 3y ago
What's the relationship between this work and rhasspy 3?
- synesthesiam 3y ago
  Home Assistant and Rhasspy 3 will be able to share voice services thanks to the shared Wyoming protocol. Rhasspy 3 will have more options, including lots of experimental services.
candiddevmike 3y ago
I built an app called Homechart that I think would be a good partnership/would appreciate help with integrating into home assistant (or you could just aquire it =D), contact details in my profile
IgorPartola 3y ago
First of all, thank you so much for HA. My family uses it daily and despite some of them being at first resistant to anything smart they now can’t imagine life without it. I have a general question about HA: the main issues I’ve had with it have had to do with specifically only two points. 1. The recorder module has brought the system to a crawl when a single rogue device on my network kept spamming MQTT with status updates about power consumption and eventually killed the SD card it was running on. Some aggressive ignore statements in the config fixed the issue but I still see this as a major pain point as there is no indication of anything happening. Is there any plan to introduce any kind of housekeeping into the recorder module besides periodic purges (which in my case were not enough to actually keep the system running)? 2. I like to have control over my computers so I did not want to use a pre-installed image of HA for RPi. But the alternative is a cumbersome process of installing it in a Python virtual env and keeping things updated which is not ideal. Are there any plans for improving this installation path? Or alternatively some sort of advanced version of the HA image which would allow someone like me full control over the base OS while keeping the benefits of the cohesive self-updating HA distribution?
- balloob 3y ago
  1. We don't currently have anything for that besides looking at the database directly. 2. We offer VM images if you want to keep flexibility but also benefit from all our work on an integrated system. You can also do a Supervised installation, but then it's your own responsibility to keep the host OS running the requirements of HA.
  - veleek 3y ago
    I’m a HA Container user and I can vouch for this path. It gives me the ability to run whatever other stuff I want to alongside HA in Docker containers with minimal effort. Works really well, updating is low effort (but not automatic) but you could probably set something up if you wanted to pretty easily.
    - windexh8er 3y ago
      I do this as well and leverage Watchtower [0] to upgrade the container for me and keep the image cache flushed. I just track the `stable` tag and make sure to keep an eye on changes. I only check for updates once a week and it has worked out great. [0] https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/
splitrocket 3y ago
First off, Thank you! Wondering if you all are thinking about multiple remote microphones, and choosing which microphone will respond to the speaker? Ideally, I'd love to have multiple microphones/speakers throughout the house, all of which can listen and answer, however, only the one that hears me the clearest/loudest/etc. actually answers. Make sense?
- balloob 3y ago
  Once we get our hardware done, this is feature that we definitely plan to include (but probably as a later software update)
  - freen 3y ago
    That is incredibly rad.
bioemerl 3y ago
No questions, only a comment, you kick ass.
LelouBil 3y ago
Are you looking into making fine tuning easy for the speech to text model ? I feel like that's the only thing missing before it's perfect. My entity names can contain the English or other text that just won't be picked up by the speech to text, so if there was a way to record a couple of pronunciations for an entity name, and home assistant would fine-tune whisper on them in the background it would be wonderful !
Havoc 3y ago
How does this setup cope with multiple microphone? ie potentially overlapping ones. If I toggle a light I wouldn’t want it to execute the command twice haha

miki123211 3y ago

Have you considered integrating RHVoice in addition to the Piper models? It's nowhere near as exciting (it uses HMMs instead of cool new shiny neural network goodness), but the use-cases are sort of similar (it was originally developed for screen-readers, which require extremely low latencies and high responsiveness.) It doesn't support as many languages, but for some of the languages it does support (Polish for example), I find the quality superior to what Piper can offer.

moritonal 3y ago

Does the API of the Agent support the phrase "In ten minutes, turn off any light bulb beginning with the letter B"? I hope you saw the recent demo of a ChatGPT plug-in for HA that might add a lot of functionality to HA if there is the ability to use it to construct action plans and then execute them. https://www.reddit.com/r/homeassistant/comments/12l10se/introducing_a_home_assistant_plugin_for_chatgpt https://www.reddit.com/r/homeassistant/comments/12l10se/intr...

balloob 3y ago
In Home Assistant a voice assistant is a combination of a speech-to-text, conversation and text-to-speech engines. A user can create as many combinations as they want. For a conversation agent, it's text command in, text response out. We already have the option to use OpenAI GPT 3.5, so it's possible in the future to hook it up to a large language model and allow it to control your house. Fun fact: we supported Almond (later Genie), the IoT AI from Stanford which they released a couple of years ago. It would convert natural language into their ThingTalk language, which could do things like you mention. It was not popular at all and Stanford ended up shutting it down.
- windexh8er 3y ago
  I remember the Almond days. Back then they were up against Vera which, IMO, was the clear leader. Then the guys from SmartThings happened to come out with their product which I saw at a MinneStar event. I asked one of the founders after the preso what their perspective on cloud connected services was and the guy couldn't believe I had an opinion on a home automation system that couldn't operate without cloud services. They're were clearly going after a much simpler consumer market and looking to sell out, of which they did. But it was a disaster for anyone that was used to the configurability and flexibility of Vera. Then Vera started to lose focus and HomeSeer seemed to come up as a viable replacement, along with the early days of HomeAssistant. It's interesting that HA is only getting better while all the rest are slowly dying off. HomeSeer v4 is still pretty rough transition from v3. I only hope HA stays the course and continues to hold their high standards. It's such a phenomenal project. From simple to complex automation, it is truly a great work of open source.

outworlder 3y ago

That's pretty good. Amazon Echo integration is one of the main Cloud dependencies I have. I also like that you are directly supporting ESPHome and I'll definitely take that route and build something nice. However, for the more casual users, are you thinking about releasing an off-the-shelf "voice assistant" device?

balloob 3y ago
Yes, that's the plan. We're still exploring different ML on the edge chips to help with wake word and audio processing. One hard requirement that we have is that the tooling is openly available, so that anyone can convert an AI model and run it on the chip. We don't accept tooling to be available as a website or under NDA. That challenge is harder than we expected!
- photonbeam 3y ago
  What does an ML-on-the-edge system chip do differently from other chips? Does it have a gpu or other accelerator on it?
  - flangola7 3y ago
    Some labs are developing analogue chips instead of digital
  - baq 3y ago
    Domain specific fixed function hardware could consume 10-100x less power than a gpu for the same workload. Think video decode or ‘YouTube battery life’ benchmarks but for speech recognition.

xrd 3y ago

Can I repurpose my Mycroft device with this? I'm worried it is now a dead rock.

synesthesiam 3y ago
Mark I or Mark II?
- xrd 3y ago
  Mark II.
  - synesthesiam 3y ago
    I used to work for Mycroft, so I'm hoping to eventually create an image that's compatible with Home Assistant pipelines. For now, though, you may want to check out OVOS: https://openvoiceos.com/ https://openvoiceos.com/
    - xrd 3y ago
      I checked it out but didn't see that it runs on my hardware. I'm worried this gets me closer not father to brick status with my device.
      - synesthesiam 3y ago
        Neon AI and OVOS are taking over development of Mycroft Core and the Mark II: https://www.reddit.com/r/Mycroftai/comments/1212h87/neon_ai_and_openvoiceos_ovos_are_taking_the_lead/ https://www.reddit.com/r/Mycroftai/comments/1212h87/neon_ai_...
- bluGill 3y ago
  I have a mark I that hasn't been plugged in for years. Iirc The SD card is bad.

chrismiceli 3y ago

What's the relationship between this work and rhasspy 3?

synesthesiam 3y ago
Home Assistant and Rhasspy 3 will be able to share voice services thanks to the shared Wyoming protocol. Rhasspy 3 will have more options, including lots of experimental services.

candiddevmike 3y ago

I built an app called Homechart that I think would be a good partnership/would appreciate help with integrating into home assistant (or you could just aquire it =D), contact details in my profile

IgorPartola 3y ago

First of all, thank you so much for HA. My family uses it daily and despite some of them being at first resistant to anything smart they now can’t imagine life without it. I have a general question about HA: the main issues I’ve had with it have had to do with specifically only two points. 1. The recorder module has brought the system to a crawl when a single rogue device on my network kept spamming MQTT with status updates about power consumption and eventually killed the SD card it was running on. Some aggressive ignore statements in the config fixed the issue but I still see this as a major pain point as there is no indication of anything happening. Is there any plan to introduce any kind of housekeeping into the recorder module besides periodic purges (which in my case were not enough to actually keep the system running)? 2. I like to have control over my computers so I did not want to use a pre-installed image of HA for RPi. But the alternative is a cumbersome process of installing it in a Python virtual env and keeping things updated which is not ideal. Are there any plans for improving this installation path? Or alternatively some sort of advanced version of the HA image which would allow someone like me full control over the base OS while keeping the benefits of the cohesive self-updating HA distribution?

balloob 3y ago
1. We don't currently have anything for that besides looking at the database directly. 2. We offer VM images if you want to keep flexibility but also benefit from all our work on an integrated system. You can also do a Supervised installation, but then it's your own responsibility to keep the host OS running the requirements of HA.
- veleek 3y ago
  I’m a HA Container user and I can vouch for this path. It gives me the ability to run whatever other stuff I want to alongside HA in Docker containers with minimal effort. Works really well, updating is low effort (but not automatic) but you could probably set something up if you wanted to pretty easily.
  - windexh8er 3y ago
    I do this as well and leverage Watchtower [0] to upgrade the container for me and keep the image cache flushed. I just track the `stable` tag and make sure to keep an eye on changes. I only check for updates once a week and it has worked out great. [0] https://containrrr.dev/watchtower/ https://containrrr.dev/watchtower/

splitrocket 3y ago

First off, Thank you! Wondering if you all are thinking about multiple remote microphones, and choosing which microphone will respond to the speaker? Ideally, I'd love to have multiple microphones/speakers throughout the house, all of which can listen and answer, however, only the one that hears me the clearest/loudest/etc. actually answers. Make sense?

balloob 3y ago
Once we get our hardware done, this is feature that we definitely plan to include (but probably as a later software update)
- freen 3y ago
  That is incredibly rad.

bioemerl 3y ago

No questions, only a comment, you kick ass.

LelouBil 3y ago

Are you looking into making fine tuning easy for the speech to text model ? I feel like that's the only thing missing before it's perfect. My entity names can contain the English or other text that just won't be picked up by the speech to text, so if there was a way to record a couple of pronunciations for an entity name, and home assistant would fine-tune whisper on them in the background it would be wonderful !

Havoc 3y ago

How does this setup cope with multiple microphone? ie potentially overlapping ones. If I toggle a light I wouldn’t want it to execute the command twice haha