3 ms·

This is very helpful, thanks. I am grateful for your explanations of the voice aspect - Amazon has kept all of this under wraps so much so that I didn't unders

This is very helpful, thanks. I am grateful for your explanations of the voice aspect - Amazon has kept all of this under wraps so much so that I didn't understand all the difficulties until I read your site just now. I can guess at the difficulties based on how often I have to repeat simple commands to Alexa, or "normal people bread" (my fam's word for non-grain, non-healthy bread) gets changed to "normal maple bread," things like that. Practical question - are there plans to commercialize a turn-key Willow setup? Basically, I want an Alexa setup without the dependence on (or financial expectations from) Amazon. Key features: * Few terminals, maximum reach. I want to be heard without having a cube every 10 feet. I'm okay shouting if need be. The ability to whisper is nice, too. * Reminders, timers, alarms, schedules. Alexa, set egg timer for three minutes. Alexa, remind me six months from now at 8pm to change the furnace filter. * Cloud-based lists: shopping shopping shopping. Alexa, add milk to the shopping list. * Home automation: plugs, lightbulbs minimum. Thermostat nice to have. * Routines. Alexa, turn on the living room lights. Alexa, fix the TV (occasionally the HDMI splitter freaks out and quits working; "fix the TV" turns off the power plug, waits five seconds, then turns it back on again.) * Music: ability to play Spotify. Nice to have is "play this everywhere" capability. * Basic math/language capabilities: Alexa, what is one point oh three raised to the 30th power (e.g., for calculating inflation over 30 years)? Alexa, define opprobrium. * Other integrations: bluetooth, especially for speakers. As for the other Alexa features - I only "reorder" things from Alexa. I never shop for new things, because shopping requires me to read reviews, see alternatives, who else bought what else, visit websites, and so on. I think Amazon missed the mark here - or rather, the product team promised senior management that the shopping experience would be easy, but it's not. Fail. (If they charged me a nominal fee for Alexa, I would happily oblige. Sssh, don't tell them that.) I also am itching to try a Raspberry Pi temperature probe, and somehow tie it into my outdoor pool heater.

kkielhofner 3y ago
My pleasure! If you have a lot of time and patience you can read my very long recent response[0] to a user asking about this where I go into arguably far too much detail. For context Wyoming is the fundamental protocol for the entire Home Assistant/ESPHome implementation. In terms of commercialization, yes and no. Speaking of experience and background - the real genesis of Willow was three years ago when I was approached by a large publicly traded company in the healthcare space to implement a voice assistant for their use cases. The underlying and fundamental technology just wasn't there yet and it never got past PoC stage. That failure really bothered me. Earlier this year everything seemed to really align and thus Willow was born. Our monetization strategy is where this all started three years ago - commercial and "enterprise" use cases. It's the only market I know and have experience with. This is actually relevant to my comments about fundamental approach and architecture. Willow was designed for this market and their use cases which needless to say has substantially higher expectations than the more-or-less consumer/"tinkerer" market. I will NEVER monetize consumer/home users directly (again, I don't understand the market) but they get to benefit from our substantially better approach. All of that said I'd certainly be interested in partnering with someone to bring Willow to the consumer market commercially. I've spent my entire career with the "sales cycle measured in months with a big customer for a lot of money" approach, not the "tons of consumers paying $5 a month" approach. Everything you're describing is what we're currently focusing on. We feel we have the fundamentals of voice and audio pretty much nailed. Then the next question becomes "Great, I have performant and accurate speech in and out. What do I actually do with it?". Taking the speech recognition transcripts and matching them with flexible grammar, functionality and application routing, etc is the next significant challenge. We have some of our own logic internally that we will be building out but at this early stage we heavily rely on the command processing and intent recognition from our supported command endpoints. Nothing you're describing is impossible (of course) but Alexa (for example) has an eight year head start. It's going to take a while to reach feature parity with that. Our intention is to not only match Alexa functionality but also far surpass it by leveraging the entire open source ecosystem. [0] - https://github.com/toverainc/willow-inference-server/discussions/135 https://github.com/toverainc/willow-inference-server/discuss...
- PennRobotics 3y ago
  I know it's difficult, but is there a way to gather feedback from the user when a voice command is below a certain threshold of understandability? Or to build a context catalog, such as time of day specific types of commands are expected? I think these two areas could be holding the commercial voice assistants back, since the majority of bug reports on the Google Home and Alexa subreddits are people complaining about being misheard.
  - kkielhofner 3y ago
    I've been thinking about this a little recently. Raw brain dump incoming... The problem with intent matching now (say with Home Assistant) is that the transcribed speech more-or-less needs to exactly (character for character) match the defined entities (names of lights, etc) as well as the grammar and structure expected by the intent definitions. Something as simple as a hyphen in "turn-off" currently breaks the matching. That's an easy one to fix, I'm just providing it as an example. In terms of addressing this, I've kicked around a few ideas: With Home Assistant at least we can pull all of the entities and supported intents. We have a variety of options to do all kinds of fuzzy matching with these now known phrases. We'd fix the transcription based on something like nearest neighbor (more or less) and send the clean command to HA. There is also a way to handle this in the prompt to Whisper. A literal text embedding model with nearest neighbor search seems kind of ridiculous but I've been curious about it. The same could be done at the audio level by essentially making an embedding of the Mel Spectrogram of the actual audio (which we could have available) and searching on that. There are also approaches with a variety of NLP implementations, language models, etc that could be combined or used separately. We've also considered some more manual approaches - things like an interface that logs sessions where you can (after the fact) find a session/command that went wrong, and basically map it to what you wanted. While this may seem tedious we get some reports that our speech recognition consistently mis-transcribes a given voice command. Implementing a lookup table to correct for this is simple. The biggest challenge to this is how to deal with the UI/UX aspects.
PennRobotics 3y ago
> Basically, I want an Alexa setup without the dependence on (or financial expectations from) Amazon Amen. +1 to timers, schedules, integration with a lighting bridge or even kitchen appliances e.g. Home Connect, custom routines, music (including Bluetooth and/or TuneIn), line out (so I can connect my own speakers), microphone relays that I can put all around the house instead of owning 5 or 6 RaspPis and the associated 15 to 30 watts Better than math, for me, would be a Wikipedia summary readout that voices the first paragraph (or a reasonably defined "summary length") from simple.wikipedia.com, although metric/imperial conversions are also very handy. Weather integration would also be delightful---including a way to tie this in to reminders e.g. "bring the plants indoors tonight", "remember rain clothes for riding home this evening", "tomorrow is probably the last good sled day until next winter" Oh. And properly bilingual. We have a Philips light named "Turtle". Alexa absolutely cannot handle this name if the input is set to either German or German/English.
- kkielhofner 3y ago
  > microphone relays that I can put all around the house instead of owning 5 or 6 RaspPis and the associated 15 to 30 watts I've always found this approach odd and this is one the reasons we primarily target the ESP-BOX-3 from Espressif[0]. For $50 you take it out of the box, flash from our web flasher, and stick it in your house. Given that it's ESP32-S3 based total power consumption is somewhere in the 100 milliwatt range. Once initially flashed all upgrades are done over the air (with < 5MB binaries) from our web management interface. All configuration changes are also centrally managed there. Make change, click "Save and Apply", goes live on any number of associated devices in seconds. The thought of managing half a dozen full Linux machines (Pis) just to get audio for stuff like this boggles my mind. Additionally, the audio quality is significantly better with the ESP-BOX-3 vs Pi, etc because the entire device and software stack is holistically engineered for high quality far-field voice capture. Acoustically tuned enclosure, dual microphones, acoustic echo cancellation, blind source separation, voice activity detection, wake word, etc. As I like to tell people - if you think you're going to slap a random microphone together with a Raspberry Pi and compete with Alexa you have another thing coming. I go into greater detail in this issue[1]. Getting high quality audio is step one. Step two is accurate transcription of that audio. Steps 3 - ? are what you're describing. We know what you said, now what do we do with it? Now that we feel steps one and two are more-or-less nailed steps three and beyond are our next focus. The good news is the final step is of course user feedback and we have that handled with text to speech and output on the BOX-3 LCD display. It's just currently the wide gulf in between these that needs focus. In short, our mouth and ears work well. It's the brain that needs work. [0] - https://www.espressif.com/en/news/ESP32-S3-BOX-3 https://www.espressif.com/en/news/ESP32-S3-BOX-3 [1] - https://github.com/toverainc/willow/issues/317#issuecomment-1774172563 https://github.com/toverainc/willow/issues/317#issuecomment-...

kkielhofner 3y ago

My pleasure! If you have a lot of time and patience you can read my very long recent response[0] to a user asking about this where I go into arguably far too much detail. For context Wyoming is the fundamental protocol for the entire Home Assistant/ESPHome implementation. In terms of commercialization, yes and no. Speaking of experience and background - the real genesis of Willow was three years ago when I was approached by a large publicly traded company in the healthcare space to implement a voice assistant for their use cases. The underlying and fundamental technology just wasn't there yet and it never got past PoC stage. That failure really bothered me. Earlier this year everything seemed to really align and thus Willow was born. Our monetization strategy is where this all started three years ago - commercial and "enterprise" use cases. It's the only market I know and have experience with. This is actually relevant to my comments about fundamental approach and architecture. Willow was designed for this market and their use cases which needless to say has substantially higher expectations than the more-or-less consumer/"tinkerer" market. I will NEVER monetize consumer/home users directly (again, I don't understand the market) but they get to benefit from our substantially better approach. All of that said I'd certainly be interested in partnering with someone to bring Willow to the consumer market commercially. I've spent my entire career with the "sales cycle measured in months with a big customer for a lot of money" approach, not the "tons of consumers paying $5 a month" approach. Everything you're describing is what we're currently focusing on. We feel we have the fundamentals of voice and audio pretty much nailed. Then the next question becomes "Great, I have performant and accurate speech in and out. What do I actually do with it?". Taking the speech recognition transcripts and matching them with flexible grammar, functionality and application routing, etc is the next significant challenge. We have some of our own logic internally that we will be building out but at this early stage we heavily rely on the command processing and intent recognition from our supported command endpoints. Nothing you're describing is impossible (of course) but Alexa (for example) has an eight year head start. It's going to take a while to reach feature parity with that. Our intention is to not only match Alexa functionality but also far surpass it by leveraging the entire open source ecosystem. [0] - https://github.com/toverainc/willow-inference-server/discussions/135 https://github.com/toverainc/willow-inference-server/discuss...

PennRobotics 3y ago
I know it's difficult, but is there a way to gather feedback from the user when a voice command is below a certain threshold of understandability? Or to build a context catalog, such as time of day specific types of commands are expected? I think these two areas could be holding the commercial voice assistants back, since the majority of bug reports on the Google Home and Alexa subreddits are people complaining about being misheard.
- kkielhofner 3y ago
  I've been thinking about this a little recently. Raw brain dump incoming... The problem with intent matching now (say with Home Assistant) is that the transcribed speech more-or-less needs to exactly (character for character) match the defined entities (names of lights, etc) as well as the grammar and structure expected by the intent definitions. Something as simple as a hyphen in "turn-off" currently breaks the matching. That's an easy one to fix, I'm just providing it as an example. In terms of addressing this, I've kicked around a few ideas: With Home Assistant at least we can pull all of the entities and supported intents. We have a variety of options to do all kinds of fuzzy matching with these now known phrases. We'd fix the transcription based on something like nearest neighbor (more or less) and send the clean command to HA. There is also a way to handle this in the prompt to Whisper. A literal text embedding model with nearest neighbor search seems kind of ridiculous but I've been curious about it. The same could be done at the audio level by essentially making an embedding of the Mel Spectrogram of the actual audio (which we could have available) and searching on that. There are also approaches with a variety of NLP implementations, language models, etc that could be combined or used separately. We've also considered some more manual approaches - things like an interface that logs sessions where you can (after the fact) find a session/command that went wrong, and basically map it to what you wanted. While this may seem tedious we get some reports that our speech recognition consistently mis-transcribes a given voice command. Implementing a lookup table to correct for this is simple. The biggest challenge to this is how to deal with the UI/UX aspects.

PennRobotics 3y ago

> Basically, I want an Alexa setup without the dependence on (or financial expectations from) Amazon Amen. +1 to timers, schedules, integration with a lighting bridge or even kitchen appliances e.g. Home Connect, custom routines, music (including Bluetooth and/or TuneIn), line out (so I can connect my own speakers), microphone relays that I can put all around the house instead of owning 5 or 6 RaspPis and the associated 15 to 30 watts Better than math, for me, would be a Wikipedia summary readout that voices the first paragraph (or a reasonably defined "summary length") from simple.wikipedia.com, although metric/imperial conversions are also very handy. Weather integration would also be delightful---including a way to tie this in to reminders e.g. "bring the plants indoors tonight", "remember rain clothes for riding home this evening", "tomorrow is probably the last good sled day until next winter" Oh. And properly bilingual. We have a Philips light named "Turtle". Alexa absolutely cannot handle this name if the input is set to either German or German/English.

kkielhofner 3y ago
> microphone relays that I can put all around the house instead of owning 5 or 6 RaspPis and the associated 15 to 30 watts I've always found this approach odd and this is one the reasons we primarily target the ESP-BOX-3 from Espressif[0]. For $50 you take it out of the box, flash from our web flasher, and stick it in your house. Given that it's ESP32-S3 based total power consumption is somewhere in the 100 milliwatt range. Once initially flashed all upgrades are done over the air (with < 5MB binaries) from our web management interface. All configuration changes are also centrally managed there. Make change, click "Save and Apply", goes live on any number of associated devices in seconds. The thought of managing half a dozen full Linux machines (Pis) just to get audio for stuff like this boggles my mind. Additionally, the audio quality is significantly better with the ESP-BOX-3 vs Pi, etc because the entire device and software stack is holistically engineered for high quality far-field voice capture. Acoustically tuned enclosure, dual microphones, acoustic echo cancellation, blind source separation, voice activity detection, wake word, etc. As I like to tell people - if you think you're going to slap a random microphone together with a Raspberry Pi and compete with Alexa you have another thing coming. I go into greater detail in this issue[1]. Getting high quality audio is step one. Step two is accurate transcription of that audio. Steps 3 - ? are what you're describing. We know what you said, now what do we do with it? Now that we feel steps one and two are more-or-less nailed steps three and beyond are our next focus. The good news is the final step is of course user feedback and we have that handled with text to speech and output on the BOX-3 LCD display. It's just currently the wide gulf in between these that needs focus. In short, our mouth and ears work well. It's the brain that needs work. [0] - https://www.espressif.com/en/news/ESP32-S3-BOX-3 https://www.espressif.com/en/news/ESP32-S3-BOX-3 [1] - https://github.com/toverainc/willow/issues/317#issuecomment-1774172563 https://github.com/toverainc/willow/issues/317#issuecomment-...