3 ms·

My pleasure! If you have a lot of time and patience you can read my very long recent response[0] to a user asking about this where I go into arguably far too m

My pleasure! If you have a lot of time and patience you can read my very long recent response[0] to a user asking about this where I go into arguably far too much detail. For context Wyoming is the fundamental protocol for the entire Home Assistant/ESPHome implementation. In terms of commercialization, yes and no. Speaking of experience and background - the real genesis of Willow was three years ago when I was approached by a large publicly traded company in the healthcare space to implement a voice assistant for their use cases. The underlying and fundamental technology just wasn't there yet and it never got past PoC stage. That failure really bothered me. Earlier this year everything seemed to really align and thus Willow was born. Our monetization strategy is where this all started three years ago - commercial and "enterprise" use cases. It's the only market I know and have experience with. This is actually relevant to my comments about fundamental approach and architecture. Willow was designed for this market and their use cases which needless to say has substantially higher expectations than the more-or-less consumer/"tinkerer" market. I will NEVER monetize consumer/home users directly (again, I don't understand the market) but they get to benefit from our substantially better approach. All of that said I'd certainly be interested in partnering with someone to bring Willow to the consumer market commercially. I've spent my entire career with the "sales cycle measured in months with a big customer for a lot of money" approach, not the "tons of consumers paying $5 a month" approach. Everything you're describing is what we're currently focusing on. We feel we have the fundamentals of voice and audio pretty much nailed. Then the next question becomes "Great, I have performant and accurate speech in and out. What do I actually do with it?". Taking the speech recognition transcripts and matching them with flexible grammar, functionality and application routing, etc is the next significant challenge. We have some of our own logic internally that we will be building out but at this early stage we heavily rely on the command processing and intent recognition from our supported command endpoints. Nothing you're describing is impossible (of course) but Alexa (for example) has an eight year head start. It's going to take a while to reach feature parity with that. Our intention is to not only match Alexa functionality but also far surpass it by leveraging the entire open source ecosystem. [0] - https://github.com/toverainc/willow-inference-server/discussions/135 https://github.com/toverainc/willow-inference-server/discuss...

PennRobotics 3y ago
I know it's difficult, but is there a way to gather feedback from the user when a voice command is below a certain threshold of understandability? Or to build a context catalog, such as time of day specific types of commands are expected? I think these two areas could be holding the commercial voice assistants back, since the majority of bug reports on the Google Home and Alexa subreddits are people complaining about being misheard.
- kkielhofner 3y ago
  I've been thinking about this a little recently. Raw brain dump incoming... The problem with intent matching now (say with Home Assistant) is that the transcribed speech more-or-less needs to exactly (character for character) match the defined entities (names of lights, etc) as well as the grammar and structure expected by the intent definitions. Something as simple as a hyphen in "turn-off" currently breaks the matching. That's an easy one to fix, I'm just providing it as an example. In terms of addressing this, I've kicked around a few ideas: With Home Assistant at least we can pull all of the entities and supported intents. We have a variety of options to do all kinds of fuzzy matching with these now known phrases. We'd fix the transcription based on something like nearest neighbor (more or less) and send the clean command to HA. There is also a way to handle this in the prompt to Whisper. A literal text embedding model with nearest neighbor search seems kind of ridiculous but I've been curious about it. The same could be done at the audio level by essentially making an embedding of the Mel Spectrogram of the actual audio (which we could have available) and searching on that. There are also approaches with a variety of NLP implementations, language models, etc that could be combined or used separately. We've also considered some more manual approaches - things like an interface that logs sessions where you can (after the fact) find a session/command that went wrong, and basically map it to what you wanted. While this may seem tedious we get some reports that our speech recognition consistently mis-transcribes a given voice command. Implementing a lookup table to correct for this is simple. The biggest challenge to this is how to deal with the UI/UX aspects.

PennRobotics 3y ago

I know it's difficult, but is there a way to gather feedback from the user when a voice command is below a certain threshold of understandability? Or to build a context catalog, such as time of day specific types of commands are expected? I think these two areas could be holding the commercial voice assistants back, since the majority of bug reports on the Google Home and Alexa subreddits are people complaining about being misheard.

kkielhofner 3y ago
I've been thinking about this a little recently. Raw brain dump incoming... The problem with intent matching now (say with Home Assistant) is that the transcribed speech more-or-less needs to exactly (character for character) match the defined entities (names of lights, etc) as well as the grammar and structure expected by the intent definitions. Something as simple as a hyphen in "turn-off" currently breaks the matching. That's an easy one to fix, I'm just providing it as an example. In terms of addressing this, I've kicked around a few ideas: With Home Assistant at least we can pull all of the entities and supported intents. We have a variety of options to do all kinds of fuzzy matching with these now known phrases. We'd fix the transcription based on something like nearest neighbor (more or less) and send the clean command to HA. There is also a way to handle this in the prompt to Whisper. A literal text embedding model with nearest neighbor search seems kind of ridiculous but I've been curious about it. The same could be done at the audio level by essentially making an embedding of the Mel Spectrogram of the actual audio (which we could have available) and searching on that. There are also approaches with a variety of NLP implementations, language models, etc that could be combined or used separately. We've also considered some more manual approaches - things like an interface that logs sessions where you can (after the fact) find a session/command that went wrong, and basically map it to what you wanted. While this may seem tedious we get some reports that our speech recognition consistently mis-transcribes a given voice command. Implementing a lookup table to correct for this is simple. The biggest challenge to this is how to deal with the UI/UX aspects.