4 ms·

Somewhat shameless but relevant plug - we use the ESP32-S3 for Willow[0]. The dual core S3 and "high speed" external PSRAM are game changers. What it is capabl

Somewhat shameless but relevant plug - we use the ESP32-S3 for Willow[0]. The dual core S3 and "high speed" external PSRAM are game changers. What it is capable of (especially considering the price point) is nothing short of incredible. Wake word activation, audio processing (AGC, AEC, etc), audio streaming, even on device speech recognition for up to 400 commands with Multinet. All remarkably performant, easily besting Alexa/Echo in terms of interactivity and response time (even when using an inference server across the internet for speech recognition). Sure we're down in ESP-IDF land and managing everything we have going on in FreeRTOS is a pain but that's not anything you wouldn't have on any microcontroller. We're also doing a lot considering and generally speaking we "just" throw/pin audio tasks (with varying priority) on core 1 while more-or-less dumping everything else on core 0. Seems to be working well so far! [0] - https://github.com/toverainc/willow https://github.com/toverainc/willow

no_time 3y ago
Wow good luck with your project. Virtual assistants are not my thing. But if I were to try again, this would certainly be my first choice.
- kkielhofner 3y ago
  Thanks!
skykooler 3y ago
Interesting, this is yet another open source project that relies on a proprietary wake word model. Why are there no open source wake word engines like there are for speech recognition?
- ktm5j 3y ago
  Probably because no one has built one yet.. feel free to do so.
- thesh4d0w 3y ago
  The process of generating the wake word data is pretty onerous - https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/wake_word_engine/ESP_Wake_Words_Customization.html https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3...
- kkielhofner 3y ago
  Short answer: it’s a tiny niche that is still very difficult, expensive, and time consuming. This isn’t Whisper or an LLM that has applications in practically anything. You’re not going to see OpenAI, Meta, etc release a microcontroller optimized wake implementation because literally no one cares (by comparison). Yet it requires a not-completely dissimilar level of effort. Someone else linked to the Espressif training dataset requirements. That said, wake word comes up more than I could have ever imagined. Everyone has an opinion and exactly zero of them have any idea what they are talking about (that I’ve encountered). It’s getting very, very old and I’ve all but given up engaging on it because it always ends with the classic “Well couldn’t you just…”. Yes, in the 15 seconds it took you to write that comment you came up with something no one in the field has ever thought of before. If you can create a completely open source wake implementation that gets even remotely close to the performance and reliability of those from Espressif, etc (while running on a microcontroller) we would be thrilled to use it. You will have created the first of its kind, and you’ll be famous! The more likely outcome (as you dismissively said) will be “yet another open source project” that goes in the graveyard of completely unusable open source wake word implementations. There are plenty. As you note - all of them. Let’s just say I won’t be waiting for it.
- jononor 3y ago
  In case anyone with practical skills in ML+deep learning (but not in audio or embedded) wants to tackle this project, I am willing to mentor on those things. Microcontroller-level audio ML is my speciality. One can find me in the Sound of AI Slack. Ping me there and we can create a channel etc. It will be a multi-month endeavour though. Getting to PoC level is quite quick, but then getting robust performance in nearfield / high SNR cases across diverse background noise is a lot more work. And then tackling low SNR and far-field like with Alexa is yet another level. Model architecture wise the problem is rather well understood, there are several good papers available from ARM etc. Deployement infrastructure also pretty good these days, for example with Tensorflow Lite Micro. In addition to ML skills, the project would need someone that is good at organizing volunteer outreach, in order to build a good sized dataset. The Espressif docs are a reasonable spec for something quite good on the voice side. But then we would also need a good dataset of background noise. Of course those with embedded/microcontroller skills are also very welcome.

no_time 3y ago

Wow good luck with your project. Virtual assistants are not my thing. But if I were to try again, this would certainly be my first choice.

kkielhofner 3y ago
Thanks!

skykooler 3y ago

Interesting, this is yet another open source project that relies on a proprietary wake word model. Why are there no open source wake word engines like there are for speech recognition?

ktm5j 3y ago
Probably because no one has built one yet.. feel free to do so.
thesh4d0w 3y ago
The process of generating the wake word data is pretty onerous - https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/wake_word_engine/ESP_Wake_Words_Customization.html https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3...
kkielhofner 3y ago
Short answer: it’s a tiny niche that is still very difficult, expensive, and time consuming. This isn’t Whisper or an LLM that has applications in practically anything. You’re not going to see OpenAI, Meta, etc release a microcontroller optimized wake implementation because literally no one cares (by comparison). Yet it requires a not-completely dissimilar level of effort. Someone else linked to the Espressif training dataset requirements. That said, wake word comes up more than I could have ever imagined. Everyone has an opinion and exactly zero of them have any idea what they are talking about (that I’ve encountered). It’s getting very, very old and I’ve all but given up engaging on it because it always ends with the classic “Well couldn’t you just…”. Yes, in the 15 seconds it took you to write that comment you came up with something no one in the field has ever thought of before. If you can create a completely open source wake implementation that gets even remotely close to the performance and reliability of those from Espressif, etc (while running on a microcontroller) we would be thrilled to use it. You will have created the first of its kind, and you’ll be famous! The more likely outcome (as you dismissively said) will be “yet another open source project” that goes in the graveyard of completely unusable open source wake word implementations. There are plenty. As you note - all of them. Let’s just say I won’t be waiting for it.
jononor 3y ago
In case anyone with practical skills in ML+deep learning (but not in audio or embedded) wants to tackle this project, I am willing to mentor on those things. Microcontroller-level audio ML is my speciality. One can find me in the Sound of AI Slack. Ping me there and we can create a channel etc. It will be a multi-month endeavour though. Getting to PoC level is quite quick, but then getting robust performance in nearfield / high SNR cases across diverse background noise is a lot more work. And then tackling low SNR and far-field like with Alexa is yet another level. Model architecture wise the problem is rather well understood, there are several good papers available from ARM etc. Deployement infrastructure also pretty good these days, for example with Tensorflow Lite Micro. In addition to ML skills, the project would need someone that is good at organizing volunteer outreach, in order to build a good sized dataset. The Espressif docs are a reasonable spec for something quite good on the voice side. But then we would also need a good dataset of background noise. Of course those with embedded/microcontroller skills are also very welcome.