3 ms·

Cool. You are right, we do have the tech stack to do something similar to the thing I am talking about here but as you point out, a lot of things can't be handl

Cool. You are right, we do have the tech stack to do something similar to the thing I am talking about here but as you point out, a lot of things can't be handled such as pauses, mood, and irony. Both in the case of input and output. Did you manage to get it robust? What was the latency etc? Did you end up using it? :)

ilaksh 3y ago
I used it for the second version of aidev.codes which was a static website generator basically. I did not have money for marketing and could not get users that wanted to pay. The voice interface was optional. New version in progress is a ChatGPT plugin and does not have any voice interface. The latency was not that bad but it does take a moment for Eleven Labs to generate even in streaming mode. I optimized it slightly by doing one sentence at a time. It might be possible to optimize it farther than I did. Also Eleven Labs is iterating rapidly and have been making announcements about new models and plans. But anyway that stuff was a bit of a distraction for me because the code generation for web applications is hard enough and I need to focus. I end up spending more than half my time on side projects to pay the bills which is delaying progress on the third attempt at this. Currently working on a contract where we generate queries to a specific database to answer questions a user types into chat. I do believe that voice interfaces will be a standard option on these things within a few years. Right now I think most projects are focused on things like core code generation and retrieval challenges. But pretty soon I think everyone will have to add the voice because users will demand it or take it as a given. I assume. Maybe they haven't seen it a lot and don't know it's possible so don't expect it. It's a question of momentum like most things. Detecting emotion at some level will be important for call centers. I think all you really need for the core part of this is datasets with yelling or angry people that use uppercase in the transcripts, exclamation marks, and maybe an [angry] annotation. But actually eventually if you think about Meta's new multimodal speech to text, eventually you will see multimodal models trained on speech dialogue and large amounts of text like LLMs directly. That will provide truly human-like responses in terms of tone.

ilaksh 3y ago

I used it for the second version of aidev.codes which was a static website generator basically. I did not have money for marketing and could not get users that wanted to pay. The voice interface was optional. New version in progress is a ChatGPT plugin and does not have any voice interface. The latency was not that bad but it does take a moment for Eleven Labs to generate even in streaming mode. I optimized it slightly by doing one sentence at a time. It might be possible to optimize it farther than I did. Also Eleven Labs is iterating rapidly and have been making announcements about new models and plans. But anyway that stuff was a bit of a distraction for me because the code generation for web applications is hard enough and I need to focus. I end up spending more than half my time on side projects to pay the bills which is delaying progress on the third attempt at this. Currently working on a contract where we generate queries to a specific database to answer questions a user types into chat. I do believe that voice interfaces will be a standard option on these things within a few years. Right now I think most projects are focused on things like core code generation and retrieval challenges. But pretty soon I think everyone will have to add the voice because users will demand it or take it as a given. I assume. Maybe they haven't seen it a lot and don't know it's possible so don't expect it. It's a question of momentum like most things. Detecting emotion at some level will be important for call centers. I think all you really need for the core part of this is datasets with yelling or angry people that use uppercase in the transcripts, exclamation marks, and maybe an [angry] annotation. But actually eventually if you think about Meta's new multimodal speech to text, eventually you will see multimodal models trained on speech dialogue and large amounts of text like LLMs directly. That will provide truly human-like responses in terms of tone.