4 ms·
Hi there! "Author" here - glad to see this picking up. This was a fascinating project to work on and I learned a ton in the process. As it's often the case, I w
by jamez 4y ago
Hi there! "Author" here - glad to see this picking up. This was a fascinating project to work on and I learned a ton in the process. As it's often the case, I would do a lot of things differently if I were starting from scratch today.
- nortonham 4y agohow did you not go insane working on this?!
- nortonham 4y agooh and zizek needs more sniffles lol
- jamez 4y agoThe main issue is that there was no sniffling symbol in the transcript. And the generated text wouldn't contain it either, because (thankfully) they are pruned out of written interviews that I used to train the model.
- steve_adams_86 4y agoThanks for the explanation. I had some assumptions but wasn’t totally sure how this was trained. How would you make it sniffle in a natural way, too? It’s not a usual speech mannerism, and the way he does it is distinct. I wouldn’t know how to efficiently represent it with text. Maybe it’s easier than I’m imagining.
- yucatansunshine 4y agoJust stop the audio output every 5 seconds and include a sniffle sounds, at least that's what it sounds like in real life haha
- jamez 4y agoThe TTS model is trained on two things: speech samples and their transcript. If you add enough sniffle-symbols every time a sniffle appears in the speech, I am confident the model would pick up on that. And then you would be able to replicate a sniffle in the generation part. The more time-consuming bit would be to add in the training data for the language model those sniffle-symbols, so that they would be organically added in the text in the text-generation phase. But seriously, it's not worth it. I think he's a brilliant man with an idiosyncratic speech, let's leave it to that.
- steve_adams_86 4y agoI agree, I personally don't hear his sniffles when I'm listening to him intently. It's irrelevant. I was mostly curious if and how, generally speaking, a model could be trained to sniffle. Now that you describe it though it seems fairly clear, so thanks!
- nortonham 4y agoin all seriousness how do you not hear them?
- nortonham 4y agoI assumed it would difficult to include, it was just something I noticed about him
- Tenal 4y ago
- YossarianFrPrez 4y agoThis is incredible! I've sent it to several people already. Is there any chance you could provide more details as to the tech stack / training / technical setup?
- muted_pigment 4y agoGood stuff. I have a question. How did you align the transcribed interview with the audio?
- jamez 4y agoI use Aeneas[1], a set of tools to do force alignment. I found it in equal measures an amazing and a hard to navigate resource. Took me a while to set up and configure everything to the point that it was usable. But when it works, it works well. [1] https://github.com/readbeyond/aeneas https://github.com/readbeyond/aeneas
- BitwiseFool 4y agoHow did you manage to get us to read this in their respective voices?
- jamez 4y agoAh! Welcome to my head.
- eggoa 4y agoHow much manual pruning/selection from the generated text do you have to do?
- fab1an 4y agoamazing work! Curious how you cloned the voices - tortoise? I've previously tried Herzog, but couldn't quite train the German accent...
- biztos 4y agoI don’t know if this is useful, but Herzog has a distinctly Bavarian accent. And of course has spent most of his adult life far from there, so it’s not quite Bavarian either. Training a Herzogbot on recordings/transcriptions of, say, Kinski would be a waste of time accent-wise.
- jamez 4y agoI haven't tried Tortoise, thanks for pointing me to it. The voices were cloned by fine tuning a VITS model with coqui.ai. I used about two hours of speech for each speaker. With more time and resources, I'm certain it's possible to make those voices considerably better.
- OgAstorga 4y agoCan I get an invite link?
- blueberrychpstx 4y agohttps://coqui.ai?referralCode=q8jfhfs&refSource=copy https://coqui.ai?referralCode=q8jfhfs&refSource=copy help us move up the list!
- jamez 4y agoNo need to be invited. Between their GitHub[1] page and the documentation[2], you'll find everything you need to get started. [1] https://github.com/coqui-ai/TTS https://github.com/coqui-ai/TTS [2] https://tts.readthedocs.io/en/latest/ https://tts.readthedocs.io/en/latest/
- hanselot 4y agoHow much input audio would you need to produce audiobook quality? Hint Hint...