4 ms·

This is pretty interesting... I once wrote a similar thing for building a custom LJSPEECH dataset out of ebook/audiobook combinations to synthesize my favorite

This is pretty interesting... I once wrote a similar thing for building a custom LJSPEECH dataset out of ebook/audiobook combinations to synthesize my favorite narrator voices using coqui-tts and the VITS model and make them "publish" books that never came out as audiobook. It was able to synchronize the book contents to timestamps, split the spoken word in to sentences and create a LJSPEECH datasets out of the combinations. I used aeneas[1], it was a bit finicky to set up, but after a while it even was able to map non-english languages (in my case german) with more than 80% accuracy. Worked out pretty well, the LJSPEECH datasets were good (I still have them here), but the TTS tech was not there yet :-) Maybe it's time to revive this project using newer modelling approaches like XTTS or something... [1]: https://www.readbeyond.it/aeneas/ https://www.readbeyond.it/aeneas/

vagrantJin 3y ago
I've thought about exactly this a few years back but lacked the technical skills to implement it. there are some great books out there as you mentioned, but even worse are great books with mediocre narration/production. eg, A Song of Ice and Fire on Audible is absolutely horrid. The Martian by Andy Weir is fantastic. Can I transplant Will wheaton or Greg Tremblay into GOT? Can I have multiple characters narrated by different voices? please revisit it if you can.
- monkeywork 3y ago
  IMHO the original narration on The Martian by RC Bray is better than Wil's. I enjoyed Wil's work on Ernest Cline's books but RC Bray and Dennis E. Taylor are (for me) top of the mountain when it comes to SF narration.
- aedocw 3y ago
  You can do this today, though you would definitely be breaking copyright (you need to strip the DRM from the epub), and if you're cloning someone's voice without their permission you're probably breaking some more laws. You're pretty safe though assuming you don't distribute it or try to make money. Check out https://github.com/aedocw/epub2tts https://github.com/aedocw/epub2tts for creating an audiobook from epub. Take a look in the utils directory for notes about fine-tuning a voice clone. I can tell you I've done some voices that are close enough to the original to be pretty shocking. Feel free to get in touch if you have any questions, it's pretty fun making your own audiobooks with the reader of your choice!

vagrantJin 3y ago

I've thought about exactly this a few years back but lacked the technical skills to implement it. there are some great books out there as you mentioned, but even worse are great books with mediocre narration/production. eg, A Song of Ice and Fire on Audible is absolutely horrid. The Martian by Andy Weir is fantastic. Can I transplant Will wheaton or Greg Tremblay into GOT? Can I have multiple characters narrated by different voices? please revisit it if you can.

monkeywork 3y ago
IMHO the original narration on The Martian by RC Bray is better than Wil's. I enjoyed Wil's work on Ernest Cline's books but RC Bray and Dennis E. Taylor are (for me) top of the mountain when it comes to SF narration.
aedocw 3y ago
You can do this today, though you would definitely be breaking copyright (you need to strip the DRM from the epub), and if you're cloning someone's voice without their permission you're probably breaking some more laws. You're pretty safe though assuming you don't distribute it or try to make money. Check out https://github.com/aedocw/epub2tts https://github.com/aedocw/epub2tts for creating an audiobook from epub. Take a look in the utils directory for notes about fine-tuning a voice clone. I can tell you I've done some voices that are close enough to the original to be pretty shocking. Feel free to get in touch if you have any questions, it's pretty fun making your own audiobooks with the reader of your choice!