3 ms·
I took a week of from work recently to reimplement Storyteller's forced alignment algorithm. Storyteller[1] is an open source, self hosted platform for creating
by smoores 1mo ago
I took a week of from work recently to reimplement Storyteller's forced alignment algorithm. Storyteller[1] is an open source, self hosted platform for creating, managing, and reading/listening to "readaloud" books — books that have audiobook narration built in and can highlight each sentence (and/or word, with this new algorithm!) as it's read aloud. Forced alignment is the process of determining where each piece of text starts and ends in the audiobook.
Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well.
[1] https://storyteller-platform.dev https://storyteller-platform.dev
- orsenthil 1mo agoThis is excellent! I wanted this. I had noticed that Kindle had limited feature like this for certain books that support audiobooks. I had tried this style of reading with audiobook and ebook without a any automated sync, but hadn't pursued it. This looks cool. :)
- zackify 1mo agoThis is great. Recently got an xteink x3 and jailbroke my kindle, making custom stuff for it. I couldn't find it exactly in the docs. Sorry if I missed it. If you align these and then pull onto koreader via opds or other way, will audio work over Bluetooth there too? Along with the highlighting.
- smoores 1mo agoYou can use a KOReader plugin, https://github.com/stradichenko/audiobook.koplugin https://github.com/stradichenko/audiobook.koplugin, which has work-in-progress support for Media Overlays (the EPUB spec that Storyteller uses for readaloud)!
- zackify 1mo agoWow I didn't even think about just using on device AI. I'm going to set this one up and try it with kokoro which has the most natural for small size that I've seen. Wonder if the paperwhite 12th Gen can handle it. And also setup the main repo for when I have the real audio book. Thanks!
- leobg 1mo agoI tried running Kokoro on my iPhone XS (several years old), and it was slower than real-time. So, i wouldn’t expect it to be usable on a Kindle. But if you find a solution, please let me know.
- deivid 1mo agoKokoro can be hacked a bit (see script https://github.com/DavidVentura/translator-rs/blob/master/scripts/kokoro_mnn/rebuild_best_mnn.sh https://github.com/DavidVentura/translator-rs/blob/master/sc... ) which makes it about 3x faster. My phone went from slightly slower than realtime to very comfortable (also lower latency on first utterance!)
- leobg 1mo agoThanks. Will try!
- zackify 1mo agoAny plans to offer a way to generate audio books on demand. like if i set kokoro, a book auto imports, it can make the file for me for the whole book after some time of local gpu work?
- chrisweekly 1mo agoCool! S an avid reader, Whispersync is one of the things that keeps me paying Amazon/Audible. Competition and viable OSS alternatives would be great!
- smoores 1mo agoI'll post again when we release the v3 apps — even in alpha, they're really awesome, I think they blow the Kindle app out of the water!