4 ms·

Ask HN: Why don't podcasts/audiobooks have closed captions for misheard words?

31 points by amichail 4y ago

This would be like automatically generated captions in YouTube but for audio only content.

annagrigoryan2 4y ago
For podcasts only human transcription works good. Tech still is very hard on people with accents, even worse with non-english languages. And human transcription costs like a lot 45 cents a word, imagine a 1 hour conversation.
shreyshnaccount 4y ago
Nice project idea
- slekker 4y ago
  rev.com
deleted 4y ago
[deleted]
donatj 4y ago
Audiobooks in the kindle app highlight the spoken text as the audio plays.
WesleyLivesay 4y ago
The simple answer for this is probably just that the vast majority of people are not looking at the screen while listening, which probably makes it hard to make a business case for it to happen.
Overtonwindow 4y ago
I believe that’s called a book, and you could use it to read along?
sokoloff 4y ago
My experience with automatic transcription is that it introduces far more errors than it would eliminate. If it’s automatic, I don’t see how it would be beneficial to listeners/readers. If it’s manual, it’s not cheap.
- doliveira 4y ago
  Audiobooks wouldn't need automatic transcription, they have the source book.
- Broken_Hippo 4y ago
  If I don't hear something - due to traffic or children or dogs barking, for example - automatic transcription still lets me pick up on some things even if it is wrong 25% of the time. In general, it usually isn't wrong enough that I can't get the context but instead, a word or two. And if I'm already wrong, I know because of context and would hopefully pick it up. In other words, it is still a benefit.
- mattmaroon 4y ago
  I don't know, I use it on YouTube a lot. It's pretty great.
imgabe 4y ago
Podcasts don’t do it because good transcription costs money. Audiobooks already do it, it’s called … the book.
- paulryanrogers 4y ago
  Podcasts are taking over one off content like daily radio programs, so the quantity is huge. Audio book subtitles could be more useful if they highlighted what was just read instead of the whole page. Possibly with TTS alternative pronunciations or spelling aloud upon request.
- throwoutway 4y ago
  And I’ve heard that 3rd party audiobook transcription violate’s the written book’s copyright (different publishers perhaps?)
  - httpsterio 4y ago
    It's not even related to the publisher, they're two different products that are separately licensed. Generally the publisher only retains some rights to the book and how they may use it and same goes for the audio book, with the difference that the recordings are owned by those that produced the recording and not the original writer (unless it's licensed as so).
elil17 4y ago
For audiobooks, there’s a legal question of whether it causes licensing issues. Audible got in a huge lawsuit over it which they settled out of court: https://amp.theguardian.com/books/2020/jan/15/audible-settles-copyright-lawsuit-publishers-captions https://amp.theguardian.com/books/2020/jan/15/audible-settle...
paxys 4y ago
Search for "<podcast name> transcript". A lot of them do publish the full text of the show on their website.
- amichail 4y ago
  Closed captions would be more convenient.
  - paxys 4y ago
    Closed captions make sense for video content because there are visuals on the screen to follow along with and associate with the text. For podcasts it'll simply be like reading a long article with one sentence displayed like every 15 seconds. If your eyes are anyways on the screen you'd rather turn off the audio and read it at your own pace.
    - amichail 4y ago
      Maybe you are doing something else like programming while listening to a podcast/audiobook?
      - paxys 4y ago
        It's still the same problem. Are you going to look at the code you are typing or the podcast text?
  - ghaff 4y ago
    Transcripts largely exist for a different purpose than you seem to want. They're typically a human-edited version of the podcast that's searchable, skimmable, and can be easily read instead of listening to the podcast.
jasode 4y ago
>automatically generated captions in YouTube Youtube's AI-powered auto gen captions are often inaccurate for uncommon words and proper nouns (names, etc). Here's an example to a deep link : https://youtu.be/z9IXfYHhKYI?t=20m33s https://youtu.be/z9IXfYHhKYI?t=20m33s The instructor actually says "UIs" (user interfaces) but the auto gen transcript is "you eyes". Therefore, accurate transcripts require humans to manually correct the errors and most content creators of podcasts don't have the time-or-money budget for that. If your question is actually asking why the iPhone/Android software apps don't dynamically generate and display "close captions" while the podcast/audiobook is playing similar to Youtube or Chrome, it's because the devices don't include an onboard trained neural network that's big & comprehensive enough to translate general audio of uncommon terminology. Today's smartphones only have RAM of ~4GB to 8GB. (I read that iOS API has max RAM allocation of 5GB per app.) I found a Google paper talking about reducing a 2GB voice recognition model to 450MB with alternative techniques but the neural net is probably not good enough for accurately transcribing uncommon words: https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp... So current state-of-technology means accurate close captions on smartphones would require a separate "subtitle track" (authored by humans) akin to *.srt files in mkv video containers -- instead of attempting dynamic speech recognition of the audio. Maybe not enough consumer demand to have mp3 or m4b files formats with transcripts embedded.
- amichail 4y ago
  Why would mobile devices need to do this themselves? Couldn't Google do this with a podcast/audiobook service that uses YouTube tech?
  - jasode 4y ago
    >Why would mobile devices need to do this themselves? Couldn't Google do this with a podcast/audiobook service To answer your question, what's the baseline premise of how podcasts should work on a mobile phone? - Should podcasts/audiobooks continue to be local files such as mp3/m4b that's copied to the mobile phone? If so, the companion technology to enable "captions" would also require a local transcript file that's copied to the phone. (Like srt files for mkv.) In that case, a Google podcast transcript service that created such text files would require the whole podcast ecosystem and app players to change and recognize timestamped transcript files. E.g. even if you embed transcript into mp3/m4b container files, Apple's Podcast player app isn't going to know about it. So would it require everyone to download 3rd-party podcast players? So this scenario -- while not impossible -- has a lot of barriers to adoption. Too few consumers probably care about it. - Or should podcasts/audio be streamed from the cloud -- like Youtube videos -- to more easily enable your captions feature request? In this scenario, Google's podcast transcription wouldn't require embedding the text in new audio file formats for distribution. But you still have the mobile apps incompatibility issue of displaying the captions for podcasts. Again, Apple's Podcast app doesn't know about streamed captions. You also have content creators locked into Google's hypothetical podcast-with-captions-streaming cloud ecosystem which many wouldn't want to do.
ghaff 4y ago
Some sites do for podcasts. But machine transcriptions are IMO useless for the purpose. (They're fine for interviews where they're just source material for something you're writing.) And they're especially useless for audio you're having trouble understanding anyway. Human transcriptions aren't that expensive, about $1/minute, but they do cost money and you still need to spend some time cleaning them up and formatting them--and you still end up with words here and there you can't make out.
baxtr 4y ago
Amazon's WhisperSync is pretty cool. I love listening to audio books but I also like to read and annotate them. WhisperSync solves for that. I don't buy any eBooks without that feature.
- mattmaroon 4y ago
  I do love that feature also. Sometimes I want to read read. Sometimes I want to read while driving. It's great.
- leobg 4y ago
  Another solution is Voice Dream Reader on iOS. I used to think I preferred an Audible audiobook because of the human voice. But when I want to take notes, I find I end up going back to Voice Dream. I even scan physical books to be able to read them in the app. Only for novels do I still prefer a human narrated book, provided there is a version with a really good narrator.
- olejorgenb 4y ago
  https://www.snipd.com/ https://www.snipd.com/ is another (automated - so it's not perfect) option
deleted 4y ago
[deleted]
seydor 4y ago
Is there an audio file format with teletext?
Willson50 4y ago
https://play.google.com/store/apps/details?id=com.google.audio.hearing.visualization.accessibility.scribe https://play.google.com/store/apps/details?id=com.google.aud...
sandreas 4y ago
There is a id3v2.3 tag field "synchronized lyrics" (Field: SYLT)[1] where you can store a "phrase" with a "timestamp". One library I know that may support writing this tag is atldotnet[2], while I did not try it out. At least there is a Type TRANSCRIPTION in the linked code segment. There is also LRC[3], a synchronized lyrics format stored in an extra file (*.lrc). [1]: https://id3.org/id3v2.3.0#sec4.10 https://id3.org/id3v2.3.0#sec4.10 [2]: https://github.com/Zeugma440/atldotnet/blob/749d9ccb0303266740181eb552724b8b6420fca2/ATL/Entities/LyricsInfo.cs https://github.com/Zeugma440/atldotnet/blob/749d9ccb03032667... [3]: https://en.wikipedia.org/wiki/LRC_(file_format) https://en.wikipedia.org/wiki/LRC_(file_format)
aaron695 4y ago
assbuttbuttass 4y ago
On Android, you can turn on "Live Caption" for autogenerated captions a la YouTube
ohiovr 4y ago
Would something like this help? https://www.youtube.com/watch?v=pPBpxoIqDSA https://www.youtube.com/watch?v=pPBpxoIqDSA
kradeelav 4y ago
My guess would be "it's expensive", same reason why live-action TV shows online didn't have subs until fairly recently. A quick hack for fellow people who are hard of hearing: I often turn on a private "google meets" session, turn video off, and have google do the automatic captions through there if it's a captionless video I really want to see. I don't expect any privacy, and you do have to have a silent home for it, but hey. Works.
t-3 4y ago
Generally, when I'm listening to podcasts or audiobooks, I don't have a screen in front of me to put CC on. Might just be me, but I can only follow one source of information at a time, so I don't like non-music audio while using the computer or reading.

31 points by amichail 4y ago

This would be like automatically generated captions in YouTube but for audio only content.

For podcasts only human transcription works good. Tech still is very hard on people with accents, even worse with non-english languages. And human transcription costs like a lot 45 cents a word, imagine a 1 hour conversation.

shreyshnaccount 4y ago

Nice project idea

slekker 4y ago
rev.com

deleted 4y ago

[deleted]

donatj 4y ago

Audiobooks in the kindle app highlight the spoken text as the audio plays.

WesleyLivesay 4y ago

The simple answer for this is probably just that the vast majority of people are not looking at the screen while listening, which probably makes it hard to make a business case for it to happen.

Overtonwindow 4y ago

I believe that’s called a book, and you could use it to read along?

sokoloff 4y ago

My experience with automatic transcription is that it introduces far more errors than it would eliminate. If it’s automatic, I don’t see how it would be beneficial to listeners/readers. If it’s manual, it’s not cheap.

doliveira 4y ago
Audiobooks wouldn't need automatic transcription, they have the source book.
Broken_Hippo 4y ago
If I don't hear something - due to traffic or children or dogs barking, for example - automatic transcription still lets me pick up on some things even if it is wrong 25% of the time. In general, it usually isn't wrong enough that I can't get the context but instead, a word or two. And if I'm already wrong, I know because of context and would hopefully pick it up. In other words, it is still a benefit.
mattmaroon 4y ago
I don't know, I use it on YouTube a lot. It's pretty great.

imgabe 4y ago

Podcasts don’t do it because good transcription costs money. Audiobooks already do it, it’s called … the book.

paulryanrogers 4y ago
Podcasts are taking over one off content like daily radio programs, so the quantity is huge. Audio book subtitles could be more useful if they highlighted what was just read instead of the whole page. Possibly with TTS alternative pronunciations or spelling aloud upon request.
throwoutway 4y ago
And I’ve heard that 3rd party audiobook transcription violate’s the written book’s copyright (different publishers perhaps?)
- httpsterio 4y ago
  It's not even related to the publisher, they're two different products that are separately licensed. Generally the publisher only retains some rights to the book and how they may use it and same goes for the audio book, with the difference that the recordings are owned by those that produced the recording and not the original writer (unless it's licensed as so).

elil17 4y ago

For audiobooks, there’s a legal question of whether it causes licensing issues. Audible got in a huge lawsuit over it which they settled out of court: https://amp.theguardian.com/books/2020/jan/15/audible-settles-copyright-lawsuit-publishers-captions https://amp.theguardian.com/books/2020/jan/15/audible-settle...

paxys 4y ago

Search for "<podcast name> transcript". A lot of them do publish the full text of the show on their website.

amichail 4y ago
Closed captions would be more convenient.
- paxys 4y ago
  Closed captions make sense for video content because there are visuals on the screen to follow along with and associate with the text. For podcasts it'll simply be like reading a long article with one sentence displayed like every 15 seconds. If your eyes are anyways on the screen you'd rather turn off the audio and read it at your own pace.
  - amichail 4y ago
    Maybe you are doing something else like programming while listening to a podcast/audiobook?
    - paxys 4y ago
      It's still the same problem. Are you going to look at the code you are typing or the podcast text?
- ghaff 4y ago
  Transcripts largely exist for a different purpose than you seem to want. They're typically a human-edited version of the podcast that's searchable, skimmable, and can be easily read instead of listening to the podcast.

jasode 4y ago

>automatically generated captions in YouTube Youtube's AI-powered auto gen captions are often inaccurate for uncommon words and proper nouns (names, etc). Here's an example to a deep link : https://youtu.be/z9IXfYHhKYI?t=20m33s https://youtu.be/z9IXfYHhKYI?t=20m33s The instructor actually says "UIs" (user interfaces) but the auto gen transcript is "you eyes". Therefore, accurate transcripts require humans to manually correct the errors and most content creators of podcasts don't have the time-or-money budget for that. If your question is actually asking why the iPhone/Android software apps don't dynamically generate and display "close captions" while the podcast/audiobook is playing similar to Youtube or Chrome, it's because the devices don't include an onboard trained neural network that's big & comprehensive enough to translate general audio of uncommon terminology. Today's smartphones only have RAM of ~4GB to 8GB. (I read that iOS API has max RAM allocation of 5GB per app.) I found a Google paper talking about reducing a 2GB voice recognition model to 450MB with alternative techniques but the neural net is probably not good enough for accurately transcribing uncommon words: https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp... So current state-of-technology means accurate close captions on smartphones would require a separate "subtitle track" (authored by humans) akin to *.srt files in mkv video containers -- instead of attempting dynamic speech recognition of the audio. Maybe not enough consumer demand to have mp3 or m4b files formats with transcripts embedded.

amichail 4y ago
Why would mobile devices need to do this themselves? Couldn't Google do this with a podcast/audiobook service that uses YouTube tech?
- jasode 4y ago
  >Why would mobile devices need to do this themselves? Couldn't Google do this with a podcast/audiobook service To answer your question, what's the baseline premise of how podcasts should work on a mobile phone? - Should podcasts/audiobooks continue to be local files such as mp3/m4b that's copied to the mobile phone? If so, the companion technology to enable "captions" would also require a local transcript file that's copied to the phone. (Like srt files for mkv.) In that case, a Google podcast transcript service that created such text files would require the whole podcast ecosystem and app players to change and recognize timestamped transcript files. E.g. even if you embed transcript into mp3/m4b container files, Apple's Podcast player app isn't going to know about it. So would it require everyone to download 3rd-party podcast players? So this scenario -- while not impossible -- has a lot of barriers to adoption. Too few consumers probably care about it. - Or should podcasts/audio be streamed from the cloud -- like Youtube videos -- to more easily enable your captions feature request? In this scenario, Google's podcast transcription wouldn't require embedding the text in new audio file formats for distribution. But you still have the mobile apps incompatibility issue of displaying the captions for podcasts. Again, Apple's Podcast app doesn't know about streamed captions. You also have content creators locked into Google's hypothetical podcast-with-captions-streaming cloud ecosystem which many wouldn't want to do.

ghaff 4y ago

Some sites do for podcasts. But machine transcriptions are IMO useless for the purpose. (They're fine for interviews where they're just source material for something you're writing.) And they're especially useless for audio you're having trouble understanding anyway. Human transcriptions aren't that expensive, about $1/minute, but they do cost money and you still need to spend some time cleaning them up and formatting them--and you still end up with words here and there you can't make out.

baxtr 4y ago

Amazon's WhisperSync is pretty cool. I love listening to audio books but I also like to read and annotate them. WhisperSync solves for that. I don't buy any eBooks without that feature.

mattmaroon 4y ago
I do love that feature also. Sometimes I want to read read. Sometimes I want to read while driving. It's great.
leobg 4y ago
Another solution is Voice Dream Reader on iOS. I used to think I preferred an Audible audiobook because of the human voice. But when I want to take notes, I find I end up going back to Voice Dream. I even scan physical books to be able to read them in the app. Only for novels do I still prefer a human narrated book, provided there is a version with a really good narrator.
olejorgenb 4y ago
https://www.snipd.com/ https://www.snipd.com/ is another (automated - so it's not perfect) option

deleted 4y ago

[deleted]

seydor 4y ago

Is there an audio file format with teletext?

Willson50 4y ago

https://play.google.com/store/apps/details?id=com.google.audio.hearing.visualization.accessibility.scribe https://play.google.com/store/apps/details?id=com.google.aud...

sandreas 4y ago

There is a id3v2.3 tag field "synchronized lyrics" (Field: SYLT)[1] where you can store a "phrase" with a "timestamp". One library I know that may support writing this tag is atldotnet[2], while I did not try it out. At least there is a Type TRANSCRIPTION in the linked code segment. There is also LRC[3], a synchronized lyrics format stored in an extra file (*.lrc). [1]: https://id3.org/id3v2.3.0#sec4.10 https://id3.org/id3v2.3.0#sec4.10 [2]: https://github.com/Zeugma440/atldotnet/blob/749d9ccb0303266740181eb552724b8b6420fca2/ATL/Entities/LyricsInfo.cs https://github.com/Zeugma440/atldotnet/blob/749d9ccb03032667... [3]: https://en.wikipedia.org/wiki/LRC_(file_format) https://en.wikipedia.org/wiki/LRC_(file_format)

aaron695 4y ago

assbuttbuttass 4y ago

On Android, you can turn on "Live Caption" for autogenerated captions a la YouTube

ohiovr 4y ago

Would something like this help? https://www.youtube.com/watch?v=pPBpxoIqDSA https://www.youtube.com/watch?v=pPBpxoIqDSA

kradeelav 4y ago

My guess would be "it's expensive", same reason why live-action TV shows online didn't have subs until fairly recently. A quick hack for fellow people who are hard of hearing: I often turn on a private "google meets" session, turn video off, and have google do the automatic captions through there if it's a captionless video I really want to see. I don't expect any privacy, and you do have to have a silent home for it, but hey. Works.

t-3 4y ago

Generally, when I'm listening to podcasts or audiobooks, I don't have a screen in front of me to put CC on. Might just be me, but I can only follow one source of information at a time, so I don't like non-music audio while using the computer or reading.