6 ms·

Lyra: A New Very Low-Bitrate Codec for Speech Compression

64 points by tmfi 6y ago

xiphias2 6y ago
For some reason the 1st Lyra example is disturbingly loud at 1 second for me on my mobile phone speaker. Speex doesn't have that problem for example.
- WHAT_IS_LOVE 6y ago
  That part sounds similar to a microphone echo, wonder if it is because the training data was contaminated with echoing samples.
jrexilius 6y ago
Maybe I missed it in the post, but is this an open source codec or are they just comparing themselves to open source codecs?
- dablweb 6y ago
  Exactly. Surprised more people are not asking this. I'm guessing they can't really OS their ML pipeline, or training data... so essentially they'd just be releasing the pre-trained ML model and the codec framework? Perhaps legal / privacy issues with that? Not sure. I can't see why they would... it would give them a huge competitive advantage on "Internet 3.0" which is live streaming and Zoom-like video conferences.
ZeroCool2u 6y ago
The performance in the examples is phenomenal. And at 3 kbps? It just blows opus and speex out of the water. Really excited to see if it holds up when they roll it out in Duo. I remember noticing the ML based improvements in Duo kicking in when talking to my Dad a while back (US<->EU video call and he was using mobile data). Was even more impressive seeing it work in the wild.
- mchusma 6y ago
  Duo video and audio calling for me have been the best for repliability and latency comparing it vs zoom and others. The ML techniques they use to fill blanks + use of AV1 are really great.
  - gazby 6y ago
    For those unfamiliar, Duo is Google's video calling app. https://duo.google.com/about/ https://duo.google.com/about/
temp-dude-87844 6y ago
libopus 1.3-26-ge85ed772 has a huge jump in quality on the 'noisy' sample at 9 kbps (CVBR or true VBR), because it moves from a 4 kHz lowpass to 8 kHz. In 2010 Nokia's listening test found [1] that SILK (at the time an independent speech codec, later incorporated into Opus) gained a quality benefit from reaching 8 kHz vs. the ITU-T and 3GPP codecs that would top out at 7 kHz for comparable modes and bitrates. So any gain in bandwidth in the 0-8 kHz range makes an appreciable difference, especially when there's distracting background noise in the lower bands and you can't filter it out so you have to encode along with your signal. While Opus can go as low as 6 kbps, at that bitrate it very clearly sounds like narrowband audio we're used to from telephones. Frustrating, but not an unfamiliar kind of degradation. Speex behaves like a classic CELP codec and will get robotic at its low end; 3 kbps is just a cruelly low bitrate for a codec whose advertised range is 2-44 kbps. Lyra does sound richer and wider-band than both Opus and Speex, but there's also a peculiar style transfer going on that's most apparent to me in the chocolate bread sample. Opus clearly sounds like a low-quality encode of the original -- it would benefit from some background noise reduction prior to the encode. But the Lyra version exaggerates the pronunciation of the phrase 'with chocolate' in a way that meaningfully differs from the speaker's original. It weakens the voiced 'th' to nothingness, and overshoots both the lead consonant and first vowel of 'choc', and then proceeds to wash the entire rest of the sentence with a peculiar brightened voice that's high, lacks consonant definition, and is close to ringing. I'm guessing it's actually style transfer, because though the result sounds not much like the speaker's original, the result is reminiscent of the speech pattern and accent that people with East Asian and Southeast Asian ancestry adopt when speaking American English. It was surprising, given that the speaker doesn't sound like that in the original. Does anyone else hear this too? [1] Rämö, Anssi & Toukomaa, Henri. (2010). Voice quality evaluation of recent open source codecs.
- ysleepy 6y ago
  Lyra distinctly swallows some syllables. The "th" in "with" of the video sample, or the second "looking" in the first sample. I found Opus to have more artifacts, but less ambiguous. I feel like there will be a false/unjustified confidence in the Lyra audio because its artifacts sound like speech, as opposed to noise, giving the listener clear sounding audio with altered content.
londons_explore 6y ago
With ML, we ought to be able to bridge the gap between sending audio (here 3000 bps to be usable) and sending a compressed transcription (20 bps to get words across at a similar rate). Surely there is some middle ground where we dedicate say 100bps to get the words over, together with a small bit of info about the emphasis, accent, tone and timing of the words.
- mchusma 6y ago
  It makes sense to me that you should send more bits at first to describe the speaker. If I load a theoretically perfect audio GAN of myself, I should be able to send many fewer bits later to reproduce my speech effectively. For most practical current applications, I'd like to see more work on reducing latency. 20ms seems to be the range on imperceivable, and the "processing latency" of 90ms. Light takes 3.34ms to travel 1000km, so for most calls transmission is not a theoretical limitation to long range low latency communication. Anyway, this is great work, but I would be curious to see what could be done with <10ms of processing time as well.
  - londons_explore 6y ago
    I believe the 90ms is the window over which the bitrate is constant... If you want to reduce the latency further, you might need to have variable bitrate (eg. perhaps more bits to encode the start of each syllable, and fewer to encode the end of the same syllable). That in turn adds complexity to other subsystems - typically variable bitrate leads to loss (bad), packet queuing and therefore variable latency (bad), or requires feedback into the video stream (ie. if the audio uses more data, encode the video with fewer bytes, which in turn means encoding must be serial rather than parallel, increasing latency).
londons_explore 6y ago
What is the purpose of putting effort into such low bitrate audio? A 3g connection on a decade old phone with a low signal strength might only get 50 kbps, but it tends to be bursty (ie. Offline for a few seconds, then a few hundred kilobytes arriving all at once). That makes it impractical for audio conferencing. It might be useful for streaming YouTube, but for that it would need to be able to encode music and sound effects reasonably too.
- WalterGR 6y ago
  (Some) new codecs use the increase in available bandwidth (the increase simply by virtue of the audio itself using less bandwidth) to improve redundancy and better handle burst loss.
- ysleepy 6y ago
  1 hour of audiobook in 2.6MB ? I quite love this.
- IshKebab 6y ago
  Lower bitrates give you more space for redundancy, forward error correction an so on. And as you said, in some situations you do have really low bandwidth available.
- ksec 6y ago
  Exactly my same thought. At least in the context of Internet Audio Conference. Even 3G AMR, I think that was pre 2000 Speech Codec started at 5Kbps. With a latency of only ~20ms. If I am reading correctly the encode due to ML nature would take at least 40ms on Lyra. I am sure there are some specific usage that would be a great use case. But most consumer consumption I cant think of one on top of my head. One should also be aware of the current roadmap in 5G and the on going work in 6G. We still have a long way to go in maximising bandwidth / transfer per capita. It seems to be the case ML codec wants to take these speech codec to new low bitrate. While it is fun doing it as a research, I much rather they push the envelop at least 6 / 8Kbps closer to perfection with even lower latency ( 10ms if not lower ).
- teddyh 6y ago
the_only_law 6y ago
As someone who has a tangential interest in audio codecs , are there any go to books for learning about them and how they work, including the math involved and the physics / biology of the sound. I’m dealing with some very simple stuff (like G.711) and just working with ffmpeg, but I’d like to learn more about the subject in general.
- castlec 6y ago
  fast fourier transforms and specifications of current codecs are a solid start.
WalterGR 6y ago
Related and recently: Satin: Microsoft’s latest AI-powered audio codec for real-time communications (microsoft.com) 13 points by panabee 5 days ago | flag | hide | past | favorite | 2 comments https://news.ycombinator.com/item?id=26218002 https://news.ycombinator.com/item?id=26218002 Direct link: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/satin-microsoft-s-latest-ai-powered-audio-codec-for-real-time/ba-p/2141382 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/... ...Satin can deliver super wide band speech starting at a bitrate of 6 kbps, and full-band stereo music starting at a bitrate of 17 kbps, with progressively higher quality at higher bitrates. Satin has been designed to provide great audio quality even under high packet loss...
floppiplopp 6y ago
So, I can assume the ok-google voice recordings are by now taking up too much costly storage space...?
- londons_explore 6y ago
  This could help get ok-google results back quicker, since I think a good chunk of the latency is in uploading the users audio data. Lots of mobile devices have a couple of seconds after being woken from sleep where the data connection is either not active, or stuck in gprs or 3g mode. It takes a few seconds to switch over to LTE, and during those few seconds, the user is sitting there waiting, especially for short queries...
hnarma 6y ago
https://news.ycombinator.com/item?id=17383061 https://news.ycombinator.com/item?id=17383061 Codec 2 w/ WaveNet @ 2.4kpbs [3ya] https://news.ycombinator.com/item?id=19520194 https://news.ycombinator.com/item?id=19520194 LPCNet @ 1.6kbps [2ya]
jscholes 6y ago
I suppose this is subjective. But listening to the short samples, I found the 6kbps Opus audio easier on the ears than Lyra. In the first sample ("pot of gold"), Lyra made the speaker sound like they had a condition which was disrupting their ability to naturally form sounds (incl. a slight slurs). Granted, it sounded a lot better than Speex which sucks.

64 points by tmfi 6y ago

For some reason the 1st Lyra example is disturbingly loud at 1 second for me on my mobile phone speaker. Speex doesn't have that problem for example.

WHAT_IS_LOVE 6y ago
That part sounds similar to a microphone echo, wonder if it is because the training data was contaminated with echoing samples.

jrexilius 6y ago

Maybe I missed it in the post, but is this an open source codec or are they just comparing themselves to open source codecs?

dablweb 6y ago
Exactly. Surprised more people are not asking this. I'm guessing they can't really OS their ML pipeline, or training data... so essentially they'd just be releasing the pre-trained ML model and the codec framework? Perhaps legal / privacy issues with that? Not sure. I can't see why they would... it would give them a huge competitive advantage on "Internet 3.0" which is live streaming and Zoom-like video conferences.

ZeroCool2u 6y ago

The performance in the examples is phenomenal. And at 3 kbps? It just blows opus and speex out of the water. Really excited to see if it holds up when they roll it out in Duo. I remember noticing the ML based improvements in Duo kicking in when talking to my Dad a while back (US<->EU video call and he was using mobile data). Was even more impressive seeing it work in the wild.

mchusma 6y ago
Duo video and audio calling for me have been the best for repliability and latency comparing it vs zoom and others. The ML techniques they use to fill blanks + use of AV1 are really great.
- gazby 6y ago
  For those unfamiliar, Duo is Google's video calling app. https://duo.google.com/about/ https://duo.google.com/about/

temp-dude-87844 6y ago

libopus 1.3-26-ge85ed772 has a huge jump in quality on the 'noisy' sample at 9 kbps (CVBR or true VBR), because it moves from a 4 kHz lowpass to 8 kHz. In 2010 Nokia's listening test found [1] that SILK (at the time an independent speech codec, later incorporated into Opus) gained a quality benefit from reaching 8 kHz vs. the ITU-T and 3GPP codecs that would top out at 7 kHz for comparable modes and bitrates. So any gain in bandwidth in the 0-8 kHz range makes an appreciable difference, especially when there's distracting background noise in the lower bands and you can't filter it out so you have to encode along with your signal. While Opus can go as low as 6 kbps, at that bitrate it very clearly sounds like narrowband audio we're used to from telephones. Frustrating, but not an unfamiliar kind of degradation. Speex behaves like a classic CELP codec and will get robotic at its low end; 3 kbps is just a cruelly low bitrate for a codec whose advertised range is 2-44 kbps. Lyra does sound richer and wider-band than both Opus and Speex, but there's also a peculiar style transfer going on that's most apparent to me in the chocolate bread sample. Opus clearly sounds like a low-quality encode of the original -- it would benefit from some background noise reduction prior to the encode. But the Lyra version exaggerates the pronunciation of the phrase 'with chocolate' in a way that meaningfully differs from the speaker's original. It weakens the voiced 'th' to nothingness, and overshoots both the lead consonant and first vowel of 'choc', and then proceeds to wash the entire rest of the sentence with a peculiar brightened voice that's high, lacks consonant definition, and is close to ringing. I'm guessing it's actually style transfer, because though the result sounds not much like the speaker's original, the result is reminiscent of the speech pattern and accent that people with East Asian and Southeast Asian ancestry adopt when speaking American English. It was surprising, given that the speaker doesn't sound like that in the original. Does anyone else hear this too? [1] Rämö, Anssi & Toukomaa, Henri. (2010). Voice quality evaluation of recent open source codecs.

ysleepy 6y ago
Lyra distinctly swallows some syllables. The "th" in "with" of the video sample, or the second "looking" in the first sample. I found Opus to have more artifacts, but less ambiguous. I feel like there will be a false/unjustified confidence in the Lyra audio because its artifacts sound like speech, as opposed to noise, giving the listener clear sounding audio with altered content.

londons_explore 6y ago

With ML, we ought to be able to bridge the gap between sending audio (here 3000 bps to be usable) and sending a compressed transcription (20 bps to get words across at a similar rate). Surely there is some middle ground where we dedicate say 100bps to get the words over, together with a small bit of info about the emphasis, accent, tone and timing of the words.

mchusma 6y ago
It makes sense to me that you should send more bits at first to describe the speaker. If I load a theoretically perfect audio GAN of myself, I should be able to send many fewer bits later to reproduce my speech effectively. For most practical current applications, I'd like to see more work on reducing latency. 20ms seems to be the range on imperceivable, and the "processing latency" of 90ms. Light takes 3.34ms to travel 1000km, so for most calls transmission is not a theoretical limitation to long range low latency communication. Anyway, this is great work, but I would be curious to see what could be done with <10ms of processing time as well.
- londons_explore 6y ago
  I believe the 90ms is the window over which the bitrate is constant... If you want to reduce the latency further, you might need to have variable bitrate (eg. perhaps more bits to encode the start of each syllable, and fewer to encode the end of the same syllable). That in turn adds complexity to other subsystems - typically variable bitrate leads to loss (bad), packet queuing and therefore variable latency (bad), or requires feedback into the video stream (ie. if the audio uses more data, encode the video with fewer bytes, which in turn means encoding must be serial rather than parallel, increasing latency).

londons_explore 6y ago

What is the purpose of putting effort into such low bitrate audio? A 3g connection on a decade old phone with a low signal strength might only get 50 kbps, but it tends to be bursty (ie. Offline for a few seconds, then a few hundred kilobytes arriving all at once). That makes it impractical for audio conferencing. It might be useful for streaming YouTube, but for that it would need to be able to encode music and sound effects reasonably too.

WalterGR 6y ago
(Some) new codecs use the increase in available bandwidth (the increase simply by virtue of the audio itself using less bandwidth) to improve redundancy and better handle burst loss.
ysleepy 6y ago
1 hour of audiobook in 2.6MB ? I quite love this.
IshKebab 6y ago
Lower bitrates give you more space for redundancy, forward error correction an so on. And as you said, in some situations you do have really low bandwidth available.
ksec 6y ago
Exactly my same thought. At least in the context of Internet Audio Conference. Even 3G AMR, I think that was pre 2000 Speech Codec started at 5Kbps. With a latency of only ~20ms. If I am reading correctly the encode due to ML nature would take at least 40ms on Lyra. I am sure there are some specific usage that would be a great use case. But most consumer consumption I cant think of one on top of my head. One should also be aware of the current roadmap in 5G and the on going work in 6G. We still have a long way to go in maximising bandwidth / transfer per capita. It seems to be the case ML codec wants to take these speech codec to new low bitrate. While it is fun doing it as a research, I much rather they push the envelop at least 6 / 8Kbps closer to perfection with even lower latency ( 10ms if not lower ).
teddyh 6y ago

the_only_law 6y ago

As someone who has a tangential interest in audio codecs , are there any go to books for learning about them and how they work, including the math involved and the physics / biology of the sound. I’m dealing with some very simple stuff (like G.711) and just working with ffmpeg, but I’d like to learn more about the subject in general.

castlec 6y ago
fast fourier transforms and specifications of current codecs are a solid start.

WalterGR 6y ago

Related and recently: Satin: Microsoft’s latest AI-powered audio codec for real-time communications (microsoft.com) 13 points by panabee 5 days ago | flag | hide | past | favorite | 2 comments https://news.ycombinator.com/item?id=26218002 https://news.ycombinator.com/item?id=26218002 Direct link: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/satin-microsoft-s-latest-ai-powered-audio-codec-for-real-time/ba-p/2141382 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/... ...Satin can deliver super wide band speech starting at a bitrate of 6 kbps, and full-band stereo music starting at a bitrate of 17 kbps, with progressively higher quality at higher bitrates. Satin has been designed to provide great audio quality even under high packet loss...

floppiplopp 6y ago

So, I can assume the ok-google voice recordings are by now taking up too much costly storage space...?

londons_explore 6y ago
This could help get ok-google results back quicker, since I think a good chunk of the latency is in uploading the users audio data. Lots of mobile devices have a couple of seconds after being woken from sleep where the data connection is either not active, or stuck in gprs or 3g mode. It takes a few seconds to switch over to LTE, and during those few seconds, the user is sitting there waiting, especially for short queries...

hnarma 6y ago

https://news.ycombinator.com/item?id=17383061 https://news.ycombinator.com/item?id=17383061 Codec 2 w/ WaveNet @ 2.4kpbs [3ya] https://news.ycombinator.com/item?id=19520194 https://news.ycombinator.com/item?id=19520194 LPCNet @ 1.6kbps [2ya]

jscholes 6y ago

I suppose this is subjective. But listening to the short samples, I found the 6kbps Opus audio easier on the ears than Lyra. In the first sample ("pot of gold"), Lyra made the speaker sound like they had a condition which was disrupting their ability to naturally form sounds (incl. a slight slurs). Granted, it sounded a lot better than Speex which sucks.