9 ms·

A Real-Time Wideband Neural Vocoder at 1.6 Kb/S Using LPCNet

203 points by weinzierl 8y ago

mmastrac 8y ago
I've been waiting for someone to do this* with audio and/or video. Amazing work. Also worth reading this related link: https://www.rowetel.com/?p=6639 https://www.rowetel.com/?p=6639 There's some really exciting progress that could be made in this space. The page mentions that they could use it as a post-processing filter for Opus for better decoding to avoid changing the bitstream. It could also be useful as a way to accommodate for packet loss and recover "just enough" to avoid interrupting the conversation. * encoding audio through a neural network for network transmission
loudmax 8y ago
For those too impatient to read the details, check out the "Hear for yourself" examples toward the bottom of the page. They're reproducing decent sounding speech at 1.6 kbps. 1.6 kbps is nuts! I like to re-encode audio books or podcasts in Opus at 32 kbps and I consider that stingy. The fact that speech is even comprehensible at 1.6 kbps is impressive. As the article explains, their technique is analogous to speech-to-text, then text-to-speech. The original recordings are a little stiff, and the encoded speech is a little more stiff. It isn't perfect, but it's decent. It'll be interesting to hear this technique applied to normal conversation. If regular speech holds up as well as their samples, it should be perfectly adequate for conversational speech. At 1.6 kbps, which is absurd. Also, I wonder how well this technique could be applied to music. My guess is that it won't do justice to great musicians ... but it might be good enough simple pop tunes.
- dietrichepp 8y ago
  I’ve done experiments with Opus that produce intelligible (but ugly) speech at 2.3 kbit/s. It involves downsampling the Opus stream at the packet level—e.g. transmit only one out of every three packets. It was surprisingly easy. Nothing as sophisticated as what’s going on here. Also based on work by Xiph. Possibly using the same LPC used here.
- jmvalin 8y ago
  Actually, this won't work at all for music because it makes fundamental assumptions that the signal is speech. For normal conversations, it should work, though for now the models are not yet as robust as I'd like (in case of noise and reverberation). That's next on the list of things to improve.
  - petercooper 8y ago
    I tried it with music and the results were spooky. Very ethereal and ghostly. It was only with some classical music though, I might have to do a pop song next and share the results!
  - petercooper 8y ago
    Here we go! This is the first minute or so of Penny Lane by The Beatles converted down to a 10KB .bin and then back to a .wav: http://no.gd/pennylane.wav http://no.gd/pennylane.wav .. unsurprisingly the vocals remain recognizable, but the music barely at all.
    - bch 8y ago
      As imagined by Marilyn Manson...
      - petercooper 8y ago
        Pretty much! It shows off how the codec works to a great extent though as it seems to be misinterpreting parts of the music to be the pitch of the speech, so Paul's voice sounds weird at the start of most lines but okay throughout the lines. I've also run a BBC news report through the program with better results although it demonstrates that any background noise at all can throw things off significantly: https://twitter.com/peterc/status/1111736029558517760 https://twitter.com/peterc/status/1111736029558517760 .. so at this low bitrate, it really is only good for plain speech without any other noise.
        jmvalin 8y ago
- semi-extrinsic 8y ago
  For comparison, adaptive GSM encodings, which are in use for cellphones today, are also in the single-digit kbps. https://en.wikipedia.org/wiki/Adaptive_Multi-Rate_audio_codec https://en.wikipedia.org/wiki/Adaptive_Multi-Rate_audio_code...
- Eli_P 8y ago
  I use --vbr --bitrate 16 and it feels indistinguishable from the original for podcasts. As opusenc takes only wav for input and does not use multiple cores, I had to write scripts for parallel re-encoding of stuff.
  - klodolph 8y ago
    I like to use Makefiles for parallel encoding. OPUSFLAGS = --vbr --bitrate 16 all: $(patsubst %.wav,%.opus,$(wildcard *.wav)) .PHONY: all %.opus: %.wav opusenc $(OPUSFLAGS) $< $@ Make -j4 or whatever. There are a few other ways to do this (e.g. xargs).
rasz 8y ago
3 Gflops, we are deep beyond diminishing returns here. Opus seems good enough.
- gok 8y ago
  3 GFLOP/sec sounds like a lot but it's considerably less math than the radio DSPs inside any modern phone's baseband is doing during a phone call.
  - opportune 8y ago
    I don't know much about phone tech, are the basebands really doing math or just instrumenting? My assumption would be that there is just some sensor writing to a buffer at a high frequency but that whatever processes that buffer operates at a lower frequency.
    - anisppp 8y ago
      Your question is hard to parse? What is instrumenting? If it helps though... the word “baseband” itself is the lower frequency containing just the bandwidth of the signal. Ie that is the lower frequency...
      - gok 8y ago
        True but here I intended "baseband" to mean https://en.wikipedia.org/wiki/Baseband_processor https://en.wikipedia.org/wiki/Baseband_processor
- nine_k 8y ago
  I wonder if a particular network can be implemented more economically if we only run it in "transformation" mode, without the need do train it.
- jmvalin 8y ago
  Keep in mind that the very first CELP speech codec (in 1984) used to take 90 seconds to encode just 1 second of speech... on a Cray supercomputer. Ten years later, people had that running in their cell phones. It's not just that hardware keeps getting faster, but algorithms are also getting more efficient. LPCNet is already 1/100 the complexity of the original WaveNet (which is just 2 years old) and I'm pretty sure it's still far from optimal.
deleted 8y ago
[deleted]
a-dub 8y ago
might be fun to try to port this to the dadamachines doppler... :)
0-_-0 8y ago
OK, so... Would it be possible to do something similar for video?
- cooper12 8y ago
  Similar as in the same approach, or as in "apply neural networks to all the things"? Because if it's the former, this approach was very specifically tailored to human speech, taking into account how much it can compress/interpolate qualities like pitch and the spectral envelope. That's far too specific to apply to video. As for the latter, you'd have to perhaps feed Google Scholar the right incantations or ask someone with knowledge. As far as I know, video codecs already have a huge bag of tricks they use (for example the B-frames borrowed in this post). Even then, the key points in this codec were that firstly it's meant for use at very low bitrates, where existing codecs break down, and then secondly it's a vocoder, so it's converting audio to an intermediate form and resynthesizing it. That kind of lossiness is acceptable for audio, but I'm not sure how it would work acceptably for video.
  - 0-_-0 8y ago
    I should have been more specific. I meant that instead of compressing video to minimise pixel difference, minimise feature difference instead.
Causality1 8y ago
Local text to speech quality blows this out of the water with a much smaller bandwidth footprint than 1.6kb/s. To me this sounds sort of like "I figured out how to eat a pinecone in less than three seconds". Impressive but not useful.
- est31 8y ago
  The difference is that it uses the voice of the original person. And STT is rather erroneous. The WER is in the single percent range :).
- kstrauser 8y ago
  You really can't think of any uses for being able to push 40 simultaneous speech connections through a single 64Kbps voice channel?
- CamperBob2 8y ago
  That was a fairly ignorant comment because the whole idea behind a speech codec is to compress and reproduce speech in a manner that allows one to at least recognize who's speaking. But I wonder if there isn't a gem in there somewhere. The essential expressive characteristics of a person's voice change much more slowly than the frame rate of any codec, and predictive coding alone doesn't cover all of the possibilities. There are also not that many unique human voices in the world. If you had several thousand people read a given passage and asked me to listen to them and tell me which voice belongs to a close friend or relative, I doubt I could do it. So, if a codec could adequately pigeonhole the speaker's inflection, accent, timbre, pacing, and other characteristics and send that information only once per transmission, or whenever it actually changes, then a text-to-speech solution with appropriate metadata describing how to 'render' the voice might work really well. Put another way, I doubt the nerves between your brain and your larynx carry anything close to 1.6 kbps of bandwidth. Near-optimal compression might be achieved by modeling the larynx accurately alongside the actual nerve signals that drive it, rather than by trying to represent both with a traditional frame-based predictive model.
  - teraflop 8y ago
walrus01 8y ago
For those interested in low bandwidth audio codecs, take a look at the voice codec used for Iridium handheld satellite phones, which was finalized in about 1998. Fully twenty plus years ago. It doesn't sound the best, but consider the processing power constraints it was designed with... https://en.wikipedia.org/wiki/Iridium_Communications#Voice_and_data_calls https://en.wikipedia.org/wiki/Iridium_Communications#Voice_a...
- jmvalin 8y ago
  Iridium appears to be using a vocoder called AMBE. Its quality is similar to the one of the MELP codec from the demo and it also runs at 2.4 kb/s. LPCNet at 1.6 is a significant improvement over that -- if you can afford the complexity of course (at least it'll work on a phone now).
  - walrus01 8y ago
    Based on my previous experience with Iridium I believe it actually operates at a data rate up to about 3 to 3.2 kb/s. 2400 bps of it is actual usable voice payload, the remaining 600 bps is FEC. Iridium data (not the new next-generation network) service is around the same speed, it's 2400 bps + whatever compression v42bis can gain you. For plain text and stuff it can be a bit faster, something that's already incompressible by v42bis will be right around 2400 baud.
th0ma5 8y ago
Would be really cool when this is in FreeDV and the HF bands benefit from this!
- Scoundreller 8y ago
  My dream is for telecom monopolies to get broken up by technologies like this.
danielvf 8y ago
For comparison, your standard police/fire/medical digital radio in the US sends voice at 4.4Kb/s. So this is a approximately a third of that.
- jdc 8y ago
  So maybe this line of work will mean more spectrum available in the future.
  - metildaa 8y ago
    Only if Motorola gets out of the way and supports modern codecs and standards. Current public safety radio networks are using ancient TDMA tech that Motorola has cobbled together, along with audio codecs that shred voice quality. The only good part is the durability of the pricey radio, some are even intrinsically safe.
    - davidf560 8y ago
      Public safety digital radio networks are primarily APCO Project 25 (P25) which use IMBE/AMBE developed by DVSI. Motorola's original digital radios used a proprietary vocoder called VSELP (also used by iDEN/Nextel). When APCO standardized public safety digital radios, they rejected VSELP and chose IMBE from DVSI instead. Personally I think VSELP sounds better than IMBE, and I'm not sure IMBE was chosen due to technical superiority or if it was political reasons (i.e. picking a non-Motorola solution due to Motorola's dominance). Also, APCO Project 25 Phase 1 was not TDMA, however Phase 2 is. https://en.wikipedia.org/wiki/Project_25 https://en.wikipedia.org/wiki/Project_25 Public safety radio is a true mission critical service that moves slowly - equipment lasts years or decades and is expensive and not frequently replaced or upgraded, hence new technology adoption is slow. Vocoder choice is driven by a standards committee for interoperability (which has seen more emphasis since 9/11), and of course committees aren't typically known for working fast. Public safety radio is definitely not a place for a "move fast and break things" mentality.
- rasz 8y ago
  and uses codec with up to couple hundred MIPS (AMBE 1/+2) computation cost, not GFLOPS
nategri 8y ago
Really thought this was gonna be human-neural-activity-to-speech and I feel like a doofus.
wanderfowl 8y ago
This is really amazing work. Nice to see LPC pushed to its limits, and I can't wait to see what's next for speech compression. Here's hoping the mobile companies pick up on something similar soon.
tomcam 8y ago
Voice quality at that bit rate is absolutely astounding to me. This is one of my favorite things – to see such elegantly applied research.
qwerty456127 8y ago
The examples sound excellent. Top (equal or better) of any text-to-speech synthesizer I've ever heard. I would love to start using it for audio books and for VoIP to save space and traffic as soon as possible. And a Linux-native text-to-speech synthesizer capable of producing speech of this quality is a thing I dream of (now the only option I know is booting to Windows and using Ivona voices)
- metildaa 8y ago
  Mozilla Deepspeech (Speech tp Text) and Mozilla TTS are both useful at this point: https://research.mozilla.org/machine-learning/ https://research.mozilla.org/machine-learning/
zackbloom 8y ago
Just to put this in perspective, a traditional phone line encodes 56 Kb/s of data, which was believed to be the size channel to send the human voice with a reasonable quality. They are doing it in 1.6 Kb/s!
- na85 8y ago
  Aren't "traditional" aka POTS lines analog, and therefore not doing any encoding whatsoever?
  - metildaa 8y ago
    There are band filters and such on legacy, fully analog systems. G.711 (which is standard now for non-cellular call audio) is a step down, but Opus at 16Kbps sounds better to me than a classic, full analog system due to the lack of band cutoff & smarter encoding.

203 points by weinzierl 8y ago

I've been waiting for someone to do this* with audio and/or video. Amazing work. Also worth reading this related link: https://www.rowetel.com/?p=6639 https://www.rowetel.com/?p=6639 There's some really exciting progress that could be made in this space. The page mentions that they could use it as a post-processing filter for Opus for better decoding to avoid changing the bitstream. It could also be useful as a way to accommodate for packet loss and recover "just enough" to avoid interrupting the conversation. * encoding audio through a neural network for network transmission

loudmax 8y ago

For those too impatient to read the details, check out the "Hear for yourself" examples toward the bottom of the page. They're reproducing decent sounding speech at 1.6 kbps. 1.6 kbps is nuts! I like to re-encode audio books or podcasts in Opus at 32 kbps and I consider that stingy. The fact that speech is even comprehensible at 1.6 kbps is impressive. As the article explains, their technique is analogous to speech-to-text, then text-to-speech. The original recordings are a little stiff, and the encoded speech is a little more stiff. It isn't perfect, but it's decent. It'll be interesting to hear this technique applied to normal conversation. If regular speech holds up as well as their samples, it should be perfectly adequate for conversational speech. At 1.6 kbps, which is absurd. Also, I wonder how well this technique could be applied to music. My guess is that it won't do justice to great musicians ... but it might be good enough simple pop tunes.

dietrichepp 8y ago
I’ve done experiments with Opus that produce intelligible (but ugly) speech at 2.3 kbit/s. It involves downsampling the Opus stream at the packet level—e.g. transmit only one out of every three packets. It was surprisingly easy. Nothing as sophisticated as what’s going on here. Also based on work by Xiph. Possibly using the same LPC used here.
jmvalin 8y ago
Actually, this won't work at all for music because it makes fundamental assumptions that the signal is speech. For normal conversations, it should work, though for now the models are not yet as robust as I'd like (in case of noise and reverberation). That's next on the list of things to improve.
- petercooper 8y ago
  I tried it with music and the results were spooky. Very ethereal and ghostly. It was only with some classical music though, I might have to do a pop song next and share the results!
- petercooper 8y ago
  Here we go! This is the first minute or so of Penny Lane by The Beatles converted down to a 10KB .bin and then back to a .wav: http://no.gd/pennylane.wav http://no.gd/pennylane.wav .. unsurprisingly the vocals remain recognizable, but the music barely at all.
  - bch 8y ago
    As imagined by Marilyn Manson...
    - petercooper 8y ago
      Pretty much! It shows off how the codec works to a great extent though as it seems to be misinterpreting parts of the music to be the pitch of the speech, so Paul's voice sounds weird at the start of most lines but okay throughout the lines. I've also run a BBC news report through the program with better results although it demonstrates that any background noise at all can throw things off significantly: https://twitter.com/peterc/status/1111736029558517760 https://twitter.com/peterc/status/1111736029558517760 .. so at this low bitrate, it really is only good for plain speech without any other noise.
      - jmvalin 8y ago
semi-extrinsic 8y ago
For comparison, adaptive GSM encodings, which are in use for cellphones today, are also in the single-digit kbps. https://en.wikipedia.org/wiki/Adaptive_Multi-Rate_audio_codec https://en.wikipedia.org/wiki/Adaptive_Multi-Rate_audio_code...
Eli_P 8y ago
I use --vbr --bitrate 16 and it feels indistinguishable from the original for podcasts. As opusenc takes only wav for input and does not use multiple cores, I had to write scripts for parallel re-encoding of stuff.
- klodolph 8y ago
  I like to use Makefiles for parallel encoding. OPUSFLAGS = --vbr --bitrate 16 all: $(patsubst %.wav,%.opus,$(wildcard *.wav)) .PHONY: all %.opus: %.wav opusenc $(OPUSFLAGS) $< $@ Make -j4 or whatever. There are a few other ways to do this (e.g. xargs).

rasz 8y ago

3 Gflops, we are deep beyond diminishing returns here. Opus seems good enough.

gok 8y ago
3 GFLOP/sec sounds like a lot but it's considerably less math than the radio DSPs inside any modern phone's baseband is doing during a phone call.
- opportune 8y ago
  I don't know much about phone tech, are the basebands really doing math or just instrumenting? My assumption would be that there is just some sensor writing to a buffer at a high frequency but that whatever processes that buffer operates at a lower frequency.
  - anisppp 8y ago
    Your question is hard to parse? What is instrumenting? If it helps though... the word “baseband” itself is the lower frequency containing just the bandwidth of the signal. Ie that is the lower frequency...
    - gok 8y ago
      True but here I intended "baseband" to mean https://en.wikipedia.org/wiki/Baseband_processor https://en.wikipedia.org/wiki/Baseband_processor
nine_k 8y ago
I wonder if a particular network can be implemented more economically if we only run it in "transformation" mode, without the need do train it.
jmvalin 8y ago
Keep in mind that the very first CELP speech codec (in 1984) used to take 90 seconds to encode just 1 second of speech... on a Cray supercomputer. Ten years later, people had that running in their cell phones. It's not just that hardware keeps getting faster, but algorithms are also getting more efficient. LPCNet is already 1/100 the complexity of the original WaveNet (which is just 2 years old) and I'm pretty sure it's still far from optimal.

deleted 8y ago

[deleted]

a-dub 8y ago

might be fun to try to port this to the dadamachines doppler... :)

0-_-0 8y ago

OK, so... Would it be possible to do something similar for video?

cooper12 8y ago
Similar as in the same approach, or as in "apply neural networks to all the things"? Because if it's the former, this approach was very specifically tailored to human speech, taking into account how much it can compress/interpolate qualities like pitch and the spectral envelope. That's far too specific to apply to video. As for the latter, you'd have to perhaps feed Google Scholar the right incantations or ask someone with knowledge. As far as I know, video codecs already have a huge bag of tricks they use (for example the B-frames borrowed in this post). Even then, the key points in this codec were that firstly it's meant for use at very low bitrates, where existing codecs break down, and then secondly it's a vocoder, so it's converting audio to an intermediate form and resynthesizing it. That kind of lossiness is acceptable for audio, but I'm not sure how it would work acceptably for video.
- 0-_-0 8y ago
  I should have been more specific. I meant that instead of compressing video to minimise pixel difference, minimise feature difference instead.

Causality1 8y ago

Local text to speech quality blows this out of the water with a much smaller bandwidth footprint than 1.6kb/s. To me this sounds sort of like "I figured out how to eat a pinecone in less than three seconds". Impressive but not useful.

est31 8y ago
The difference is that it uses the voice of the original person. And STT is rather erroneous. The WER is in the single percent range :).
kstrauser 8y ago
You really can't think of any uses for being able to push 40 simultaneous speech connections through a single 64Kbps voice channel?
CamperBob2 8y ago
That was a fairly ignorant comment because the whole idea behind a speech codec is to compress and reproduce speech in a manner that allows one to at least recognize who's speaking. But I wonder if there isn't a gem in there somewhere. The essential expressive characteristics of a person's voice change much more slowly than the frame rate of any codec, and predictive coding alone doesn't cover all of the possibilities. There are also not that many unique human voices in the world. If you had several thousand people read a given passage and asked me to listen to them and tell me which voice belongs to a close friend or relative, I doubt I could do it. So, if a codec could adequately pigeonhole the speaker's inflection, accent, timbre, pacing, and other characteristics and send that information only once per transmission, or whenever it actually changes, then a text-to-speech solution with appropriate metadata describing how to 'render' the voice might work really well. Put another way, I doubt the nerves between your brain and your larynx carry anything close to 1.6 kbps of bandwidth. Near-optimal compression might be achieved by modeling the larynx accurately alongside the actual nerve signals that drive it, rather than by trying to represent both with a traditional frame-based predictive model.
- teraflop 8y ago

walrus01 8y ago

For those interested in low bandwidth audio codecs, take a look at the voice codec used for Iridium handheld satellite phones, which was finalized in about 1998. Fully twenty plus years ago. It doesn't sound the best, but consider the processing power constraints it was designed with... https://en.wikipedia.org/wiki/Iridium_Communications#Voice_and_data_calls https://en.wikipedia.org/wiki/Iridium_Communications#Voice_a...

jmvalin 8y ago
Iridium appears to be using a vocoder called AMBE. Its quality is similar to the one of the MELP codec from the demo and it also runs at 2.4 kb/s. LPCNet at 1.6 is a significant improvement over that -- if you can afford the complexity of course (at least it'll work on a phone now).
- walrus01 8y ago
  Based on my previous experience with Iridium I believe it actually operates at a data rate up to about 3 to 3.2 kb/s. 2400 bps of it is actual usable voice payload, the remaining 600 bps is FEC. Iridium data (not the new next-generation network) service is around the same speed, it's 2400 bps + whatever compression v42bis can gain you. For plain text and stuff it can be a bit faster, something that's already incompressible by v42bis will be right around 2400 baud.

th0ma5 8y ago

Would be really cool when this is in FreeDV and the HF bands benefit from this!

Scoundreller 8y ago
My dream is for telecom monopolies to get broken up by technologies like this.

danielvf 8y ago

For comparison, your standard police/fire/medical digital radio in the US sends voice at 4.4Kb/s. So this is a approximately a third of that.

jdc 8y ago
So maybe this line of work will mean more spectrum available in the future.
- metildaa 8y ago
  Only if Motorola gets out of the way and supports modern codecs and standards. Current public safety radio networks are using ancient TDMA tech that Motorola has cobbled together, along with audio codecs that shred voice quality. The only good part is the durability of the pricey radio, some are even intrinsically safe.
  - davidf560 8y ago
    Public safety digital radio networks are primarily APCO Project 25 (P25) which use IMBE/AMBE developed by DVSI. Motorola's original digital radios used a proprietary vocoder called VSELP (also used by iDEN/Nextel). When APCO standardized public safety digital radios, they rejected VSELP and chose IMBE from DVSI instead. Personally I think VSELP sounds better than IMBE, and I'm not sure IMBE was chosen due to technical superiority or if it was political reasons (i.e. picking a non-Motorola solution due to Motorola's dominance). Also, APCO Project 25 Phase 1 was not TDMA, however Phase 2 is. https://en.wikipedia.org/wiki/Project_25 https://en.wikipedia.org/wiki/Project_25 Public safety radio is a true mission critical service that moves slowly - equipment lasts years or decades and is expensive and not frequently replaced or upgraded, hence new technology adoption is slow. Vocoder choice is driven by a standards committee for interoperability (which has seen more emphasis since 9/11), and of course committees aren't typically known for working fast. Public safety radio is definitely not a place for a "move fast and break things" mentality.
rasz 8y ago
and uses codec with up to couple hundred MIPS (AMBE 1/+2) computation cost, not GFLOPS

nategri 8y ago

Really thought this was gonna be human-neural-activity-to-speech and I feel like a doofus.

wanderfowl 8y ago

This is really amazing work. Nice to see LPC pushed to its limits, and I can't wait to see what's next for speech compression. Here's hoping the mobile companies pick up on something similar soon.

tomcam 8y ago

Voice quality at that bit rate is absolutely astounding to me. This is one of my favorite things – to see such elegantly applied research.

qwerty456127 8y ago

The examples sound excellent. Top (equal or better) of any text-to-speech synthesizer I've ever heard. I would love to start using it for audio books and for VoIP to save space and traffic as soon as possible. And a Linux-native text-to-speech synthesizer capable of producing speech of this quality is a thing I dream of (now the only option I know is booting to Windows and using Ivona voices)

metildaa 8y ago
Mozilla Deepspeech (Speech tp Text) and Mozilla TTS are both useful at this point: https://research.mozilla.org/machine-learning/ https://research.mozilla.org/machine-learning/

zackbloom 8y ago

Just to put this in perspective, a traditional phone line encodes 56 Kb/s of data, which was believed to be the size channel to send the human voice with a reasonable quality. They are doing it in 1.6 Kb/s!

na85 8y ago
Aren't "traditional" aka POTS lines analog, and therefore not doing any encoding whatsoever?
- metildaa 8y ago
  There are band filters and such on legacy, fully analog systems. G.711 (which is standard now for non-cellular call audio) is a step down, but Opus at 16Kbps sounds better to me than a classic, full analog system due to the lack of band cutoff & smarter encoding.