10 ms·

Lyra V2 – a better, faster, and more versatile speech codec

andrewmcwatters 4y ago
Subjectively beating Opus on quality-to-bit rate is quite impressive, but I noticed the samples had some interesting audible artifacts. I wonder where these come from, and if they're related in any way to this codec using machine learning techniques.
petercooper 4y ago
That sample at 3200 bits per second is fantastic for such a low bitrate. I also love how that works out at 1.44MB/hr.. one floppy disk per hour!
- nine_k 4y ago
  Imagine a portable podcast player for fans of vintage tech. (Won't work with music, of course.)
  - jagger27 4y ago
    Books-on-floppy would work too. Like a digital version of books on tape, with about as many disk changes.
  - ace2358 4y ago
    Please I wish something like that came out. Same with modern tech with cassette and minidisc. Something new but with that kind of hardware. I love the physical mediums!
  - petercooper 4y ago
    A Diskman, if you will..
- odiroot 4y ago
  I still remember the times when I regularly visited my friend with 2 floppies just to get one ~4 minutes MP3 song back home.
learndeeply 4y ago
It would be nice to compare it to a higher quality sample, currently the samples sound like they were recorded through a telephone (4 khz).
- simlevesque 4y ago
  It's meant to be a phone codec.
  - learndeeply 4y ago
    "HD" phone codecs seem higher quality than the example given.
  - bscphil 4y ago
    That's true, but in this era of high quality voice and video calls, it's not that uncommon for someone to want to play a song or even a live instrument, so some capability for handling that intelligently seems important.
  - kevingadd 4y ago
    The mics built into modern smartphones aren't limited to the fidelity level of ancient telephones, so it seems reasonable to hope for more.
    - BenjiWiebe 4y ago
      I believe the limitation was actually not the mics. You can fit a lot more phone calls into a given bandwidth if you heavily restrict the bandwidth of each phone call.
bumblebritches5 4y ago
rektide 4y ago
Very impressive. It'd be interesting to see what the lift would be to get encoding & decoding running in webassembly/wasm. Further, it'd be really neat to try to take something like the tflife_model_wrapper[1] and to get it backed by something like tsjs-tflite[2] perhaps running atop for example tfjs-backend-webgpu[3]. Longer run, the web-nn[4] spec should hopefully simplify/bake-in some of these libraries to the web platform, make running inference much easier. But there's still an interesting challenge & question, that I'm not sure how to tackle; how to take native code, compile it to wasm, but to have some of the implementation provided else-where. At the moment, Lyra V2 can already use XNNPACK[4], which does have a pretty good wasm implementation. But trying to swap out implementations, so for example we might be able to use the GPU or other accelerators, could still have some good benefits on various platforms. [1] https://github.com/google/lyra/pull/89/files#diff-ed2f131a63fbdc64ba2e03e3df47a4ca301c5874401ebd4f15c6ce710634ece6 https://github.com/google/lyra/pull/89/files#diff-ed2f131a63... [2] https://www.npmjs.com/package/@tensorflow/tfjs-tflite https://www.npmjs.com/package/@tensorflow/tfjs-tflite [3] https://www.npmjs.com/package/@tensorflow/tfjs-backend-webgpu https://www.npmjs.com/package/@tensorflow/tfjs-backend-webgp... [4] https://www.w3.org/TR/webnn/ https://www.w3.org/TR/webnn/ [5] https://github.com/google/XNNPACK https://github.com/google/XNNPACK
- encryptluks2 4y ago
  Why would you want to run codecs I'm WASM? Makes no sense to me.
  - rektide 4y ago
    Cause the web is awesome and anyone could use something built for it with zero friction.
  - jagger27 4y ago
    Forwards and backwards compatibility and not having to rely on every vendor to ship support for the codec you want to use in your software.
freediver 4y ago
This is an amazing contribution by Google. I wonder if there is a simple WebRTC demo app available with this codec plugged in?
dale_glass 4y ago
Is there an open codec that concentrates on low CPU usage? I'm fine with it not being very bandwidth efficient. Opus is a very good codec, but it's not amazing CPU-wise. I work on a VR world, and audio encoding is usually our most limiting factor when running on an VPS. We have the capability to negotiate codecs, so the high cpu/low bandwidth use case is already covered. What I'm looking for specifically: * Low CPU usage * Support for high bitrate, suitable for music and sounds other than voice * Low latency
- CharlesW 4y ago
  I couldn't find data on CPU requirements for encode/decode versus, say, Opus, but Apple uses AAC-LD for similar scenarios.
- rasz 4y ago
  How is Audio encoding the most limiting factor in a VR project? :o Afaik Opus encoder eats something like 30-50MHz of one cpu core.
  - CodesInChaos 4y ago
    It sounds plausible that it's the most expensive thing on the server side, if you have cheap simulation/behaviour and many concurrent users. But unless it's a non commercial project, the cost shouldn't be a big deal, so it's still a bit strange.
    - dale_glass 4y ago
      We work on a community-led fork of the dead commercial High Fidelity project. The server requirements are indeed very light except for audio. Physics are actually farmed out to the clients themselves, it's a bit of a quirky idea, but it actually works if one isn't concerned with accuracy.
      - rasz 4y ago
        Ventrilo/teampspeak servers run great on shared hosting. https://www.myteamspeak.com/addons/9ddfa0b2-25c2-4302-8a43-07f8819af9a9 https://www.myteamspeak.com/addons/9ddfa0b2-25c2-4302-8a43-0... gives you positional audio support on teamspeak server
squarefoot 4y ago
Check out also Codec2, Open Source as well, which offers really good quality down to 700 bit/s. It has been ported to small MCUs such as the ESP32, STM32 etc. also supported by Arduino libraries. https://www.rowetel.com/?page_id=452 https://www.rowetel.com/?page_id=452 https://github.com/deulis/ESP32_Codec2 https://github.com/deulis/ESP32_Codec2 https://www.arduino.cc/reference/en/libraries/codec2/ https://www.arduino.cc/reference/en/libraries/codec2/
- throw0101a 4y ago
  > Check out also Codec2, Open Source as well, which offers really good quality down to 700 bit/s Yes, but it tops out at 3200b/s: * https://en.wikipedia.org/wiki/Codec_2 https://en.wikipedia.org/wiki/Codec_2 Lyra V2 seems to start there and then goes up to teens of kb/s, at which point Opus can perhaps take over the job.
  - squarefoot 4y ago
    Yes, that is done on purpose. The goal of the project is to be used in HF/VHF voice communications so it makes sense. That low bandwidth usage allows it to be employed from ordinary HAM gear down to cheap LoRa modules, a feature that opens huge possibilities like building point to point or multipoint encrypted communications with portable devices not tied to cellphone towers, which in some areas of the world would be so useful these days.
    - FatCatsClub 4y ago
      >employed from ordinary HAM gear down to cheap LoRa modules, a feature that opens huge possibilities like building point to point or multipoint encrypted communications I hope you aren't suggesting that encryption should be used on amateur bands.
      - dmos62 4y ago
        Is that not a good idea?
hoppyhoppy2 4y ago
What's the difference between googleblog.com and blog.google ?
- deleted 4y ago
  [deleted]
- rootw0rm 4y ago
  blog.google.com = blogger.l.google.com = 172.217.14.105 googleblog.com = 172.217.14.65
dcl 4y ago
As an ML person I totally get the how NN works to accomplish this and it's very cool. What's really cool is how they get this to work in real time with little/no latency.
londons_explore 4y ago
Things missing that I think would have to be added before this could become a widely used standard: * Variable bitrate. For many uses, the goal isn't 'fill this channel', but instead 'transmit this audio stream at the best quality possible'. That means filling the channel sometimes, but other times transmitting less data (ie. When there is silence, or when the entropy of the speech being sent is low - for example the user is saying something very predictable). * Handling all types of audio. Even something designed for phone calls will occasionally be asked by users to transmit music, sound effects, etc. The codec should do an acceptable job at those other tasks.
Conan_Kudo 4y ago
Alas, the sources require Bazel to build. That's going to limit adoption because Bazel is difficult to deal with and most projects don't use it.
- ajconway 4y ago
  1. Install numpy (pip3 install numpy) 2. Download a bazel binary (https://github.com/bazelbuild/bazel/releases https://github.com/bazelbuild/bazel/releases or use package manager) 3. bazel build -c opt :encoder_main 4. bazel-bin/encoder_main --input_path=testdata/sample1_16kHz.wav --output_dir=$HOME/temp --bitrate=3200 Done!

Subjectively beating Opus on quality-to-bit rate is quite impressive, but I noticed the samples had some interesting audible artifacts. I wonder where these come from, and if they're related in any way to this codec using machine learning techniques.

petercooper 4y ago

That sample at 3200 bits per second is fantastic for such a low bitrate. I also love how that works out at 1.44MB/hr.. one floppy disk per hour!

nine_k 4y ago
Imagine a portable podcast player for fans of vintage tech. (Won't work with music, of course.)
- jagger27 4y ago
  Books-on-floppy would work too. Like a digital version of books on tape, with about as many disk changes.
- ace2358 4y ago
  Please I wish something like that came out. Same with modern tech with cassette and minidisc. Something new but with that kind of hardware. I love the physical mediums!
- petercooper 4y ago
  A Diskman, if you will..
odiroot 4y ago
I still remember the times when I regularly visited my friend with 2 floppies just to get one ~4 minutes MP3 song back home.

learndeeply 4y ago

It would be nice to compare it to a higher quality sample, currently the samples sound like they were recorded through a telephone (4 khz).

simlevesque 4y ago
It's meant to be a phone codec.
- learndeeply 4y ago
  "HD" phone codecs seem higher quality than the example given.
- bscphil 4y ago
  That's true, but in this era of high quality voice and video calls, it's not that uncommon for someone to want to play a song or even a live instrument, so some capability for handling that intelligently seems important.
- kevingadd 4y ago
  The mics built into modern smartphones aren't limited to the fidelity level of ancient telephones, so it seems reasonable to hope for more.
  - BenjiWiebe 4y ago
    I believe the limitation was actually not the mics. You can fit a lot more phone calls into a given bandwidth if you heavily restrict the bandwidth of each phone call.

bumblebritches5 4y ago

rektide 4y ago

Very impressive. It'd be interesting to see what the lift would be to get encoding & decoding running in webassembly/wasm. Further, it'd be really neat to try to take something like the tflife_model_wrapper[1] and to get it backed by something like tsjs-tflite[2] perhaps running atop for example tfjs-backend-webgpu[3]. Longer run, the web-nn[4] spec should hopefully simplify/bake-in some of these libraries to the web platform, make running inference much easier. But there's still an interesting challenge & question, that I'm not sure how to tackle; how to take native code, compile it to wasm, but to have some of the implementation provided else-where. At the moment, Lyra V2 can already use XNNPACK[4], which does have a pretty good wasm implementation. But trying to swap out implementations, so for example we might be able to use the GPU or other accelerators, could still have some good benefits on various platforms. [1] https://github.com/google/lyra/pull/89/files#diff-ed2f131a63fbdc64ba2e03e3df47a4ca301c5874401ebd4f15c6ce710634ece6 https://github.com/google/lyra/pull/89/files#diff-ed2f131a63... [2] https://www.npmjs.com/package/@tensorflow/tfjs-tflite https://www.npmjs.com/package/@tensorflow/tfjs-tflite [3] https://www.npmjs.com/package/@tensorflow/tfjs-backend-webgpu https://www.npmjs.com/package/@tensorflow/tfjs-backend-webgp... [4] https://www.w3.org/TR/webnn/ https://www.w3.org/TR/webnn/ [5] https://github.com/google/XNNPACK https://github.com/google/XNNPACK

encryptluks2 4y ago
Why would you want to run codecs I'm WASM? Makes no sense to me.
- rektide 4y ago
  Cause the web is awesome and anyone could use something built for it with zero friction.
- jagger27 4y ago
  Forwards and backwards compatibility and not having to rely on every vendor to ship support for the codec you want to use in your software.

freediver 4y ago

This is an amazing contribution by Google. I wonder if there is a simple WebRTC demo app available with this codec plugged in?

dale_glass 4y ago

Is there an open codec that concentrates on low CPU usage? I'm fine with it not being very bandwidth efficient. Opus is a very good codec, but it's not amazing CPU-wise. I work on a VR world, and audio encoding is usually our most limiting factor when running on an VPS. We have the capability to negotiate codecs, so the high cpu/low bandwidth use case is already covered. What I'm looking for specifically: * Low CPU usage * Support for high bitrate, suitable for music and sounds other than voice * Low latency

CharlesW 4y ago
I couldn't find data on CPU requirements for encode/decode versus, say, Opus, but Apple uses AAC-LD for similar scenarios.
rasz 4y ago
How is Audio encoding the most limiting factor in a VR project? :o Afaik Opus encoder eats something like 30-50MHz of one cpu core.
- CodesInChaos 4y ago
  It sounds plausible that it's the most expensive thing on the server side, if you have cheap simulation/behaviour and many concurrent users. But unless it's a non commercial project, the cost shouldn't be a big deal, so it's still a bit strange.
  - dale_glass 4y ago
    We work on a community-led fork of the dead commercial High Fidelity project. The server requirements are indeed very light except for audio. Physics are actually farmed out to the clients themselves, it's a bit of a quirky idea, but it actually works if one isn't concerned with accuracy.
    - rasz 4y ago
      Ventrilo/teampspeak servers run great on shared hosting. https://www.myteamspeak.com/addons/9ddfa0b2-25c2-4302-8a43-07f8819af9a9 https://www.myteamspeak.com/addons/9ddfa0b2-25c2-4302-8a43-0... gives you positional audio support on teamspeak server

squarefoot 4y ago

Check out also Codec2, Open Source as well, which offers really good quality down to 700 bit/s. It has been ported to small MCUs such as the ESP32, STM32 etc. also supported by Arduino libraries. https://www.rowetel.com/?page_id=452 https://www.rowetel.com/?page_id=452 https://github.com/deulis/ESP32_Codec2 https://github.com/deulis/ESP32_Codec2 https://www.arduino.cc/reference/en/libraries/codec2/ https://www.arduino.cc/reference/en/libraries/codec2/

throw0101a 4y ago
> Check out also Codec2, Open Source as well, which offers really good quality down to 700 bit/s Yes, but it tops out at 3200b/s: * https://en.wikipedia.org/wiki/Codec_2 https://en.wikipedia.org/wiki/Codec_2 Lyra V2 seems to start there and then goes up to teens of kb/s, at which point Opus can perhaps take over the job.
- squarefoot 4y ago
  Yes, that is done on purpose. The goal of the project is to be used in HF/VHF voice communications so it makes sense. That low bandwidth usage allows it to be employed from ordinary HAM gear down to cheap LoRa modules, a feature that opens huge possibilities like building point to point or multipoint encrypted communications with portable devices not tied to cellphone towers, which in some areas of the world would be so useful these days.
  - FatCatsClub 4y ago
    >employed from ordinary HAM gear down to cheap LoRa modules, a feature that opens huge possibilities like building point to point or multipoint encrypted communications I hope you aren't suggesting that encryption should be used on amateur bands.
    - dmos62 4y ago
      Is that not a good idea?

hoppyhoppy2 4y ago

What's the difference between googleblog.com and blog.google ?

deleted 4y ago
[deleted]
rootw0rm 4y ago
blog.google.com = blogger.l.google.com = 172.217.14.105 googleblog.com = 172.217.14.65

dcl 4y ago

As an ML person I totally get the how NN works to accomplish this and it's very cool. What's really cool is how they get this to work in real time with little/no latency.

londons_explore 4y ago

Things missing that I think would have to be added before this could become a widely used standard: * Variable bitrate. For many uses, the goal isn't 'fill this channel', but instead 'transmit this audio stream at the best quality possible'. That means filling the channel sometimes, but other times transmitting less data (ie. When there is silence, or when the entropy of the speech being sent is low - for example the user is saying something very predictable). * Handling all types of audio. Even something designed for phone calls will occasionally be asked by users to transmit music, sound effects, etc. The codec should do an acceptable job at those other tasks.

Conan_Kudo 4y ago

Alas, the sources require Bazel to build. That's going to limit adoption because Bazel is difficult to deal with and most projects don't use it.

ajconway 4y ago
1. Install numpy (pip3 install numpy) 2. Download a bazel binary (https://github.com/bazelbuild/bazel/releases https://github.com/bazelbuild/bazel/releases or use package manager) 3. bazel build -c opt :encoder_main 4. bazel-bin/encoder_main --input_path=testdata/sample1_16kHz.wav --output_dir=$HOME/temp --bitrate=3200 Done!