17 ms·
Lyra audio codec enables high-quality voice calls at 3 kbps bitrate
- motiejus 6y agoNothing about licensing or patents. I assume the worst (read: unusable for small businesses)? 10+ years ago I worked in a small voip shop, where we had very high quality (low jitter), but low bandwidth connection. I researched many codecs of the time (2010-ish). We liked speex, because it can be used "without strings attached". Also, I can choose the quality depending on the bandwidth. Although for low bandwidth g729 was better. Which we couldn't use because of royalties (but allowed myself to test it). We chose alaw/ulaw when bandwidth was not a concern, and speex when it was. Since it does not mention usability outside of google, I also find this comparison unfair or incomplete: if you are comparing a proprietary codec, compare it to g729. If you are comparing a codec to speex, it should be open/free. Edit: grammar
- lights0123 6y agoI wouldn't be as pessimistic—they invented VP9 and were important to AV1, and continue to promote both and use them everywhere.
- marcodiego 6y agoWith good enough licensing it could possibly replace speex. It is sad that we have to think about licensing and patents of technologies instead of only how good or advanced they are.
- 1996 6y ago> Nothing about licensing or patents. I assume the worst (read: unusable for small businesses)? If there's a free software implementation, and a company offering the service based in the EU (or shop around and find any other jurisdiction where software patents don't matter), it's often YOLO - but call that "legal arbitrage" if you want to sound fancy :)
- bscphil 6y agoThese days the correct comparison would be to Opus, which is similarly unencumbered and performs fantastically at low bitrates (and has a speech specific mode for even lower bitrates, because it's a hybrid of two codecs). It's also extremely low latency, so there's now no reason to accept trade-offs. (For the same bitrate as alaw/ulaw, you can get high quality full band music with Opus.) These days it's more or less the standard for realtime voice. WebRTC uses it, most of the popular realtime voice applications use it as well, as does Signal.
- IshKebab 6y agoThey did compare to Opus.
- kreetx 6y agoWhat are the current choices for "CD quality" speech compression (lossy but indiscernible) at the moment? Just had a discussion with a friend of keeping an always-on speech recorder on and wondered about disk space consumption.
- p1mrx 6y agoBasically Opus (Vorbis successor) or AAC (MP3 successor).
- Aloha 6y agoI'm over here preferring g711/ulaw because I prefer the hard roll off at 4kc.
- dkjaudyeqooe 6y agoA fair point if you're evaluating actual usage rather than just quality, but Google isn't going to be only one with this sort of codec. We can expect better free versions to come along in time.
- pabs3 6y agoThe G.729 patents are now royalty-free btw: https://web.archive.org/web/20170202062530/http://www.sipro.com/G729.html https://web.archive.org/web/20170202062530/http://www.sipro.... https://en.wikipedia.org/wiki/G.729 https://en.wikipedia.org/wiki/G.729
- LinuxBender 6y agoI am curious about this as well. With all the debates in the threads here about this codec, I think it would be worth having the folks at Mumble [1] (murmur server) incorporate it as an option so people could fire up an instance and put it to the test. That of course only works if the license is compatible. They compare the performance to Opus which is used currently by Mumble. [1] - https://www.mumble.info/ https://www.mumble.info/
- faebi 6y agoMaybe in the future, all we need is a speech example, some AI and the continious transmission of text for low data voice transmission?
- dheera 6y agoI think this will be the rough direction, but not exactly text, rather some other efficient, machine-readable embedding of speech that is also able to carry tone and rhythm effectively and pronounciation accurately and unambiguously.
- cecja 6y agoWhy speak then?
- wmf 6y agoBasically yes. "Features, or distinctive speech attributes, are extracted from speech every 40ms and are then compressed for transmission. The features themselves are log mel spectrograms, a list of numbers representing the speech energy in different frequency bands, which have traditionally been used for their perceptual relevance because they are modeled after human auditory response."
- viraptor 6y agoOnly if you're ready to kill all the intonation nuance. If you're ok with that, why not stick to just reading text? At least we can use emotes in there.
- tyingq 6y agoIf the demos are actually representative, it does seem impressive. Could save a lot of bandwidth for VoiP if it replaced 8kb/s G729.
- lxgr 6y agoIsn‘t VoIP at such low data rates already dominated by the overhead of UDP, IP and whatever lower layer? Multiplexing it with a low-bandwidth video stream would be possible, though. I was thinking this could be most relevant for something like digital wireless transmissions.
- nousermane 6y agoTo say the least, yeah. At 3kbps and 20ms framing, it's only 7.5 bytes of payload per frame. RTP, UDP, IP, and Ethernet overhead are what - 60-ish bytes?
- tyingq 6y agoG.729 is 21-30kbps with transport overhead, depending on a few factors. So shaving off 5kbps would still be meaningful. Or better quality at the same bandwidth might enable in-band DTMF or fax, neither works on G.729 now.
- lxgr 6y agoIn-band fax will certainly not work over a lossy voice codec, unless your fax modem is able to mimic human speech patterns.
- eznzt 6y agoIs it any good for languages other than English?
- wmf 6y ago"As with any ML based system, the model must be trained to make sure that it works for everyone. We’ve trained Lyra with thousands of hours of audio with speakers in over 70 languages using open-source audio libraries and then verifying the audio quality with expert and crowdsourced listeners. ... Lyra trains on a wide dataset, including speakers in a myriad of languages, to make sure the codec is robust to any situation it might encounter."
- sophiebits 6y agoPrevious discussion: https://news.ycombinator.com/item?id=26279891 https://news.ycombinator.com/item?id=26279891
- hatsunearu 6y agohttp://www.rowetel.com/wordpress/?page_id=452 http://www.rowetel.com/wordpress/?page_id=452 Take a look at this too. Also runnable on low power devices. And there was some work of using AI to enhance the codec2 encoded bits too.
- deleted 6y ago[deleted]
- ddevault 6y agoYeah, with Codec 2 setting the gold standard, I don't find this very impressive. I find this more intelligible at one-third of the Lyra bitrate: http://www.rowetel.com/downloads/codec2/hts2a_1300.wav http://www.rowetel.com/downloads/codec2/hts2a_1300.wav Codec 2 does a better job of isolating the parts of sound which are most necessary to intelligible speech, without necessarily caring too much about preserving the original qualities of the speaker's voice or environment. Fun fact: Codec 2 can be used to transmit voice over IRC: https://github.com/asiekierka/voirc https://github.com/asiekierka/voirc
- londons_explore 6y agoI had to listen to your wav sample 4 times before understanding what it was saying... To me, that isn't intelligible... Perhaps with practice one could learn to understand it, but that isn't really what I want from my audio codec.
- vidarh 6y agoThe biggest challenge with evaluating all of these, is that once you've listened to a comprehensible version of one of these samples, they all sound more intelligible. I had problems with the example too. After hearing the original it's now easy. It makes it really hard to properly assess the intelligibility for developers without decent sized panels of people to help evaluating them.
- alvarlagerlof 6y agoCompletely unrelated but damn so they need to update the illustration at the top of that page. It's hideous.
- temp-dude-87844 6y agoWhen Google's announcement [1] was posted a few days ago, I listened to their samples and heard an odd effect in the "chocolate bread" sample (the video chat example) [1], which is not mirrored in this article. On that sample, I felt [2] that the Lyra version exaggerates the pronunciation of the phrase 'with chocolate' in a way that meaningfully differs from the speaker's original. It weakens the voiced 'th' to nothingness, and overshoots both the lead consonant and first vowel of 'choc', and then proceeds to wash the entire rest of the sentence with a peculiar brightened voice that's high, lacks consonant definition, and is close to ringing. I'm guessing it's actually style transfer, because though the result sounds not much like the speaker's original, the result is reminiscent of the speech pattern and accent that people with East Asian and Southeast Asian ancestry adopt when speaking American English. It was surprising, given that the speaker doesn't sound like that in the original. I wonder if others hear this too. While Lyra sounds richer and wider-band than Opus or Speex at these bitrates, the degradations and artifacts of those codecs are universally recognized (through years of familiarity with telephones) as compression artifacts and not innate features of the speaker themselves. Therefore listeners can be expected to be sympathetic to the quality issues and not attribute the whole of the sound on the speaker's person. If AI-trained voice synthesizer codecs become the norm, and it performs well on most speakers, that expectation will go away, and the resulting audio will be attributed wholly to the speaker. That increases the impact of mistakes and misrepresentations introduced by the codec, unbeknowst to the speaker and listener. [1] https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-... [2] https://news.ycombinator.com/item?id=26282519 https://news.ycombinator.com/item?id=26282519
- ampdepolymerase 6y agoAre the speech models sufficiently generic across all languages?
- qayxc 6y agoThat remains to be seen. In my experience the performance with anything other than (US) English is mediocre at best and less common the language, the worse results get. So while Spanish, French, or German might get there eventually, don't even try Polish, Czech, or Farsi (Persian) dialects.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- skyde 6y agothis is super impressive
- boneitis 6y agoI found the Lyra codecs in both examples easily the most difficult to comprehend, even compared against the scratchy Speex. Am I the only one? It is a little odd to me to see the praise here and on the previous discussion. To be fair, I am convinced I have APD (and, to be fair again, I have never got it checked out). E: Just realized there is a third example. Perhaps it is not as strong a statement due to Opus' doubled bitrate, but it is still far scratchier. Yet, it is more decipherable than the Lyra codec to me.
- deleted 6y ago[deleted]
- bscphil 6y agoI personally wouldn't say "difficult to comprehend". I would say that the Lyra audio is "cleaner" but that the artifacts that there are, are louder and more annoying in Lyra. There's a very bad ringing effect and some flutter. If you personally find these artifacts distracting or confusing, I could very easily see the Lyra examples being harder to understand. I'm almost certain that Lyra has increased the volume on the first sample too. It's quite audible, although I haven't confirmed this with Audacity. Through good quality headphones, I actually find the Lyra artifacts rather piercing and think I'd pretty quickly get fatigued through having it in my ears over a long conversation. Maybe they would handle this better with a bit of a lowpass filter added.
- acdha 6y agoI noticed the loudness, too, followed by a bit of a letdown with the subsequent odd artifacting. I was wondering whether that was also a factor in the user-perception ratings similar to how FM radio stations all started deploying compression to sound louder because it increased the odds of people favorably picking out their station while scanning through the spectrum.
- contravariant 6y agoIt sounds 'clean' but the remaining artefacts are quite weird. Somehow the intonation/timing feels off, which is just sounds plain odd. The compression artefacts are 'ugly' but at least recogniseable as compression artefacts.
- wrongdonf 6y agoWe are getting to the point where compression is so good, you aren’t actually hearing the other person. Wild
- moonbug 6y agoyou'll never know if you're speaking to the Blight.
- kristofferR 6y agoThe Lyra examples are way harder to understand than the rest of them though. It injects sounds that wasn't there. It sounds like he said "Someuve" instead of the "Some" clearly audible in the other versions.
- walrus01 6y agothe 3kbps example here with the 'bread with chocolate filling inside' video is frankly amazing, how good it is compared to the original. https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-... It is unfortunate that for now this appears to be proprietary, closed source and being treated as a google competitive advantage over others, unlike opus which is fully open.
- londons_explore 6y agoIf they add it to WebRTC as they suggest, it will get auto-included in nearly all videoconferencing applications (most use webrtc under the covers, and a simple git pull will get it included in the next release).
- walrus01 6y agoThey briefly mention the existence and current role of the webrtc codecs, but I don't see where they suggest they intend to contribute it or open it up as a library others can use.
- londons_explore 6y agoIf they don't release it as part of webrtc, they won't be able to use it in browser based videoconferencing. Google Meet/Hangouts on desktop rely on that. While it is theoretically possible to process audio with something compiled with webassembly, data can't be marshalled into/out of a webassembly worker without the main browser threads help, and that tends to be too janky to use for realtime audio on most platforms. That pretty much forces googles hand - if they want to use it in their web-based products, it must be opensource and available to all competitors. They might say this feature only works with their mobile apps though.
- lankalanka 6y agoIt would take some time to see how it goes. It's possible to run on browsers without being a member of WebRTC codec family. The codec can be deployed as a WebAssemble module and WebRTC acts as a data channel for it. Zoom used to deploy a similar approach while staying on its proprietary codec.
- walrus01 6y agoMicrosoft also just announced a 6kbps advanced audio codec: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/satin-microsoft-s-latest-ai-powered-audio-codec-for-real-time/ba-p/2141382 https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...
- perryizgr8 6y agoAnd it sounds significantly better to me, which is expected since they use double the bitrate.
- londons_explore 6y agoThis is blogspam that doesn't add meaningfully to the original[1]. [1]: https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-...
- etaioinshrdlu 6y agoImportant to remember that this type of codec can be used as a backup for higher-bandwidth codecs. You don’t necessarily need to hear it’s artifacts all the time. The higher level codec also only needs to encode the differences between the prediction and groundtruth. The same thinking applies to video especially of faces. Neural nets are a huge leap forward for this type of data compression and will likely be used pretty much everywhere in the future with great success.
- rectang 6y agoWhat I want to know is whether Lyra takes any longer to encode than the alternatives. Because as far as I can tell, nobody cares in the slightest about latency. Phone calls are getting to be like writing postcards to each other. Speak in a whole paragraph. Wait several seconds for the latency to clear. Then the other party responds with a whole paragraph, waits several seconds for the latency to clear... Improvements to fidelity are nice-to-have, but I would like some real-time in my real-time communications, please.
- reaperducer 6y agoI had the pleasure of using a real landline just before the pandemic. Honest wire-to-wire connection between two ranches, so no silly VOIP steps between. It was fantastic. You don't appreciate how much latency is destroying our ability to communicate verbally until you go back to the old way. One example is arguing. It's no wonder people used to be able to argue with one another on a telephone. You could raise your voice and still hear the other side and adjust your speech in real time. Today it's just one party shouting over the other to drown the opponent out.
- rectang 6y agoBetween miserable latency, not-so-great fidelity, and the fecklessness of phone companies in the face of the robocall epidemic, I have come to hate phone calls. I'm rooting for something to replace phone communications. Any chance that Matrix can do better on any of those fronts? Especially on fidelity and latency since they're germane to the high-level subject of this discussion.
- colordrops 6y agoOne conspiracy theory is that tech companies have lobbied to prevent real action on robo-calls, in order to get people like you to hate calls and migrate to online services.
- eeZah7Ux 6y ago
- Vadoff 6y agoHow come the clean reference wav file is 168KB, while the clean Lyra (@3kbps) wav file is significantly larger at 328KB?
- walrus01 6y agoSince a browser can't play lyra I think they took the lyra output and put it inside something lossless like a 44kHz stereo wav so that people can listen to it.
- bscphil 6y agoWhen they converted back to wav from Lyra, they used a 32 bit 16 Khz wav instead of 16 bit 16 Khz wav like the source. The size of the Lyra file is almost exactly 2x as big as the reference. Note that this isn't cheating in any way, the source is the source, so it's just a quirk from their conversion process. Probably the tooling around Lyra is pretty rudimentary and the decoder could only output a 32 bit file.
- jimktrains2 6y agoI wonder how this compares with codec2[0] which is decent at around 3kbps and can go even lower. [0] https://en.m.wikipedia.org/wiki/Codec_2 https://en.m.wikipedia.org/wiki/Codec_2
- donpark 6y agoAccording to someone who listened to both, Lyra beats Codec2 on quality. YMMV
- trevorishere 6y agoI'd be curious to know how this compares to Microsoft's Satin codec, now used in Teams, which is ML-driven.
- charlesdaniels 6y agoThis reminds me of a technology in Vernor Vinge's "zone of thought" series. I think they called it "evocations", where at the beginning of a call, a model is transmitted that allows the other end to re-construct what the sender would look/sound like from severely abridged data. It sure sounds plausible - the semantically meaningful parts of a conversation (video or audio) would appear to have significantly less entropy that all of the details captured by a mic/webcam. The fact that things like JPEG and MP3 exist are proof enough, and those (to my knowledge) aren't even feature-based. Maybe N years from now, your {Skype,FaceTime,Zoom,Jitsi} call starts by transmitting a pre-trained auto-encoder that can reproduce your speech and visual appears with a "good enough" margin of error from a few kbps worth of data.
- ryukafalz 6y agoIt's been done, at least for video: https://www.youtube.com/watch?v=NqmMnjJ6GEg&feature=emb_title https://www.youtube.com/watch?v=NqmMnjJ6GEg&feature=emb_titl... Not for audio yet, I think?
- bscphil 6y agoI've just taken a minute to confirm what my ears told me in Audacity. Please have a look at this screenshot: https://cloudflare-ipfs.com/ipfs/Qma41RMzieQ6ZGdGem9rLxnxEL15oEs9tmiF288KmazfTs/ref_lyra_opus.png https://cloudflare-ipfs.com/ipfs/Qma41RMzieQ6ZGdGem9rLxnxEL1... The Lyra version is clearly much louder. This is a serious problem and it borders on being reasonable to call it "cheating". It's well known in the audio biz that if you ask people to compare two experiences, and one of them is a bit louder than the other, people will say that the louder one was better, or came through more clearly, or whatever it is you're trying to market for. For the purpose of comparing artifacts in two samples, it's absolutely crucial that they be the same volume. You might as well compare two image compression codecs where one of them "enhances" the colors of the original image. Note: I took the clips for this comparison from the "clean speech" examples at the original source on Googleblog, not the blogspam.
- dkjaudyeqooe 6y agoIsn't that due to audio (frequency) compression coming out of the generative model? I guess that can be tweaked either way but they're going to tend towards that exactly because it sounds louder and thus clearer.
- bscphil 6y agoThere are a couple of effects here: 1. Lossy codecs will use a low-pass filter to get rid of hard to compress higher frequencies. This is often inaudible, but even when it is, it should lower the volume, unless you're applying some kind of compensation for it. 2. It's true that lossy codecs compress different frequencies differently, but that's not usually done in such a way that amounts to applying EQ to the frequencies. 3. Even if the relative balance of frequencies did shift as a result of applying lossy compression, this is still done in a way that the overall loudness of the audio does not change. In this case the Lyra output has changed significantly and in an easily audible way (about +6 dB). You could easily get the same effect in Opus just by amplifying (or applying compression to) the result, but Opus is doing things correctly.
- rectang 6y ago
- annoyingnoob 6y agoLooks interesting. What is the use case for a 3k codec in 2021? Having the Lyra sample louder than the others is cheating and gives a false impression.
- walrus01 6y agoThe original Iridium satellite phone voice codec did voice calls at approximately 2800 bps 22 years ago, but the quality is quite poor, and it's very rudimentary. Encoding power and CPUs were quite different in 1998/1999.
- esturk 6y agoAs one post alluded to the side effect of creating a "synthetic accent", I wonder what kind of social implications this will create. Over time, as these "synthetic accents" gain wide spread adoption, would the main stream pronunciation also adopt such an accent? It's certainly interesting to think about.
- RasmusLarsen 6y agoI'm not going to say it's not impressive given the limitations, but am I an outlier if I say it sounds unacceptably bad for a voice call? If someone was on discord/zoom/hangouts with that quality I'd ask them to check if they had some hardware issues or their connection was borked.
- jdkee 6y agoHow does this bit rate compare to POTS? "Restricted to a narrow frequency range of 300–3,300 Hz, called the voiceband, which is much less than the human hearing range of 20–20,000 Hz" https://en.wikipedia.org/wiki/Plain_old_telephone_service https://en.wikipedia.org/wiki/Plain_old_telephone_service
- mfkp 6y agoAfter reading vidarh's comment below ("The biggest challenge with evaluating all of these, is that once you've listened to a comprehensible version of one of these samples, they all sound more intelligible."), I recommend trying to listen to the Lyra version first, then listen to the original next. It actually does sound very strange when you don't have the true audio parsed in your head already. I'm not sure if I would like having to do the extra work to translate this in my head on every conversation. It's more apparent in the video example from the Google post, I won't spoil it but there's a word that starts with B that sounds very funny in the Lyra version (listen to that one first): https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-...
- jbluepolarbear 6y agoI did as you said and watched the Lyra version first. I couldn’t understand her and the tone wasn’t matching her lips and mouth. It sounds like someone with a speech impediment is dubbing someone who is not. I wonder how much miscommunication is caused by poor compression and latency.
- IshKebab 6y agoDefinitely a real effect, but Lyra is still clearly an improvement and I assume their MOS testing accounts for that.
- jbluepolarbear 6y agoI listened to the first 4 samples of the woman speaking and Lyra was the most clear and close to the loudness of the original, but it was also the most different. Lyra sounds like a computer voice of the women speaking the sentence back. I don’t know how to explain it better, I pick out the slight buildup/double “p” and it’s not present in the Lyra version. Almost like it’s removing human speech imperfections.
- guywhocodes 6y agoFeels like one could get something comparable with RNNoise into DCT and then just gziping the frames.
- vortico 6y agoWhy in the world do we need 3 kbps audio for voice? It's so hard to hear people speaking over mobile phones. Why don't we use 32-64 kbps 48 kHz for all voice communication? GSM CSD offers 9-14 kbps down/up, 3G offers 384 kbps, and EDGE offers 473 kbps. Why limit to 3 kbps?
- Manfred 6y agoFor situations with low bandwidth. Communication in space or far outside of civilization.
- mkl 6y agoThere are many reasons. - Not everyone has such good connectivity. - So you can handle many streams at once, like in a big meeting, without a server mixing them. - So you can have decent quality video on the same limited connection. - So you can archive large amounts of speech. - To advance the state of the art. - etc.
- kilroy123 6y agoI think the old Iridium satellite constellation only went as high 3 kbps. It was heavily depended on by the US military. I'm sure there's still a need for this.
- drudu 6y agoif you can't answer the question yourself, you wouldn't ever be able to understand the answer. maybe i want to have two calls at once / maybe i have a shitty gsm/csd link / maybe we are transmitting over a link that has less than 10kbps bandwidth. asking this question implies you have no imagination or awareness beyond what you already know. i'll get banned from hn for this but goddamn you're a dumb person
- dang 6y agoWe've banned this account. Please do not create accounts to break HN's rules with. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- kragen 6y agoThe LPCNet decoder for Codec2 enables high-quality voice calls at 1700 bits per second since 02019: http://www.rowetel.com/?p=6639 http://www.rowetel.com/?p=6639 Why doesn't this article mention LPCNet or Codec2?
- MarkusWandel 6y agoThis is subjective, but there is loss of real speech information here. I don't hear the accents of these speech samples daily. I can easily follow the uncompressed versions. But the heavily compressed versions delete cues and I could not initially understand them. This is an effect even with regular telephony these days. A smartphone, carefully held a little way from your ear because you don't trust it to mask touch events when used as a phone, and using 16kbps audio, is not as understandable as an old fashioned hardline phone. Ironically, higher fidelity audio via app (e.g. Whatsapp phone calls) scores better, despite the occasional glitches.
- peterhil 6y agoI find the glitches in Lyra codec very unpleasant compared to the other codecs. They remind of the visual glitches of AI generated images, and I really hope Lyra does not become a common codec!
- drudu 6y agoTo clarify, do you find Lyra @ 3kbs to be more unpleasant overall than Speex at @ 3kbps?
- ajb 6y agoFunny thing: voice codecs below 2.4kbps are export controlled by ITAR, right along side military technology and nuclear devices. David Rowe, who made the codec2 open source voice codec, had to get confirmation from his govt that they would not enforce it on his project: https://www.mail-archive.com/freetel-codec2@lists.sourceforge.net/msg00183.html https://www.mail-archive.com/freetel-codec2@lists.sourceforg... I'm not sure exactly why - someone mentioned that it might be because deep submarines have very little bandwidth and need to use them, but I don't have a reference. Nevertheless, wouldn't be surprised if google didn't want the hassle.
- Saris 6y agoHmm, I have to say it's impressive to do this at 3 kbps, but even the Lyra sample I found very hard to understand, and misheard most of the words. The original was the only one I could understand fully.