19 ms·

MLow: Meta's low bitrate audio codec

572 points by mikece 2y ago

PaulHoule 2y ago
Sometimes it sounds great but there are moments I think I'm listening to a harp and not somebody's voice.
- plus 2y ago
  It's not exactly reasonable to expect super high fidelity audio at the bitrate constraints they're targeting here, and it certainly sounds a lot better than the Opus examples they're comparing against.
  - cobbal 2y ago
    The more complicated the codec, the more fascinating the failure modes. I love watching a digital TV with a bad signal, because the motion tracking in the codec causes people to wear previous, glitched frames as a skin while they move.
    - ugjka 2y ago
      Look up datamoshing on youtube
    - cnity 2y ago
      Good observation, and probably part of what makes "glitchy" AI generated video so captivating to watch.
  - 77pt77 2y ago
    Are they comparing against opus using nolace? Because that makes all the difference!
Tostino 2y ago
That is a marked improvement compared to the other examples provided. Nice to see it also has less compute resources required for that higher quality output.
zekica 2y ago
Honest question: why do we need to optimize for <10kbps? It's really impressive what they are able to achieve at 6kbps, but LTE already supports >32kbps and there we have AMR-WB or Opus (Opus even has in-band FEC at these bitrates so packet loss is not that catastrophic). Maybe it's useful in satellite direct-to-phone use-cases?
- ThrowawayTestr 2y ago
  > why do we need to optimize for <10kbps? Because some people have really slow internet
- hokumguru 2y ago
  There exist a few billion people without LTE. Meta doesn’t only operate in the western world.
  - noprocrasted 2y ago
    Are there really many situations where a 10kbps connection would actually be stable enough to be usable? Usually when you get these kinds of speeds it means the underlying connection is well and truly compromised, and any kind of real-time audio would fail anyway because you're drowning in a sea of packet loss and retransmissions. Even in cases where you do get a stable 10kbps connection from upstream, how are you going to manage getting any usable traffic through it when everything nowadays wastes bandwidth and competes with you (just look at any iOS device's background network activity - and that's before running any apps which usually embed dozens of malicious SDKs all competing for bandwidth)?
    - bogwog 2y ago
      I don't know what you consider "stable enough", but the 30% packet loss demo in the article is pretty impressive.
    - zeroxfe 2y ago
      > Are there really many situations where a 10kbps connection would actually be stable enough to be usable? Yes there are. We ran on stable low bandwidth connections for a very long time before we had stable high bandwidth connections. A large part of the underdeveloped world has very low bandwidth, and use 5 - 10 Kbps voice channels.
gorkish 2y ago
The lack of any reference or comparison to Codec2 immediately leads me to question the real value and motivation of this work. The world doesn't need another IP-encumbered audio codec in this space.
- muizelaar 2y ago
  They also don't compare with Lyra (https://github.com/google/lyra https://github.com/google/lyra)
  - cvg 2y ago
    Nice. Google's soundstream already has some great quality. Some examples at 6kbps here: https://google-research.github.io/seanet/soundstream/examples/ https://google-research.github.io/seanet/soundstream/example...
  - gorkish 2y ago
    Or speex narrowband or others. I think the tendency to pick Opus is just because it has a newer date on it -- its design goals were not necessarily to optimize for low bitrate; Opus just happened to still sound OK when the knob was turned down that far. One other point I intended to make that is not reflected in many listening/comparison tests offered by these presentations -- in the typical applications of low bitrate codecs, they absolutely must be able to gracefully degrade. We see Mlow performing at 6kbps here; how does it perform with 5% bit errors? Can it be tuned for lower bitrates like 3kpbs? A codec with a 6kbps floor that garbles into nonsense with a single bit flip would be dead-on-arrival for most real world applications. If you have to double the bitrate with FEC to make it reliable, have you really designed a low bitrate codec? The only example we heard of mlow was 30% loss on a 14kbps stream = 9.8kbps. Getting 6kbps through such a channel is a trivial exercise.
    - DragonStrength 2y ago
      My understanding was Opus was specifically developed with the idea of replacing both Speex and Vorbis. "Better quality than Speex" is literally one of their selling points, so I'd be interested to hear more details.
nickels 2y ago
Could it be used for voice over satellite, ie Emergency SOS via satellite on iPhones?
- lxgr 2y ago
  iPhones use Globalstar, which theoretically supports voice bitrates of (I believe) 9.6 kbps, although only using dedicated satphones with large, external antennas. Apple's current solution requires several seconds to transmit a location stamp of only a handful of bytes, so I think we're some either iPhone or satellite upgrades away from real-time voice communication over that. Starlink has demonstrated a direct-to-device video call already, though, so we seem to be quickly approaching that point! My strong suspicion is that Apple has bigger plans for Globalstar than just text messaging.
  - zekica 2y ago
    Starlink is in a better position as their satellites are in a low earth orbit - 30 times closer than geostationary. It correlates to 1000 times (30dB) stronger signal on both sides.
    - lxgr 2y ago
      Globalstar is LEO as well, although a bit higher (~1400 km) than Iridium (~780 km) and Starlink (below Iridium; various altitudes). In terms of SNR, they're very comparable. Newer GEO direct-to-device satellites also have huge reflectors and often much higher transmit power levels that can compensate for the greater distance somewhat. Terrestar and Thuraya have had quite small phones available since the late 2000s already, and they're both (large) GEO.
      - deleted 2y ago
        [deleted]
    - ianburrell 2y ago
      Iridium and Globalstar aren't geostationary. They are LEO not much higher than Starlink. Starlink is doing direct-to-cell. Talking to existing phones requires a large antenna. The bandwidth for each device is slow, not enough for mobile data, but better than Iridium. I think they recently showed off voice calls.
chronogram 2y ago
No mention of NoLACE make the comparison samples a bit less useful: https://opus-codec.org/demo/opus-1.5/ https://opus-codec.org/demo/opus-1.5/
- jamal-kumar 2y ago
  That does sound very nice
- sitkack 2y ago
  This is really cool and I very very very much appreciate that xiph puts so much work into standardization. https://datatracker.ietf.org/wg/mlcodec/documents/ https://datatracker.ietf.org/wg/mlcodec/documents/ It would be nice if Meta donated this to the world so we have less anchors for patent trolls and can transition the future we deserve.
77pt77 2y ago
Where is the source code?
lxgr 2y ago
All these new, low-bitrate codecs are amazing, but ironically I suspect that they won't actually be very useful in most of the scenarios Meta is using them: To keep latency low in real-time communications, the packet rate needs to be relatively high, and at some point the overhead of UDP, IP, and lower layers starts dominating over the actual payload. As an example, consider (S)RTP (over UDP and IP): RTP adds at least 12 bytes of overhead (let's ignore the SRTP authentication tag for now); UDP adds 8 byte, and IPv4 adds 20, for a total of 40. At at typical packet rate of 50 per second (for a serialization delay of 1/50 = 20ms), that's 16 kbps of overhead alone! It might still be acceptable to reduce the packet rate to 25 per second, which would cut this in half for an overhead of 8 kbps, but the overhead would still be dominating the total transmission rate. Where codecs like this can really shine, though, is circuit-switched communication (some satphones use bitrates of around 2 kbps, which currently sound awful!), or protocol-aware VoIP systems that can employ header compression such as that used by LTE and 5G in IMS (most of the 40 bytes per frame are extremely predictable).
- deleted 2y ago
  [deleted]
- tgtweak 2y ago
  I think this is likely incorrect based on how much voice/audio distribution meta does today with facebook (and facebook live), instagram and whatsapp - moreso with whatsapp voice message and calling given it's considerable market share in countries with intermittent and low-reliability network connectivity. The fact it is more packet-loss robust and jitter-robust means that you can rely on protocols that have less error correction, segmenting and receive-reply overhead as well. I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality". Looking at wireshark review of whatsapp on an active call there was around 380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps. quick edit to clarify why this is: you can see starting ptime (audio size per packet) set to 20ms here, but maxptime set to 150ms, which the clients can/will use opportunistically to reduce the number of packets being sent taking into consideration the latency between parties and bandwidth available. (image): https://www.twilio.com/content/dam/twilio-com/global/en/blog/legacy/2022/srtp-deep-dive/ac6tJnTHzRYFPOSgUlDXoG0991VSDV3XLoSZz4F2W5J-vpmpPSxaR9fNkNRNPDpNLobLFzX11i4WJO.png https://www.twilio.com/content/dam/twilio-com/global/en/blog...
dgmdoug 2y ago
They also don't do a comparison with Pied Piper.
- mig39 2y ago
  It might have a Weissman score in the fives, but I haven't seen a real-world implementation. Does it use middle-out compression?
  - byteknight 2y ago
    MIDDLE OUT!
barbazoo 2y ago
That's great, now they can reach even more developing countries and do damage the way they did for example in Myanmar [1]. [1] https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...
thrtythreeforty 2y ago
Are they releasing this or is this just engineering braggadocio? I can't find any other references to MLow other than this blog post. Facebook/Meta AI Research does cool stuff, and releases a substantial portion of it (I dislike Facebook but I can admit they are highly innovative in the AI space).
- sllabres 2y ago
  If you think about 'implementing then algorithm in a product' it seems so: (From the article) "We are really excited about what we have accomplished in just the last two years—from developing a new codec to successfully shipping it to billions of users around the globe"
animanoir 2y ago
Nice technology, tho Opus adds that warm sound I love...
ConanRus 2y ago
[dead]
aidenn0 2y ago
Only slightly OT: ELI5: Why is a typical phone call today less intelligible than a 8kHz 8-bit μ-law with ADPCM from the '90s did? [edit] s/sound worse/less intelligible/
- skygazer 2y ago
  Does decrease in intelligibility correlate with the instance count of concert seats in front of the loud speakers back in the oughts?
- toast0 2y ago
  Depends on your call; u-law has poor frequency response and reasonable dynamic range. Not great for music, but ok enough for voice, and it's very consistent. 90s calls were almost all circuit switched in the last mile, and multiplexed per sample on digital lines (T1 and up). This means very low latency and zero jitter; there would be a measurable but actually imperceptible delay versus an end to end analog circuit switched call; but digital sampling near the ends means there would be a lot less noise. Circuit switching also means you'd never get dropped samples --- the connection is made or its not, although sometimes only one-way. Modern calls are typically using 20 ms samples, over packet switched networks, so you're adding sampling delay, and jitter and jitter buffers. The codecs themselves have encode/decode delay, because they're doing more than a ADC/DAC with a logarithm. Most of the codecs are using significantly fewer bits for the samples than u-law, and that's not for free either. HD Voice (g.722.2 AMR-Wide Band) has a much larger frequency pass band, and sounds much better than GSM or OPUS or most of these other low bandwidth codecs. There's still delay though; even if people will tell you 20-100ms delay is imperceptible, give someone an a/b call with 0 and 20 ms delay and they'll tell you the 0 ms delay call is better.
  - rylittle 2y ago
    could you explain a little more in a more ELI5 please?
    - epanchin 2y ago
      Old way: two cups and string. New way: chinese whispers.
iamnotsure 2y ago
Please stop lossy compression.
- cheema33 2y ago
  Lossy compression has its practical uses. Under ideal circumstances nobody is going to stop you from using FLAC.
- GrantMoyer 2y ago
  Have you ever looked at the size of losslessly compressed video? It's huge. Lossy compression is the only practical way to store and stream video, since it's typically less than 1% of the size of uncompressed video. Lossless compression typically only gets down to about 50% of the size. It's amazing how much information you can throw out from a video, and barely be able to tell the difference.
- aembleton 2y ago
  Surely all digital audio is lossy to a certain extent as you have a finite number of quantization levels and a finite sampling frequency.
  - GrantMoyer 2y ago
    It depends on what you consider lossy. For time discretization, the Nyquist–Shannon sampling theorem[1] says a band-limited, continuous-time signal can be perfectly reconstructed from a discrete-time signal with a sufficiently high (but finite) sampling frequency. Human hearing is naturally band limited to about the range 20Hz to 20kHz, and audio recordings typically use sufficient sampling frequency to losslessly recontruct thos bandwidth. For quantization, any recorded analog signal is a true signal plus some level of measurement noise. Quantization of a signals can also be thought of as adding some noise to a continuous amplitude signal[2]. If the "quantization noise" is much smaller than the measurement noise of the signal, then the discretization is effectively lossless. Typical audio formats have a maximum quantization error far, far smaller than typical audio recording hardware's measurment error (and the human ear's). So typical quantized, discrete-time audio formats can be considered lossless representations of the anolog sound signals humans can hear (assuming proper capture and processing). On the other hand, no quantized or discretized signal can losslessly represent a non-band-limited, zero-noise audio signal. [1]: https://en.wikipedia.org/w/index.php?title=Nyquist%E2%80%93Shannon_sampling_theorem&diffonly=true https://en.wikipedia.org/w/index.php?title=Nyquist%E2%80%93S... [2]: https://en.wikipedia.org/wiki/Quantization_(signal_processing)#Noise_and_error_characteristics https://en.wikipedia.org/wiki/Quantization_(signal_processin...
mcoliver 2y ago
Maybe SiriusXM can pick this up. The audio quality is generally awful but especially on news/talk channels like Bloomberg and CNBC. There is no depth or richness to the voices.
- tgtweak 2y ago
  It actually comes down to the SiriusXM receiver that is being used - I've witnessed the built-in sirius/xm on on the latest GM platform (a $100,000+ Cadillac) sounding like AM radio to immediately sitting in a better-than-apple-streaming quality rendition of the exact same channel on an older lexus a few minutes apart... The mobile xm receivers (like ipods) that they used to sell also had very good quality and I never noticed any quality shortcomings even with good headphones. I think the "high" quality stream is 256kbps/16k which is fairly high compared to most streaming services that come in around 128/160.
  - sitkack 2y ago
    I am archiving some music at 40kbps using Opus and the quality is pretty amazing. I think once things get over 20+kbps all the codecs start sounding pretty good (relative to the these low bitrates). I still prefer flac if possible.
    - theoperagoer 2y ago
      Opus is fantastic!
  - wildzzz 2y ago
    My old Sirius portable receiver sounded like garbage despite the marketing material saying "Crystal Clear". My 2006 Infiniti Sirius receiver didn't sound any better despite being a massive space heater in the trunk. The later cars I've used it in sound good, at least good enough to sound as clear as FM radio or even HD radio. I think some of the channels are still in lower bitrates like the news channels for example, they've always sounded bad. There's something I've read about SiriusXM using terrestrial transmitters which may improve the signal whereas the satellite link may be of lower bandwidth.
theoperagoer 2y ago
Was hoping this would have a GitHub link ...
amelius 2y ago
Can't we have an audio codec that first sends a model of the particular voice, and then starts to send bits corresponding to the actual speech?
- roywiggins 2y ago
  You need a bunch of bandwidth upfront for that, which you might not have, and enough compute available at the other end to reconstruct it, which you really might not have.
  - amelius 2y ago
    Regarding your first point, how about reserving a small percentage of the bandwidth for a model that improves incrementally?
    - wildzzz 2y ago
      You're adding more complexity to both the transmitter and receiver. I'd be pretty pissed if I had to endure unintelligible speech for a few minutes until the model was downloaded enough to be able to hear my friend. I'd also be a little pissed if I had to store massive models for everyone in my call log. Also both devices need to be able to run this model. If you are regularly talking over a shit signal, you're probably not going to be able to afford the latest flagship phone that has the hardware necessary to run it (which is exactly what the article touches on). The ideal codec takes up almost no bandwidth, sounds like you're sitting next to the caller, and runs on the average budget smartphone/PC. The issue is that you aren't going to be able to get one of these things so you choose a codec that best balances complexity, quality, and bandwidth given the situation. Link quality improves? Improve your voice quality by increasing the codec bitrate or switching to another less complex one to save battery. If both devices are capable of running a ML codec, then use that to improve quality and fit within the given bandwidth.
- neckro23 2y ago
  This is actually an old idea, minus the AI angle (1930s). It’s what voders and vocoders were originally designed for, before Kraftwerk et al. found out you can use them to make cool robot voices.
hubraumhugo 2y ago
Is it just my perception or has Meta become cool again by sharing a ton of research and open source (or open weights) work? Facebook's reputation was at the bottom, but now it seems like they made up for it.
- mrguyorama 2y ago
  How the hell does releasing one audio codec undo years and years of privacy nightmare, being a willing bystander in an actual genocide, experimenting with the emotions of depressed kids, and collusion to depress wages?
  - pt_PT_guy 2y ago
    they also did release LLM models, and zstd, and mold, and and and... a lot of stuff
    - visarga 2y ago
      React and Pytorch compare that to Angular and TensorFlow, such a difference in culture
      - hot_gril 2y ago
        Easy vs tons-o-boilerplate.
    - cheema33 2y ago
      Don't forget React. The most popular frontend stack at the moment. Been that way for some time. And GraphQL, Relay, Stylex...
      - XlA5vEKsMISoIln 2y ago
        >React Ah yes, the <body id="app"></body> websites.
victorp13 2y ago
Does anyone happen to know if ChatGPT's voice feature uses audio compression similar to Opus? Especially the "heavy 30 percent receiver-side packet loss" example sounds a LOT like the experience I have sometimes.
dsign 2y ago
Can I use this to make music? A little bit on a tangent, a technique called Linear Predictive Coding, which was developed by telecoms in the sixties and seventies, has a calculated bandwidth of 2.5 kbit/s. The sound quality is not any good, and telephone companies of the time didn't use it for calls, but the paper I read describing the technique says the decoded speech is "understandable". LPC found its way into musical production, in a set of instruments called "vocoders" used to distort a singer's voce. There are, for example, variations of it in something called "Orange Vocoder IV". So, now I'm wondering, can MLow be used to purposefully introduce interesting distortions in speech? Or even change a singer's voice color?
- WalterSear 2y ago
  Just use Digitalis :) https://www.youtube.com/watch?v=bA23ysR2hAo https://www.youtube.com/watch?v=bA23ysR2hAo
dbcurtis 2y ago
What is the license? I searched but could not find anything.
- varenc 2y ago
  It appears to be entirely closed source at the moment. This is just the announcement of development of in-house tech. Nothing public yet that could even be licensed.
annoyingnoob 2y ago
I wonder how it sounds compared to G.729. I worked for a company 20 years ago that had a modified G.729 codec that could go below 8kbps but sounded decent. We used this for VoIP over dial-up Internet, talk about low bandwidth. Turns out some of the more interesting bits were in the jitter buffer and ways to manage the buffer. Scrappy connections deliver packets when they can and there is an art to managing the difference between the network experience and the user experience. For communications, you really need to manage the user experience.
eelioss 2y ago
I am curious about encoding times comparing with standard codecs using ffmpeg
- lucb1e 2y ago
  They claim it's 10% lower than Opus, specifically for the decoder iirc but since they speak of the 10-year-old hardware used to make million of WhatsApp calls daily, the encoder can't be computationally complex either But, yeah, some actual data (if they're not willing to provide running code) would have been a welcome addition to this PR overview
therealmarv 2y ago
Somebody knows if this is better compared to whatever Google Meet is using? With choppy near unusable slow Internet Google Meet still fulfils its purpose on Audio Calls where all other competitors fail (tested e.g. while being in Philippines on a remote island with very bad internet). However Google Meet's tech is not published anywhere afaik.
- lucb1e 2y ago
  Can hardly try that out if this PR piece does not contain any code. We can judge it as well as you can from the couple examples they showed off
mckirk 2y ago
Maybe it's just me (or maybe I've invested too much money into headphones), but I actually liked the Opus sound better at 6 kbps. The MLow samples had these... harsh and unnatural artifacts, whereas the Opus sound (though sounding like it came from a tin-can-and-string telephone and lacked all top-end) at least was 'smooth'. But I'm pretty sure that's because they are demonstrating here the very edge of what their codec can do, at higher bitrates the choice would probably be a lot clearer.
Manabu92 2y ago
[flagged]
FrostKiwi 2y ago
Not sure of the timing, is this what Meta researched when fighting audio latency for Virtual Meetings during the development of the Meta Quest 2?
andrewstuart 2y ago
This is amazing. So you could have about 9 voice streams over a 56K modem. Incredible. Great for stuffing audio recordings into little devices like greeting card audio players and microcontrollers.
justinclift 2y ago
Doesn't seem to have any mention of what they're going to do with this patent wise. The one of the biggest points for Opus was it being patent free (and "good enough" audio wise), so anyone and everyone was able to use it legally.
arendtio 2y ago
Does someone know if we can expect to see similar improvements for video streaming in the near future?
snvzz 2y ago
Looks (and sounds) cool. What is the license?
jacobgorm 2y ago
Will the source code be available?
megamix 2y ago
Meta..
sidcool 2y ago
Meta engineering really rocks. All other controversies aside.
yobid20 2y ago
Is it free and open source? Any plans on making it available for native webrtc in the browser?
deleted 2y ago
[deleted]
Rakshith 2y ago
they should all jump straight to vvc with lcevc to enhance it
vagab0nd 2y ago
How low can we go? Ultimately we just need to send "who's speaking" at the beginning and the content as text. So about 20 bps. Plenty of room at the bottom.
babi76 2y ago
[flagged]

572 points by mikece 2y ago

PaulHoule 2y ago

Sometimes it sounds great but there are moments I think I'm listening to a harp and not somebody's voice.

plus 2y ago
It's not exactly reasonable to expect super high fidelity audio at the bitrate constraints they're targeting here, and it certainly sounds a lot better than the Opus examples they're comparing against.
- cobbal 2y ago
  The more complicated the codec, the more fascinating the failure modes. I love watching a digital TV with a bad signal, because the motion tracking in the codec causes people to wear previous, glitched frames as a skin while they move.
  - ugjka 2y ago
    Look up datamoshing on youtube
  - cnity 2y ago
    Good observation, and probably part of what makes "glitchy" AI generated video so captivating to watch.
- 77pt77 2y ago
  Are they comparing against opus using nolace? Because that makes all the difference!

Tostino 2y ago

That is a marked improvement compared to the other examples provided. Nice to see it also has less compute resources required for that higher quality output.

zekica 2y ago

Honest question: why do we need to optimize for <10kbps? It's really impressive what they are able to achieve at 6kbps, but LTE already supports >32kbps and there we have AMR-WB or Opus (Opus even has in-band FEC at these bitrates so packet loss is not that catastrophic). Maybe it's useful in satellite direct-to-phone use-cases?

ThrowawayTestr 2y ago
> why do we need to optimize for <10kbps? Because some people have really slow internet
hokumguru 2y ago
There exist a few billion people without LTE. Meta doesn’t only operate in the western world.
- noprocrasted 2y ago
  Are there really many situations where a 10kbps connection would actually be stable enough to be usable? Usually when you get these kinds of speeds it means the underlying connection is well and truly compromised, and any kind of real-time audio would fail anyway because you're drowning in a sea of packet loss and retransmissions. Even in cases where you do get a stable 10kbps connection from upstream, how are you going to manage getting any usable traffic through it when everything nowadays wastes bandwidth and competes with you (just look at any iOS device's background network activity - and that's before running any apps which usually embed dozens of malicious SDKs all competing for bandwidth)?
  - bogwog 2y ago
    I don't know what you consider "stable enough", but the 30% packet loss demo in the article is pretty impressive.
  - zeroxfe 2y ago
    > Are there really many situations where a 10kbps connection would actually be stable enough to be usable? Yes there are. We ran on stable low bandwidth connections for a very long time before we had stable high bandwidth connections. A large part of the underdeveloped world has very low bandwidth, and use 5 - 10 Kbps voice channels.

gorkish 2y ago

The lack of any reference or comparison to Codec2 immediately leads me to question the real value and motivation of this work. The world doesn't need another IP-encumbered audio codec in this space.

muizelaar 2y ago
They also don't compare with Lyra (https://github.com/google/lyra https://github.com/google/lyra)
- cvg 2y ago
  Nice. Google's soundstream already has some great quality. Some examples at 6kbps here: https://google-research.github.io/seanet/soundstream/examples/ https://google-research.github.io/seanet/soundstream/example...
- gorkish 2y ago
  Or speex narrowband or others. I think the tendency to pick Opus is just because it has a newer date on it -- its design goals were not necessarily to optimize for low bitrate; Opus just happened to still sound OK when the knob was turned down that far. One other point I intended to make that is not reflected in many listening/comparison tests offered by these presentations -- in the typical applications of low bitrate codecs, they absolutely must be able to gracefully degrade. We see Mlow performing at 6kbps here; how does it perform with 5% bit errors? Can it be tuned for lower bitrates like 3kpbs? A codec with a 6kbps floor that garbles into nonsense with a single bit flip would be dead-on-arrival for most real world applications. If you have to double the bitrate with FEC to make it reliable, have you really designed a low bitrate codec? The only example we heard of mlow was 30% loss on a 14kbps stream = 9.8kbps. Getting 6kbps through such a channel is a trivial exercise.
  - DragonStrength 2y ago
    My understanding was Opus was specifically developed with the idea of replacing both Speex and Vorbis. "Better quality than Speex" is literally one of their selling points, so I'd be interested to hear more details.

nickels 2y ago

Could it be used for voice over satellite, ie Emergency SOS via satellite on iPhones?

lxgr 2y ago
iPhones use Globalstar, which theoretically supports voice bitrates of (I believe) 9.6 kbps, although only using dedicated satphones with large, external antennas. Apple's current solution requires several seconds to transmit a location stamp of only a handful of bytes, so I think we're some either iPhone or satellite upgrades away from real-time voice communication over that. Starlink has demonstrated a direct-to-device video call already, though, so we seem to be quickly approaching that point! My strong suspicion is that Apple has bigger plans for Globalstar than just text messaging.
- zekica 2y ago
  Starlink is in a better position as their satellites are in a low earth orbit - 30 times closer than geostationary. It correlates to 1000 times (30dB) stronger signal on both sides.
  - lxgr 2y ago
    Globalstar is LEO as well, although a bit higher (~1400 km) than Iridium (~780 km) and Starlink (below Iridium; various altitudes). In terms of SNR, they're very comparable. Newer GEO direct-to-device satellites also have huge reflectors and often much higher transmit power levels that can compensate for the greater distance somewhat. Terrestar and Thuraya have had quite small phones available since the late 2000s already, and they're both (large) GEO.
    - deleted 2y ago
      [deleted]
  - ianburrell 2y ago
    Iridium and Globalstar aren't geostationary. They are LEO not much higher than Starlink. Starlink is doing direct-to-cell. Talking to existing phones requires a large antenna. The bandwidth for each device is slow, not enough for mobile data, but better than Iridium. I think they recently showed off voice calls.

chronogram 2y ago

No mention of NoLACE make the comparison samples a bit less useful: https://opus-codec.org/demo/opus-1.5/ https://opus-codec.org/demo/opus-1.5/

jamal-kumar 2y ago
That does sound very nice
sitkack 2y ago
This is really cool and I very very very much appreciate that xiph puts so much work into standardization. https://datatracker.ietf.org/wg/mlcodec/documents/ https://datatracker.ietf.org/wg/mlcodec/documents/ It would be nice if Meta donated this to the world so we have less anchors for patent trolls and can transition the future we deserve.

77pt77 2y ago

Where is the source code?

lxgr 2y ago

All these new, low-bitrate codecs are amazing, but ironically I suspect that they won't actually be very useful in most of the scenarios Meta is using them: To keep latency low in real-time communications, the packet rate needs to be relatively high, and at some point the overhead of UDP, IP, and lower layers starts dominating over the actual payload. As an example, consider (S)RTP (over UDP and IP): RTP adds at least 12 bytes of overhead (let's ignore the SRTP authentication tag for now); UDP adds 8 byte, and IPv4 adds 20, for a total of 40. At at typical packet rate of 50 per second (for a serialization delay of 1/50 = 20ms), that's 16 kbps of overhead alone! It might still be acceptable to reduce the packet rate to 25 per second, which would cut this in half for an overhead of 8 kbps, but the overhead would still be dominating the total transmission rate. Where codecs like this can really shine, though, is circuit-switched communication (some satphones use bitrates of around 2 kbps, which currently sound awful!), or protocol-aware VoIP systems that can employ header compression such as that used by LTE and 5G in IMS (most of the 40 bytes per frame are extremely predictable).

deleted 2y ago
[deleted]
tgtweak 2y ago
I think this is likely incorrect based on how much voice/audio distribution meta does today with facebook (and facebook live), instagram and whatsapp - moreso with whatsapp voice message and calling given it's considerable market share in countries with intermittent and low-reliability network connectivity. The fact it is more packet-loss robust and jitter-robust means that you can rely on protocols that have less error correction, segmenting and receive-reply overhead as well. I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality". Looking at wireshark review of whatsapp on an active call there was around 380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps. quick edit to clarify why this is: you can see starting ptime (audio size per packet) set to 20ms here, but maxptime set to 150ms, which the clients can/will use opportunistically to reduce the number of packets being sent taking into consideration the latency between parties and bandwidth available. (image): https://www.twilio.com/content/dam/twilio-com/global/en/blog/legacy/2022/srtp-deep-dive/ac6tJnTHzRYFPOSgUlDXoG0991VSDV3XLoSZz4F2W5J-vpmpPSxaR9fNkNRNPDpNLobLFzX11i4WJO.png https://www.twilio.com/content/dam/twilio-com/global/en/blog...

dgmdoug 2y ago

They also don't do a comparison with Pied Piper.

mig39 2y ago
It might have a Weissman score in the fives, but I haven't seen a real-world implementation. Does it use middle-out compression?
- byteknight 2y ago
  MIDDLE OUT!

barbazoo 2y ago

That's great, now they can reach even more developing countries and do damage the way they did for example in Myanmar [1]. [1] https://www.amnesty.org/en/latest/news/2022/09/myanmar-facebooks-systems-promoted-violence-against-rohingya-meta-owes-reparations-new-report/ https://www.amnesty.org/en/latest/news/2022/09/myanmar-faceb...

thrtythreeforty 2y ago

Are they releasing this or is this just engineering braggadocio? I can't find any other references to MLow other than this blog post. Facebook/Meta AI Research does cool stuff, and releases a substantial portion of it (I dislike Facebook but I can admit they are highly innovative in the AI space).

sllabres 2y ago
If you think about 'implementing then algorithm in a product' it seems so: (From the article) "We are really excited about what we have accomplished in just the last two years—from developing a new codec to successfully shipping it to billions of users around the globe"

animanoir 2y ago

Nice technology, tho Opus adds that warm sound I love...

ConanRus 2y ago

[dead]

aidenn0 2y ago

Only slightly OT: ELI5: Why is a typical phone call today less intelligible than a 8kHz 8-bit μ-law with ADPCM from the '90s did? [edit] s/sound worse/less intelligible/

skygazer 2y ago
Does decrease in intelligibility correlate with the instance count of concert seats in front of the loud speakers back in the oughts?
toast0 2y ago
Depends on your call; u-law has poor frequency response and reasonable dynamic range. Not great for music, but ok enough for voice, and it's very consistent. 90s calls were almost all circuit switched in the last mile, and multiplexed per sample on digital lines (T1 and up). This means very low latency and zero jitter; there would be a measurable but actually imperceptible delay versus an end to end analog circuit switched call; but digital sampling near the ends means there would be a lot less noise. Circuit switching also means you'd never get dropped samples --- the connection is made or its not, although sometimes only one-way. Modern calls are typically using 20 ms samples, over packet switched networks, so you're adding sampling delay, and jitter and jitter buffers. The codecs themselves have encode/decode delay, because they're doing more than a ADC/DAC with a logarithm. Most of the codecs are using significantly fewer bits for the samples than u-law, and that's not for free either. HD Voice (g.722.2 AMR-Wide Band) has a much larger frequency pass band, and sounds much better than GSM or OPUS or most of these other low bandwidth codecs. There's still delay though; even if people will tell you 20-100ms delay is imperceptible, give someone an a/b call with 0 and 20 ms delay and they'll tell you the 0 ms delay call is better.
- rylittle 2y ago
  could you explain a little more in a more ELI5 please?
  - epanchin 2y ago
    Old way: two cups and string. New way: chinese whispers.

iamnotsure 2y ago

Please stop lossy compression.

cheema33 2y ago
Lossy compression has its practical uses. Under ideal circumstances nobody is going to stop you from using FLAC.
GrantMoyer 2y ago
Have you ever looked at the size of losslessly compressed video? It's huge. Lossy compression is the only practical way to store and stream video, since it's typically less than 1% of the size of uncompressed video. Lossless compression typically only gets down to about 50% of the size. It's amazing how much information you can throw out from a video, and barely be able to tell the difference.
aembleton 2y ago
Surely all digital audio is lossy to a certain extent as you have a finite number of quantization levels and a finite sampling frequency.
- GrantMoyer 2y ago
  It depends on what you consider lossy. For time discretization, the Nyquist–Shannon sampling theorem[1] says a band-limited, continuous-time signal can be perfectly reconstructed from a discrete-time signal with a sufficiently high (but finite) sampling frequency. Human hearing is naturally band limited to about the range 20Hz to 20kHz, and audio recordings typically use sufficient sampling frequency to losslessly recontruct thos bandwidth. For quantization, any recorded analog signal is a true signal plus some level of measurement noise. Quantization of a signals can also be thought of as adding some noise to a continuous amplitude signal[2]. If the "quantization noise" is much smaller than the measurement noise of the signal, then the discretization is effectively lossless. Typical audio formats have a maximum quantization error far, far smaller than typical audio recording hardware's measurment error (and the human ear's). So typical quantized, discrete-time audio formats can be considered lossless representations of the anolog sound signals humans can hear (assuming proper capture and processing). On the other hand, no quantized or discretized signal can losslessly represent a non-band-limited, zero-noise audio signal. [1]: https://en.wikipedia.org/w/index.php?title=Nyquist%E2%80%93Shannon_sampling_theorem&diffonly=true https://en.wikipedia.org/w/index.php?title=Nyquist%E2%80%93S... [2]: https://en.wikipedia.org/wiki/Quantization_(signal_processing)#Noise_and_error_characteristics https://en.wikipedia.org/wiki/Quantization_(signal_processin...

mcoliver 2y ago

Maybe SiriusXM can pick this up. The audio quality is generally awful but especially on news/talk channels like Bloomberg and CNBC. There is no depth or richness to the voices.

tgtweak 2y ago
It actually comes down to the SiriusXM receiver that is being used - I've witnessed the built-in sirius/xm on on the latest GM platform (a $100,000+ Cadillac) sounding like AM radio to immediately sitting in a better-than-apple-streaming quality rendition of the exact same channel on an older lexus a few minutes apart... The mobile xm receivers (like ipods) that they used to sell also had very good quality and I never noticed any quality shortcomings even with good headphones. I think the "high" quality stream is 256kbps/16k which is fairly high compared to most streaming services that come in around 128/160.
- sitkack 2y ago
  I am archiving some music at 40kbps using Opus and the quality is pretty amazing. I think once things get over 20+kbps all the codecs start sounding pretty good (relative to the these low bitrates). I still prefer flac if possible.
  - theoperagoer 2y ago
    Opus is fantastic!
- wildzzz 2y ago
  My old Sirius portable receiver sounded like garbage despite the marketing material saying "Crystal Clear". My 2006 Infiniti Sirius receiver didn't sound any better despite being a massive space heater in the trunk. The later cars I've used it in sound good, at least good enough to sound as clear as FM radio or even HD radio. I think some of the channels are still in lower bitrates like the news channels for example, they've always sounded bad. There's something I've read about SiriusXM using terrestrial transmitters which may improve the signal whereas the satellite link may be of lower bandwidth.

theoperagoer 2y ago

Was hoping this would have a GitHub link ...

amelius 2y ago

Can't we have an audio codec that first sends a model of the particular voice, and then starts to send bits corresponding to the actual speech?

roywiggins 2y ago
You need a bunch of bandwidth upfront for that, which you might not have, and enough compute available at the other end to reconstruct it, which you really might not have.
- amelius 2y ago
  Regarding your first point, how about reserving a small percentage of the bandwidth for a model that improves incrementally?
  - wildzzz 2y ago
    You're adding more complexity to both the transmitter and receiver. I'd be pretty pissed if I had to endure unintelligible speech for a few minutes until the model was downloaded enough to be able to hear my friend. I'd also be a little pissed if I had to store massive models for everyone in my call log. Also both devices need to be able to run this model. If you are regularly talking over a shit signal, you're probably not going to be able to afford the latest flagship phone that has the hardware necessary to run it (which is exactly what the article touches on). The ideal codec takes up almost no bandwidth, sounds like you're sitting next to the caller, and runs on the average budget smartphone/PC. The issue is that you aren't going to be able to get one of these things so you choose a codec that best balances complexity, quality, and bandwidth given the situation. Link quality improves? Improve your voice quality by increasing the codec bitrate or switching to another less complex one to save battery. If both devices are capable of running a ML codec, then use that to improve quality and fit within the given bandwidth.
neckro23 2y ago
This is actually an old idea, minus the AI angle (1930s). It’s what voders and vocoders were originally designed for, before Kraftwerk et al. found out you can use them to make cool robot voices.

hubraumhugo 2y ago

Is it just my perception or has Meta become cool again by sharing a ton of research and open source (or open weights) work? Facebook's reputation was at the bottom, but now it seems like they made up for it.

mrguyorama 2y ago
How the hell does releasing one audio codec undo years and years of privacy nightmare, being a willing bystander in an actual genocide, experimenting with the emotions of depressed kids, and collusion to depress wages?
- pt_PT_guy 2y ago
  they also did release LLM models, and zstd, and mold, and and and... a lot of stuff
  - visarga 2y ago
    React and Pytorch compare that to Angular and TensorFlow, such a difference in culture
    - hot_gril 2y ago
      Easy vs tons-o-boilerplate.
  - cheema33 2y ago
    Don't forget React. The most popular frontend stack at the moment. Been that way for some time. And GraphQL, Relay, Stylex...
    - XlA5vEKsMISoIln 2y ago
      >React Ah yes, the <body id="app"></body> websites.

victorp13 2y ago

Does anyone happen to know if ChatGPT's voice feature uses audio compression similar to Opus? Especially the "heavy 30 percent receiver-side packet loss" example sounds a LOT like the experience I have sometimes.

dsign 2y ago

Can I use this to make music? A little bit on a tangent, a technique called Linear Predictive Coding, which was developed by telecoms in the sixties and seventies, has a calculated bandwidth of 2.5 kbit/s. The sound quality is not any good, and telephone companies of the time didn't use it for calls, but the paper I read describing the technique says the decoded speech is "understandable". LPC found its way into musical production, in a set of instruments called "vocoders" used to distort a singer's voce. There are, for example, variations of it in something called "Orange Vocoder IV". So, now I'm wondering, can MLow be used to purposefully introduce interesting distortions in speech? Or even change a singer's voice color?

WalterSear 2y ago
Just use Digitalis :) https://www.youtube.com/watch?v=bA23ysR2hAo https://www.youtube.com/watch?v=bA23ysR2hAo

dbcurtis 2y ago

What is the license? I searched but could not find anything.

varenc 2y ago
It appears to be entirely closed source at the moment. This is just the announcement of development of in-house tech. Nothing public yet that could even be licensed.

annoyingnoob 2y ago

I wonder how it sounds compared to G.729. I worked for a company 20 years ago that had a modified G.729 codec that could go below 8kbps but sounded decent. We used this for VoIP over dial-up Internet, talk about low bandwidth. Turns out some of the more interesting bits were in the jitter buffer and ways to manage the buffer. Scrappy connections deliver packets when they can and there is an art to managing the difference between the network experience and the user experience. For communications, you really need to manage the user experience.

eelioss 2y ago

I am curious about encoding times comparing with standard codecs using ffmpeg

lucb1e 2y ago
They claim it's 10% lower than Opus, specifically for the decoder iirc but since they speak of the 10-year-old hardware used to make million of WhatsApp calls daily, the encoder can't be computationally complex either But, yeah, some actual data (if they're not willing to provide running code) would have been a welcome addition to this PR overview

therealmarv 2y ago

Somebody knows if this is better compared to whatever Google Meet is using? With choppy near unusable slow Internet Google Meet still fulfils its purpose on Audio Calls where all other competitors fail (tested e.g. while being in Philippines on a remote island with very bad internet). However Google Meet's tech is not published anywhere afaik.

lucb1e 2y ago
Can hardly try that out if this PR piece does not contain any code. We can judge it as well as you can from the couple examples they showed off

mckirk 2y ago

Maybe it's just me (or maybe I've invested too much money into headphones), but I actually liked the Opus sound better at 6 kbps. The MLow samples had these... harsh and unnatural artifacts, whereas the Opus sound (though sounding like it came from a tin-can-and-string telephone and lacked all top-end) at least was 'smooth'. But I'm pretty sure that's because they are demonstrating here the very edge of what their codec can do, at higher bitrates the choice would probably be a lot clearer.

Manabu92 2y ago

[flagged]

FrostKiwi 2y ago

Not sure of the timing, is this what Meta researched when fighting audio latency for Virtual Meetings during the development of the Meta Quest 2?

andrewstuart 2y ago

This is amazing. So you could have about 9 voice streams over a 56K modem. Incredible. Great for stuffing audio recordings into little devices like greeting card audio players and microcontrollers.

justinclift 2y ago

Doesn't seem to have any mention of what they're going to do with this patent wise. The one of the biggest points for Opus was it being patent free (and "good enough" audio wise), so anyone and everyone was able to use it legally.

arendtio 2y ago

Does someone know if we can expect to see similar improvements for video streaming in the near future?

snvzz 2y ago

Looks (and sounds) cool. What is the license?

jacobgorm 2y ago

Will the source code be available?

megamix 2y ago

Meta..

sidcool 2y ago

Meta engineering really rocks. All other controversies aside.

yobid20 2y ago

Is it free and open source? Any plans on making it available for native webrtc in the browser?

deleted 2y ago

[deleted]

Rakshith 2y ago

they should all jump straight to vvc with lcevc to enhance it

vagab0nd 2y ago

How low can we go? Ultimately we just need to send "who's speaking" at the beginning and the content as text. So about 20 bps. Plenty of room at the bottom.

babi76 2y ago

[flagged]