5 ms·
All these new, low-bitrate codecs are amazing, but ironically I suspect that they won't actually be very useful in most of the scenarios Meta is using them: To
by lxgr 2y ago
All these new, low-bitrate codecs are amazing, but ironically I suspect that they won't actually be very useful in most of the scenarios Meta is using them:
To keep latency low in real-time communications, the packet rate needs to be relatively high, and at some point the overhead of UDP, IP, and lower layers starts dominating over the actual payload.
As an example, consider (S)RTP (over UDP and IP): RTP adds at least 12 bytes of overhead (let's ignore the SRTP authentication tag for now); UDP adds 8 byte, and IPv4 adds 20, for a total of 40. At at typical packet rate of 50 per second (for a serialization delay of 1/50 = 20ms), that's 16 kbps of overhead alone!
It might still be acceptable to reduce the packet rate to 25 per second, which would cut this in half for an overhead of 8 kbps, but the overhead would still be dominating the total transmission rate.
Where codecs like this can really shine, though, is circuit-switched communication (some satphones use bitrates of around 2 kbps, which currently sound awful!), or protocol-aware VoIP systems that can employ header compression such as that used by LTE and 5G in IMS (most of the 40 bytes per frame are extremely predictable).
- deleted 2y ago[deleted]
- tgtweak 2y agoI think this is likely incorrect based on how much voice/audio distribution meta does today with facebook (and facebook live), instagram and whatsapp - moreso with whatsapp voice message and calling given it's considerable market share in countries with intermittent and low-reliability network connectivity. The fact it is more packet-loss robust and jitter-robust means that you can rely on protocols that have less error correction, segmenting and receive-reply overhead as well. I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality". Looking at wireshark review of whatsapp on an active call there was around 380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps. quick edit to clarify why this is: you can see starting ptime (audio size per packet) set to 20ms here, but maxptime set to 150ms, which the clients can/will use opportunistically to reduce the number of packets being sent taking into consideration the latency between parties and bandwidth available. (image): https://www.twilio.com/content/dam/twilio-com/global/en/blog/legacy/2022/srtp-deep-dive/ac6tJnTHzRYFPOSgUlDXoG0991VSDV3XLoSZz4F2W5J-vpmpPSxaR9fNkNRNPDpNLobLFzX11i4WJO.png https://www.twilio.com/content/dam/twilio-com/global/en/blog...
- lxgr 2y agoWhat part of that calculation is incorrect in your view? > 380 UDP packets sent from source to recipient during a 1 minute call, and a handful of TCP packets to whatsapp's servers. That would yield a transmission overhead of about 2.2kbps. That sounds like way too many packets! 380 packets per second, at 40 bytes of overhead per packet, would be almost 120 kbps. My calculation only assumes just 50, and that’s already at a quite high packet rate. > you can rely on protocols that have less error correction You could, but there's no way to get a regular smartphone IP stack running over Wi-Fi or mobile data to actually expose that capability to you. Even just getting the OS's UDP stack (to say nothing of middleboxes) to ignore UDP checksums and let you use those extra four bytes for data can be tricky. Non-IP protocols, or even just IP or UDP header compression, are completely out of reach for an OTT application. (Networks might transparently do it; I'm pretty sure they'd still charge based on the gross data rate though, and as soon as the traffic leaves their core network, it'll be back to regular RTP over UDP over IP). What they could do (and I suspect they might already be doing) is to compress RTP headers (or use something other than RTP) and/or pick even lower packet rates. > I don't think it's unreasonable to assume this could reduce their total audio-sourced bandwidth consumption by a considerable amount while maintaining/improving reliability and perceived "quality". I definitely don't agree on the latter assertion – packet loss resilience is a huge deal for perceived quality! I'm just a bit more pessimistic on the former, unless they do the other optimizations mentioned above.
- roman-holovin 2y agoI read it as in 380 packets per whole call, which was a minute long, not 380 packets per second during 1 minute.
- mikepavone 2y agoThat's about 160 ms of audio per packet. That's a lot of latency to add before you even hit the network
- ant6n 2y ago
- vel0city 2y agoAnother interesting use case for these kinds of ultra-low bitrate voice compression systems are digital radio systems. AMBE+2 and similar common voice codecs used on radio systems sound pretty miserable and don't handle dropped packets nearly as gracefully as compared to these newer codecs.
- toast0 2y agoLatency is the mind killer, but if available bandwidth is low, you save a ton of overhead by bundling 2-5 of your 20ms samples. Enough that the codec savings start to make sense, even though 100ms packets adds a ton of latency. Fancier systems can adapt codecs and samples per packet based on current conditions. The one I work on is a static codec and 60 ms of audio per packet, which isn't ideal, but allows us to run in low bandwidth much better than 20 ms per packet. Edit to add: Meta can also afford to add a bit more sampling delay, because they've got very wide distribution of forwarding servers (they can do forwarding in their content appliances embedded in many ISPs), which reduces network delay vs competing services that have limited ability to host forwarding around the globe. Peer to peer doesn't always work and isn't always lower delay than going through a nearby forwarding server.
- akira2501 2y ago> you save a ton of overhead by bundling 2-5 of your 20ms samples. You pay that price back when the packets are dropped. We use a lot of advanced codec managers for broadcast, and while they do combining like this, they also offer the ability to repeat frames within subsequent packets. So you may get a packet with frames [1,2,3] then a packet with frames [2,3,4]. The best codecs actively monitor the connection and adjust all these parameters in real time for you.
- lukevp 2y agoWhy would you need 50 packets per second vs 10? Is 100ms not acceptable but 20ms is?
- tgtweak 2y agoDefault configuration for SIP used to be 20ms, the rationale behind it was actually sourced in the fact that most SIP was done on LANs and inter-campus WAN which had generally high bitrate connectivity and low latency. The lower the packet time window the sooner the recipient could "hear" your voice, and if there were to be packet loss, there would be less of an impact if that packet were dropped - you'd only lose 20ms of audio vs 100ms. The same applies for high bitrate but high latency (3g for example) connectivity - you want to take advantage of the bandwidth to mitigate some of the network level latency that would impact the audio delay - being "wasteful" to ensure lower latency and higher packet loss tolerance. Pointedly - if you had a 75ms of one-way latency (150ms RTT) between two parties, and you used a 150ms audio segment length (ptime) you'd be getting close to the 250ms generally accepted max audio delay for smooth two-way communication. the recipient is hearing your first millisecond of audio 226ms later at best. If any packet does get lost, the recipient would lose 150ms of your message vs 20ms. Modern voice apps and voip use dynamic ptime (usually via "maxptime" which specifies the highest/worst case) in their protocol for this reason - it allows the clients to optimize for all combinations of high/low bandwidth, high/low latency and high/low packet loss in realtime - as network conditions can often change during the course of a call especially while driving around or roaming between wifi and cellular.
- lxgr 2y ago> the rationale behind it was actually sourced in the fact that most SIP was done on LANs and inter-campus WAN which had generally high bitrate connectivity and low latency In addition to that, early VoIP applications mostly used uncompressed G.711 audio, both for interoperability with circuit switched networks and because efficient voice compression codecs weren't yet available royalty-free. G.711 is 64 kbps, so 12 kbps of overhead are less than 25% – not much point in cutting that down to, say, 10% at the expense of doubling effective latency in a LAN use case.
- 2y ago
- yalok 2y agothis codec is for RTC comms - it supports 20ms frame rate. They did mention it's launched in their calling products: "We have already fully launched MLow to all Instagram and Messenger calls and are actively rolling it out on WhatsApp—and we’ve already seen incredible improvement in user engagement driven by better audio quality."
- saurik 2y agoI don't know of any setups which would support muxing in exactly the way I am thinking, but another interesting use case is if you have multiple incoming audio streams which you don't want to be mixed by the server -- potentially because they are end-to-end encrypted -- and so a single packet can contain the data from multiple streams. Doing end-to-end encrypted audio calls is finally becoming pretty widespread, and I could see Facebook being in a good position for their products to do custom muxing.
- sjacob 2y agoWhat skills/concepts would one need to acquire in order to fully grasp all that you've detailed here?
- dan-robertson 2y agoI’m not totally certain about your argument for the specific amount of overhead (if the receiver/sender are on mobile networks, maybe something happens to the packet headers for the first/last legs before the real internet). But doesn’t the OP already give an example where the low bit-rate codec is good: if you can compress things more then you have more of an opportunity to add forward error correction, which greatly improves the quality of calls on lossy connections. I wonder if smaller packets are less likely to be lost, and if there are important cases where multiple streams may be sent like group-calls.
- woah 2y agoAccording to the blog post, this is all practical research that Meta is doing to improve their services. Maybe that is BS, but I kind of doubt that given that Meta is in fact one of the biggest providers of voice and video calls on low bandwidth devices. What makes you think that they have somehow just been misleading themselves the whole time?
- mmmBacon 2y agoThey do this to optimize user experience for 3G and 2G networks not LTE or 5G. A huge number of people are on those networks.
- jhugo 2y agoHaving traveled widely in three of the four most populous countries I’m a bit dubious of this claim. Having decent 4G in the middle of nowhere in India or Indonesia is pretty standard these days. I haven’t been to Africa, but a quick search suggests more than two thirds of the population of the continent has 4G coverage, and more than 80% has 3G. Where are these huge numbers of people on 2G?
- eimrine 2y agoIn Ukraine we use to have a lot of 2G devices: energy meters, terminals in grocery shops, panic buttons and anti-theft devices.
- kragen 2y agowhat you're saying is that tcp/ip sucks at 9600 baud and below, nothing to do with voice really. this is a well-known fact, and solutions to the problem include cslip, vj header compression in ppp, and csp, as well as the lte and 5g stuff you mention
- trustmebroo 2y agowebrtc engineer here... You are right that OPUS already hits the ball for most audio applications. 20kbps is pretty good, and 30 kbps is almost perfect. Meta has tons of Indian and South Asian users with insanely poor network. It's already using 60ms/120ms packets (16.6/8.3 packet rate) for those people, but still, some calls are still bad. What this new codec does is to further make the call accessible by bringing down the bitrate, which seems to help their DAU. Nitpick: measuring packet rate on your device doesn't usually work. I think meta (as well as most other voip providers) is using DTX, so when user doesn't talk, packet rate is 2/sec.