23 ms·

In the article, > This trick enables Lyra to not only run on cloud servers, but also on-device on mid-range phones in real time (with a processing latency of 9

by kixiQu 6y ago

In the article, > This trick enables Lyra to not only run on cloud servers, but also on-device on mid-range phones in real time (with a processing latency of 90ms, which is in line with other traditional speech codecs). Does that not cover it?

regularfry 6y ago
It covers it, but it's not exactly brilliant. 200ms is the point at which conversation breaks down. If half that budget has gone on the codec, not much has to happen on the wire for it to be noticeable.
- mlyle 6y ago
  If your 200ms number is round trip, 90ms + 90ms is more than half that budget.
  - volta83 6y ago
    The pipeline is basically: - stream your voice through an encoder: X ms - send encoded packages over the network: 20-100ms latency (fiber vs mobile phone) - potential decoding + encoding (if receiver does not support the senders codec, e.g., a landline phone using old codec) - stream packages through a decoder: Y ms If you are aiming for 60ms audio latency, which is what I would consider "good", then in the best scenario (20ms network latency; both using same codec) the latency of the encoder+decoder has to be max 40ms (e.g. 20ms for encoder, and 20ms for decoder). It should be obvious that a decoder that does not meet the 20 ms budget, but takes 90 ms instead which is > 3x the budget, can produce better audio (ideally 3-4x better). Latency wise, everything below 60 ms is really good, 60ms is good, and the 60-200ms range goes from good to unusable. That is, 200ms, which is what this new codec would hit under ideal conditions, has a latency that humans consider "unusable" because it is too high to be able to have a fluent conversation. For me, personally, if latency is higher than 120ms, I really don't care about how good a codec "sounds". I use a phone to talk to people, and if we start speaking over each other, cutting each other, etc. because latency is too high, then the function of the phone is gone. Its like having a super nice car that cannot drive. Sure its nice, but when I want to use a car, I actually want to drive with it somewhere. If it cannot drive anywhere, then it is not very useful to me.
  - regularfry 6y ago
    It's not, that's mouth-to-ear.
jtsiskin 6y ago
The “mid-range phones” has me suspicious, I wish they defined that better. And 90ms is much higher than what Opus is supposed to achieve
amluto 6y ago
That covers it, but 90ms sucks. Opus has much lower latency.
bscphil 6y ago
Worth noting that the paper itself explicitly says 90ms algorithmic delay. Seems plausible to me that there might be additional processing required on top of that, and then you have to add network transit time... suffice it to say that the vast majority of use cases are going to be better served by Opus, for the time being. > The overall algorithmic delay is 90 ms https://arxiv.org/pdf/2102.09660.pdf https://arxiv.org/pdf/2102.09660.pdf

by kixiQu 6y ago

regularfry 6y ago

It covers it, but it's not exactly brilliant. 200ms is the point at which conversation breaks down. If half that budget has gone on the codec, not much has to happen on the wire for it to be noticeable.

mlyle 6y ago
If your 200ms number is round trip, 90ms + 90ms is more than half that budget.
- volta83 6y ago
  The pipeline is basically: - stream your voice through an encoder: X ms - send encoded packages over the network: 20-100ms latency (fiber vs mobile phone) - potential decoding + encoding (if receiver does not support the senders codec, e.g., a landline phone using old codec) - stream packages through a decoder: Y ms If you are aiming for 60ms audio latency, which is what I would consider "good", then in the best scenario (20ms network latency; both using same codec) the latency of the encoder+decoder has to be max 40ms (e.g. 20ms for encoder, and 20ms for decoder). It should be obvious that a decoder that does not meet the 20 ms budget, but takes 90 ms instead which is > 3x the budget, can produce better audio (ideally 3-4x better). Latency wise, everything below 60 ms is really good, 60ms is good, and the 60-200ms range goes from good to unusable. That is, 200ms, which is what this new codec would hit under ideal conditions, has a latency that humans consider "unusable" because it is too high to be able to have a fluent conversation. For me, personally, if latency is higher than 120ms, I really don't care about how good a codec "sounds". I use a phone to talk to people, and if we start speaking over each other, cutting each other, etc. because latency is too high, then the function of the phone is gone. Its like having a super nice car that cannot drive. Sure its nice, but when I want to use a car, I actually want to drive with it somewhere. If it cannot drive anywhere, then it is not very useful to me.
- regularfry 6y ago
  It's not, that's mouth-to-ear.

jtsiskin 6y ago

The “mid-range phones” has me suspicious, I wish they defined that better. And 90ms is much higher than what Opus is supposed to achieve

amluto 6y ago

That covers it, but 90ms sucks. Opus has much lower latency.

bscphil 6y ago

Worth noting that the paper itself explicitly says 90ms algorithmic delay. Seems plausible to me that there might be additional processing required on top of that, and then you have to add network transit time... suffice it to say that the vast majority of use cases are going to be better served by Opus, for the time being. > The overall algorithmic delay is 90 ms https://arxiv.org/pdf/2102.09660.pdf https://arxiv.org/pdf/2102.09660.pdf