6 ms·

>If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts Is this realistic? I don't know m

>If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts Is this realistic? I don't know much about providing software for a wide range of hardware, but I would imagine this would require either Netflix or the integrator to write drivers for Netflix to communicate to the hardware dsp right (or standardize)? It seems more feasible to piggyback on a platform that has done that already and is already widely deployed.

asdfasgasdgasdg 6y ago
If I'm reading correctly, they're saying that Netflix is not doing realtime audio, so they need not have this level of precision. They should buffer more than 40ms of audio data.
- tomnipotent 6y ago
  Except buffering wasn't the problem, an OS-managed thread was.
  - PaulDavisThe1st 6y ago
    It was the interaction between the expected amount of required buffering and the thread scheduling behavior that caused the problem. As typically happens when there's a video programmer around, the audio timing is considered less important, and there's a belief that "we can always get the audio we need for the video". Ergo, not much need for buffering. In this case, the odd/incorrect behavior of the thread scheduler exposed the optimistic assumption that you can rely on waking up every 15ms and handling the audio.
    - marcan_42 6y ago
      And further, video latency is much less noticeable than audio latency. ~nobody will notice one dropped frame, but ~everyone will notice 16ms of audio missing, because you get a blatant click. Therefore, it is always important to prioritize audio buffering over video buffering.
      - gpderetta 6y ago
        oh, yes, I had to use some videoconferencing software recently that appears to drop audio before video. It was basically unusable.
  - wolrah 6y ago
    The reason it caused a problem is because Netflix wasn't loading more than a frame worth of data at a time. That's a strategy that makes sense for a game streaming app like Stadia or Steam Link where every frame of latency matters, but for a movie/TV streaming app it really doesn't. The point is that there was never any good reason to make the Netflix app so timing-dependent. Had Netflix not done this they wouldn't have even noticed. --- That said, if a company was releasing an Android 5.0 powered device in late 2017 when Android 8.0 had already been released I can't help but say they brought this on themselves by being idiotically out of date. There's no excuse for launching a device with an OS three years and three versions out of date. Everybody in this story is doing really dumb things.
    - tomnipotent 6y ago
      > wasn't loading more than a frame worth of data at a time So the real question should be, does Netflix have a valid reason for only delivering a single frame of video and audio at a time? Is it bad software design, or good software design for an unknown problem (to us)? I've seen plenty of things that looked stupid out of context, then made sense later on after understanding why the "stupid" things was done in the first place.
      - jfim 6y ago
        It's probably easier to keep the audio and video in sync with a simple implementation. They can't really drift out of sync if all that's being one is handling one frame at a time.
        ghusbands 6y ago
        Though it sounds like they aren't even getting that advantage, because the audio thread is separate, and the described problem is starvation of the audio buffer alone.
      - Closi 6y ago
        It probably makes it easier to implement things like variable frame rates.
- scoutt 6y ago
  I think they are doing so ("The Netflix application is complex, but at its simplest it streams data from a Netflix server, buffers several seconds worth of video and audio data on the device"). The system was built to fill the (hardware) buffers for 15ms worth of audio. The problem is that the 15ms window was not invoked in time (sometimes it took 40ms). Some suggested Netflix that in that case they could fill audio data for 40ms into the HW, but they didn't wanted to because it required low-level modifications.
amluto 6y ago
I see no reason that interrupts are need at all. On a properly working system, the system clock will track the audio clock quite well. If the audio hardware tells you that it will need more data in x ms, it will be very close to x ms as seen by the CPU. If you use a deadline scheduler, you don’t want to wait for an interrupt — you tell the scheduler that you need to run before a specific deadline, and that that. A hardware interrupt is pure overhead, especially on platforms like x86 with silly interrupt latency. (A good kernel can account for the time it takes to wake from idle and compensate when programming a timer (as could a good CPU, but I’ve never heard of hardware that does this) and can wake a user program a bit early if the CPU is otherwise idle and would take a short nap. A sound hardware interrupt can’t do these tricks.)
- PaulDavisThe1st 6y ago
  This is both true and false at the same time. (1) It is absolutely not true that the system clock will track the audio clock "quite well". Real world numbers would typically involve drift measurable in seconds within a low integer number of hours. (2) It absolutely is true that on most sensible audio interface designs, you don't need interrupts except early after device open. At that point, you use them to set up a DLL (Delay Locked Loop) that will enable you to use the system clock to know where the audio hardware is reading/writing to in the buffer used to move data to/from the hardware. Once the DLL is correctly configured, you can just be woken by the system timer, determine the current audio hardware state and read/write data appropriately.
  - marcan_42 6y ago
    But on (2): unless you're a low level app directly driving the hardware, you shouldn't do this. And you especially shouldn't do this without good guarantees that system timers are indeed well behaved. Effectively, using a DLL and system timers instead of straight IRQs is a low level implementation detail, but that belongs either in the driver or in a very low level piece of code anyway (like JACK or some other audio daemon), not user applications. I was sort of throwing all of this together when I said hardware "events"; locking a DLL with a different time base to the playback hardware buffer pointer stats is still effectively that, and should be done, but only at a level low enough where you know it'll work. Which definitely isn't Android's high level thread scheduling framework.
marcan_42 6y ago
It isn't on Android, but as I said, Netflix isn't a real-time app, so Android's deficiencies in the low latency audio department should not concern them. They should just buffer more audio per wakeup.
- saghm 6y ago
  > as I said, Netflix isn't a real-time app I honestly had a tough time parsing the sentence when you said that due to the triple negative, so it's possible that others might have misunderstood it.
  - sitkack 6y ago
    Multiple negatives are common in some languages and absolutely do connote a distinct meaning from the simplified language. https://linguistics.stackexchange.com/questions/15334/what-is-the-reason-for-the-double-negation-found-in-some-languages https://linguistics.stackexchange.com/questions/15334/what-i...
    - saghm 6y ago
      Yep, I don't disagree! I think they can still be tough to parse in English though, even as a native speaker
  - marcan_42 6y ago
    That sentence was indeed a bit awkward, but I hope the meaning gets across since it wouldn't really make much sense the other way around, I think :-)
  - Toutouxc 6y ago
    I'm not a native speaker and the sentence is honestly absolutely unparseable for me and reads like a contradiction. Is it even... correct?
    - saghm 6y ago
      I think it's grammatically correct, yes! But being grammatically correct isn't always the same as being easy to understand; some correct things can be hard to understand, and some incorrect things can be pretty easy to understand!

asdfasgasdgasdg 6y ago

If I'm reading correctly, they're saying that Netflix is not doing realtime audio, so they need not have this level of precision. They should buffer more than 40ms of audio data.

tomnipotent 6y ago
Except buffering wasn't the problem, an OS-managed thread was.
- PaulDavisThe1st 6y ago
  It was the interaction between the expected amount of required buffering and the thread scheduling behavior that caused the problem. As typically happens when there's a video programmer around, the audio timing is considered less important, and there's a belief that "we can always get the audio we need for the video". Ergo, not much need for buffering. In this case, the odd/incorrect behavior of the thread scheduler exposed the optimistic assumption that you can rely on waking up every 15ms and handling the audio.
  - marcan_42 6y ago
    And further, video latency is much less noticeable than audio latency. ~nobody will notice one dropped frame, but ~everyone will notice 16ms of audio missing, because you get a blatant click. Therefore, it is always important to prioritize audio buffering over video buffering.
    - gpderetta 6y ago
      oh, yes, I had to use some videoconferencing software recently that appears to drop audio before video. It was basically unusable.
- wolrah 6y ago
  The reason it caused a problem is because Netflix wasn't loading more than a frame worth of data at a time. That's a strategy that makes sense for a game streaming app like Stadia or Steam Link where every frame of latency matters, but for a movie/TV streaming app it really doesn't. The point is that there was never any good reason to make the Netflix app so timing-dependent. Had Netflix not done this they wouldn't have even noticed. --- That said, if a company was releasing an Android 5.0 powered device in late 2017 when Android 8.0 had already been released I can't help but say they brought this on themselves by being idiotically out of date. There's no excuse for launching a device with an OS three years and three versions out of date. Everybody in this story is doing really dumb things.
  - tomnipotent 6y ago
    > wasn't loading more than a frame worth of data at a time So the real question should be, does Netflix have a valid reason for only delivering a single frame of video and audio at a time? Is it bad software design, or good software design for an unknown problem (to us)? I've seen plenty of things that looked stupid out of context, then made sense later on after understanding why the "stupid" things was done in the first place.
    - jfim 6y ago
      It's probably easier to keep the audio and video in sync with a simple implementation. They can't really drift out of sync if all that's being one is handling one frame at a time.
      - ghusbands 6y ago
        Though it sounds like they aren't even getting that advantage, because the audio thread is separate, and the described problem is starvation of the audio buffer alone.
    - Closi 6y ago
      It probably makes it easier to implement things like variable frame rates.
scoutt 6y ago
I think they are doing so ("The Netflix application is complex, but at its simplest it streams data from a Netflix server, buffers several seconds worth of video and audio data on the device"). The system was built to fill the (hardware) buffers for 15ms worth of audio. The problem is that the 15ms window was not invoked in time (sometimes it took 40ms). Some suggested Netflix that in that case they could fill audio data for 40ms into the HW, but they didn't wanted to because it required low-level modifications.

amluto 6y ago

I see no reason that interrupts are need at all. On a properly working system, the system clock will track the audio clock quite well. If the audio hardware tells you that it will need more data in x ms, it will be very close to x ms as seen by the CPU. If you use a deadline scheduler, you don’t want to wait for an interrupt — you tell the scheduler that you need to run before a specific deadline, and that that. A hardware interrupt is pure overhead, especially on platforms like x86 with silly interrupt latency. (A good kernel can account for the time it takes to wake from idle and compensate when programming a timer (as could a good CPU, but I’ve never heard of hardware that does this) and can wake a user program a bit early if the CPU is otherwise idle and would take a short nap. A sound hardware interrupt can’t do these tricks.)

PaulDavisThe1st 6y ago
This is both true and false at the same time. (1) It is absolutely not true that the system clock will track the audio clock "quite well". Real world numbers would typically involve drift measurable in seconds within a low integer number of hours. (2) It absolutely is true that on most sensible audio interface designs, you don't need interrupts except early after device open. At that point, you use them to set up a DLL (Delay Locked Loop) that will enable you to use the system clock to know where the audio hardware is reading/writing to in the buffer used to move data to/from the hardware. Once the DLL is correctly configured, you can just be woken by the system timer, determine the current audio hardware state and read/write data appropriately.
- marcan_42 6y ago
  But on (2): unless you're a low level app directly driving the hardware, you shouldn't do this. And you especially shouldn't do this without good guarantees that system timers are indeed well behaved. Effectively, using a DLL and system timers instead of straight IRQs is a low level implementation detail, but that belongs either in the driver or in a very low level piece of code anyway (like JACK or some other audio daemon), not user applications. I was sort of throwing all of this together when I said hardware "events"; locking a DLL with a different time base to the playback hardware buffer pointer stats is still effectively that, and should be done, but only at a level low enough where you know it'll work. Which definitely isn't Android's high level thread scheduling framework.

marcan_42 6y ago

It isn't on Android, but as I said, Netflix isn't a real-time app, so Android's deficiencies in the low latency audio department should not concern them. They should just buffer more audio per wakeup.

saghm 6y ago
> as I said, Netflix isn't a real-time app I honestly had a tough time parsing the sentence when you said that due to the triple negative, so it's possible that others might have misunderstood it.
- sitkack 6y ago
  Multiple negatives are common in some languages and absolutely do connote a distinct meaning from the simplified language. https://linguistics.stackexchange.com/questions/15334/what-is-the-reason-for-the-double-negation-found-in-some-languages https://linguistics.stackexchange.com/questions/15334/what-i...
  - saghm 6y ago
    Yep, I don't disagree! I think they can still be tough to parse in English though, even as a native speaker
- marcan_42 6y ago
  That sentence was indeed a bit awkward, but I hope the meaning gets across since it wouldn't really make much sense the other way around, I think :-)
- Toutouxc 6y ago
  I'm not a native speaker and the sentence is honestly absolutely unparseable for me and reads like a contradiction. Is it even... correct?
  - saghm 6y ago
    I think it's grammatically correct, yes! But being grammatically correct isn't always the same as being easy to understand; some correct things can be hard to understand, and some incorrect things can be pretty easy to understand!