12 ms·

This is still bad engineering on Netflix's part. You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback

This is still bad engineering on Netflix's part. You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case, which Netflix very much isn't. If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts (through as many abstraction layers as you want, but a traceable event chain nonetheless, no sleeping guesswork or timers - and ideally all those threads should be marked real-time, never allocate any memory, and never lock or block on anything else). This is what serious real time audio systems like JACK do. And if you're not, you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Not just for robustness, but also for battery life. Sure, there was an Android bug here, but the way Netflix designed this is fragile and, as someone who has messed with audio/video programming enough, wrong. Had they done things properly, they would've been insulated from this OS bug. This kind of bug is best used as a learning experience: what happened here consistently and resulted in completely destroyed playback on one device, is something that has always been happening sporadically and hurting all of your users whenever something causes the CPU to stall or slow for long enough to miss the deadline anyway. Instead of just fixing the bug, make your software robust against these cases, and then all your users win.

nemothekid 6y ago
>If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts Is this realistic? I don't know much about providing software for a wide range of hardware, but I would imagine this would require either Netflix or the integrator to write drivers for Netflix to communicate to the hardware dsp right (or standardize)? It seems more feasible to piggyback on a platform that has done that already and is already widely deployed.
- asdfasgasdgasdg 6y ago
  If I'm reading correctly, they're saying that Netflix is not doing realtime audio, so they need not have this level of precision. They should buffer more than 40ms of audio data.
  - tomnipotent 6y ago
    Except buffering wasn't the problem, an OS-managed thread was.
    - PaulDavisThe1st 6y ago
      It was the interaction between the expected amount of required buffering and the thread scheduling behavior that caused the problem. As typically happens when there's a video programmer around, the audio timing is considered less important, and there's a belief that "we can always get the audio we need for the video". Ergo, not much need for buffering. In this case, the odd/incorrect behavior of the thread scheduler exposed the optimistic assumption that you can rely on waking up every 15ms and handling the audio.
      - marcan_42 6y ago
        And further, video latency is much less noticeable than audio latency. ~nobody will notice one dropped frame, but ~everyone will notice 16ms of audio missing, because you get a blatant click. Therefore, it is always important to prioritize audio buffering over video buffering.
        gpderetta 6y ago
        oh, yes, I had to use some videoconferencing software recently that appears to drop audio before video. It was basically unusable.
    - wolrah 6y ago
      The reason it caused a problem is because Netflix wasn't loading more than a frame worth of data at a time. That's a strategy that makes sense for a game streaming app like Stadia or Steam Link where every frame of latency matters, but for a movie/TV streaming app it really doesn't. The point is that there was never any good reason to make the Netflix app so timing-dependent. Had Netflix not done this they wouldn't have even noticed. --- That said, if a company was releasing an Android 5.0 powered device in late 2017 when Android 8.0 had already been released I can't help but say they brought this on themselves by being idiotically out of date. There's no excuse for launching a device with an OS three years and three versions out of date. Everybody in this story is doing really dumb things.
- amluto 6y ago
  I see no reason that interrupts are need at all. On a properly working system, the system clock will track the audio clock quite well. If the audio hardware tells you that it will need more data in x ms, it will be very close to x ms as seen by the CPU. If you use a deadline scheduler, you don’t want to wait for an interrupt — you tell the scheduler that you need to run before a specific deadline, and that that. A hardware interrupt is pure overhead, especially on platforms like x86 with silly interrupt latency. (A good kernel can account for the time it takes to wake from idle and compensate when programming a timer (as could a good CPU, but I’ve never heard of hardware that does this) and can wake a user program a bit early if the CPU is otherwise idle and would take a short nap. A sound hardware interrupt can’t do these tricks.)
  - PaulDavisThe1st 6y ago
    This is both true and false at the same time. (1) It is absolutely not true that the system clock will track the audio clock "quite well". Real world numbers would typically involve drift measurable in seconds within a low integer number of hours. (2) It absolutely is true that on most sensible audio interface designs, you don't need interrupts except early after device open. At that point, you use them to set up a DLL (Delay Locked Loop) that will enable you to use the system clock to know where the audio hardware is reading/writing to in the buffer used to move data to/from the hardware. Once the DLL is correctly configured, you can just be woken by the system timer, determine the current audio hardware state and read/write data appropriately.
    - marcan_42 6y ago
      But on (2): unless you're a low level app directly driving the hardware, you shouldn't do this. And you especially shouldn't do this without good guarantees that system timers are indeed well behaved. Effectively, using a DLL and system timers instead of straight IRQs is a low level implementation detail, but that belongs either in the driver or in a very low level piece of code anyway (like JACK or some other audio daemon), not user applications. I was sort of throwing all of this together when I said hardware "events"; locking a DLL with a different time base to the playback hardware buffer pointer stats is still effectively that, and should be done, but only at a level low enough where you know it'll work. Which definitely isn't Android's high level thread scheduling framework.
- marcan_42 6y ago
  It isn't on Android, but as I said, Netflix isn't a real-time app, so Android's deficiencies in the low latency audio department should not concern them. They should just buffer more audio per wakeup.
  - saghm 6y ago
    > as I said, Netflix isn't a real-time app I honestly had a tough time parsing the sentence when you said that due to the triple negative, so it's possible that others might have misunderstood it.
    - sitkack 6y ago
      Multiple negatives are common in some languages and absolutely do connote a distinct meaning from the simplified language. https://linguistics.stackexchange.com/questions/15334/what-is-the-reason-for-the-double-negation-found-in-some-languages https://linguistics.stackexchange.com/questions/15334/what-i...
      - saghm 6y ago
        Yep, I don't disagree! I think they can still be tough to parse in English though, even as a native speaker
    - marcan_42 6y ago
      That sentence was indeed a bit awkward, but I hope the meaning gets across since it wouldn't really make much sense the other way around, I think :-)
    - Toutouxc 6y ago
      I'm not a native speaker and the sentence is honestly absolutely unparseable for me and reads like a contradiction. Is it even... correct?
      - saghm 6y ago
        I think it's grammatically correct, yes! But being grammatically correct isn't always the same as being easy to understand; some correct things can be hard to understand, and some incorrect things can be pretty easy to understand!
ActorNightly 6y ago
Id generalize this to: If your application is timing critical, and you aren't using things specifically desiged for time critical applications (like RTOS), then you should be doing as much low level implementation yourself as possible.
- PaulDavisThe1st 6y ago
  Here's a better generalization: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...
- josefx 6y ago
  Or the other way: don't make your application timing critical if it does not have to be. As others point out the hardware provides a buffer and the app only fills in one frame in advance instead of filling the buffer completely when possible.
Jasper_ 6y ago
Android does not have a functioning audio API. And really doesn't have the ability to mark threads as real-time that will stick. I've worked on Android game ports and done audio stack work, the whole thing is a shitshow and barely functions as documented. What their support team recommends in one version is quickly broken by hacks done by chip vendors and random bugs added to OS versions. Nobody at Google even tests basic scheduler behavior such that an extra 40ms can sneak in completely unnoticed. I've seen stuff far worse than this, they once broke their own cert test and nobody noticed for half a year.
- marcan_42 6y ago
  Sure, but this is all painful for real-time use cases, like audio production and games. Playing video is not one of those, and there the more you buffer the better the battery life thanks to reduced interrupt load and longer CPU idle states.
- jfim 6y ago
  Out of curiosity, are the audio APIs any better on iOS? Core audio seemed to work pretty well the last time I used it on OS X, so I assume it does, but I'd be curious to hear your take on it.
  - Jasper_ 6y ago
    I haven't used it, to be quite honest, the games I worked on had the Mac/iOS ports done by a different team, but since I was usually called in to help diagnose customer issues in live and never had to touch the Mac/iOS audio stuff, it probably worked well enough!
  - dannyw 6y ago
    Much better. In general, "creative" APIs (e.g. audio playback, camera/image processing) tend to have well-rounded APIs on iOS.
    - leokennis 6y ago
      Probably unrelated observation. I use a MacBook for personal use and a Windows laptop for work. For many things, they both work well in general (Windows is better for "document stuff" and file management, Mac a little better in window management etc., but comparable in general). But when it comes to media stuff (think: cropping a lossless audio file and compressing the result into an AAC or MP3, resizing 20 images, transcoding a movie from one format to the other, removing one page from a PDF), on Mac it can usually be done with built in tools. On Windows you often need either a prohibitively expensive software suite or go on an ad infested crapware hunt resulting in a borderline unusable tool.
golergka 6y ago
> And if you're not, you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Author addressed this point and agrees with it: > Why don’t you just copy more data each time the handler is called? This was a fair criticism, but changing this behavior involved deeper changes than I was prepared to make In general, "real time audio" usually refers to low-latency audio, which uninteractive video playback is not.
- pierrefermat1 6y ago
  Yes he addressed it but just said he was unwilling to do anything about it as it resulted in too much change/work That does answer the issue of why it was implemented like that in the first place
  - golergka 6y ago
    Well, it's been my experience that you only find out the proper way to implement something after you do it one or a couple of times.
tpetry 6y ago
Saying in hindsight when the error has been found that something is a wrong approach and should have never been done like this is easy. But everyone of us is daily coding simple „tricks“ to not implement a more complex solution because the simple approach seems to be working, until some months/years later it does not anymore. And then someone will point their figures at you/us and say it‘s obvious it should have never been done like that...
- ghusbands 6y ago
  This isn't just hindsight - anyone who knows audio coding or really anything about thread-sleeping will have the same reaction of disgust to the design of that audio buffering. It is a design that is bad in ways it never needed to be, that would not have saved time in coding.
  - herodoturtle 6y ago
    Honest question (I don't know much about audio coding), but from what I've read here on HN, Netflix offer some of the best remuneration packages - so don't they have some of the best developers working there? Surely they'd have picked this up if it were that straight forward?
    - oarsinsync 6y ago
      > offer some of the best remuneration packages - so don't they have some of the best developers working there? Having worked in several top-tier for employee-compensation firms, I can wholeheartedly assure you that best pay does not result in best developers (or any other job function). It does generally result in a higher “floor” of employee, but that’s about it.
    - wolco2 6y ago
      Have you seen how broken the hiring process is at these bigger places. If you spend your time learning about the details of audio playback what kind of additional time do you have to learn leetcode?
    - 6y ago
rendall 6y ago
> You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case, which Netflix very much isn't. I don't understand this criticism, but I am likely missing something. From what I gathered from the article, Ninja is specifically for firmware. It buffers frames, and delivers those frames whenever requested. > your processing loop needs to be directly driven by audio hardware buffer events/interrupts Again, I could be missing the thrust of the criticism, but isn't that exactly what's happening, here? Ninja supplies an endpoint, and it's the hardware that calls it?
- josefx 6y ago
  > It buffers frames, and delivers those frames whenever requested. Going by the article they use a plain android thread that runs at fixed 15ms intervals and buffers exactly one frame in advance. For 30 fps this is more often than needed (yay battery life) and for 60 fps that just asks for trouble when your OS doesn't guarantee real time behavior. > Ninja supplies an endpoint, and it's the hardware that calls it? As far as I understand it provides a cross platform endpoint, called by a platform specific wrapper. In the Android case this was a background worker and Android considered background workers not real time critical.
  - rendall 6y ago
    Thanks for taking the time to explain! Much appreciated. > They use a plain android thread that runs at fixed 15ms intervals and buffers exactly one frame in advance From the article: "...buffers several seconds worth of video and audio data on the device, then delivers video and audio frames one-at-a-time to the device’s playback hardware" And: "under normal playback the thread copies one frame of data into the Android playback API, then tells the thread scheduler to wait 15 ms and invoke the handler again" And: "When you create an Android thread, you can request that the thread be run repeatedly, as if in a loop, but it is the Android Thread scheduler that calls the handler, not your own application." But from OP ...you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Not just for robustness, but also for battery life. I suspect there is a misapprehension here. "...under normal playback the thread copies one frame of data into the Android playback API, then tells the thread scheduler to wait 15 ms and invoke the handler again". I suspect that one can call that handler more often than every 15ms to fill the playback buffer, and then in normal playback one adds frames one at a time to keep the buffer filled and no faster, but on quick reading it sounds like as the OP stated.
jdblair 6y ago
Author here: Fair criticism. There are some details that are missing in the interest of keeping the story simple. 1) The audio and video playback is driven by the hardware, using the audio stream as the master clock. 2) Ninja does fill the Android and hardware buffers and keep them full. In the pathological case, the hardware and Android buffers had emptied. 3) When the Android buffer isn't full, the thread yields to Android (to be a good citizen) but asks to be invoked again right away. The 40ms "background thread" delay broke this behavior. The comment about "changing this behavior involved deeper changes than I was prepared to make" was when I explored changing this behavior (copying multiple samples per invocation) and decided it was more likely to introduce more bugs.
- jdblair 6y ago
  The "buffering" parts of the graph show the "fill the buffer" behavior described in (3). The time between calls is very short and large amounts of data are moved.
  - marcan_42 6y ago
    How does the buffering proceed quickly, but then the (slower than usual) periodic calls only copy a limited amount of data? That's the part that doesn't make sense to me. Normally if you're filling buffers, you copy as much data as is needed to keep the buffer full, no matter when you're invoked.
    - carlmr 6y ago
      That is indeed interesting. The pipeline should be filling up if the OS scheduler delays the buffering for 40ms, giving enough data on the next call. Except if the buffering isn't greedy enough or aborted by the OS in some way.
    - jdblair 6y ago
      The handler always copies the same amount of data. It has to be called more often to copy a lot of data, which is what usually happens when Android buffer isn't full. I agree with you that this isn't usually how this is done. Doing it this way means that Android has more freedom to schedule other tasks during the buffering phase.
rkagerer 6y ago
Can't say I'm surprised; I have a lot of other small beefs with the Netflix app that collectively hint to me the developers who coded it may not have been top of their game. Some examples: - Unacceptably high latency for any sort of seek operation - even if it's just going back a few seconds to what you just watched and ought to still be somewhere in a local buffer. - Trying to seek multiple times sometimes forces you to wait for the previous operation to finish instead of "accumulating" your taps to determine a new time target - Grabbing the playback marker and dragging it left and right is very imprecise, it should converge into "logarithmic" behaviour for fine movements - Flakey Chromecast button. More than half the time I have to kill the app and relaunch it to get the cast button to show up. - Occasionally flakey UI layout (random bugs which present when moving between screens or searching, or flickering due to layout engine doing multiple repaints over the same controls) - Likes to plaster up things I don't want. They seem to prioritize "discovery" over "user intent", which would work if only they surfaced articles I'm interested in (which is rare). When I just want to find a "comedy" or get to the series I'm watching, it sometimes feels like I'm fighting against the UI - Constantly moving my cheese - it's like the UI designers have ADD and can't commit Amazon Prime is just as bad (maybe worse). I wish these companies didn't control the vertical - i.e. just provide content, and let other vendors make the client software. /rant
- inbx0 6y ago
  Good points, but I have to say that even with all that, Netflix app/site is by far the best of the streaming services that I've used. HBO Nordic, Viaplay and YouTube movies have pretty bad UI/UX problems and their streams become borderline unwatchable with subpar internet connections, whereas Netflix has in my experience still been able to push through video in those conditions with good enough quality that I can still enjoy it.
- throwaway2245 6y ago
  > Unacceptably high latency for any sort of seek operation - even if it's just going back a few seconds to what you just watched and ought to still be somewhere in a local buffer. Unacceptable for what use-case? I had assumed this was somewhat intentional on Netflix's part, or at least not a priority for them.
kaali1 6y ago
The way to get the best latency on a device is not for your processing loop to be directly driven by an audio hardware events/interrupts. This is for many reasons: 1) events/interrupts can be delayed (usually by (bad) drivers disabling interrupt for a long time). 2) latency is not adjustable. You may want to give your processing more time if you know more work needs to be done, or the system is under variable load. 3) you will usually finish generating your audio too early. Eg: if the interrupts is every 2ms, but your processing is 1ms long, you are adding an unnecessary 1ms latency by starting as soon as the interrupt is received. 4) the device will consume more power as it can not know when it will next wakeup (theoretically, I do not think it makes a difference in practice) Modern mobile devices use a DMA to copy the content of the application processor (AP) to the audio DSP. This DMA will copy periodically the content of a AP buffer to a DSP buffer, this is called a DMA burst. You want to track this burst and wakup just enough time before it to have the time to generate your audio data and write it to the AP buffer + a safety margin. This allows you to track the system performance & application load to adjust your safety margin and optimize latency. It also allows the scheduler to know when your app will need to be woken up far in advance leading. The Android Aaudio API [1] implement what I just described, as well as memory mapping the AP buffer to the application process to achieve 0 copy. It is the way to achieve the lowest latency on Android. I believe Apple low latency APIs uses a similar no interrupt design. Source: worked 3 years on the Android audio framework at Google. Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer. [1]: https://developer.android.com/ndk/guides/audio/aaudio/aaudio https://developer.android.com/ndk/guides/audio/aaudio/aaudio
- richardwhiuk 6y ago
  The Android Aaudio API is newer than the device in question. The previous options are either SLES or AudioDriver, both of which are bad in their own ways.
  - kaali1 6y ago
    You are right, Aaudio is available since Oreo only and the device is 3 version older. Additionally Netflix 's video playback is not a low latency use-case, so it shouldn't use Aaudio even on recent devices. My comment was about the parent's affirmation that low latency apps should be scheduled by interrupt.
ubercow13 6y ago
>You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case Why not? According to [1], using timers is how Windows, CoreAudio, and PulseAudio all work under the hood, and on Windows and in PulseAudio it replaced the previous interrupt-based implementations. On the app-end of the APIs, Windows' WASAPI code example uses Sleep polling [2], and PulseAudio's write callback is optional and VLC doesn't use it [3], foobar2000 has a polling-based output mode[4], Windows has specific APIs for audio thread scheduling [5], etc. Is this a specific deficiency of Android? [1] https://fedoraproject.org/wiki/Features/GlitchFreeAudio https://fedoraproject.org/wiki/Features/GlitchFreeAudio [2] https://docs.microsoft.com/en-us/windows/win32/coreaudio/rendering-a-stream https://docs.microsoft.com/en-us/windows/win32/coreaudio/ren... [3] http://www.videolan.org/developers/vlc/modules/audio_output/pulse.c http://www.videolan.org/developers/vlc/modules/audio_output/... [4] http://wiki.hydrogenaud.io/index.php?title=Foobar2000:Components_0.9/foo_out_wasapi#Output_modes:_push_and_event-driven http://wiki.hydrogenaud.io/index.php?title=Foobar2000:Compon... [5] https://docs.microsoft.com/en-us/windows/win32/procthread/multimedia-class-scheduler-service https://docs.microsoft.com/en-us/windows/win32/procthread/mu...
scoutt 6y ago
I agree this is bad engineering: don't use Android threads for real-time stuff. This should be running at module level. A better approach, if the micro you are using allows it, is to have an interrupt when (for example) the I2S buffer is empty. I would then point the DMA to fetch the next buffer (already processed and mixed) and fire the DMA transfer. > at least 100ms This calls for huge latency problems. But I understand the approach if your buffer-filling/reading procedures are slow or unreliable. I disagree with the timer thing. There are systems that provide precise timers for media (for example Win32 multimedia timers[1], that I never used but I know they exist). [1] https://docs.microsoft.com/en-us/windows/win32/multimedia/about-multimedia-timers https://docs.microsoft.com/en-us/windows/win32/multimedia/ab...
- ubercow13 6y ago
  >This calls for huge latency problems It doesn't mean there is 100ms of latency, it just means that 100ms of audio is buffered so that you have ~100ms of leeway about when you app's audio thread is scheduled. Changes to the audio stream such, as stop/start/volume control, can be achieved with much lower latency using buffer rewriting or by applying changes lower down the stack where the buffers are smaller, or both. By default PulseAudio will buffer ~2000ms of audio from clients [1] [1] https://freedesktop.org/software/pulseaudio/doxygen/structpa__buffer__attr.html#aa7e8f3348cbda863b6f1dd55a9024b7a https://freedesktop.org/software/pulseaudio/doxygen/structpa...
  - scoutt 6y ago
    It DOES mean there is 100ms of latency. Anything above 10ms (some say max. 20ms) for real-time audio processing (especially musical instruments) is prohibitive. Imagine an electronic drum set: if you hit the snare and the audio is output 100ms later from the speakers, I bet you'll notice it :)
    - deleted 6y ago
      [deleted]
    - ubercow13 6y ago

nemothekid 6y ago

>If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts Is this realistic? I don't know much about providing software for a wide range of hardware, but I would imagine this would require either Netflix or the integrator to write drivers for Netflix to communicate to the hardware dsp right (or standardize)? It seems more feasible to piggyback on a platform that has done that already and is already widely deployed.

asdfasgasdgasdg 6y ago
If I'm reading correctly, they're saying that Netflix is not doing realtime audio, so they need not have this level of precision. They should buffer more than 40ms of audio data.
- tomnipotent 6y ago
  Except buffering wasn't the problem, an OS-managed thread was.
  - PaulDavisThe1st 6y ago
    It was the interaction between the expected amount of required buffering and the thread scheduling behavior that caused the problem. As typically happens when there's a video programmer around, the audio timing is considered less important, and there's a belief that "we can always get the audio we need for the video". Ergo, not much need for buffering. In this case, the odd/incorrect behavior of the thread scheduler exposed the optimistic assumption that you can rely on waking up every 15ms and handling the audio.
    - marcan_42 6y ago
      And further, video latency is much less noticeable than audio latency. ~nobody will notice one dropped frame, but ~everyone will notice 16ms of audio missing, because you get a blatant click. Therefore, it is always important to prioritize audio buffering over video buffering.
      - gpderetta 6y ago
        oh, yes, I had to use some videoconferencing software recently that appears to drop audio before video. It was basically unusable.
  - wolrah 6y ago
    The reason it caused a problem is because Netflix wasn't loading more than a frame worth of data at a time. That's a strategy that makes sense for a game streaming app like Stadia or Steam Link where every frame of latency matters, but for a movie/TV streaming app it really doesn't. The point is that there was never any good reason to make the Netflix app so timing-dependent. Had Netflix not done this they wouldn't have even noticed. --- That said, if a company was releasing an Android 5.0 powered device in late 2017 when Android 8.0 had already been released I can't help but say they brought this on themselves by being idiotically out of date. There's no excuse for launching a device with an OS three years and three versions out of date. Everybody in this story is doing really dumb things.
amluto 6y ago
I see no reason that interrupts are need at all. On a properly working system, the system clock will track the audio clock quite well. If the audio hardware tells you that it will need more data in x ms, it will be very close to x ms as seen by the CPU. If you use a deadline scheduler, you don’t want to wait for an interrupt — you tell the scheduler that you need to run before a specific deadline, and that that. A hardware interrupt is pure overhead, especially on platforms like x86 with silly interrupt latency. (A good kernel can account for the time it takes to wake from idle and compensate when programming a timer (as could a good CPU, but I’ve never heard of hardware that does this) and can wake a user program a bit early if the CPU is otherwise idle and would take a short nap. A sound hardware interrupt can’t do these tricks.)
- PaulDavisThe1st 6y ago
  This is both true and false at the same time. (1) It is absolutely not true that the system clock will track the audio clock "quite well". Real world numbers would typically involve drift measurable in seconds within a low integer number of hours. (2) It absolutely is true that on most sensible audio interface designs, you don't need interrupts except early after device open. At that point, you use them to set up a DLL (Delay Locked Loop) that will enable you to use the system clock to know where the audio hardware is reading/writing to in the buffer used to move data to/from the hardware. Once the DLL is correctly configured, you can just be woken by the system timer, determine the current audio hardware state and read/write data appropriately.
  - marcan_42 6y ago
    But on (2): unless you're a low level app directly driving the hardware, you shouldn't do this. And you especially shouldn't do this without good guarantees that system timers are indeed well behaved. Effectively, using a DLL and system timers instead of straight IRQs is a low level implementation detail, but that belongs either in the driver or in a very low level piece of code anyway (like JACK or some other audio daemon), not user applications. I was sort of throwing all of this together when I said hardware "events"; locking a DLL with a different time base to the playback hardware buffer pointer stats is still effectively that, and should be done, but only at a level low enough where you know it'll work. Which definitely isn't Android's high level thread scheduling framework.
marcan_42 6y ago
It isn't on Android, but as I said, Netflix isn't a real-time app, so Android's deficiencies in the low latency audio department should not concern them. They should just buffer more audio per wakeup.
- saghm 6y ago
  > as I said, Netflix isn't a real-time app I honestly had a tough time parsing the sentence when you said that due to the triple negative, so it's possible that others might have misunderstood it.
  - sitkack 6y ago
    Multiple negatives are common in some languages and absolutely do connote a distinct meaning from the simplified language. https://linguistics.stackexchange.com/questions/15334/what-is-the-reason-for-the-double-negation-found-in-some-languages https://linguistics.stackexchange.com/questions/15334/what-i...
    - saghm 6y ago
      Yep, I don't disagree! I think they can still be tough to parse in English though, even as a native speaker
  - marcan_42 6y ago
    That sentence was indeed a bit awkward, but I hope the meaning gets across since it wouldn't really make much sense the other way around, I think :-)
  - Toutouxc 6y ago
    I'm not a native speaker and the sentence is honestly absolutely unparseable for me and reads like a contradiction. Is it even... correct?
    - saghm 6y ago
      I think it's grammatically correct, yes! But being grammatically correct isn't always the same as being easy to understand; some correct things can be hard to understand, and some incorrect things can be pretty easy to understand!

ActorNightly 6y ago

Id generalize this to: If your application is timing critical, and you aren't using things specifically desiged for time critical applications (like RTOS), then you should be doing as much low level implementation yourself as possible.

PaulDavisThe1st 6y ago
Here's a better generalization: http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing http://www.rossbencina.com/code/real-time-audio-programming-...
josefx 6y ago
Or the other way: don't make your application timing critical if it does not have to be. As others point out the hardware provides a buffer and the app only fills in one frame in advance instead of filling the buffer completely when possible.

Jasper_ 6y ago

Android does not have a functioning audio API. And really doesn't have the ability to mark threads as real-time that will stick. I've worked on Android game ports and done audio stack work, the whole thing is a shitshow and barely functions as documented. What their support team recommends in one version is quickly broken by hacks done by chip vendors and random bugs added to OS versions. Nobody at Google even tests basic scheduler behavior such that an extra 40ms can sneak in completely unnoticed. I've seen stuff far worse than this, they once broke their own cert test and nobody noticed for half a year.

marcan_42 6y ago
Sure, but this is all painful for real-time use cases, like audio production and games. Playing video is not one of those, and there the more you buffer the better the battery life thanks to reduced interrupt load and longer CPU idle states.
jfim 6y ago
Out of curiosity, are the audio APIs any better on iOS? Core audio seemed to work pretty well the last time I used it on OS X, so I assume it does, but I'd be curious to hear your take on it.
- Jasper_ 6y ago
  I haven't used it, to be quite honest, the games I worked on had the Mac/iOS ports done by a different team, but since I was usually called in to help diagnose customer issues in live and never had to touch the Mac/iOS audio stuff, it probably worked well enough!
- dannyw 6y ago
  Much better. In general, "creative" APIs (e.g. audio playback, camera/image processing) tend to have well-rounded APIs on iOS.
  - leokennis 6y ago
    Probably unrelated observation. I use a MacBook for personal use and a Windows laptop for work. For many things, they both work well in general (Windows is better for "document stuff" and file management, Mac a little better in window management etc., but comparable in general). But when it comes to media stuff (think: cropping a lossless audio file and compressing the result into an AAC or MP3, resizing 20 images, transcoding a movie from one format to the other, removing one page from a PDF), on Mac it can usually be done with built in tools. On Windows you often need either a prohibitively expensive software suite or go on an ad infested crapware hunt resulting in a borderline unusable tool.

golergka 6y ago

> And if you're not, you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Author addressed this point and agrees with it: > Why don’t you just copy more data each time the handler is called? This was a fair criticism, but changing this behavior involved deeper changes than I was prepared to make In general, "real time audio" usually refers to low-latency audio, which uninteractive video playback is not.

pierrefermat1 6y ago
Yes he addressed it but just said he was unwilling to do anything about it as it resulted in too much change/work That does answer the issue of why it was implemented like that in the first place
- golergka 6y ago
  Well, it's been my experience that you only find out the proper way to implement something after you do it one or a couple of times.

tpetry 6y ago

Saying in hindsight when the error has been found that something is a wrong approach and should have never been done like this is easy. But everyone of us is daily coding simple „tricks“ to not implement a more complex solution because the simple approach seems to be working, until some months/years later it does not anymore. And then someone will point their figures at you/us and say it‘s obvious it should have never been done like that...

ghusbands 6y ago
This isn't just hindsight - anyone who knows audio coding or really anything about thread-sleeping will have the same reaction of disgust to the design of that audio buffering. It is a design that is bad in ways it never needed to be, that would not have saved time in coding.
- herodoturtle 6y ago
  Honest question (I don't know much about audio coding), but from what I've read here on HN, Netflix offer some of the best remuneration packages - so don't they have some of the best developers working there? Surely they'd have picked this up if it were that straight forward?
  - oarsinsync 6y ago
    > offer some of the best remuneration packages - so don't they have some of the best developers working there? Having worked in several top-tier for employee-compensation firms, I can wholeheartedly assure you that best pay does not result in best developers (or any other job function). It does generally result in a higher “floor” of employee, but that’s about it.
  - wolco2 6y ago
    Have you seen how broken the hiring process is at these bigger places. If you spend your time learning about the details of audio playback what kind of additional time do you have to learn leetcode?
  - 6y ago

rendall 6y ago

> You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case, which Netflix very much isn't. I don't understand this criticism, but I am likely missing something. From what I gathered from the article, Ninja is specifically for firmware. It buffers frames, and delivers those frames whenever requested. > your processing loop needs to be directly driven by audio hardware buffer events/interrupts Again, I could be missing the thrust of the criticism, but isn't that exactly what's happening, here? Ninja supplies an endpoint, and it's the hardware that calls it?

josefx 6y ago
> It buffers frames, and delivers those frames whenever requested. Going by the article they use a plain android thread that runs at fixed 15ms intervals and buffers exactly one frame in advance. For 30 fps this is more often than needed (yay battery life) and for 60 fps that just asks for trouble when your OS doesn't guarantee real time behavior. > Ninja supplies an endpoint, and it's the hardware that calls it? As far as I understand it provides a cross platform endpoint, called by a platform specific wrapper. In the Android case this was a background worker and Android considered background workers not real time critical.
- rendall 6y ago
  Thanks for taking the time to explain! Much appreciated. > They use a plain android thread that runs at fixed 15ms intervals and buffers exactly one frame in advance From the article: "...buffers several seconds worth of video and audio data on the device, then delivers video and audio frames one-at-a-time to the device’s playback hardware" And: "under normal playback the thread copies one frame of data into the Android playback API, then tells the thread scheduler to wait 15 ms and invoke the handler again" And: "When you create an Android thread, you can request that the thread be run repeatedly, as if in a loop, but it is the Android Thread scheduler that calls the handler, not your own application." But from OP ...you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Not just for robustness, but also for battery life. I suspect there is a misapprehension here. "...under normal playback the thread copies one frame of data into the Android playback API, then tells the thread scheduler to wait 15 ms and invoke the handler again". I suspect that one can call that handler more often than every 15ms to fill the playback buffer, and then in normal playback one adds frames one at a time to keep the buffer filled and no faster, but on quick reading it sounds like as the OP stated.

jdblair 6y ago

Author here: Fair criticism. There are some details that are missing in the interest of keeping the story simple. 1) The audio and video playback is driven by the hardware, using the audio stream as the master clock. 2) Ninja does fill the Android and hardware buffers and keep them full. In the pathological case, the hardware and Android buffers had emptied. 3) When the Android buffer isn't full, the thread yields to Android (to be a good citizen) but asks to be invoked again right away. The 40ms "background thread" delay broke this behavior. The comment about "changing this behavior involved deeper changes than I was prepared to make" was when I explored changing this behavior (copying multiple samples per invocation) and decided it was more likely to introduce more bugs.

jdblair 6y ago
The "buffering" parts of the graph show the "fill the buffer" behavior described in (3). The time between calls is very short and large amounts of data are moved.
- marcan_42 6y ago
  How does the buffering proceed quickly, but then the (slower than usual) periodic calls only copy a limited amount of data? That's the part that doesn't make sense to me. Normally if you're filling buffers, you copy as much data as is needed to keep the buffer full, no matter when you're invoked.
  - carlmr 6y ago
    That is indeed interesting. The pipeline should be filling up if the OS scheduler delays the buffering for 40ms, giving enough data on the next call. Except if the buffering isn't greedy enough or aborted by the OS in some way.
  - jdblair 6y ago
    The handler always copies the same amount of data. It has to be called more often to copy a lot of data, which is what usually happens when Android buffer isn't full. I agree with you that this isn't usually how this is done. Doing it this way means that Android has more freedom to schedule other tasks during the buffering phase.

rkagerer 6y ago

Can't say I'm surprised; I have a lot of other small beefs with the Netflix app that collectively hint to me the developers who coded it may not have been top of their game. Some examples: - Unacceptably high latency for any sort of seek operation - even if it's just going back a few seconds to what you just watched and ought to still be somewhere in a local buffer. - Trying to seek multiple times sometimes forces you to wait for the previous operation to finish instead of "accumulating" your taps to determine a new time target - Grabbing the playback marker and dragging it left and right is very imprecise, it should converge into "logarithmic" behaviour for fine movements - Flakey Chromecast button. More than half the time I have to kill the app and relaunch it to get the cast button to show up. - Occasionally flakey UI layout (random bugs which present when moving between screens or searching, or flickering due to layout engine doing multiple repaints over the same controls) - Likes to plaster up things I don't want. They seem to prioritize "discovery" over "user intent", which would work if only they surfaced articles I'm interested in (which is rare). When I just want to find a "comedy" or get to the series I'm watching, it sometimes feels like I'm fighting against the UI - Constantly moving my cheese - it's like the UI designers have ADD and can't commit Amazon Prime is just as bad (maybe worse). I wish these companies didn't control the vertical - i.e. just provide content, and let other vendors make the client software. /rant

inbx0 6y ago
Good points, but I have to say that even with all that, Netflix app/site is by far the best of the streaming services that I've used. HBO Nordic, Viaplay and YouTube movies have pretty bad UI/UX problems and their streams become borderline unwatchable with subpar internet connections, whereas Netflix has in my experience still been able to push through video in those conditions with good enough quality that I can still enjoy it.
throwaway2245 6y ago
> Unacceptably high latency for any sort of seek operation - even if it's just going back a few seconds to what you just watched and ought to still be somewhere in a local buffer. Unacceptable for what use-case? I had assumed this was somewhat intentional on Netflix's part, or at least not a priority for them.

kaali1 6y ago

The way to get the best latency on a device is not for your processing loop to be directly driven by an audio hardware events/interrupts. This is for many reasons: 1) events/interrupts can be delayed (usually by (bad) drivers disabling interrupt for a long time). 2) latency is not adjustable. You may want to give your processing more time if you know more work needs to be done, or the system is under variable load. 3) you will usually finish generating your audio too early. Eg: if the interrupts is every 2ms, but your processing is 1ms long, you are adding an unnecessary 1ms latency by starting as soon as the interrupt is received. 4) the device will consume more power as it can not know when it will next wakeup (theoretically, I do not think it makes a difference in practice) Modern mobile devices use a DMA to copy the content of the application processor (AP) to the audio DSP. This DMA will copy periodically the content of a AP buffer to a DSP buffer, this is called a DMA burst. You want to track this burst and wakup just enough time before it to have the time to generate your audio data and write it to the AP buffer + a safety margin. This allows you to track the system performance & application load to adjust your safety margin and optimize latency. It also allows the scheduler to know when your app will need to be woken up far in advance leading. The Android Aaudio API [1] implement what I just described, as well as memory mapping the AP buffer to the application process to achieve 0 copy. It is the way to achieve the lowest latency on Android. I believe Apple low latency APIs uses a similar no interrupt design. Source: worked 3 years on the Android audio framework at Google. Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer. [1]: https://developer.android.com/ndk/guides/audio/aaudio/aaudio https://developer.android.com/ndk/guides/audio/aaudio/aaudio

richardwhiuk 6y ago
The Android Aaudio API is newer than the device in question. The previous options are either SLES or AudioDriver, both of which are bad in their own ways.
- kaali1 6y ago
  You are right, Aaudio is available since Oreo only and the device is 3 version older. Additionally Netflix 's video playback is not a low latency use-case, so it shouldn't use Aaudio even on recent devices. My comment was about the parent's affirmation that low latency apps should be scheduled by interrupt.

ubercow13 6y ago

>You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case Why not? According to [1], using timers is how Windows, CoreAudio, and PulseAudio all work under the hood, and on Windows and in PulseAudio it replaced the previous interrupt-based implementations. On the app-end of the APIs, Windows' WASAPI code example uses Sleep polling [2], and PulseAudio's write callback is optional and VLC doesn't use it [3], foobar2000 has a polling-based output mode[4], Windows has specific APIs for audio thread scheduling [5], etc. Is this a specific deficiency of Android? [1] https://fedoraproject.org/wiki/Features/GlitchFreeAudio https://fedoraproject.org/wiki/Features/GlitchFreeAudio [2] https://docs.microsoft.com/en-us/windows/win32/coreaudio/rendering-a-stream https://docs.microsoft.com/en-us/windows/win32/coreaudio/ren... [3] http://www.videolan.org/developers/vlc/modules/audio_output/pulse.c http://www.videolan.org/developers/vlc/modules/audio_output/... [4] http://wiki.hydrogenaud.io/index.php?title=Foobar2000:Components_0.9/foo_out_wasapi#Output_modes:_push_and_event-driven http://wiki.hydrogenaud.io/index.php?title=Foobar2000:Compon... [5] https://docs.microsoft.com/en-us/windows/win32/procthread/multimedia-class-scheduler-service https://docs.microsoft.com/en-us/windows/win32/procthread/mu...

scoutt 6y ago

I agree this is bad engineering: don't use Android threads for real-time stuff. This should be running at module level. A better approach, if the micro you are using allows it, is to have an interrupt when (for example) the I2S buffer is empty. I would then point the DMA to fetch the next buffer (already processed and mixed) and fire the DMA transfer. > at least 100ms This calls for huge latency problems. But I understand the approach if your buffer-filling/reading procedures are slow or unreliable. I disagree with the timer thing. There are systems that provide precise timers for media (for example Win32 multimedia timers[1], that I never used but I know they exist). [1] https://docs.microsoft.com/en-us/windows/win32/multimedia/about-multimedia-timers https://docs.microsoft.com/en-us/windows/win32/multimedia/ab...

ubercow13 6y ago
>This calls for huge latency problems It doesn't mean there is 100ms of latency, it just means that 100ms of audio is buffered so that you have ~100ms of leeway about when you app's audio thread is scheduled. Changes to the audio stream such, as stop/start/volume control, can be achieved with much lower latency using buffer rewriting or by applying changes lower down the stack where the buffers are smaller, or both. By default PulseAudio will buffer ~2000ms of audio from clients [1] [1] https://freedesktop.org/software/pulseaudio/doxygen/structpa__buffer__attr.html#aa7e8f3348cbda863b6f1dd55a9024b7a https://freedesktop.org/software/pulseaudio/doxygen/structpa...
- scoutt 6y ago
  It DOES mean there is 100ms of latency. Anything above 10ms (some say max. 20ms) for real-time audio processing (especially musical instruments) is prohibitive. Imagine an electronic drum set: if you hit the snare and the audio is output 100ms later from the speakers, I bet you'll notice it :)
  - deleted 6y ago
    [deleted]
  - ubercow13 6y ago