5 ms·

Author here: Fair criticism. There are some details that are missing in the interest of keeping the story simple. 1) The audio and video playback is driven by

Author here: Fair criticism. There are some details that are missing in the interest of keeping the story simple. 1) The audio and video playback is driven by the hardware, using the audio stream as the master clock. 2) Ninja does fill the Android and hardware buffers and keep them full. In the pathological case, the hardware and Android buffers had emptied. 3) When the Android buffer isn't full, the thread yields to Android (to be a good citizen) but asks to be invoked again right away. The 40ms "background thread" delay broke this behavior. The comment about "changing this behavior involved deeper changes than I was prepared to make" was when I explored changing this behavior (copying multiple samples per invocation) and decided it was more likely to introduce more bugs.

jdblair 6y ago
The "buffering" parts of the graph show the "fill the buffer" behavior described in (3). The time between calls is very short and large amounts of data are moved.
- marcan_42 6y ago
  How does the buffering proceed quickly, but then the (slower than usual) periodic calls only copy a limited amount of data? That's the part that doesn't make sense to me. Normally if you're filling buffers, you copy as much data as is needed to keep the buffer full, no matter when you're invoked.
  - carlmr 6y ago
    That is indeed interesting. The pipeline should be filling up if the OS scheduler delays the buffering for 40ms, giving enough data on the next call. Except if the buffering isn't greedy enough or aborted by the OS in some way.
  - jdblair 6y ago
    The handler always copies the same amount of data. It has to be called more often to copy a lot of data, which is what usually happens when Android buffer isn't full. I agree with you that this isn't usually how this is done. Doing it this way means that Android has more freedom to schedule other tasks during the buffering phase.
    - marcan_42 6y ago
      Okay, so the real problem here is there was effectively a yield() inside a buffer fill loop with a small block size, and Android decided to make each of those cost 40ms. The article made it sound like you were always waiting 15ms, but you were actually waiting 0ms when there was more work to do. I still think the blocksize should've been larger for various reasons in this case (it's a trade-off still, usually larger block sizes are mildly more CPU-efficient, besides preventing pathological scheduling cases like this one) but this explanation makes more sense.
      - jdblair 6y ago
        Yes, that's all correct. The reason for the small block size has to do design decisions made in the rest of the streaming stack, which is shared between all TV devices.
bambax 6y ago
Excellent article btw! But why did the problem only appear on this particular TV and not any other using the same version of Android? Was this the only TV to use Lollipop?
- jdblair 6y ago
  Yes. This was the only TV device on L. It was already 2 years old at this point, which for pure Android TV would not be allowed by Google. Since this was an AOSP device, the operator could stay on an older version.
  - social_quotient 6y ago
    Quick simple question for you. What does the TV testing landscape look like at a Netflix? Is it hundreds of various TVs in a room on a variety of networks or am I imagining too much here?
    - jdblair 6y ago
      TV and set top box testing is shared between our partners (manufacturers and operators) and Netflix. Our partners must run our test suite before submitting for certification. We don't have all the devices under active testing all the time, but almost all of the devices have been tested at Netflix at some point. Most operator devices require VPNs to the operator network to function. So yes, hundreds of devices on a variety of networks, but at any given moment most of the devices are in storage.

jdblair 6y ago

The "buffering" parts of the graph show the "fill the buffer" behavior described in (3). The time between calls is very short and large amounts of data are moved.

marcan_42 6y ago
How does the buffering proceed quickly, but then the (slower than usual) periodic calls only copy a limited amount of data? That's the part that doesn't make sense to me. Normally if you're filling buffers, you copy as much data as is needed to keep the buffer full, no matter when you're invoked.
- carlmr 6y ago
  That is indeed interesting. The pipeline should be filling up if the OS scheduler delays the buffering for 40ms, giving enough data on the next call. Except if the buffering isn't greedy enough or aborted by the OS in some way.
- jdblair 6y ago
  The handler always copies the same amount of data. It has to be called more often to copy a lot of data, which is what usually happens when Android buffer isn't full. I agree with you that this isn't usually how this is done. Doing it this way means that Android has more freedom to schedule other tasks during the buffering phase.
  - marcan_42 6y ago
    Okay, so the real problem here is there was effectively a yield() inside a buffer fill loop with a small block size, and Android decided to make each of those cost 40ms. The article made it sound like you were always waiting 15ms, but you were actually waiting 0ms when there was more work to do. I still think the blocksize should've been larger for various reasons in this case (it's a trade-off still, usually larger block sizes are mildly more CPU-efficient, besides preventing pathological scheduling cases like this one) but this explanation makes more sense.
    - jdblair 6y ago
      Yes, that's all correct. The reason for the small block size has to do design decisions made in the rest of the streaming stack, which is shared between all TV devices.

bambax 6y ago

Excellent article btw! But why did the problem only appear on this particular TV and not any other using the same version of Android? Was this the only TV to use Lollipop?

jdblair 6y ago
Yes. This was the only TV device on L. It was already 2 years old at this point, which for pure Android TV would not be allowed by Google. Since this was an AOSP device, the operator could stay on an older version.
- social_quotient 6y ago
  Quick simple question for you. What does the TV testing landscape look like at a Netflix? Is it hundreds of various TVs in a room on a variety of networks or am I imagining too much here?
  - jdblair 6y ago
    TV and set top box testing is shared between our partners (manufacturers and operators) and Netflix. Our partners must run our test suite before submitting for certification. We don't have all the devices under active testing all the time, but almost all of the devices have been tested at Netflix at some point. Most operator devices require VPNs to the operator network to function. So yes, hundreds of devices on a variety of networks, but at any given moment most of the devices are in storage.