16 ms·
The Case of the Extra 40ms
- leafmeal 6y agoI appreciate the humbleness of this writeup. The dev asked others for help explaining the code, and eventually the bug was solved by someone on another team completely! And there's no shame in that (nor should there be), but my pride or fear of inadequacy might have made me feel otherwise.
- rendall 6y agoSame. I realized after reading this that I do not ask for help enough
- di 6y agoWait, a 40ms bug that isn't due to Nagle’s algorithm? Is that even possible?
- Bayart 6y agoI expected something about TCP_NODELAY and Nagle's Algorithm, a recurring theme on HN [1][2]. I'm left surprised, but I also can't shake the idea that that arbitrary sleep might just be tailored after TCP. [1] https://news.ycombinator.com/item?id=24785405 https://news.ycombinator.com/item?id=24785405 [2] https://news.ycombinator.com/item?id=25133127 https://news.ycombinator.com/item?id=25133127
- aratno 6y agoImmediately searched for “Nagle”, “TCP”, etc but was surprised to see this is something else! Not the typical TCP_NODELAY issue.
- bobnamob 6y agoSaw 40ms was immediately expecting classic Nagles problems. For those with the same suspicion this isn't that.
- abbeyj 6y agoI was sure that I had read this story before but checking the date showed that this blog post was just published a few days ago. A little searching turned up the place that I had originally seen it: a comment on this very site. https://news.ycombinator.com/item?id=24799332 https://news.ycombinator.com/item?id=24799332
- deleted 6y ago[deleted]
- brundolf 6y agoHah, and my own deja vu was from that rachelbythebay post that this comment is underneath
- deleted 6y ago[deleted]
- ricardo81 6y agoIt reminded me of something I see on Windows also, with Netflix at least on Edge. Not sure how to reproduce entirely but I think it's if I pause and move off the browser tab and play, there's a good chance the frame rate will begin to stutter.
- jdblair 6y agoIndeed, writing that post is what inspired me to write the tech blog article.
- nabeelms 6y agoMy first thought when I saw the title was that it was due to Nagle's algo.
- pantelisk 6y agoI have struggled a lot with video streams decoding on Android. Every chip manufacturer does things differently. Some have a 11 frame buffer, some a 6 frame buffer, some will pipe data out after only a few frames. Let aside the tens of different YUV formats from diagonally interlaced streams to split-byte ones lest you want to work with the output directly and not let the decoder render to a surface directly (to avoid the latency of reading from the surface). Back then I ended up doing a lot of crazy things like piping "empty" NAL frames to the decoder, to force a flush among other things. If you really cared for low-latency specific streams it was very difficult to work with the fragmentation of the ecosystem. That was a few years ago, the versions that came at around 2016++ (after API 23) were the versions that became more stable, and relied less on OEM specific magic.
- jimmySixDOF 6y agoInteresting. Oculus team at FB has been playing with frame timings etc and just released 'Phase Sync' in Quest HMDs (Android) [1] so even though it's a highly customized ROM build, it is likely they face the same type of problems. [1] https://uploadvr.com/facebook-phase-sync-quest-latency/ https://uploadvr.com/facebook-phase-sync-quest-latency/
- soylentgraham 6y agoCan definitely concur with this; decoding on each compatible phone for gearvr was a nightmare, not to mention that decoding to a buffer instead of a texture surface was noticably faster. My code still has enums of magic numbers for S5's and note3 weird yuv formats
- dboreham 6y ago> that Android Threads are a userspace construct Whoa!
- mehrdadn 6y agoDoes anyone have more reading on this? Sounds interesting. I know Windows 7+ x64 has a user-mode scheduler but I hadn't heard of a Linux-based OS supporting this. How does it handle blocking calls and such?
- tmurray 6y agothey're not a userspace construct in any meaningful way. libutils/Thread (capital-T Thread) is a thin wrapper around pthread_t, and most Android code uses pthread_t directly. what this hit is probably a bug related to timerslack_ns--there's some code to tweak timerslack values on a per-thread basis instead of a per-process basis even though audio should never be coming from processes low enough on the importance list to get a high timerslack value--but L bugs fixed in M are before my time.
- dboreham 6y agoMy money was on: someone ported NT thread priority inversion to Linux.
- angry_octet 6y agoIt’s a scheduler bug (exposed by a race), but not due to priority inversion, as it wasn’t waiting on anything —- it was just being treated as a low priority task. Android itself just isn’t very suited to real time tasks.
- deleted 6y ago[deleted]
- watt 6y agoIf it's running on a TV, it better be. At that point it's not an plucky O/S running on underpowered hardware, trying to save battery.
- redis_mlc 6y ago> plucky O/S That's cute! I guess we haven't really moved beyond BeOs from 1990, have we? https://en.wikipedia.org/wiki/BeOS https://en.wikipedia.org/wiki/BeOS https://en.wikipedia.org/wiki/Haiku_(operating_system) https://en.wikipedia.org/wiki/Haiku_(operating_system)
- gumby 6y agoI have tried a bunch of steaming platforms (Netflix, Amazon, hbo max, Disney plus, Apple TV) and Netflix has the best playback experience (and overall experience), followed by Apple. They definitely take engineering more seriously than the others. We have considered ditching Netflix (content is meh for this household) but after using one of the other apps we frequently bail and find something on NF simply because the overall experience is better.
- paxys 6y agoInterestingly I have found the various Prime Video apps to consistently be the worst in terms of UI and performance. I imagine Amazon as a company also takes engineering and infrastructure seriously, but the results don't show it. HBO, Disney, Hulu, Apple etc. are all somewhere in the middle, and yes Netflix is FAR ahead of the pack.
- nl 6y agoI agree with this (Haven't tried HBO though). Disney streaming is pretty good, but the UI isn't as good as Netflix for use across the room. Why do they make the title cards have second level text that is so small it can't be read, and then not have the continually updated preview that Netflix does? I agree Amazon is pretty disappointing. It seems more like a dodgy pirate video player than a major competitor.
- seedless-sensat 6y agoMaybe I shouldn't be surprised, but Prime Video's Chromecast integration is just terrible.
- gpderetta 6y agoStill better than actually using a fire stick directly. At least the UI latency on the phone is bearable. My, admitedly old, fire sticks have input latency measurable in seconds. The firestick remote replacement app (because I can't ever find the actual remote) is also a joke: it disconnects all the time, fails to reconnect and it is generally not very usable.
- marcan_42 6y agoThis is still bad engineering on Netflix's part. You can't and shouldn't rely on your audio handler getting called on time via a timer in order to keep playback stable, especially not on a non latency sensitive use case, which Netflix very much isn't. If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts (through as many abstraction layers as you want, but a traceable event chain nonetheless, no sleeping guesswork or timers - and ideally all those threads should be marked real-time, never allocate any memory, and never lock or block on anything else). This is what serious real time audio systems like JACK do. And if you're not, you need to be capable of buffer filling way more than one frame's worth of audio - at least 100ms, preferably much more. Not just for robustness, but also for battery life. Sure, there was an Android bug here, but the way Netflix designed this is fragile and, as someone who has messed with audio/video programming enough, wrong. Had they done things properly, they would've been insulated from this OS bug. This kind of bug is best used as a learning experience: what happened here consistently and resulted in completely destroyed playback on one device, is something that has always been happening sporadically and hurting all of your users whenever something causes the CPU to stall or slow for long enough to miss the deadline anyway. Instead of just fixing the bug, make your software robust against these cases, and then all your users win.
- nemothekid 6y ago>If you're doing real time audio, your processing loop needs to be directly driven by audio hardware buffer events/interrupts Is this realistic? I don't know much about providing software for a wide range of hardware, but I would imagine this would require either Netflix or the integrator to write drivers for Netflix to communicate to the hardware dsp right (or standardize)? It seems more feasible to piggyback on a platform that has done that already and is already widely deployed.
- asdfasgasdgasdg 6y agoIf I'm reading correctly, they're saying that Netflix is not doing realtime audio, so they need not have this level of precision. They should buffer more than 40ms of audio data.
- fastest963 6y agoThey didn't say how they fixed it. I doubt the vendor updated the Android version. Did Netflix make a change to ensure they're always making the thread in the foreground?
- rendall 6y agoMy impression is that the bug workaround was for the integrator to make sure the application was in the foreground before calling the Ninja handler
- rkachowski 6y agoyeah I also hit this impression. despite all the investigation and detail it seems in the end it was an exercise in hand washing and blame shifting.
- spixy 6y ago> The integrator back-ported the patch (linked above) to L. It is literally only a few lines so this was straightforward. Some of the hardest problems take 1 or 2 lines to fix :)
- matchbok 6y agoMan, Android is just not a well-designed OS. This is silly.
- bcheung 6y agoThis bug write-up reads like a suspense novel. I cheated and jumped to the ending. It's finding bugs like these that give you an immense sense of satisfaction for discovering it once you get through all the frustration. Well done!
- paxys 6y agoAt least tell us how you fixed it!
- aluminussoma 6y agoAndroid frustrates me on many levels, but I appreciate having the source code to dig into. If the same issue happened in an iOS device, Netflix would need to do the workaround.
- random5634 6y agoInteresting - I wonder if not having the source code means coreaudio and video playback on apple is actually better behaved across the entire ecosystem of devices vs android. I've never seen a music production app I don't think on android now that I think of it.
- zarmin 6y ago> I've never seen a music production app I don't think on android now that I think of it. Have you looked for any? https://play.google.com/store/apps/details?id=com.imageline.FLM&hl=en_US&gl=US https://play.google.com/store/apps/details?id=com.imageline....
- perryizgr8 6y agoSo how did they finally fix it? Did they do the workaround of loading more than 1 frame of data? Or did the vendor upgrade to a more recent AOSP? Or did they patch the fix into their version?
- Codesleuth 6y agoI get this exact same problem on my SHIELD TV 2017 with all latest updates. Makes me think I should be reporting it somewhere.
- rendall 6y ago"The integrator and Netflix had already completed the rigorous Netflix certification process..." That part confused me. Netflix completed its own certification process? Does the author mean himself, as "Netflix"?
- JohnBooty 6y agoThis story really exemplifies an aspect of my job I love: I can’t predict all of the issues that our partners will throw at me, and I know that to fix them I have to understand multiple systems Reading things like this makes me realize I've got to get out of this business at some point. That is not a part of the job that I love. That is a part of the job that makes me want to bash my brains out with a rock. I actually do sort of like that kind of problem-solving, in the abstract, buttttttt in reality I usually need to do it under pressure for unappreciative clients/management who just know that things are taking longer than expected. This is of course a communication challenge, and I am a good communicator in general, but... I'm kind of burned out on it.
- ralfd 6y agoI bet that he really meant "This story really exemplifies an aspect of my job I hate" and won't miss at a future next job. The positive spin was only there because it got on the company blog.
- JohnBooty 6y agoHahaha. I was wondering the same thing. Did he really, truly mean that? Was it sarcasm? Was it something PR asked him to add? Was it something PR added without asking him?
- jrochkind1 6y agoI am curious what tools were used to make the nice visualization. I lack skills in making visualizations, and do regularly run into cases similar to this one where it would help me understand or diagnose something to chart some somewhat complex data like this.
- leafmeal 6y agoAny spreadsheet software should get you pretty far. Most languages have some graph/chart making libraries too.
- layoutIfNeeded 6y agoThe vendor was right: you’re using unreasonably short buffers for a non-interactive use case (i.e. video playback). With such short buffers (16.6ms) and a user-space timer (i.e. basically usleep) driving the whole thing I’m actually surprised that this didn’t broke earlier...
- hallgrim 6y agoHuh. I wonder if this is also the reason why my unity game stuttered below 30fps even in menus on a galaxy s6 until I put it to sleep and woke it back up.
- gambiting 6y agoUgh, if only other companies cared that much :-( Amazon Prime app on LG TVs has had lip sync issues for at least 4 generations of their TVs, problem has been reported both to LG and samsung, and no one cares enough to fix it. When using the app on my LG CX the audio is about 500ms behind, it's an insane amount of missing audio sync and no one cares to fix it? No other apps do that, it's specifically amazon prime, but I guess it doesn't sell their Fire Sticks so hey, who cares?
- mastercheif 6y agoThe YouTube app also has horrendous lip-sync issues on LG WebOS. They don't care.
- gambiting 6y agoYeah I saw, but Netflix is pretty much perfect(I think it has maybe a frame or two of audio delay, but not enough to notice most of the time), so clearly it is technically possible, just that these companies don't care about sitting down and fixing it.
- gambiting 6y ago*LG and Amazon, no idea why I said samsung :P
- bambax 6y agoA little OT: often when chasing bugs, the diagnosis represents 99.99% of the work, and the fix is completely trivial. I recently spent over a week trying to find why a Windows PC was acting erratically in very specific circumstances and seemed to work fine the rest of the time. It was (here also) a problem with timers. Once the problem was identified, correcting it took 15 seconds. I sometimes worry that in healthcare it's the other way around: doctors do an immediate diagnosis and then months (or years) are spent trying to fix what they think needs fixing. Isn't it possible that we are doing it wrong and are not actually getting to the bottom of things?
- yunruse 6y agoThe reason we can make medical diagnoses so efficient is because we have had the collective experience, and that the human body does not vary radically from one person to another such that it would be hard to keep track. While a doctor can’t be expected to know every intricacy of the human body, it has been catalogued and studied extensively over thousands of years, and this information has been made available for quick reference. Between experience and the rigorous education, doctors will know a significant amount of detail such that they can diagnose (or confidently know how to delegate this) fairly quickly. It’s not perfect, of course, and you are right that sometimes the symptoms are fixed rather than the causes. Compare, however, to software: unlike the human body which has numerous small variations but a common base, software is free to take many different forms and frameworks. This software continues to change over time, so while documentation certainly can exist, it can be immensely fragmented and describe complex behaviours. Plus, when software developers fix an issue it often tends to stay fixed; there is a use for a postmortem but no immediate necessity. But in people, this has to be diagnosed every time it is encountered, so it is natural that documentation will occur. (This isn’t the perfect metaphor, of course, but I hope it roughly corresponds.)
- tomaszs 6y agoI am curious what caused opening the thread in the background. Seems like a race condition of sorts. Anyways fascinating read. I love to debug and read about such cases.
- evnix 6y agoThere are a few websites who spend so much on CDNs, static site generation and other crazy techniques to lower a few milliseconds. But then you have the same websites, cluttered with Ads all over and those super annoying pop ups, ultimately slowing down the experience.
- rossmohax 6y agoQuote from the blog: > I walked upstairs and found the engineer who wrote the audio and video pipeline in Ninja, and he gave me a guided tour of the code. That's what I miss working from home. Random, spontaneous, productive interactions with coworkers.
- detaro 6y agoWhile it's not quite the same, remote has the benefit that you can do that with people not working in the same building too. (e.g. where I work spontaneous screen share sessions are totally normal)
- obblekk 6y agoThis is a really fun read - I like the core insight of graphing the log statement timings to see what's going on.
- cratermoon 6y agoI love this. I'd like to know a bit more about how it was fixed, but the story of diagnosing and proving the problem is great. Sadly, Medium is now more full of crap than a herd of circus elephants, with tech blogs written by people who have nothing under their belt but a code camp, but this is a diamond in the rough.
- jrochkind1 6y ago> Next I started reading the Ninja source code. I wanted to find the precise code that delivers the audio data. I recognized a lot, but I started to lose the plot in the playback code and I needed help. > I walked upstairs and found the engineer who wrote the audio and video pipeline in Ninja, and he gave me a guided tour of the code. THERE IS NO SUBSTITUTE FOR DEVELOPER CONTINUITY. Yes, you want your code as good as possible, your docs as good as possible, your knowledge transfer as good as possible. But there is ultimately no substitute! Domain knowledge matters, and takes years to develop. Familiarity with the codebase matters, and in a complex and changing codebase is a process that is never complete. Having access to the people who wrote the things in the first place matters. These facts are often neglected in a world where developers are treated like swappable commodities, and it's considered normal to job hop frequently. Organizations that treat developers well and give them paths of advancement/reward within the organization to encourage developers to stick around for a while will reap benefits.
- titanomachy 6y agoGood observation. I've experienced the consequences of turnover on my current team. The product was mostly built 5 years ago, but no one on my team has been here more than 1.5 years. Getting even basic things done requires extensive codebase archaeology. It sucks. I don't necessarily think you need the people who wrote the code originally though. As long as there's enough overlap between generations, and deliberate effort to train people on different parts of the system, knowledge can be preserved. That's probably not possible if everyone changes teams after 2 years though, except for relatively simple or highly standardized codebases.