4 ms·
I think this problem would have been debugged and solved much quicker if they'd done a CPU scheduling trace. Then they could see, microsecond by microsecond,
by londons_explore 5y ago
I think this problem would have been debugged and solved much quicker if they'd done a CPU scheduling trace. Then they could see, microsecond by microsecond, exactly which processes were doing which work, and what incoming requests are still waiting.
Then, let a human go in and say "How come request#77 hasn't yet been processed at this point, even though CPU#3 is working on printing unused debug data for a low priority request and #77 is well after its deadline!??".
Then you debug deeper and deeper, adjusting parameters and patching algorithms till you can get a CPU trace that a human can look at and think "yeah, I couldn't adjust this schedule by hand to get this work done better".
In this process, most people/teams will find at least 10x performance gains if they've never done it before, and usually still 2x if you limit changes to one layer of the stack ('eg. Im just tweaking the application code - we won't touch the runtime, VM, OS or hypervisor parameters').
- ghusbands 5y agoThat does not cover almost anything in the article. It's a long article, so maybe you could quote the bit you're responding to. A CPU scheduling trace wouldn't easily show you the details of the kernel-level group throttling that was causing a lot of issues, for example. They weren't having an issue with threads fighting other threads, they were having an issue with threads being penalised now for activity from several seconds ago, drastically reducing the amount of available CPU. The article clearly shows a lot of debugging and diagnostic patching ability, so it's unlikely they missed the simple options. Rather, they probably didn't mention them because they were obvious to try and didn't help.
- londons_explore 5y ago> threads being penalised now for activity from several seconds ago, Exactly... They would have found this out much quicker with a trace. They would have seen "how come this application level request is being handled on thread number X, yet that thread is not running on any core, and many cores are idle"? Then quickly they could see the reason that thread isn't scheduled by enabling extra tracing detail seeing the internal data structures used by the scheduler to see why something is schedulable or not at that instant.
- ghusbands 5y agoI think you're suffering hindsight bias, here. A trace is rarely as clear as that, and it's hard to see the details it's not designed to expose. Your original message would probably be better received if you'd omitted the "I think this problem would have been debugged and solved much quicker [...]" and its insulting implications and instead started with "Sometimes, I find that CPU activity traces can really help with diagnosing this sort of problem".
- The_rationalist 5y agoPlease stop advocating for politeness over correctness. Sure hindsight help but regardless, a company such as Twitter should have experts at tracing that have tools and knowledge that goes beyond the average developer knowledge about tracing methodologies. Excusing that is an appeal to a lowering of technical excellence worldwide, which is majorly important and matter more than hypothetical feelings.
- londons_explore 5y ago> a company such as Twitter should have experts at tracing In a big company, getting the person with the most skills to solve a problem to be the one actually tasked with solving the problem is very hard. This particular problem had many avenues to find a solution - and while I think my proposed route would have been quicker, if you aren't aware of those tools or techniques, then other avenues might be much quicker. When starting an investigation like this, you don't know where you're going to end up either - if it turned out that the performance cliff was caused by CPU thermal throttling, it would be hard to see in a scheduling trace - everything would just seem universally slow all of a sudden.
- neerajsi 5y agoOn Windows, we have the xperf and wpa toolset that makes looking at holistic scheduling performance, including processor power management and device io tractable. Even then, the skillset to analyze an issue like the one presented here takes months to acquire and only a few engineers can do it. We have dedicated teams to do this performance analysis work, and they're always in high demand.
- neerajsi 5y agoI don't know why the negative reaction. I've done the kind of analysis you've described many times and essentially been able to quickly identify such issues over the years. We had a similar problem in Windows when we first implemented the Dynamic Fair Share thread scheduler. It took a couple months to have the right tooling to do a proper scheduler trace, but with that available the problem was better understood in a week. I eventually rewrote the scheduler component and added a control law to give better burstable behavior than the hard cap quota that this article seems to be describing.