12 ms·
24-core CPU and I can’t type an email – part two
- agumonkey 8y agoI wonder how much cpu time has been liberated through the CFG scan patch.
- atesti 8y agoWhat exactly did his it department do using wmi to query this Win32_PerfFormattedData_PerfProc_ProcessAddressSpace_Costly? I can't find this in the article
- garaetjjte 8y ago> "By the way, the reason that the hangs kept happening at 10:30 am is just because that’s when our IT team ran their inventory scans. If you want to trigger a scan manually, you can go to Control Panel | Configuration Manager | Actions | Select ‘Hardware Inventory Cycle’ and then Run Now."
- brucedawson 8y agoThe actual high-level query is in part one and in the first reply, but I think your real question is "why did they want to scan the address space of every process on the system?" I think that the answer is that they didn't. That was just one of the bazillion counters that came along for the ride. I believe that that counter has been removed in the latest OS, thus squishing this bug in another way. I don't understand WMI, but it sounds really weird. One peculiarity is that once some program asks for counters "WMI refreshes the list of counters every 2 minutes until the WMI helper process closes due to inactivity." So that's great. IT asks for some data, they get memory scans that they don't want, and those scans are repeated every two minutes for a while (ten minutes?) even though nobody is looking at the results.
- dongerfan 8y agoCongratulations on your article, as well.
- snaky 8y ago> the lock was being acquired and released ~49,000 times and was held for, on average, less than one ms at a time. But for some reason, even though the lock was released 49,000 times the Chrome process was never able to acquire it. Well, locking is hard. > The good news is that even though there is occasional unfairness, there is unlikely to be persistent unfairness. In order for a thread to steal the lock, it needs to hit the tiny window where the lock is available. In practice, a thread is unlikely to be this lucky repeatedly. https://blogs.msdn.microsoft.com/oldnewthing/20170705-00/?p=96535 https://blogs.msdn.microsoft.com/oldnewthing/20170705-00/?p=... > The fact is, any time anybody makes up a new locking mechanism, THEY ALWAYS GET IT WRONG. Don't do it. Take heed. You got it wrong. Admit it. Locking is _hard_. https://yarchive.net/comp/linux/locking.html https://yarchive.net/comp/linux/locking.html
- AstralStorm 8y agoActually this is exactly because someone in Windows used the plain old mutex. I'd call that "your granddad's lock". It's old, crotchety, unfair and shouldn't be used in situations where any concurrency can be expected. This despite the kernel having a nice RCU mechanism inside as well as waitfree queues.
- snaky 8y agoIf there was a spinlock, the problem stay. > the rule simply is that you MUST NOT release and immediately re-acquire the same spinlock on the same core, because as far as other cores are concerned, that's basically the same as never releasing it in the first place. https://yarchive.net/comp/linux/spinlocks.html https://yarchive.net/comp/linux/spinlocks.html Other mechanisms do exist of course.
- geezerjay 8y ago> I'd call that "your granddad's lock". It's old, crotchety, unfair and shouldn't be used in situations where any concurrency can be expected. What alternatives do you suggest?
- gpderetta 8y agoI guess: s/concurrency/contention/
- jwilk 8y agoPart one: https://news.ycombinator.com/item?id=17780127 https://news.ycombinator.com/item?id=17780127
- lmilcin 8y agoI have on my desk a machine that would be an equivalent of a large datacenter two decades ago and then sometimes it is barely able to keep up with me pressing keys on the keyboard. I think the next major step will be for the humanity to learn make software more efficient and effective rather than throw more CPU cycles at the problem.
- snarfy 8y agoIt will happen naturally. Throwing hardware at the problem is still the most economical thing to do. Features and speed of development still beat performance. When that's no longer true we'll focus on performance again.
- Kurtz79 8y agoIn the last few years we switched from throwing more CPU cycles at the problem to throwing more CPUs at the (not necessarily parallelizable) problem.
- maccard 8y agoEach of the 40 cores on my workstation runs at 3.5 GHz. I had a quick look and I found reference to Google using pentium 2 CPUs in their data centres in 1999. According to Wikipedia, they max out at 450MHz. So my workstation is 8 times faster per core, and has 40 more cores. It's hard to justify the differences really.
- Kurtz79 8y ago20 years is equivalent to a couple of geological ages in tech, definitely not the "last few years" I was referring to. This article is from 2014 but hardly anything has changed. https://www.comsol.com/blogs/havent-cpu-clock-speeds-increased-last-years/ https://www.comsol.com/blogs/havent-cpu-clock-speeds-increas... https://superuser.com/questions/543702/why-are-newer-generations-of-processors-faster-at-the-same-clock-speed https://superuser.com/questions/543702/why-are-newer-generat... https://i.stack.imgur.com/z94Of.png https://i.stack.imgur.com/z94Of.png
- raattgift 8y agoCould you build on the always-unfair system along these lines? (C11ish, sorry) void do_work(the_work_t *w) { /* for simplicity here, rather than e.g. w->contenders */ static _Atomic int contenders = 0; contenders++; for ( ; work_remaining(w); ) { take_mutex(w->m); do_some_work(w); drop_mutex(w->m); if (contenders > 1) reschedule_this_thread(); } contenders--; } This depends on the OS providing a cheap and fast reschedule_this_thread() mechanism that effectively guarantees that if there is only one other contending thread with work, that thread will end up holding the mutex. (If there are multiple such threads, an arbitrary one of them will end up with the mutex, rather than the thread that just dropped the mutex.) One could of course only check for other contenders every few times through the for loop if reschedule_this_thread() is expensive or slow, or if contenders is especially hot. contenders is explicitly not a locking mechanism and should not influence the policy of any code running while the mutex is held. It should also be a per-mutex counter.
- brucedawson 8y agoThe trick is how you implement reschedule this thread. What you want in this specific case is to take it off-core long enough for another thread to wake up and take the lock. That is far too squishy a goal to be something that you can implement. If you sleep for some number of nanoseconds then you are wasting performance and/or not sleeping long enough. In this particular case the lock was a kernel lock in kernel code so the OS would have to fix this, by making the locks fair (or occasionally fair).
- raattgift 8y agoI've been spoiled by a better OS. :D How about: while work: if (contenders > 1) { reduce_priority; pri_reduced = true } take_lock if (pri_reduced) { unreduce_priority; pri_reduced = false } do_work_quantum drop_lock endwhile (I mean empirically, although an educated guess would do). Of course, if reducing priority is too fast, this likely doesn't help; alternatively it could be too slow and what you get back in system latency is taken away in lowered throughput. That's probably not OK if you don't need the system latency to be low. I wonder if (dramatically, even) reducing the priority of some of the original workload exposing the problem, not just when racing for a lock but when doing the actual work quanta, would help. My thought here is that your email-sending is higher-prirority and will at least push some of the workload out of the way in reasonable time, giving you back some responsiveness. I'm surprised if Windows doesn't offer up a high-throughput/latency-tolerant QOS for threads.
- snarfybarfy 8y agoYou should have asked your pointy haired boss. He could have explained the problem in much simpler terms. Would you expect 24 employees to write ONE email without 4 team leads and one department head? Obviously NO!! Your processors obviously need more management. I think Intel has the right offering for you, aka management engine.
- Scarblac 8y agoNew from Intel: the Scrum processor.
- bluejekyll 8y agoYou joke, but seriously, this is generally a goal of distributed and/or parallel computing. Reduce interdependencies, stop constant cross chatter and try to do as much computing in isolation as possible. Scrum’s not necessarily a bad anology. Here’s a different thought that this conversation has me now thinking about, what if we did think about development teams as CPU cores? We might discover weak points in the architecture of an organization, and recognize more quickly where we need to address bottlenecks. The bandwidth of the bus betweeen the cores might be too limited. The pipeline of work (backlog) might not be deep enough, and has a ton of branch statements (spikes) that may throw out the entire pipeline...
- nextos 8y agoWhy not just embracing simplicity? I have a golden rule for my systems. After booting into X and opening one terminal with htop and hiding kernel threads, everything should fit very comfortably in one screen. This constraint forces yourself to have a very simple setup. A few daemons, a window manager and a terminal. I do the rest of my computing in Emacs and Firefox. I have several Arch or NixOS setups that could work on a 128 MB RAM setup, excluding Firefox. Plus, if something breaks, I know how to fix it.
- bluejekyll 8y agoOrganizations like to grow, though. Wouldn’t that imply that organizational structure would have a maximum size?
- coldcode 8y agoMy MBP only has 8 cores but often times during the day it locks up and blows up a hurricane of fan noise. Why? Idiotic Jamf management software and a virus checker that never finds anything. Some days you only get 1 core for work when it gets stuck and have to reboot to get it back. It's not always the computer, OS or the software you use.
- posixplz 8y agoYep, Jamf and McAfee are both trash-quality software that significantly reduce the overall security of a system. McAfee is a well-known story: their AV unpacks and analyses potential malware via a kernel module. Wtf? Jamf's code quality and security model are abysmal. It blows my mind that apple recommends the use of Jamf to large Mac shops. Having peeked under the hood at previous employers, I was extremely disappointed. The product is insecure by design - not what one wants for device management that's given root privileges on machines containing corporate crown jewels. I highly doubt their operational paradigm for Jamf cloud has changed in the last year, either. I apologize for going completely off-topic, but these products make my blood boil. Especially because I now work for a shop that uses both Jamf and McAfee for Mac "security". On a positive note, I removed the corporate mandated malware with ease - no boot to single-user required.
- sfink 8y agoThere was a time that running Windows absolutely required you to use an antivirus solution. It was crazy not to. Those days are past. Now you have to be crazy to run one. Even setting security aside, I think antivirus packages are the number one source of instability and weird performance problems.
- ubermonkey 8y agoIt's tangential here, but, seriously, let me ask: For those of us here with > 15 years in the business, when's the last time you really felt a BIG uptick in performance or responsiveness from a new computer upgrade? I buy a new laptop every 3-4 years, and I buy a nice one, but generally speaking I feel like we've been kind of flat in terms of usable power for a while. 6 or 8 years ago I switched to an SSD, and THAT was a big deal -- easily the most dramatic uptick in performance since I moved from an AT clone to a 386 in 1991. But since then? Not so much. My laptop is smaller. It runs longer on the battery, and is generally cooler. The screen is better. But in terms of how long it takes to boot, or open large data sets, or whatever? Not so much different than 5 years ago.
- stefanfisk 8y ago2014 I replaced my MBA 11" with a maxed out MBP 15", and the performance difference was crazy. Buuuut, my next laptop (if I stick with Apple) will probably be a MB 12" with medium to low specs, with most of my demanding dev stuff moved to a Linux server.
- thedz 8y agoI actually feel a big difference when I upgrade, usually on the same 3-4 year cadence. Admittedly, the most recent cycle felt somewhat the same, but generally I see a boost in quality of life. Not things like how fast a script runs on a large dataset, but like how responsive my machine is _while_ that script is running, or _while_ I'm doing some intensive task.
- wenc 8y agoUI and user-facing elements are largely dependent on perception and I/O (human and computer), and the band of improvement there is narrow. Also, I/O is often the bottleneck, which is why most people perceive a speed increase moving from spinning platters to SSDs. If you move stuff to RAM, I bet you'd see another perceptible speed increase. (I've recently started using the RAM-backed /dev/shm on Linux for transient, throwaway data generated by my programs, and boy it is fast.) I have a Linux machine running on a Core 2 Duo machine (2005) at home and an Core i7 (2017) at work. An "ls -la" feels about the same on both. But when I try to train an ML model, the difference is stark and perceptible.
- fencepost 8y agoI'll have to go through with some of the same steps to see if a recent problem I had is related to this. In my case it was actually being triggered by a piece of monitor management software that came with the LG ultrawide monitor that I use. No significant CPU load, no memory issues, plenty of cores in an older HP workstation with a Xeon Processor, 48 gig of RAM and a Samsung SSD. When that display management software was running a text editor couldn't even keep up with displaying text as it was typed. Edit: rereading some of the original 2 articles, yeah, I'm not going to be running the same kinds of tests - even if I did it'd take too long to develop the knowledge base to be able to interpret my results adequately.
- equalunique 8y agoI really enjoyed the first iteration of this article and am happy to see a part two.