46 ms·

24-core CPU and I can’t move my mouse

1011 points by joebaf 9y ago

microcolonel 9y ago
The terrifying result of having ABI compatibility with the first commercially successful system of its kind from the '80s. Granted, they aren't doing themselves any favours with their new straight-jacket style application ABI.
- willvarfar 9y ago
  You are trotting out the same tired line without reading the article carefully? ;) This is a regression. The article shows it worked fine up to Windows 7.
  - Aeolun 9y ago
    7 is the best
    - option_greek 9y ago
      Regardless of how many windows/any other OS versions I use later on, I will remember Win7 fondly. Its just so.. less cumbersome. May be the drastic changes that went into its preceeding and succeeding versions have a role to play in this.
      - krylon 9y ago
        There is this old quote that Algol 60 was an improvement not only on its predecessors but also its successors. The same could be said about Windows 7, I guess. ;-)
- jbg_ 9y ago
  How is ABI compatibility with early systems the cause of this bug? You'll note that it wasn't present in Windows 7.
- LeoPanthera 9y ago
  64-bit versions of Windows 10 (and 8/7 maybe?) are no longer compatible with 16-bit code.
  - ripdog 9y ago
    Yep, no 64-bit Windows has 16-bit compatibility.
- deleted 9y ago
  [deleted]
yourstruly33 9y ago
These posts are getting boring.. the only thing that they're saying is: I can use ETW and to me being told that I'm smart and capable is paramount.
- digitalnomad91 9y ago
  Then why are you commenting? Kind of a crappy attitude you have there.
  - alacombe 9y ago
    not really, I was kinda expecting the post would also contain interactions with Microsoft, leading to the resolution of the bug in update XX. The post kinda left me unsatisfied.
    - kirillkh 9y ago
      You can fix those problems by writing a post that would contain said interactions, resolution and update.
    - digitalnomad91 9y ago
      Sure, and you're entitled to feel that way, but that doesn't make his attitude any less crappy.
    - BinaryIdiot 9y ago
      While I too am anxious to hear the outcome after Microsoft investigates the author still shared something they were interested in of which holds a value to the HN community as many of us love digging into a system and finding issues. This was the main purpose of the article. Regardless the author may not know how critical Microsoft has classified this issue. It could be something they don't resolve for years. Since the Microsoft investigation and possible patch has no barring on the content of this article I wouldn't expect it to be held until such a time. I'm sure we'll get a follow up on HN when or if it gets patched. Even better maybe posting this article helps get it fixed sooner :)
    - 9y ago
yuhong 9y ago
I can't imagine that moving the Win32k stuff back to CSRSS would help much in this case, right? Though it is still a good thing especially for terminal servers where hopefully one CSRSS process crashing just terminate the session.
- sjmulder 9y ago
  > moving the Win32k stuff back to CSRSS Out of curiosity, was there some big move of functionality from CSRSS into Win32k earlier? When and what?
  - yuhong 9y ago
    In NT4.
isatty 9y ago
I read mouse as house and was confused for the longest time.
- mjsweet 9y ago
  Same happened to me... initially I thought it must be some 1st world problem.
aramadia 9y ago
When I saw his workstation specs, I thought, thats the exact same one I have at work! Then I checked the bottom, yep, he's at Google too.
- unixhero 9y ago
  Lucky bastards
- willvarfar 9y ago
  Does yours run Windows? Do you get to choose? If you could choose, which OS would you use? Any clear favourites with your peers for development? Just collecting anecdotes ;)
albertzeyer 9y ago
I was wondering what ETW is, and found in one of his other posts: ETW (Event Tracing for Windows). He seems to be one of the main contributors of https://github.com/google/UIforETW https://github.com/google/UIforETW Seems to be quite useful. --- About the post: tl;dr: NtGdiCloseProcess has a system-wide global lock which is used quite often e.g. during a build of Chrome which spawns a lot of processes. This problem seems to be introduced between Windows 7 and Windows 10. I thought that there would be a solution or a fix but it seems this is not yet fixed. "This problem has been reported to Microsoft and they are investigating."
- yuhong 9y ago
  I wonder if this is the same lock most win32k call takes.
- masklinn 9y ago
  > NtGdiCloseProcess has a system-wide global lock which is used quite often "Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates. This is compounded by parallelised build processes as not only do they spawn lots of processes leading to lots of process destruction: it both fucks up the UI and completely serialises process destruction so your nice 24 process flights for faster processing end up taking multiple seconds to all shut down, and the more cores (and thus processes) you have the worse it is, and the worse your stutters.
  - qb45 9y ago
    > "Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates. Do you happen to know anything about it? I'm scratching my head how it's possible that process termination serializes with GUI... Maybe it hogs some lock on process descriptors which SendMessage also needs to grab for a moment to find the target process? I hope you didn't mean to say that every SendMessage call is completely serialized with each other.
weinzierl 9y ago
I grew up on the Commodore 64 (1 Core, 1 hyper-thread :-), almost 1 MHz clock freq, almost 64 K usable RAM). The machine was usually pretty responsive, but when I typed too quickly in my word processor it sometimes got stuck and ate a few characters. I used to think: "If computers were only fast enough so I could type without interruption...". If you'd asked me back then for a top ten list what I wished computers could do, this would certainly have been on the list. Now, 30 years later, whenever my cursor gets stuck I like to think: "If computers were only fast enough so I could type without interruption..."
- fredley 9y ago
  It's all just trade offs. However fast or powerful your machine is, software will use as much of that resource as possible, up to the point where it occasionally interferes with input (but not too much or you'll switch to something else).
  - anigbrowl 9y ago
    But why? What good is an operating system on a multi-core device that allows anything to get that close to the performance envelope? This is a fine example of competition driving change for change's sake rather than real innovation and everything ending up worse as a result. I like new features as much as the next person, but not when they compromise core functionality. Not being able to type is inexcusable.
    - TeMPOraL 9y ago
      I agree. However at this point, I don't see anything an OS can do to help. I see plenty of typing slowdowns every other day now. But I'm not sure just how many of them are the OS's fault. When your typing seems to lag, there are two places that can be slowing it down - the input side (reacting to hardware events) and the output side (drawing and updating the UI). I suppose keyboard buffers are pretty well isolated, and native UI controls tend to work fine too. The problem is, everyone now goes for non-native controls. You type in e.g. Firefox, and it is slow not because of your OS, but because Firefox does all its UI drawing by itself. And God help you if the application you want to use is done in Electron. There's so many layers of non-nativeness on top of that, that the OS has close to zero say in what's being done. There's no way to help that - resource quotas will only make the problem worse, and giving such program free reign will only take everything else down. All in all, it's just - again - problem of people writing shitty software, because of laziness and time-to-market reasons. Blame "Worse is Better".
alacombe 9y ago
So what now ? Should/does the author wait getting some traction on Hacker News and hopefully be noticed by dev at Microsoft, or is there some way to provide Microsoft directly these data, skipping Tier-1 support ? The author seem to be working at Google, so he might get some leverage from that, but what about Random Joe ? Keep enjoying the bug "forever", I guess ?
- jsnell 9y ago
  From the article: > This problem has been reported to Microsoft and they are investigating.
- BinaryIdiot 9y ago
  Did you read the article? The bug was already submitted to Microsoft and they are investigating. So the answer to your question is: wait until they're done investigation and possibly release a patch.
  - alacombe 9y ago
    I skipped the conclusion, though, I'd still be interested about the channel that could be used to report such issues.
- ximeng 9y ago
  There are many many MS bugs that last for years. Pretty much any time you search for a bug in MS software you will find someone in a support forum giving generic advice (known issue that we will work on, not a bug, please reinstall Windows and applications) and users complaining. Typical example: https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10374741-stop-excel-from-changing-large-numbers-actually https://excel.uservoice.com/forums/304921-excel-for-windows-... basic problem that has been there many many years. For this one there are workarounds but many others there are not. The only real long-term solution is supporting competitive products.
  - intoverflow2 9y ago
eecc 9y ago
so basically a fork-bomb. I think Linux can still buckle under one, nothing that obscene...
- mattnewton 9y ago
  No, the article says this is a regression in how processes close.
- masklinn 9y ago
  A fork bomb exhausts all system resources, this is not that, it doesn't actually use any resources, rather it locks the system out of them by serialising both process termination and UI updates.
  - eecc 9y ago
    Well, a fork bomb doesn't really use anything except it makes the process table management extremely time consuming. Not quite the same, but still conceptually similar
yanpanlau 9y ago
Just use linux
- jlebrech 9y ago
  that's asking for downvotes, I voted this up nonetheless.
- gurkendoktor 9y ago
  Staying responsive (under load) is also still a work in progress on Linux: https://www.phoronix.com/scan.php?page=news_item&px=BFQ-Queued-Linux-4.12 https://www.phoronix.com/scan.php?page=news_item&px=BFQ-Queu...
  - dispat0r 9y ago
    That is a disk scheduler and mostly for spinning disks. I haven't seen a stuck mouse cursor for a long time in linux.
    - hypervis0r 9y ago
      > I haven't seen a stuck mouse cursor for a long time in linux. That's because the only cursor you have in Linux is the terminal one.
    - goodplay 9y ago
      I always found this peculiar about linux; even when the system is swapping hard and application windows/window managers completely freeze, the cursor always remains responsive and movement rendered without so much as a hitch. I wonder if xorg has some sort of kernel support to enable this.
      - gens 9y ago
        Not on my computer. Something like a WM will freeze while it waits in queue to read those 0.2kB it needs of the disk (poor software design, if you ask me). But the mouse will freeze when IO buffers get filled. Just writing a huge file to a (slow-ish) usb stick will make my whole computer freeze, including the mouse, because the kernel usb code doesn't limit the buffer to some sane size (there was a kernel patch, and there is an option, and even with it turned on the problem is still there) (note that the probable reasoning for that is the fact that usb sucks). A mouse cursor, AFAIK, is a gpu thing, not that it matters (wayland, i remember something that, will use normal gpu rendering to render the mouse). In the UNIX haters handbook there is a section about X where it is written that displays used to have 2-3 planes (IIRC, 2 planes + a cursor plane. I do recommend reading the good book, as it is funny). I remember some talk about better kernel buffers management just for things like this. Memory management in the kernel is one of those actually hard things.
SigSegOwl 9y ago
Thats why my machine gets that slow after running for weeks... explains everything!
blunte 9y ago
This is great work. I hope MS can improve Windows 10 by fixing this. I just added a new Win10 laptop with much better specs than my 3 year old rMBP, and I'm shocked by how much apparently random latency I experience with the UI in Windows 10 compared to the Mac. That's not to mention the issues of sloppier track pad (which constantly detects my left hand while I type) or the ungodly slow unzip (via 7z). If only Apple would give us more than 16GB of RAM (in a laptop)... what a frustrating world for developers.
- ShinTakuya 9y ago
  I mean the trackpad issue is generally down to the manufacturer and the drivers they provide, but I take your point otherwise. A pro tip with 7z is to use the two pane interface to extract and not dragging files to Explorer. The latter option extracts the files to a temp directory before copying them whereas the former extracts directly to the destination. The Windows file systems are pretty cruddy in general though so decompressing tonnes of small files in general will take longer than on OS X or Linux.
  - SyneRyder 9y ago
    > I mean the trackpad issue is generally down to the manufacturer and the drivers they provide... Was just wondering this myself. I tried a Surface Laptop this week, and its trackpad has everything I love from the 2012-era Mac trackpads. A satisfyingly snappy physical mouse click, gestures & acceleration, and a perfect size. (I can't stand Force Touch on the new MacBook Pro, and I don't like their new giant trackpads.)
    - zokier 9y ago
      MS is trying to fix the driver situation with their "precision touchpad" initiative: https://arstechnica.com/gadgets/2016/10/pc-oems-ditch-the-custom-touchpad-drivers-give-us-precision-touchpad/ https://arstechnica.com/gadgets/2016/10/pc-oems-ditch-the-cu...
    - cerved 9y ago
      No complaints on the sp4 trackpad here
sgift 9y ago
Probably not really feasible, but I'd be interested if something comparable happens when your build process uses many threads instead of many processes. I still think using processes instead of threads is a hack, though I know the mainstream opinion says nowadays processes are the way to go and threads are a hack.
titzer 9y ago
Full disclosure: I work for Google on Chrome. A Chrome build is truly a computational load to be reckoned with. Without the distributed build, a from-scratch build of Chrome will take at least 30 minutes on a Macbook Pro--maybe an hour(!). TBH I don't remember toughing out a full build without resorting to goma. Even on a hefty workstation, a full build is a go-for-lunch kind of interruption. It will absolutely own a machine. How did we get here? Well, C++ and its stupid O(n^2) compilation complexity. As an application grows, the number of header files grows because, as any sane and far-thinking programmer would do, we split the complexity up over multiple header files, factor it into modules, and try to create encapsulation with getter/setters. However, to actually have the C++ compiler do inlining at compile time (LTO be damned), we have to put the definitions of inline functions into header files, which greatly increases their size and processing time. Moreover, because the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file! Don't even get me started on templates. Oh, and at the end of the day, the linker has to clean up the whole mess, discarding the vast majority of the compiler's output due to so many duplicated functions. And this blowup can be huge. A debug build of V8, which is just a small subsystem of Chrome, will generate about 1.4GB of .o files which link to a 75MB .so file and 1.2MB startup executable--that's a 18x blowup. Ugh. I've worked with a lot of build systems over the years, including Google's internal build system open sourced as Bazel. While these systems have scaled C++ development far further than ever thought possible, and are remarkable in the engineering achievement therein, we just need to step back once in a while and ask ourselves: Damn, are we doing this wrong?
- bluedino 9y ago
  Have you worked with Firefox? How does the build compare?
  - robin_reala 9y ago
    Firefox is C and Rust, so I’m not sure there’s such a direct comparison.
    - 9y ago
elorant 9y ago
We've come to the point where building the browser from scratch takes more than building the OS itself.
- laumars 9y ago
  That's because we've come to the point where the browser is now more akin to an OS than a networked document reader.
  - bittermang 9y ago
    I'm reading this thread on a Chromebook and really getting a kick...
  - oblio 9y ago
    That's because the browser had to do things the sneaky way. Everybody wants to control the app platform. Every time a cross platform solutions appears everyone tries to shut it down. See: Java, Flash, etc. The only one that platform owners couldn't really shut down (though they are getting better at it), is the web. It was considered too "dumb" to shut down and too useful to completely avoid. And bit by bit, just like the boiling frog story, people added interactivity features until we arrived where we are now. A bloated, perhaps crippled cross platform solution, but the best we have now for running things on things from embedded, mobile, desktop, etc., etc.
    - Nition 9y ago
      Sounds a lot like WeChat. Messaging app that's added more and more features to now become almost an OS on top of the smartphone OS in China.
- qb45 9y ago
  Somebody said Chrome takes 30 minutes to build on one workstation and I don't seriously think you would build Windows 10 that fast.
- Beltiras 9y ago
  Interesting metric actually. Building Windows takes 12 hours [1]. It's a bit harder to find metrics on Linux. You can build the kernel in 60 seconds apparently [2] but that is not a complete operating system. [1] https://stackoverflow.com/questions/226377/operating-system-compile-time https://stackoverflow.com/questions/226377/operating-system-... [2] http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjU http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjU
nomercy400 9y ago
Does this also occur on other OSes, like Linux, or MacOS? Can you move your Chrome build to another OS and not experience this problem?
hoodoof 9y ago
Saw the headline and thought "must be Windows". I'm not a Windows hater, but one of my long standing gripes about Windows is that it just seems to have terrible multitasking compared to OSX. I'm sure there are reasons but it just seems utterly symbolic of Microsoft that they never managed to get Windows to multitask in a rock solid, smooth and reliable way like OSX.
- hota_mazi 9y ago
  It seems to have improved these past years but beach balls used to be comically widespread on OS X. I'm not at all convinced OS X is in any better shape than Windows. And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article.
  - hultner 9y ago
    You do know that OS X have DTrace?
    - nreilly 9y ago
      And then with DTrace - instruments! https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/index.html https://developer.apple.com/library/content/documentation/De...
    - dijit 9y ago
      not just "it has DTrace" it has one of the best implementations/supports of DTrace in any operating system. It's quite impressive.
  - deleted 9y ago
    [deleted]
  - rbanffy 9y ago
    > And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article It certainly helps that Intel really likes Windows, but I haven't seen Mac users complaining as much as Windows users do, so, it may be for lack of demand (and the bundling of dtrace, for ages).
nailer 9y ago
While the issue closing processes slowly is unique to Windows 10, I've found similar situations of not being able to control my OS on RHEL, Fedora, Ubuntu and MacOS. At this point I genuinely believe latency will be the death of general purpose computing. iOS and Android very rarely get out of control. Apple is already pushing iOS devices as laptop replacements. But we lose a lot on these devices with their locked down OSs and inability to install the software we want.
- m_fayer 9y ago
  All the midrange Android devices I own get out of control all the time if you switch between heavy apps too quickly. And every once in a while while I wait for the system to become responsive again, some low level process like "Android System" will crash and restart itself. I shudder to think what will happen once we start throwing desktop-class loads at Android using it's current application stack.
- SmellyGeekBoy 9y ago
  I've seen Android get "out of control" on lower end devices. If you stick with higher end devices and/or first-party the experience is usually very slick. I don't think I've ever had a OnePlus device freeze up or stutter on me, for instance.
  - Dayshine 9y ago
    My OnePlus X certainly does. When charging or after 10/15 minutes of intense graphical use it starts getting sluggish and unresponsive.
    - martinald 9y ago
      Probably the CPU/GPU scaling back because the phone is getting too hot?
- peoplewindow 9y ago
  Android isn't locked down. But it also isn't exactly smooth. my Android device frequently shows me white screens or startup splashes when doing something as simple as switching tasks. It clearly can't keep multiple apps in memory at once. Somehow iOS manages to produce a much smoother experience. I don't think it's a fundamental issue though. Just a matter of priorities.
paulio 9y ago
He's also written some stuff regarding Visual Studio perf. https://randomascii.wordpress.com/2014/04/15/self-inflicted-denial-of-service-in-visual-studio-search/ https://randomascii.wordpress.com/2014/04/15/self-inflicted-... Close to my heart as I use Visual Studio all day. Horrid piece of software. I'll probably get downvoted for this.
- qb45 9y ago
  > I'll probably get downvoted for this. Obliged :) No, seriously, why are people doing that?
- bitwize 9y ago
  It's against HN policy to downvote-bait.
- SmellyGeekBoy 9y ago
  I live in two worlds at the moment - supporting and maintaining a large PHP application (for which I use phpStorm) and developing .NET applications (for which I, obviously, use VS). I've been a VS user for nigh-on 10 years now and always found the experience really fantastic. I think it's certainly more coherent that the IntelliJ-based IDEs (which, to be fair, are also very good). Anyway, I'm genuinely surprised to see VS described as "horrid" - what specifically do you dislike about it?
  - ygra 9y ago
    VS for .NET is a different beast than VS for C++, the latter being a far less tool-friendly environment. There's also the point that newer versions sometimes improve a lot over older ones, but licensing costs may keep people on older versions for too long (that, and the fact that with large C++ projects an upgrade is rarely straight-forward). This then ends up with people hating VS when all they've used is VS2008 on C++. Not that I don't have any complaints (although those may very well be due to certain extensions), but I've always found VS to be much more responsive, stable, and featureful on the .NET side than C++; much of that certainly due to complexities of the respective languages.
  - 9y ago
bitwize 9y ago
The Amiga prioritized user-input interrupts before all other interrupts, so if there ever was a time you couldn't move the mouse on the Amiga, it meant that the system was well and truly crashed. 30 years on and the peecee industry still doesn't know how to design a fucking system.
- amiga-workbench 9y ago
  I was just thinking that, my A1200 never skipped a beat.
ComodoHacker 9y ago
I remember every other book on Windows programming saying "Process creation/destruction is expensive, use thread pools (or at least process pools) instead, that's the way to go on Windows". Perhaps this mindset is ingrained for Windows QA team too - they don't have [enough] test cases for such scenarios.
- usefulcat 9y ago
  Seems like a feedback loop. It's expensive, so most apps avoid doing that, meaning there's less need to check for performance regressions. Then if there is a regression, it further increases the incentive for developers to avoid it, so in the future even fewer apps will do that, making it even more of an unusual use case, so even less need to test for it..
- koffiezet 9y ago
  This is probably also why Cygwin and even the WSL subsystem in general are a lot slower when running more complex shell scripts, which is typically spawns tons of processes. I wrote a pretty simple shell script to test WSL process spawn speed, which loops over a simple echo piped to a grep, and add 1 to a counter until it reaches 1000. On my windows machine, in a Linux VM, I consistently get times like this: real 0m1.381s user 0m0.073s sys 0m1.472s On the same machine in WSL, I get results like this consistently: real 0m14.878s user 0m0.469s sys 0m12.109s That is 10 times slower... I don't have cygwin installed anymore, but when I tested it initially when trying out WSL, it was even slower...
stargrazer 9y ago
I think the best summary of how to do this is to take a look at Herb Sutter's three article set on "Minimizing Compile-Time Dependencies". https://herbsutter.com/gotw/ https://herbsutter.com/gotw/ "The Compilation Firewall", or pimpl idiom, is a clever mechanism for getting code out of the header: https://herbsutter.com/gotw/_100/ https://herbsutter.com/gotw/_100/
- cheez 9y ago
  I use this liberally. Generally, it is a runtime performance issue only on repeated allocations for which you can optimize if needed. Once-only allocations can be ignored when using compiler firewalls.
  - lallysingh 9y ago
    And the second cache miss: 1 for the cache miss, 1 for the actual object.
    - cheez 9y ago
      Right. Usually the entire application is not performance critical. Only certain sections are. For that part of the application, profile and optimize.
hl5 9y ago
I wonder if the "privacy" features in Win10 play a role here. Seems like some extra process accounting could cause delays not present in previous versions.
dankohn1 9y ago
I feel like the (brilliant post) is missing the context that if you were debugging the latency on Linux, you would have the source code to continue the investigation until you found and fixed the problem, as opposed to just teeing it up for Microsoft.
- hghwng 9y ago
  Most of the problems are solvable, if proper tools are used. Microsoft provides "PDB" files, which contains symbols for ease of debugging. You can get them from Microsoft's symbol server. Load the symbol and the binary in IDA, and the generated pseudo code is enough for most scenarios. In theory, debugging programs on Linux should be easier. However, for some distributions (like Arch Linux) debug symbols are not provided. You have to compile the program on your own if you want to debug. It's especially painful if the target program has a large codebase!
martyvis 9y ago
For my own amusement, when ever I get a new OS build on my machine, I'd open up a task manager and watch CPU load just by wiggling the mouse a lot or maybe simply pressing page up and down. I'm pretty sure it's always pretty easy to generate 25% CPU doing very little. Another thing would be just opening a local file from within a running application and wondering why multiple seconds and hence billions of CPU cycles seems to be consumed what one expects should be doing a fairly menial task. (I am pretty sure in DOS 3.3 with Norton Commander it was quicker )
- stinos 9y ago
  I'm pretty sure it's always pretty easy to generate 25% CPU doing very little not for me, just tried mouse wiggling, scrolling, page up/down in browsers/file managers on Windows 7, 10 and Ubuntu 16 and none of those have such behaviour. Likewise for opening a file dialog. And I didn't expect otherwise actually, what OS are you talking about?
- ptero 9y ago
  I wonder if this accurately captures the time spent on the task. I am surprised opening a file takes that long. I primarily live in a Linux environment, but I also run some Java code for R&D data collection on Windows also. It opens 10-20 files on startup which does not lead to a perceptible delay. I have not measured exact performance, but if opening (all 20) took more than half a second or so I would have surely noticed this.
- zlynx 9y ago
  That used to be true back when the CPU did all of the GUI rendering. But now most all of it is offloaded to the GPU. Any GPU that can render Quake 3 Arena at 120 FPS (and that's ALL of them even Intel IGPs) can wiggle a window around very easily. Not sure about file opens. Simple applications like GVIM don't seem to have seconds of delay for me, but I know what you mean with things like spreadsheet or word processor files. I guess it is all of the unzipping and XML processing.
  - Narishma 9y ago
    He said wiggling the mouse, not a window.
ericfrederich 9y ago
I had the same exact problem. A machine that is overkill spec wise but the mouse and keyboard would freeze up every minute on the minute. I tracked it down to my desktop wallpaper being on a rotation. Seriously... how bad does that have to be implemented in Windows to actually hang the mouse and keyboard?
- Eridrus 9y ago
  For all the hate Windows gets, UI lockup hasn't really been a pervasive issue for me in years, whereas on my Linux desktop that's just a typical day.
  - _arvin 9y ago
    My Linux desktop never locks up. I'd suggest messing around with a different kernel, double checking graphic drivers, etc.
    - Eridrus 9y ago
      I was probably a bit imprecise; I do sometimes get full lock ups where I have to reboot, mostly when running TensorFlow locally, but usually I just get lock ups that last < 10 seconds when compiling things in the background. It's at the level of "annoying, but doesn't impact my work", so I just live with it.
      - _arvin 9y ago
        Ah I see. Which scheduler are using? $ cat /sys/block/sda/queue/scheduler noop deadline cfq [bfq] I've been very pleased with the responsiveness of BFQ when multitasking. And actually, I shouldn't say I never get lock-ups. There's been a couple times where the DE just freezes, and I'll have to hit Ctrl+Alt+F2 to switch to another tty and restart the display manager. But I attribute that to running bleeding-edge version of things and enabling experimental features, so that's fair. Lastly, my MacBook Pro (2010 6,2) would also experience random freezes on Ubuntu (would have to power off/on) and upgrading the kernel from the Ubuntu default to latest mainline solved that problem completely.
faragon 9y ago
Why the OS don't prioritize UI threads on one or two cores?
- mac01021 9y ago
  How does it know which threads are UI threads?
  - vivekseth 9y ago
    I think it should at least know which thread handles mouse movement.
- usefulcat 9y ago
  Won't help if the problem is as the author describes (contention for a system-wide lock).
- asveikau 9y ago
  I think the windows scheduler might already do this, but the issue is that UI message queues take a highly contended lock.
snarfy 9y ago
My 16mhz 68000 based Amiga 500 had smoother mouse movement than my 3.2ghz 8 core desktop.
- rbanffy 9y ago
  IIRC, on the Amigas the CPU was barely involved in the process of reading the mouse events and moving the cursor sprite on the screen.
  - snarfy 9y ago
    Yep, hardware sprites.
    - TazeTSchnitzel 9y ago
      Modern GPUs still have hardware sprites for the cursor, and OSes still use them! However, the path between the mouse and updating that sprite has gotten more complex, alas.
      - rbanffy 9y ago
        I imagine a modern day Amiga would run all mouse and window server code on the GPU, only telling the CPU when the app needs to update its bitmap.
onetokeoverthe 9y ago
There's too many spinning wheels. Stop it! People are going to just stop using the internet. If I could, I would. But I can't. So please just build simple websites!
coldcode 9y ago
Try running Carthage update for an iOS app with 80 frameworks. Ever seen a Mac hit 55GB of ram? MacOS kills it. It takes many of these runs, and all this does is fetch prebuilt binaries. Yes, everything I said here is beyond stupid.
- sixstringtheory 9y ago
  Is it necessary to update _all_ of them at once? I think that's the same problem as OP... rebuilding the whole world is slow, so why not do it incrementally? Also, 80 frameworks ⊙⊙ I'm assuming some of these are internal, and written in Swift, meaning they can't be compiled into static libs (easily).
cat199 9y ago
Can take the alternate approach - Run your builds in a hugely underpowered VM, and wait much longer.. your regular usage will be largely unimpacted, although the builds take longer. Source: Currently running a ~1000 package dpb(1)[1] build of my needed openbsd ports on a dual-core KVM machine hosted on a 8-9 year old amd64x2 2.2ghz. 3 Days and counting, will probably be done around next weekend. From there, incremental updates are mostly slight, and can complete overnight from a cron job. .. [1] https://man.openbsd.org/dpb https://man.openbsd.org/dpb
- metalliqaz 9y ago
  Out of curiosity why do you keep such an old machine around? I have found that newer machines can be had for free, and 7-year-old Xeon servers can be had for <$200 from IT recyclers.
  - gwern 9y ago
    The electricity consumption numbers are also relevant.
wjd2030 9y ago
I just wanted to say good work. I'm impressed you dug so deep. A fix here could really impact the entire Windows 10 user base.
fooker 9y ago
Use the gold linker if your setup permits it. This issue went away for me when I switched.
iainmerrick 9y ago
The first time I had to compile a Linux kernel for Android, it took only a few seconds (on the ridiculously overpowered "build machine" my employer supplied). I was sure I must have done something wrong, but no, that was the entire build. It takes longer for Android to reboot than it does to build the kernel. It does feel like there's something seriously wrong with the massive C++ codebases we use these days for key infrastructure like browsers, and the massive compilation times we put up with.
ksk 9y ago
Speaking of W10, there was another annoying W10 bug where if you started typing immediately after using the touchpad there was a random delay. If you care about latency and responsiveness, it makes you want to scream at the people who implement these features.
- mrstone 9y ago
  That's not a bug. It's some kind of feature by Symantec that can be disabled in the settings.
  - ksk 9y ago
    Haha, it never crossed my mind that someone would implement this purposely. I don't have a W10 laptop so I've only noticed this on other peoples machines.
    - NuDinNou 9y ago
      I use this feature on my laptop. Without it I would accidentally touch the touchpad and insert characters where there aren't supposed to be.
BurningFrog 9y ago
> the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file! Without knowing anything about modern C++ and its compilers, this seems fixable. I'm thinking a header compiler "hint" indicating the size of an object. When compiling the full class, you get an error if the number is wrong.
goda90 9y ago
My workstation was recently "upgraded" from a Window 8.1 workstation with a 4th gen i5 to a Windows 10 laptop with 5th gen i7. The extra RAM and SSD over HDD is great, but whether it's because it went from a quad-core to a dual-core hyperthreaded CPU, or because of the jump to Windows 10, the mouse lag is considerably more noticeable now. I've convinced them to upgrade my laptop again, but now this article doesn't give me much hope for my new work toy.
quotemstr 9y ago
Why is everyone talking about how C++ compilation is slow or something instead of talking about the real problem, which is that GDI is doing resource cleanup on process exit, under lock, for console-mode processes that have probably never used any GDI resources?
- yuhong 9y ago
  mxatone mentions that "Win32k locking design is just bad" in https://twitter.com/mxatone/status/884436870955913216 https://twitter.com/mxatone/status/884436870955913216
  - quotemstr 9y ago
    It's not even bad. It's neglecting any kind of "has this process ever touched win32k.sys" test.
bane 9y ago
Seems like a good way to deal with this (outside of trying to convince Microsoft to fix it) is to spin up a VM and just give it a few cores, and do the build inside the VM to isolate this behavior.
arwhatever 9y ago
Wasn't this the title of a Bruce Springsteen song?
drumttocs8 9y ago
When I converted from Windows 7 to 10 I noticed that I started getting audio latency/glitches/squelches from my external audio interface- a Focusrite Scarlett 2i4. The mouse would also sometimes hang. Doing some basic tests, the problem seemed to be coming from network drivers... but I could never resolve it. I wonder if the author's discovery has anything to do with the issues.
hindsightbias 9y ago
What Knuth said: "Premature optimization is the root of all evil" What most developers hear: "Optimization is the root of all evil"
rebootthesystem 9y ago
Thank you for bringing attention to this. Experiencing this on our W10 workstations. I hope MS does something about this immediately. It's maddening.
lobo_tuerto 9y ago
I have a new Ryzen 7 CPU (you know, 8 cores, 16 threads), with 64GB RAM and for HDD a Samsung 960 PRO M.2 drive. But when I went and plugged in an external Seagate 4TB drive, and tried to "dd zero" the s#it out of it, my whole system became unresponsive after a while, obviously I had to reset the machine as it wouldn't "kill -9" the process that made the system unresponsive. Trying to type was a no go as keys would sometimes become "stuck". Moving the mouse around was an exercise in predictability too. All this happened in the latest Ubuntu 17.04 64bit... #sadstory

1011 points by joebaf 9y ago

The terrifying result of having ABI compatibility with the first commercially successful system of its kind from the '80s. Granted, they aren't doing themselves any favours with their new straight-jacket style application ABI.

willvarfar 9y ago
You are trotting out the same tired line without reading the article carefully? ;) This is a regression. The article shows it worked fine up to Windows 7.
- Aeolun 9y ago
  7 is the best
  - option_greek 9y ago
    Regardless of how many windows/any other OS versions I use later on, I will remember Win7 fondly. Its just so.. less cumbersome. May be the drastic changes that went into its preceeding and succeeding versions have a role to play in this.
    - krylon 9y ago
      There is this old quote that Algol 60 was an improvement not only on its predecessors but also its successors. The same could be said about Windows 7, I guess. ;-)
jbg_ 9y ago
How is ABI compatibility with early systems the cause of this bug? You'll note that it wasn't present in Windows 7.
LeoPanthera 9y ago
64-bit versions of Windows 10 (and 8/7 maybe?) are no longer compatible with 16-bit code.
- ripdog 9y ago
  Yep, no 64-bit Windows has 16-bit compatibility.
deleted 9y ago
[deleted]

yourstruly33 9y ago

These posts are getting boring.. the only thing that they're saying is: I can use ETW and to me being told that I'm smart and capable is paramount.

digitalnomad91 9y ago
Then why are you commenting? Kind of a crappy attitude you have there.
- alacombe 9y ago
  not really, I was kinda expecting the post would also contain interactions with Microsoft, leading to the resolution of the bug in update XX. The post kinda left me unsatisfied.
  - kirillkh 9y ago
    You can fix those problems by writing a post that would contain said interactions, resolution and update.
  - digitalnomad91 9y ago
    Sure, and you're entitled to feel that way, but that doesn't make his attitude any less crappy.
  - BinaryIdiot 9y ago
    While I too am anxious to hear the outcome after Microsoft investigates the author still shared something they were interested in of which holds a value to the HN community as many of us love digging into a system and finding issues. This was the main purpose of the article. Regardless the author may not know how critical Microsoft has classified this issue. It could be something they don't resolve for years. Since the Microsoft investigation and possible patch has no barring on the content of this article I wouldn't expect it to be held until such a time. I'm sure we'll get a follow up on HN when or if it gets patched. Even better maybe posting this article helps get it fixed sooner :)
  - 9y ago

yuhong 9y ago

I can't imagine that moving the Win32k stuff back to CSRSS would help much in this case, right? Though it is still a good thing especially for terminal servers where hopefully one CSRSS process crashing just terminate the session.

sjmulder 9y ago
> moving the Win32k stuff back to CSRSS Out of curiosity, was there some big move of functionality from CSRSS into Win32k earlier? When and what?
- yuhong 9y ago
  In NT4.

isatty 9y ago

I read mouse as house and was confused for the longest time.

mjsweet 9y ago
Same happened to me... initially I thought it must be some 1st world problem.

aramadia 9y ago

When I saw his workstation specs, I thought, thats the exact same one I have at work! Then I checked the bottom, yep, he's at Google too.

unixhero 9y ago
Lucky bastards
willvarfar 9y ago
Does yours run Windows? Do you get to choose? If you could choose, which OS would you use? Any clear favourites with your peers for development? Just collecting anecdotes ;)

albertzeyer 9y ago

I was wondering what ETW is, and found in one of his other posts: ETW (Event Tracing for Windows). He seems to be one of the main contributors of https://github.com/google/UIforETW https://github.com/google/UIforETW Seems to be quite useful. --- About the post: tl;dr: NtGdiCloseProcess has a system-wide global lock which is used quite often e.g. during a build of Chrome which spawns a lot of processes. This problem seems to be introduced between Windows 7 and Windows 10. I thought that there would be a solution or a fix but it seems this is not yet fixed. "This problem has been reported to Microsoft and they are investigating."

yuhong 9y ago
I wonder if this is the same lock most win32k call takes.
masklinn 9y ago
> NtGdiCloseProcess has a system-wide global lock which is used quite often "Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates. This is compounded by parallelised build processes as not only do they spawn lots of processes leading to lots of process destruction: it both fucks up the UI and completely serialises process destruction so your nice 24 process flights for faster processing end up taking multiple seconds to all shut down, and the more cores (and thus processes) you have the worse it is, and the worse your stutters.
- qb45 9y ago
  > "Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates. Do you happen to know anything about it? I'm scratching my head how it's possible that process termination serializes with GUI... Maybe it hogs some lock on process descriptors which SendMessage also needs to grab for a moment to find the target process? I hope you didn't mean to say that every SendMessage call is completely serialized with each other.

weinzierl 9y ago

I grew up on the Commodore 64 (1 Core, 1 hyper-thread :-), almost 1 MHz clock freq, almost 64 K usable RAM). The machine was usually pretty responsive, but when I typed too quickly in my word processor it sometimes got stuck and ate a few characters. I used to think: "If computers were only fast enough so I could type without interruption...". If you'd asked me back then for a top ten list what I wished computers could do, this would certainly have been on the list. Now, 30 years later, whenever my cursor gets stuck I like to think: "If computers were only fast enough so I could type without interruption..."

fredley 9y ago
It's all just trade offs. However fast or powerful your machine is, software will use as much of that resource as possible, up to the point where it occasionally interferes with input (but not too much or you'll switch to something else).
- anigbrowl 9y ago
  But why? What good is an operating system on a multi-core device that allows anything to get that close to the performance envelope? This is a fine example of competition driving change for change's sake rather than real innovation and everything ending up worse as a result. I like new features as much as the next person, but not when they compromise core functionality. Not being able to type is inexcusable.
  - TeMPOraL 9y ago
    I agree. However at this point, I don't see anything an OS can do to help. I see plenty of typing slowdowns every other day now. But I'm not sure just how many of them are the OS's fault. When your typing seems to lag, there are two places that can be slowing it down - the input side (reacting to hardware events) and the output side (drawing and updating the UI). I suppose keyboard buffers are pretty well isolated, and native UI controls tend to work fine too. The problem is, everyone now goes for non-native controls. You type in e.g. Firefox, and it is slow not because of your OS, but because Firefox does all its UI drawing by itself. And God help you if the application you want to use is done in Electron. There's so many layers of non-nativeness on top of that, that the OS has close to zero say in what's being done. There's no way to help that - resource quotas will only make the problem worse, and giving such program free reign will only take everything else down. All in all, it's just - again - problem of people writing shitty software, because of laziness and time-to-market reasons. Blame "Worse is Better".

alacombe 9y ago

So what now ? Should/does the author wait getting some traction on Hacker News and hopefully be noticed by dev at Microsoft, or is there some way to provide Microsoft directly these data, skipping Tier-1 support ? The author seem to be working at Google, so he might get some leverage from that, but what about Random Joe ? Keep enjoying the bug "forever", I guess ?

jsnell 9y ago
From the article: > This problem has been reported to Microsoft and they are investigating.
BinaryIdiot 9y ago
Did you read the article? The bug was already submitted to Microsoft and they are investigating. So the answer to your question is: wait until they're done investigation and possibly release a patch.
- alacombe 9y ago
  I skipped the conclusion, though, I'd still be interested about the channel that could be used to report such issues.
ximeng 9y ago
There are many many MS bugs that last for years. Pretty much any time you search for a bug in MS software you will find someone in a support forum giving generic advice (known issue that we will work on, not a bug, please reinstall Windows and applications) and users complaining. Typical example: https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/10374741-stop-excel-from-changing-large-numbers-actually https://excel.uservoice.com/forums/304921-excel-for-windows-... basic problem that has been there many many years. For this one there are workarounds but many others there are not. The only real long-term solution is supporting competitive products.
- intoverflow2 9y ago

eecc 9y ago

so basically a fork-bomb. I think Linux can still buckle under one, nothing that obscene...

mattnewton 9y ago
No, the article says this is a regression in how processes close.
masklinn 9y ago
A fork bomb exhausts all system resources, this is not that, it doesn't actually use any resources, rather it locks the system out of them by serialising both process termination and UI updates.
- eecc 9y ago
  Well, a fork bomb doesn't really use anything except it makes the process table management extremely time consuming. Not quite the same, but still conceptually similar

yanpanlau 9y ago

Just use linux

jlebrech 9y ago
that's asking for downvotes, I voted this up nonetheless.
gurkendoktor 9y ago
Staying responsive (under load) is also still a work in progress on Linux: https://www.phoronix.com/scan.php?page=news_item&px=BFQ-Queued-Linux-4.12 https://www.phoronix.com/scan.php?page=news_item&px=BFQ-Queu...
- dispat0r 9y ago
  That is a disk scheduler and mostly for spinning disks. I haven't seen a stuck mouse cursor for a long time in linux.
  - hypervis0r 9y ago
    > I haven't seen a stuck mouse cursor for a long time in linux. That's because the only cursor you have in Linux is the terminal one.
  - goodplay 9y ago
    I always found this peculiar about linux; even when the system is swapping hard and application windows/window managers completely freeze, the cursor always remains responsive and movement rendered without so much as a hitch. I wonder if xorg has some sort of kernel support to enable this.
    - gens 9y ago
      Not on my computer. Something like a WM will freeze while it waits in queue to read those 0.2kB it needs of the disk (poor software design, if you ask me). But the mouse will freeze when IO buffers get filled. Just writing a huge file to a (slow-ish) usb stick will make my whole computer freeze, including the mouse, because the kernel usb code doesn't limit the buffer to some sane size (there was a kernel patch, and there is an option, and even with it turned on the problem is still there) (note that the probable reasoning for that is the fact that usb sucks). A mouse cursor, AFAIK, is a gpu thing, not that it matters (wayland, i remember something that, will use normal gpu rendering to render the mouse). In the UNIX haters handbook there is a section about X where it is written that displays used to have 2-3 planes (IIRC, 2 planes + a cursor plane. I do recommend reading the good book, as it is funny). I remember some talk about better kernel buffers management just for things like this. Memory management in the kernel is one of those actually hard things.

SigSegOwl 9y ago

Thats why my machine gets that slow after running for weeks... explains everything!

blunte 9y ago

This is great work. I hope MS can improve Windows 10 by fixing this. I just added a new Win10 laptop with much better specs than my 3 year old rMBP, and I'm shocked by how much apparently random latency I experience with the UI in Windows 10 compared to the Mac. That's not to mention the issues of sloppier track pad (which constantly detects my left hand while I type) or the ungodly slow unzip (via 7z). If only Apple would give us more than 16GB of RAM (in a laptop)... what a frustrating world for developers.

ShinTakuya 9y ago
I mean the trackpad issue is generally down to the manufacturer and the drivers they provide, but I take your point otherwise. A pro tip with 7z is to use the two pane interface to extract and not dragging files to Explorer. The latter option extracts the files to a temp directory before copying them whereas the former extracts directly to the destination. The Windows file systems are pretty cruddy in general though so decompressing tonnes of small files in general will take longer than on OS X or Linux.
- SyneRyder 9y ago
  > I mean the trackpad issue is generally down to the manufacturer and the drivers they provide... Was just wondering this myself. I tried a Surface Laptop this week, and its trackpad has everything I love from the 2012-era Mac trackpads. A satisfyingly snappy physical mouse click, gestures & acceleration, and a perfect size. (I can't stand Force Touch on the new MacBook Pro, and I don't like their new giant trackpads.)
  - zokier 9y ago
    MS is trying to fix the driver situation with their "precision touchpad" initiative: https://arstechnica.com/gadgets/2016/10/pc-oems-ditch-the-custom-touchpad-drivers-give-us-precision-touchpad/ https://arstechnica.com/gadgets/2016/10/pc-oems-ditch-the-cu...
  - cerved 9y ago
    No complaints on the sp4 trackpad here

sgift 9y ago

Probably not really feasible, but I'd be interested if something comparable happens when your build process uses many threads instead of many processes. I still think using processes instead of threads is a hack, though I know the mainstream opinion says nowadays processes are the way to go and threads are a hack.

titzer 9y ago

Full disclosure: I work for Google on Chrome. A Chrome build is truly a computational load to be reckoned with. Without the distributed build, a from-scratch build of Chrome will take at least 30 minutes on a Macbook Pro--maybe an hour(!). TBH I don't remember toughing out a full build without resorting to goma. Even on a hefty workstation, a full build is a go-for-lunch kind of interruption. It will absolutely own a machine. How did we get here? Well, C++ and its stupid O(n^2) compilation complexity. As an application grows, the number of header files grows because, as any sane and far-thinking programmer would do, we split the complexity up over multiple header files, factor it into modules, and try to create encapsulation with getter/setters. However, to actually have the C++ compiler do inlining at compile time (LTO be damned), we have to put the definitions of inline functions into header files, which greatly increases their size and processing time. Moreover, because the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file! Don't even get me started on templates. Oh, and at the end of the day, the linker has to clean up the whole mess, discarding the vast majority of the compiler's output due to so many duplicated functions. And this blowup can be huge. A debug build of V8, which is just a small subsystem of Chrome, will generate about 1.4GB of .o files which link to a 75MB .so file and 1.2MB startup executable--that's a 18x blowup. Ugh. I've worked with a lot of build systems over the years, including Google's internal build system open sourced as Bazel. While these systems have scaled C++ development far further than ever thought possible, and are remarkable in the engineering achievement therein, we just need to step back once in a while and ask ourselves: Damn, are we doing this wrong?

bluedino 9y ago
Have you worked with Firefox? How does the build compare?
- robin_reala 9y ago
  Firefox is C and Rust, so I’m not sure there’s such a direct comparison.
  - 9y ago

elorant 9y ago

We've come to the point where building the browser from scratch takes more than building the OS itself.

laumars 9y ago
That's because we've come to the point where the browser is now more akin to an OS than a networked document reader.
- bittermang 9y ago
  I'm reading this thread on a Chromebook and really getting a kick...
- oblio 9y ago
  That's because the browser had to do things the sneaky way. Everybody wants to control the app platform. Every time a cross platform solutions appears everyone tries to shut it down. See: Java, Flash, etc. The only one that platform owners couldn't really shut down (though they are getting better at it), is the web. It was considered too "dumb" to shut down and too useful to completely avoid. And bit by bit, just like the boiling frog story, people added interactivity features until we arrived where we are now. A bloated, perhaps crippled cross platform solution, but the best we have now for running things on things from embedded, mobile, desktop, etc., etc.
  - Nition 9y ago
    Sounds a lot like WeChat. Messaging app that's added more and more features to now become almost an OS on top of the smartphone OS in China.
qb45 9y ago
Somebody said Chrome takes 30 minutes to build on one workstation and I don't seriously think you would build Windows 10 that fast.
Beltiras 9y ago
Interesting metric actually. Building Windows takes 12 hours [1]. It's a bit harder to find metrics on Linux. You can build the kernel in 60 seconds apparently [2] but that is not a complete operating system. [1] https://stackoverflow.com/questions/226377/operating-system-compile-time https://stackoverflow.com/questions/226377/operating-system-... [2] http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjU http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjU

nomercy400 9y ago

Does this also occur on other OSes, like Linux, or MacOS? Can you move your Chrome build to another OS and not experience this problem?

hoodoof 9y ago

Saw the headline and thought "must be Windows". I'm not a Windows hater, but one of my long standing gripes about Windows is that it just seems to have terrible multitasking compared to OSX. I'm sure there are reasons but it just seems utterly symbolic of Microsoft that they never managed to get Windows to multitask in a rock solid, smooth and reliable way like OSX.

hota_mazi 9y ago
It seems to have improved these past years but beach balls used to be comically widespread on OS X. I'm not at all convinced OS X is in any better shape than Windows. And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article.
- hultner 9y ago
  You do know that OS X have DTrace?
  - nreilly 9y ago
    And then with DTrace - instruments! https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/index.html https://developer.apple.com/library/content/documentation/De...
  - dijit 9y ago
    not just "it has DTrace" it has one of the best implementations/supports of DTrace in any operating system. It's quite impressive.
- deleted 9y ago
  [deleted]
- rbanffy 9y ago
  > And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article It certainly helps that Intel really likes Windows, but I haven't seen Mac users complaining as much as Windows users do, so, it may be for lack of demand (and the bundling of dtrace, for ages).

nailer 9y ago

While the issue closing processes slowly is unique to Windows 10, I've found similar situations of not being able to control my OS on RHEL, Fedora, Ubuntu and MacOS. At this point I genuinely believe latency will be the death of general purpose computing. iOS and Android very rarely get out of control. Apple is already pushing iOS devices as laptop replacements. But we lose a lot on these devices with their locked down OSs and inability to install the software we want.

m_fayer 9y ago
All the midrange Android devices I own get out of control all the time if you switch between heavy apps too quickly. And every once in a while while I wait for the system to become responsive again, some low level process like "Android System" will crash and restart itself. I shudder to think what will happen once we start throwing desktop-class loads at Android using it's current application stack.
SmellyGeekBoy 9y ago
I've seen Android get "out of control" on lower end devices. If you stick with higher end devices and/or first-party the experience is usually very slick. I don't think I've ever had a OnePlus device freeze up or stutter on me, for instance.
- Dayshine 9y ago
  My OnePlus X certainly does. When charging or after 10/15 minutes of intense graphical use it starts getting sluggish and unresponsive.
  - martinald 9y ago
    Probably the CPU/GPU scaling back because the phone is getting too hot?
peoplewindow 9y ago
Android isn't locked down. But it also isn't exactly smooth. my Android device frequently shows me white screens or startup splashes when doing something as simple as switching tasks. It clearly can't keep multiple apps in memory at once. Somehow iOS manages to produce a much smoother experience. I don't think it's a fundamental issue though. Just a matter of priorities.

paulio 9y ago

He's also written some stuff regarding Visual Studio perf. https://randomascii.wordpress.com/2014/04/15/self-inflicted-denial-of-service-in-visual-studio-search/ https://randomascii.wordpress.com/2014/04/15/self-inflicted-... Close to my heart as I use Visual Studio all day. Horrid piece of software. I'll probably get downvoted for this.

qb45 9y ago
> I'll probably get downvoted for this. Obliged :) No, seriously, why are people doing that?
bitwize 9y ago
It's against HN policy to downvote-bait.
SmellyGeekBoy 9y ago
I live in two worlds at the moment - supporting and maintaining a large PHP application (for which I use phpStorm) and developing .NET applications (for which I, obviously, use VS). I've been a VS user for nigh-on 10 years now and always found the experience really fantastic. I think it's certainly more coherent that the IntelliJ-based IDEs (which, to be fair, are also very good). Anyway, I'm genuinely surprised to see VS described as "horrid" - what specifically do you dislike about it?
- ygra 9y ago
  VS for .NET is a different beast than VS for C++, the latter being a far less tool-friendly environment. There's also the point that newer versions sometimes improve a lot over older ones, but licensing costs may keep people on older versions for too long (that, and the fact that with large C++ projects an upgrade is rarely straight-forward). This then ends up with people hating VS when all they've used is VS2008 on C++. Not that I don't have any complaints (although those may very well be due to certain extensions), but I've always found VS to be much more responsive, stable, and featureful on the .NET side than C++; much of that certainly due to complexities of the respective languages.
- 9y ago

bitwize 9y ago

The Amiga prioritized user-input interrupts before all other interrupts, so if there ever was a time you couldn't move the mouse on the Amiga, it meant that the system was well and truly crashed. 30 years on and the peecee industry still doesn't know how to design a fucking system.

amiga-workbench 9y ago
I was just thinking that, my A1200 never skipped a beat.

ComodoHacker 9y ago

I remember every other book on Windows programming saying "Process creation/destruction is expensive, use thread pools (or at least process pools) instead, that's the way to go on Windows". Perhaps this mindset is ingrained for Windows QA team too - they don't have [enough] test cases for such scenarios.

usefulcat 9y ago
Seems like a feedback loop. It's expensive, so most apps avoid doing that, meaning there's less need to check for performance regressions. Then if there is a regression, it further increases the incentive for developers to avoid it, so in the future even fewer apps will do that, making it even more of an unusual use case, so even less need to test for it..
koffiezet 9y ago
This is probably also why Cygwin and even the WSL subsystem in general are a lot slower when running more complex shell scripts, which is typically spawns tons of processes. I wrote a pretty simple shell script to test WSL process spawn speed, which loops over a simple echo piped to a grep, and add 1 to a counter until it reaches 1000. On my windows machine, in a Linux VM, I consistently get times like this: real 0m1.381s user 0m0.073s sys 0m1.472s On the same machine in WSL, I get results like this consistently: real 0m14.878s user 0m0.469s sys 0m12.109s That is 10 times slower... I don't have cygwin installed anymore, but when I tested it initially when trying out WSL, it was even slower...

stargrazer 9y ago

I think the best summary of how to do this is to take a look at Herb Sutter's three article set on "Minimizing Compile-Time Dependencies". https://herbsutter.com/gotw/ https://herbsutter.com/gotw/ "The Compilation Firewall", or pimpl idiom, is a clever mechanism for getting code out of the header: https://herbsutter.com/gotw/_100/ https://herbsutter.com/gotw/_100/

cheez 9y ago
I use this liberally. Generally, it is a runtime performance issue only on repeated allocations for which you can optimize if needed. Once-only allocations can be ignored when using compiler firewalls.
- lallysingh 9y ago
  And the second cache miss: 1 for the cache miss, 1 for the actual object.
  - cheez 9y ago
    Right. Usually the entire application is not performance critical. Only certain sections are. For that part of the application, profile and optimize.

hl5 9y ago

I wonder if the "privacy" features in Win10 play a role here. Seems like some extra process accounting could cause delays not present in previous versions.

dankohn1 9y ago

I feel like the (brilliant post) is missing the context that if you were debugging the latency on Linux, you would have the source code to continue the investigation until you found and fixed the problem, as opposed to just teeing it up for Microsoft.

hghwng 9y ago
Most of the problems are solvable, if proper tools are used. Microsoft provides "PDB" files, which contains symbols for ease of debugging. You can get them from Microsoft's symbol server. Load the symbol and the binary in IDA, and the generated pseudo code is enough for most scenarios. In theory, debugging programs on Linux should be easier. However, for some distributions (like Arch Linux) debug symbols are not provided. You have to compile the program on your own if you want to debug. It's especially painful if the target program has a large codebase!

martyvis 9y ago

For my own amusement, when ever I get a new OS build on my machine, I'd open up a task manager and watch CPU load just by wiggling the mouse a lot or maybe simply pressing page up and down. I'm pretty sure it's always pretty easy to generate 25% CPU doing very little. Another thing would be just opening a local file from within a running application and wondering why multiple seconds and hence billions of CPU cycles seems to be consumed what one expects should be doing a fairly menial task. (I am pretty sure in DOS 3.3 with Norton Commander it was quicker )

stinos 9y ago
I'm pretty sure it's always pretty easy to generate 25% CPU doing very little not for me, just tried mouse wiggling, scrolling, page up/down in browsers/file managers on Windows 7, 10 and Ubuntu 16 and none of those have such behaviour. Likewise for opening a file dialog. And I didn't expect otherwise actually, what OS are you talking about?
ptero 9y ago
I wonder if this accurately captures the time spent on the task. I am surprised opening a file takes that long. I primarily live in a Linux environment, but I also run some Java code for R&D data collection on Windows also. It opens 10-20 files on startup which does not lead to a perceptible delay. I have not measured exact performance, but if opening (all 20) took more than half a second or so I would have surely noticed this.
zlynx 9y ago
That used to be true back when the CPU did all of the GUI rendering. But now most all of it is offloaded to the GPU. Any GPU that can render Quake 3 Arena at 120 FPS (and that's ALL of them even Intel IGPs) can wiggle a window around very easily. Not sure about file opens. Simple applications like GVIM don't seem to have seconds of delay for me, but I know what you mean with things like spreadsheet or word processor files. I guess it is all of the unzipping and XML processing.
- Narishma 9y ago
  He said wiggling the mouse, not a window.

ericfrederich 9y ago

I had the same exact problem. A machine that is overkill spec wise but the mouse and keyboard would freeze up every minute on the minute. I tracked it down to my desktop wallpaper being on a rotation. Seriously... how bad does that have to be implemented in Windows to actually hang the mouse and keyboard?

Eridrus 9y ago
For all the hate Windows gets, UI lockup hasn't really been a pervasive issue for me in years, whereas on my Linux desktop that's just a typical day.
- _arvin 9y ago
  My Linux desktop never locks up. I'd suggest messing around with a different kernel, double checking graphic drivers, etc.
  - Eridrus 9y ago
    I was probably a bit imprecise; I do sometimes get full lock ups where I have to reboot, mostly when running TensorFlow locally, but usually I just get lock ups that last < 10 seconds when compiling things in the background. It's at the level of "annoying, but doesn't impact my work", so I just live with it.
    - _arvin 9y ago
      Ah I see. Which scheduler are using? $ cat /sys/block/sda/queue/scheduler noop deadline cfq [bfq] I've been very pleased with the responsiveness of BFQ when multitasking. And actually, I shouldn't say I never get lock-ups. There's been a couple times where the DE just freezes, and I'll have to hit Ctrl+Alt+F2 to switch to another tty and restart the display manager. But I attribute that to running bleeding-edge version of things and enabling experimental features, so that's fair. Lastly, my MacBook Pro (2010 6,2) would also experience random freezes on Ubuntu (would have to power off/on) and upgrading the kernel from the Ubuntu default to latest mainline solved that problem completely.

faragon 9y ago

Why the OS don't prioritize UI threads on one or two cores?

mac01021 9y ago
How does it know which threads are UI threads?
- vivekseth 9y ago
  I think it should at least know which thread handles mouse movement.
usefulcat 9y ago
Won't help if the problem is as the author describes (contention for a system-wide lock).
asveikau 9y ago
I think the windows scheduler might already do this, but the issue is that UI message queues take a highly contended lock.

snarfy 9y ago

My 16mhz 68000 based Amiga 500 had smoother mouse movement than my 3.2ghz 8 core desktop.

rbanffy 9y ago
IIRC, on the Amigas the CPU was barely involved in the process of reading the mouse events and moving the cursor sprite on the screen.
- snarfy 9y ago
  Yep, hardware sprites.
  - TazeTSchnitzel 9y ago
    Modern GPUs still have hardware sprites for the cursor, and OSes still use them! However, the path between the mouse and updating that sprite has gotten more complex, alas.
    - rbanffy 9y ago
      I imagine a modern day Amiga would run all mouse and window server code on the GPU, only telling the CPU when the app needs to update its bitmap.

onetokeoverthe 9y ago

There's too many spinning wheels. Stop it! People are going to just stop using the internet. If I could, I would. But I can't. So please just build simple websites!

coldcode 9y ago

Try running Carthage update for an iOS app with 80 frameworks. Ever seen a Mac hit 55GB of ram? MacOS kills it. It takes many of these runs, and all this does is fetch prebuilt binaries. Yes, everything I said here is beyond stupid.

sixstringtheory 9y ago
Is it necessary to update _all_ of them at once? I think that's the same problem as OP... rebuilding the whole world is slow, so why not do it incrementally? Also, 80 frameworks ⊙⊙ I'm assuming some of these are internal, and written in Swift, meaning they can't be compiled into static libs (easily).

cat199 9y ago

Can take the alternate approach - Run your builds in a hugely underpowered VM, and wait much longer.. your regular usage will be largely unimpacted, although the builds take longer. Source: Currently running a ~1000 package dpb(1)[1] build of my needed openbsd ports on a dual-core KVM machine hosted on a 8-9 year old amd64x2 2.2ghz. 3 Days and counting, will probably be done around next weekend. From there, incremental updates are mostly slight, and can complete overnight from a cron job. .. [1] https://man.openbsd.org/dpb https://man.openbsd.org/dpb

metalliqaz 9y ago
Out of curiosity why do you keep such an old machine around? I have found that newer machines can be had for free, and 7-year-old Xeon servers can be had for <$200 from IT recyclers.
- gwern 9y ago
  The electricity consumption numbers are also relevant.

wjd2030 9y ago

I just wanted to say good work. I'm impressed you dug so deep. A fix here could really impact the entire Windows 10 user base.

fooker 9y ago

Use the gold linker if your setup permits it. This issue went away for me when I switched.

iainmerrick 9y ago

The first time I had to compile a Linux kernel for Android, it took only a few seconds (on the ridiculously overpowered "build machine" my employer supplied). I was sure I must have done something wrong, but no, that was the entire build. It takes longer for Android to reboot than it does to build the kernel. It does feel like there's something seriously wrong with the massive C++ codebases we use these days for key infrastructure like browsers, and the massive compilation times we put up with.

ksk 9y ago

Speaking of W10, there was another annoying W10 bug where if you started typing immediately after using the touchpad there was a random delay. If you care about latency and responsiveness, it makes you want to scream at the people who implement these features.

mrstone 9y ago
That's not a bug. It's some kind of feature by Symantec that can be disabled in the settings.
- ksk 9y ago
  Haha, it never crossed my mind that someone would implement this purposely. I don't have a W10 laptop so I've only noticed this on other peoples machines.
  - NuDinNou 9y ago
    I use this feature on my laptop. Without it I would accidentally touch the touchpad and insert characters where there aren't supposed to be.

BurningFrog 9y ago

> the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file! Without knowing anything about modern C++ and its compilers, this seems fixable. I'm thinking a header compiler "hint" indicating the size of an object. When compiling the full class, you get an error if the number is wrong.

goda90 9y ago

My workstation was recently "upgraded" from a Window 8.1 workstation with a 4th gen i5 to a Windows 10 laptop with 5th gen i7. The extra RAM and SSD over HDD is great, but whether it's because it went from a quad-core to a dual-core hyperthreaded CPU, or because of the jump to Windows 10, the mouse lag is considerably more noticeable now. I've convinced them to upgrade my laptop again, but now this article doesn't give me much hope for my new work toy.

quotemstr 9y ago

Why is everyone talking about how C++ compilation is slow or something instead of talking about the real problem, which is that GDI is doing resource cleanup on process exit, under lock, for console-mode processes that have probably never used any GDI resources?

yuhong 9y ago
mxatone mentions that "Win32k locking design is just bad" in https://twitter.com/mxatone/status/884436870955913216 https://twitter.com/mxatone/status/884436870955913216
- quotemstr 9y ago
  It's not even bad. It's neglecting any kind of "has this process ever touched win32k.sys" test.

bane 9y ago

Seems like a good way to deal with this (outside of trying to convince Microsoft to fix it) is to spin up a VM and just give it a few cores, and do the build inside the VM to isolate this behavior.

arwhatever 9y ago

Wasn't this the title of a Bruce Springsteen song?

drumttocs8 9y ago

When I converted from Windows 7 to 10 I noticed that I started getting audio latency/glitches/squelches from my external audio interface- a Focusrite Scarlett 2i4. The mouse would also sometimes hang. Doing some basic tests, the problem seemed to be coming from network drivers... but I could never resolve it. I wonder if the author's discovery has anything to do with the issues.

hindsightbias 9y ago

What Knuth said: "Premature optimization is the root of all evil" What most developers hear: "Optimization is the root of all evil"

rebootthesystem 9y ago

Thank you for bringing attention to this. Experiencing this on our W10 workstations. I hope MS does something about this immediately. It's maddening.

lobo_tuerto 9y ago

I have a new Ryzen 7 CPU (you know, 8 cores, 16 threads), with 64GB RAM and for HDD a Samsung 960 PRO M.2 drive. But when I went and plugged in an external Seagate 4TB drive, and tried to "dd zero" the s#it out of it, my whole system became unresponsive after a while, obviously I had to reset the machine as it wouldn't "kill -9" the process that made the system unresponsive. Trying to type was a no go as keys would sometimes become "stuck". Moving the mouse around was an exercise in predictability too. All this happened in the latest Ubuntu 17.04 64bit... #sadstory