17 ms·
Why is my CPU usage always 100%?
- veltas 2y agoIt doesn't feel like reading 4 times is necessarily a portable solution, if there will be more versions at different speeds and different I/O architectures; or how this will work under more load, and whether the original change was done to fix some other performance problem OP is not aware of, but not sure what else can be done. Unfortunately many vendors like Marvell can seriously under-document crucial features like this. If anything it would be good to put some of this info in the comment itself, not very elegant but how else practically are we meant to keep track of this, is the mailing list part of the documentation? Doesn't look like there's a lot of discussion on the mailing list, but I don't know if I'm reading the thread view correctly.
- _nalply 2y agoI also wondered about this, but there's a crucial differnce, no idea if it matters: in that loop it reads the register, so the register is read at least 4 times.
- adrian_b 2y agoThis is a workaround for a hardware bug of a certain CPU. Therefore it cannot really be portable, because other timers in other devices will have different memory maps and different commands for reading. The fault is with the designers of these timers, who have failed to provide a reliable way to read their value. It in hard to believe that this still happens in this century, because reading correct values despite the fact that the timer is incremented or decremented continuously is an essential goal in the design of any timer that may be read, and how to do it has been well known for more than 3 quarters of century. The only way to make such a workaround somewhat portable is to parametrize it, e.g. with the number of retries for direct reading or with the delay time when reading the auxiliary register. This may be portable between different revisions of the same buggy timer, but the buggy timers in other unrelated CPU designs will need different workarounds anyway.
- stkdump 2y ago> how to do it has been well known for more than 3 quarters of century Don't leave me hanging! How to do it?
- adrian_b 2y agoDirect reading without the risk of reading incorrect values is possible only when the timer is implemented using a synchronous counter instead of an asynchronous counter and the synchronous counter must be fast enough to ensure a stable correct value by the time when it is read, and the reading signal must be synchronized with the timer clock signal. Synchronous counters are more expensive in die area than asynchronous counters, especially at high clock frequencies. Moreover, it may be difficult to also synchronize the reading signal with the timer clock. Therefore the second solution may be preferable, which uses a separate capture register for reading the timer value. This was implemented in the timer described in TFA, but it was done in a wrong way. The capture register must either ensure that the capture is already complete by the time when it is possible to read its value after giving a capture command, or it must have some extra bit that indicates when its value is valid. In this case, one can read the capture register until the valid bit is on, having a complete certainty that the end value is correct. When adding some arbitrary delay between the capture command and reading the capture register, you can never be certain that the delay value is good. Even when the chosen delay is 100% effective during testing, it can result in failures on other computers or when the ambient temperature is different.
- veltas 2y ago> This is a workaround for a hardware bug of a certain CPU. What about different variants, revisions, and speeds of this CPU?
- Karliss 2y agoThe related part of doc has one more note "This request requires up to three timer clock cycles. If the selected timer is working at slow clock, the request could take longer." From the way doc is formatted it's not fully clear what "this request" refers to. It might explain where 3-5 attempts come from, and that it might not be pulled completely out of thin air. But the part about taking up to but sometimes more clock cycles makes it impossible to have a "proper" solution without guesswork or further clarifications from vendor. "working at slow clock" part, might explain why some other implementations had different code path for 32.768 KHz clocks. According to docs there are two available clock sources "Fast clock" and "32768 Hz" which could mean that "slow clock" refers to specific hardware functionality is not just a vague phrase. As for portability concerns, this is already low level hardware specific register access. If Marvell releases new SOC not only there is no assurance that will require same timing, it might was well have different set of registers which require completely different read and setup procedure not just different timing. One thing that slightly confuses me - the old implementation had 100 cycles of "cpu_relax()" which is unrelated to specific timer clock, but neither is reading of TMR_CVWR register. Since 3-5 of cycles of that worked better than 100 cycles of cpu_relex, it clearly takes more time unless cpu_relax part got completely optimized out. At least I didn't find any references mentioning that timer clock affects read time of TMR_CVWR.
- veltas 2y agoIt sounds like this is an old CPU(?), so no need to worry about the future here. > I didn't find any references mentioning that timer clock affects read time of TMR_CVWR. Reading the register might be related to the timer's internal clock, as it would have to wait for the timer's bus to respond. This is essentially implied if Marvell recommend re-reading this register, or if their reference implementation did so. My main complaint is it's all guesswork, because Marvell's docs aren't that good.
- MBCook 2y agoThe Chumby hardware I’m thinking of is from 2010 or so. So if that’s it, it would certainly be old. And it would explain a possible relation with the OLPC having a similar chip. https://en.wikipedia.org/wiki/Chumby https://en.wikipedia.org/wiki/Chumby
- begueradj 2y agoOops, this is not valid.
- M95D 2y agoI'm sure a few more software updates will take care of this little problem...
- zaik 2y agoYou're probably thinking about memory and caching. There are no advantages to keeping the CPU at 100% when no workload needs to be done.
- josephg 2y agoOnly when your computer actually has work to do. Otherwise your CPU is just a really expensive heater. Modern computers are designed to idle at 0% then temporarily boost up when you have work to do. Then once the task is done, they can drop back to idle and cool down again.
- PUSH_AX 2y agoNot that I disagree, but when exactly in modern operating systems are there moments where there are zero instructions being executed? Surely there are always processes doing background things?
- pintxo 2y agoWith multi-core cpus, some of them can be fully off, while others handle any background tasks.
- _flux 2y agoThere are a lot of such moments, but they are just short. When you're playing music, you download a bit of data from the network or the SSD/HDD by first issuing a request and then waiting (i.e. doing nothing) to get the short piece of data back. Then you decode it and upload a short bit of the sound to your sound card and then again you wait for new space to come up, before you send more data. One of the older ways (in x86 side) to do this was to invoke the HLT instruction https://en.wikipedia.org/wiki/HLT_(x86_instruction) https://en.wikipedia.org/wiki/HLT_(x86_instruction) : you stop the processor, and then the processor wakes up when an interrupt wakes it up. An interrupt might come from the sound card, network card, keyboard, GPU, timer (e.g. 100 times a second to schedule an another process, if some process exists that is waiting for CPU), and during the time you wait for the interrupt to happen you just do nothing, thus saving energy. I suspect things are more complicated in the world of multiple CPUs.
- g-b-r 2y agoI expected it to be about holding down the spacebar :/
- sneela 2y agoThis is a wonderful write-up and a very enjoyable read. Although my knowledge about systems programming on ARM is limited, I know that it isn't easy to read hardware-based time counters; at the very least, it's not as simple as the x86 rdtsc [1]. This is probably why the author writes: > This code is more complicated than what I expected to see. I was thinking it would just be a simple register read. Instead, it has to write a 1 to the register, and then delay for a while, and then read back the same register. There was also a very noticeable FIXME in the comment for the function, which definitely raised a red flag in my mind. Regardless, this was a very nice read and I'm glad they got down to the issue and the problem fixed. [1]: https://www.felixcloutier.com/x86/rdtsc https://www.felixcloutier.com/x86/rdtsc.
- pm215 2y agoBear in mind that the blog post is about a 32 bit SoC that's over a decade old, and the timer it is reading is specific to that CPU implementation. In the intervening time both timers and performance counters have been architecturally standardised, so on a modern CPU there is a register roughly equivalent to the one x86 rdtsc uses and which you can just read; and kernels can use the generic timer code for timers and don't need to have board specific functions to do it. But yeah, nice writeup of the kinds of problem you can run into in embedded systems programming.
- InsomniacL 2y ago> Chumby’s kernel did a total of 5 reads of the CVWR register. The other two kernels did a total of 3 reads. > I opted to use 4 as a middle ground reminded me of xkcd: Standards https://xkcd.com/927/ https://xkcd.com/927/
- thrdbndndn 2y agoI don't get the fix. Why reading it multiple times will fix the issue? Is it just because reading takes time, therefore reading multiple time makes the needed time from writing to reading passes? If so, it sounds like a worse solution than just extending waiting delay longer like the author did initially. If not, then I would like to know the reason. (Needless to say, a great article!)
- rep_lodsb 2y agoIt's possible that actually reading the register takes (significantly) more time than an empty countdown loop. A somewhat extreme example of that would be on x86, where accessing legacy I/O ports for e.g. the timer goes through a much lower-clocked emulated ISA bus. However, a more likely explanation is the use of "volatile" (which only appears in the working version of the code). Without it, the compiler might even have completely removed the loop?
- deng 2y ago> However, a more likely explanation is the use of "volatile" (which only appears in the working version of the code). Without it, the compiler might even have completely removed the loop? No, because the loop calls cpu_relax(), which is a compiler barrier. It cannot be optimized away. And yes, reading via the memory bus is much, much slower than a barrier. It's absolutely likely that reading 4 times from main memory on such an old embedded system takes several hundred cycles.
- rep_lodsb 2y agoYou're right, didn't account for that. Though even when declared volatile, the counter variable would be on the stack, and thus already in the CPU cache (at least 32K according to the datasheet)? Looking at the assembly code for both versions of this delay loop might clear it up.
- deng 2y agoThe only thing volatile does is to assure that the value is read from memory each time (which implicitly also forbids optimizations). Whether that memory is in a CPU cache is purely a hardware issue and outside the C specification. If you read something like a hardware register, you yourself need to take care in some way that a hardware cache will not give you old values (by mapping it into a non-cached memory area, or by forcing a cache update). If you for-loop over something that acts as a compiler barrier, all that 'volatile' on the counter variable will do is potentially make the for-loop slower. There's really just very few reasons to ever use 'volatile'. In fact, the Linux kernel even has its own documentation why you should usually not use it: https://www.kernel.org/doc/html/latest/process/volatile-considered-harmful.html https://www.kernel.org/doc/html/latest/process/volatile-cons...
- TrickyReturn 2y agoProbably running Slack...
- perayi5293 2y ago[flagged]
- deleted 2y ago[deleted]
- raincole 2y agoJust twice so far. Not even from the same account. Why so hostile?
- rbanffy 2y agoEven HN boosts submissions admins deem good enough that didn't get sufficient exposure the last time they were published. And, since this one got quite a bit of engagement, it demonstrates that multiple submissions are OK. Otherwise a good article would have been lost.
- Aissen 2y agoI, for one, welcome this repost of a very interesting technical article that I had missed, and arguably did not get the audience it deserved 8 months ago.
- rbanffy 2y agoIn the late 1990's I worked in a company that had a couple mainframes in their fleet and once I looked into a resource usage screen (Omegamon, perhaps? Is it that old?) and noticed the CPU was pegged at 100%. I asked the operator if that was normal. His answer was "Of course. We paid for that CPU, might as well use it". Funny though that mainframes are designed for that - most, if not all, non-application work is offloaded to other processors in the system so that the CPU can run applications as fast as it can.
- defrost 2y agoHaving a number of running processes take the CPU usage to 100% is one thing, have an under utilised CPU with almost no processes running report that usage is at 100% is another thing, the subject of the article here.
- rbanffy 2y agoI didn't intend this as an example of the issue the article mentions (a misreporting of usage because of a hardware design issue). It was just a fun example of how different hardware behaves differently. One can also say Omegamon (or whatever tool) was misreporting, because it didn't account for the processor time of the various supporting systems that dealt with peripheral operations. After all, they also paid for the disk controllers, disks, tape drives, terminal controllers and so on, so they could want to drive those to close to 100% as well.
- defrost 2y agoSure, no drama - I came across as a little dry and clipped as I was clarifying on the fly as it were. I had my time squeezing the last cycle possible from a Cyber 205 waaaay back in the day.
- datadrivenangel 2y agoSome mainframes have the ability to lock clock speed and always run at exactly 100%, so you can often have hard guarantees about program latency and performance.
- WediBlino 2y agoAn old manager of mine once spent the day trying to kill a process that was running at 99% on Windows box. When I finally got round to see what he was doing I was disappointed to find he was attempting to kill the 'system idle' process.
- TacticalCoder 2y ago[dead]
- belter 2y agoDid he have a pointy hair?
- cassepipe 2y agoI abandonned Windows 8 for linux because of an bug (?) where my HDD was showing it was 99% busy all the time. I had removed every startup program that could be and analysed thouroughly for any viruses, to no avail. Had no debugging skills at the time and wasn't sure the hardware could stand windows 10. That's how linux got me.
- saintfire 2y agoI had this happen with an nvme drive. Tried changing just about every setting that affected the slot. Everything worked fine on my Linux install ootb
- dr-detroit 2y ago[dead]
- margana 2y agoWhy is this such a huge issue if it merely shows it's busy, but the performance of it indicates that it actually isn't? Switching to Linux can be a good choice for a lot of people, the reason just seems a bit odd here. Maybe it was simply the straw that broke the camel's back.
- 2y ago
- a1o 2y agoThis was very well written, I somehow read every single line and didn't skip to the end. Great work too!
- amelius 2y agoTo diagnose, why not run "time top" and look at the user and sys outputs?
- RajT88 2y agoTIL there are still Chumby's alive in the wild. My Insignia Chumby 8 didn't last.
- evanjrowley 2y agoThis headline reminded me of Mumptris, an implementation of Tetris in the old mainframe-oriented language MUMPS, which by design, uses 100% CPU to reduce latency: https://news.ycombinator.com/item?id=4085593 https://news.ycombinator.com/item?id=4085593
- Suppafly 2y agoIsn't this one of those problems that switching to linux is supposed to fix?
- DougN7 2y agoHe’s on linux
- Suppafly 2y agoExactly, that's the joke. If it had been an issue on Windows the default response from folks here would be to switch to Linux instead of trying to get to the root of the issue. Guess I should have included an /s on my comment.
- dzaima 2y agoWell it's not like you could get to the root of the issue on windows, with it being closed-source and all.
- NotYourLawyer 2y agoThat’s an awful lot of effort to deal with an issue that was basically just cosmetic. I suspect at some point the author was just nerd sniped though.
- dougg3 2y agoTo be fair, other non-cosmetic stuff uses the CPU percentage. This same bug was preventing fast user suspend on the OLPC until they worked around it. It was also a fun challenge.
- ndesaulniers 2y agoGreat read! Eerily similar to some bugs I've had, but the root cause has been a compiler bug. Debugging a kernel that doesn't boot is... interesting. QEMU+GDB to the rescue.
- dmitrygr 2y agoCuriously, instead of "set capture reg, wait for clock edge, read", the "read reg twice, until same result is obtained" approach is ignored. This is strange as it is usually much faster - reading a 3.25MHz counter at 200MHz+ twice is very likely to see the same value twice. For a 32KHz counter, it is basically guaranteed. u32 val; do { val = readl(...); } while (val != readl(...)); return val; compiles to a nice 6-instr little function on arm/thumb too, with no delays readclock: LDR R2, =... 1: LDR R0, [R2] LDR R1, [R2] CMP R0, R1 BNE 1b BX LR
- markhahn 2y agovery nice investigation. shame about the unnecessary use of cat :)
- dougg3 2y agoHa! Guilty as charged. I have a coworker who's probably very disappointed in me right now!
- askvictor 2y agoMy recurring issue (on a variety of laptops, both Linux and Windows): the fans will start going full-blast, everything slows down, then as soon as I open a task manager CPU usage drops from 100% to something negligible.
- crazydoggers 2y agoYou my friend, most likely have mining malware on your systems. They’ll shutdown when they detect task manager is opened so you don’t notice them.
- michaelcampbell 2y agoThat was my thought too; one way to get another data point is to just run the task manager as soon as you boot and let it stay there. If the fan behavior NEVER comes back while doing that, another point in the "mining malware" favor (though of course, not definitive). Though he did say a VAREITY of laptops, both Windows and Linux. Can someone be _that_ unlucky?
- RicardoLuis0 2y agoor possibly the malware has spread to multiple of their devices?
- michaelcampbell 2y agoAlways possible; I haven't seen any legit mining malware in a long time. Mostly because it's so easy to spot with the CPU issue, and that CPU based mining even on someone else's machine isn't worth the trouble any more. But perhaps my circle/context is not that of a lot of people.
- askvictor 2y agoIf it was malware I'd expect it to happen more often; it's usually when I do have a lot of thing going on (browser tabs, VSCode sessions), so I spark up the task manager to work out the problem process, but CPU usage drops before I can investigate. Plus I'd be surprised if I got the same thing on both linux and windows
- WalterBright 2y agoI noticed that one time. Looked at the process list, and what was running was a program that enabled streaming. But since I wasn't streaming anything, I wondered what it was doing reading the disk drive. So I uninstalled it. Not having any programs that are not good citizens.
- rbohac 2y agoThis was a well written article! It was nice to read the process of troubleshooting with the rabbit holes included. Glad you stuck it out!
- deleted 2y ago[deleted]
- steventhedev 2y agoAside from the technical beauty of this post, what is the practical impact of this? Fan speeds should ideally be looking at temperature sensors, CPU idling is working albeit with interrupt waits as pointed out here. The only impact seems to be surprise that the CPU is working harder than it really is when looking at this number. It's far better to look at the system load (which was 0.0 - already a strong hint this system is working below capacity). It has a formal definition (average waiting cpu task queue depth over 1, 5, 10 minutes) and succinctly captures the concept of "this machine is under load". Many years ago, a coworker deployed a bad auditd config. CPU usage was below 10%, but system load was 20x the number of cores. We moved all our alerts to system load and used that instead.