8 ms·

“Unexplainable” core dump (2011)

193 points by curling_grad 4y ago

ksherlock 4y ago
Another good source of "impossible" bugs is overclocking. https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35...
- dnedic 4y ago
  All chips degrade over time, so it doesn't have to be overlocking, after running a CPU for many years it may turn out that it can no longer reliably work at a stock voltage levels. It is very unlikely but it does happen.
throwawaylinux 4y ago
In this CPU, AMD first introduced the "stack engine". This is likely a bug in that feature. It has a speculative stack address delta register in the front-end that is updated directly with push/pop instructions, and that delta is dispatched with the stack memory uop to be added to the original stack address register when doing address generation in the load/store units. The delta has to be small, because you don't want big adders in the front end and because the delta has to be sent down with the push/pop memory uops. That means it can overflow or underflow, at which point it has to be reset by sending a synchronize operation to the back-end to update the original stack register (agner has a better description). So the delta register is probably 10-12 bits on Barcelona, and this bug is probably a corner case where the stack register update is happening, hence 1024 bytes off. Perhaps there is a window where a uop can get the old delta value, but the new base value (or vice versa) when a sync is operation is concurrent. Setting that MSR value possibly disables that stack engine feature entirely, or it could be it disables some aggressive and complicated detail where the bug is, e.g., allowing stack operations to run concurrently while flush operations are in progress. It's not a coincidence there just happens to exist a way to disable this at runtime. The way processors are designed means that everything must be able to be observed, debugged, and fixed in the field. That means everything has to have fine-grained ability to control, disable, enable safer fallback paths, and even engage additional logic to reduce the state space in some cases (e.g., serialize pipeline while a particular operation occurs). Usually these bug fixes decrease performance (except in cases where a performance bug is found and the fix actually increases performance), so you want the switches to be very fine-grained. So it's possible they fixed the stack engine bug without disabling it entirely. It would be like shipping software and providing support and bug fixes for it for the next 5-10 years without patching the software, only updating the config file. It's quite amazing. For every one of these issues that hits the field, there will be many found internally during the internal hardware bring up and verification (which will be ongoing for at least part of the life of the CPU).
- rcxdude 4y ago
  Yup, these are often colloquially called "chicken bits" because you put them in if you're afraid your new feature won't work (It will generally just fall back to the previous battle-tested implementation). I often wonder how far back you can pare a modern CPU with these.
jalbertoni 4y ago
Reminds me of what we dubbed "the cosmic ray incident." During college, we were arranged in groups of 2 or 3 to do some pair programming for the more complicated exercises. I had attached the debugger and had set a variable to 99, so the loop would execute one more time and we could test if our changes would work. Went through the instructions step by step, and suddenly we got a segmentation fault. Between setting it to 99 and it accessing some data later on, the value changed to 107. Quite a bit of confusion ensued. I made a backup of the executable before recompiling. Running them again, both the new version and the backup worked perfectly. The files matched bit for bit. To this day we have no idea what caused that bit flip.
throwaway2037 4y ago
I recognize this StackOverflow user name. I have read many of their answers. They are usually excellent. See more here: https://stackoverflow.com/users/50617/employed-russian https://stackoverflow.com/users/50617/employed-russian
jeffrallen 4y ago
If you are lucky, you'll debug a CPU bug once in a lifetime. If you're really lucky, you'll never see one at all!
deathanatos 4y ago
I think the closest I ever came to this was when I stumbled into a bug in git. We were asking git to do something utterly trivial, like clone, and it was segfaulting. We installed the debug symbols or something (as I recall you can install these separately in Debian) and started trying to see where the function was crashing. The crash was in a parser, and the code was doing, char *foo = strstr(input, "something constant"); and segfaulting later (but not chasing null!), during the first use of foo. I figured that input must, therefore, be a bad pointer, since the string literal is by definition fine, and the only way to get a bad pointer out of strstr was to give it bad input in the first place. So, in gdb, I print the input point. It's valid. "Damn" I think to myself "we've gotten unlucky, and whatever triggers the cascade of UB hasn't happened this run. Freakin' heisenbug." So, I told gdb to just continue running the program: it crashed. Same backtrace: dereferencing foo triggered a segfault, and not with a null pointer. If the input pointer to strstr is valid, how can the output be a bad pointer? strstr is documented as, These functions return a pointer to the beginning of the located substring, or NULL if the substring is not found. So, what gives? I attempted to debug strstr … that was almost a mistake. strstr is, unfortunately heavily optimized. It might have been this one: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S;h=c6aa8f45a60d175826c3d4e6eb483495a2cbf424;hb=f2698954ff9c2f9626d4bcb5a30eb5729714e0b0 https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... I'm okay with a debugger, but I'm not going the be able to follow where vectorized code is going wrong. Almost stupidly, I told strstr to (f)inish the function's execution, at which point gdb printed the return value: it was a valid pointer! "Odd. Is this run going to succeed?" I continue execution: it crashes. I restart the debugging, and run to the completion of strstr: the pointer is correct, but again, crash. How. The segfaulting address isn't the pointer strstr is returning, either, and nothing modifies foo after the strstr. I single step out of strstr, and immediately print foo. The resulting pointer is wrong. strstr is working fine … but what, the assignment operator is broken? At this point, I'm sure I'm crazy. I disassemble the source, and lo and behold, the disassembly is trivially wrong. Forgive me as I can't amd64 unless I'm looking at it, but the disassembly is something like, call strstr cltq <store foo> We look the odd instruction up; it sign extends eax into rax. That's … not a valid operation on a supposed char pointer … there's not even technically a value in eax at this point. strstr returned a pointer in rax. eax is just the lower 32 bits of that pointer, and sign extending that back into rax makes no sense. It's like the compiler thought the return from strstr was a signed int … and that's where it hits me: C is shooting us in the foot again. > and if no declaration is visible for this identifier, the identifier is implicitly declared exactly as if. in the innermost block containing the function call. the declaration > extern int identifier(); > appeared. The default return for an undeclared function, in C89, is int. (I'm actually not sure what C11 says about this. The C89 rule appears to be gone, but there doesn't seem to be anything in its place, which is bizarre. "Undeclared" does not appear, except for an unrelated footnote.) Slap in the proper include for strstr, recompile. No segfault, disassembly is correct. Upgrade to latest version, disassemble: disassembly is correct. Someone else fixed the bug, in the meantime. Should have just tried upgrading in the first place… (After typing this up: had to dig up the debugging session. It was strchr, not strstr. Potato, potato. Someday … this should be a blog post…)
m-ee 4y ago
I had something similar to some of the things described here, with a dumber cause. Started getting hardfaults in my firmware after a context switch. Thought bad array access, incorrect FPU settings, misbehaving interrupt. All dead ends. Realized that it was only happening when a particular motor was running, then further isolated to when it was only running at higher duty cycles. I had tested the motor in isolation but at low speeds, so I didn't see issues until I tried to run the full application. Turns out the EE had royally screwed up the current sense circuit, and the MCU on the ADC was seeing voltages lower than -1V, well beyond the absolute max ratings for the chip. I guess ST doesn't make guarantees about what happens when you violate those ratings, but a corrupt stack pointer is not what I would have expected.
ndemir 4y ago
This reminded me a problem that I have seen almost 17-18 years ago. There was a file that we were trying to download but it was just being interrupted during download process. When we tried to download the file on another computer, it was fine. Long story short; we changed the ethernet card of that computer and it started downloading file. I don't remember all the details; but probably something wrong witj ethernet card driver.
jacooper 4y ago
Debugging that must've been a PITA for sure.
- cybrox 4y ago
  If you want to experience this in a slightly more controlled way, I'd recommend you give the game "Turing complete" a spin. It lets you build your own turing complete processor, and define a simple assembly language, starting from NAND gates and you can create your own arbitrarily wild edge cases for specific opcode combinations.
  - macintux 4y ago
    https://turingcomplete.game/ https://turingcomplete.game/ to save other curious parties like myself the search.
markus_zhang 4y ago
Sounds fun, wish I had the talent to do some serious debugging.
Izkata 4y ago
> Our code and compilers are constantly changing, and the problem disappeared as suddenly as it appeared ... only to happen again 2 years later in a completely unrelated executable. It does not encourage me how much this sounds like the short story "Coding Machines". The original post even happened right about 2 years after the short story was posted, then in that comment reoccured after another 2 years. https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
- moffkalast 4y ago
  Damn, what a story. Though in reality it definitely would've been some guy on the compiler dev team adding it before publishing binaries. I wonder if you could set it up to inject a halt into compiled code that runs if some conditions are met, crashing most of the word's infrastructure on a predetermined date.
- twiss 4y ago
  Great story. It reminded me of Ken Thompson's Reflections on Trusting Trust, I assume it was inspired by that: https://cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Reflect...
- inputError 4y ago
  [dead]
- armchairhacker 4y ago
  I have to say this is a really good story. I think whoever wrote it really understood computers and SE culture, it’s extremely realistic except for the metal switch part and AI. (Or…maybe not except?)
- account-5 4y ago
  Excellent story, mildly terrified!
- highspeedbus 4y ago
  That was a great read, thanks.
ericbarrett 4y ago
This is great. Reminds me of a crash I saw early in my career. It was a null-pointer exception, except it occurred right after confirming the address was non-null. This was on a single core with a non-preemptible kernel. So the processor just took the wrong branch! There was simply no other explanation.
- jrpelkonen 4y ago
  Interesting, how did you fix it? Negate the comparison with an appropriate comment?
  - ericbarrett 4y ago
    There was no fix! It was a one-off hardware error.
- JoeAltmaier 4y ago
  Got to read the assembly to really know what happened. E.g. if the architecture has pointers with non-address bits (modes or segments or whatever) and those bits were set yet the rest of the address was 'null', and the check was for 'all bits zero' then you could conceivably get that situation.
- mcculley 4y ago
  What hardware/platform was this? I worked on AIX on POWER a long time ago and it had to map the zero page read-only just to support speculative execution of dereferencing the NULL pointer, if I remember right. If you were on a platform that did this wrong, speculative execution could have been dereferencing the NULL pointer.
  - ericbarrett 4y ago
    It was a DEC Alpha chip, over two decades ago. (Or MIPS?) Pretty sure there was no speculative execution. In any case, the senior engineer I was working with would have known that—getting to "it just did the wrong thing" was our final and least satisfying conclusion!
dekhn 4y ago
One of the best bugs I've seen had a description fairly similar to this. Hot routine run at scale (floating point math for ads ML training) fails at a rate about 0.000000001. Turned out to be a very obscure bug in the context switching code in the linux kernel, the FP registers weren't being restored properly.
- logicchop 4y ago
  I suspect that windows still has a subtle FP restoration bug. We do large scale validation of floating point data and occasionally get ever so subtly different results.
  - tremon 4y ago
    Given that you say "subtly", have you ruled out rounding/precision errors? I wouldn't be surprised if some processors would play fast-and-loose with the number of significant bits they really honour.
  - dekhn 4y ago
    I would be interested in hear the use cases for large scale validation of floating point data. I used to work with processors that occasionally corrupted operations due to hardware manufacturing defects and these kinds of problems are exceptionally hard to debug, so I'm curious what techniques are used. In our case, we built programs that ran enormous numbers of semi-random programs on the accelerator and compared it to reliable results computed offline. About 1 in 1000 chips would - reproducibly - fail certain operations. Identifying this helped solve problems many of our researchers reported on specific accelerator clusters- they would get a Nan in their gradients which would kill training, and it was almost always explainable by a single processor (out of ~thousands) occasionally corrupting a float.
  - kevingadd 4y ago
    This is a known thing that happens with bad drivers, they can mess with user-mode FP flags. For a while (IIRC) Cisco's VPN software was corrupting FP state and causing Firefox to hang/crash, for example. At a previous company we had to run a dll in our web servers (provided by the payment processing company) for PCI compliance reasons, and we later discovered it was messing with our FP flags and as a result serialization code was producing invalid floats. That was a fun one.
dekhn 4y ago
One of the best bugs I've seen had a description fairly similar to this. Hot routine run at scale (floating point math for ads ML training) fails at a rate about 0.000000001. Turned out to be a very obscure bug in the context switching code in the linux kernel, the FP registers weren't being restored properly. Another one, the debugging was aided by the fact the developers ensured that everything was accessed through const pointers, so it wasn't their code corrupting their memory.
- sidewndr46 4y ago
  It has been a while, but a switch to kernel mode followed by a switch back to the same user mode process doesn't actually mess with FP registers. The idea being, the kernel should not be using those anyways. Also minor point: a const pointer is a pointer which always points at the same address. You can still change what is pointed at. You probably meant "a pointer to const"
  - dekhn 4y ago
    Not a switch back to the same process- context switching during normal process switching. People have been using the term "const pointer" to refer to "a pointer to const" for 20+ years (as long as I've been doing C++), although that's probably more out of laziness than incorrectness. Certainly the language definition didn't do anybody favors.
    - sidewndr46 4y ago
      OK, that's interesting. Have any details on how FP wasn't being restored?
- lisper 4y ago
  I had one of these back in the 90s that turned out to be a compiler bug. It was code that ran a mobile robot with an arm. Exact same code running on a Sun workstation never failed, but running on an embedded system running vxWorks crashed intermittently, but only when the arm was moving. Entire heap was corrupted, so by the time the crash occurred there was no hope of getting a stack trace or any hint of what went wrong upstream. Turned out to be two mis-ordered instructions that accessed a value on the stack after the stack pointer had been popped. On vxWorks, interrupts used the same stack as the currently running process, so if an interrupt occurred exactly between these two instructions it would clobber that value, and chaos ensued. Took a full year to figure it out. Good times.

193 points by curling_grad 4y ago

Another good source of "impossible" bugs is overclocking. https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35923 https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35...

dnedic 4y ago
All chips degrade over time, so it doesn't have to be overlocking, after running a CPU for many years it may turn out that it can no longer reliably work at a stock voltage levels. It is very unlikely but it does happen.

throwawaylinux 4y ago

In this CPU, AMD first introduced the "stack engine". This is likely a bug in that feature. It has a speculative stack address delta register in the front-end that is updated directly with push/pop instructions, and that delta is dispatched with the stack memory uop to be added to the original stack address register when doing address generation in the load/store units. The delta has to be small, because you don't want big adders in the front end and because the delta has to be sent down with the push/pop memory uops. That means it can overflow or underflow, at which point it has to be reset by sending a synchronize operation to the back-end to update the original stack register (agner has a better description). So the delta register is probably 10-12 bits on Barcelona, and this bug is probably a corner case where the stack register update is happening, hence 1024 bytes off. Perhaps there is a window where a uop can get the old delta value, but the new base value (or vice versa) when a sync is operation is concurrent. Setting that MSR value possibly disables that stack engine feature entirely, or it could be it disables some aggressive and complicated detail where the bug is, e.g., allowing stack operations to run concurrently while flush operations are in progress. It's not a coincidence there just happens to exist a way to disable this at runtime. The way processors are designed means that everything must be able to be observed, debugged, and fixed in the field. That means everything has to have fine-grained ability to control, disable, enable safer fallback paths, and even engage additional logic to reduce the state space in some cases (e.g., serialize pipeline while a particular operation occurs). Usually these bug fixes decrease performance (except in cases where a performance bug is found and the fix actually increases performance), so you want the switches to be very fine-grained. So it's possible they fixed the stack engine bug without disabling it entirely. It would be like shipping software and providing support and bug fixes for it for the next 5-10 years without patching the software, only updating the config file. It's quite amazing. For every one of these issues that hits the field, there will be many found internally during the internal hardware bring up and verification (which will be ongoing for at least part of the life of the CPU).

rcxdude 4y ago
Yup, these are often colloquially called "chicken bits" because you put them in if you're afraid your new feature won't work (It will generally just fall back to the previous battle-tested implementation). I often wonder how far back you can pare a modern CPU with these.

jalbertoni 4y ago

Reminds me of what we dubbed "the cosmic ray incident." During college, we were arranged in groups of 2 or 3 to do some pair programming for the more complicated exercises. I had attached the debugger and had set a variable to 99, so the loop would execute one more time and we could test if our changes would work. Went through the instructions step by step, and suddenly we got a segmentation fault. Between setting it to 99 and it accessing some data later on, the value changed to 107. Quite a bit of confusion ensued. I made a backup of the executable before recompiling. Running them again, both the new version and the backup worked perfectly. The files matched bit for bit. To this day we have no idea what caused that bit flip.

throwaway2037 4y ago

I recognize this StackOverflow user name. I have read many of their answers. They are usually excellent. See more here: https://stackoverflow.com/users/50617/employed-russian https://stackoverflow.com/users/50617/employed-russian

jeffrallen 4y ago

If you are lucky, you'll debug a CPU bug once in a lifetime. If you're really lucky, you'll never see one at all!

deathanatos 4y ago

I think the closest I ever came to this was when I stumbled into a bug in git. We were asking git to do something utterly trivial, like clone, and it was segfaulting. We installed the debug symbols or something (as I recall you can install these separately in Debian) and started trying to see where the function was crashing. The crash was in a parser, and the code was doing, char *foo = strstr(input, "something constant"); and segfaulting later (but not chasing null!), during the first use of foo. I figured that input must, therefore, be a bad pointer, since the string literal is by definition fine, and the only way to get a bad pointer out of strstr was to give it bad input in the first place. So, in gdb, I print the input point. It's valid. "Damn" I think to myself "we've gotten unlucky, and whatever triggers the cascade of UB hasn't happened this run. Freakin' heisenbug." So, I told gdb to just continue running the program: it crashed. Same backtrace: dereferencing foo triggered a segfault, and not with a null pointer. If the input pointer to strstr is valid, how can the output be a bad pointer? strstr is documented as, These functions return a pointer to the beginning of the located substring, or NULL if the substring is not found. So, what gives? I attempted to debug strstr … that was almost a mistake. strstr is, unfortunately heavily optimized. It might have been this one: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/strstr-sse2-unaligned.S;h=c6aa8f45a60d175826c3d4e6eb483495a2cbf424;hb=f2698954ff9c2f9626d4bcb5a30eb5729714e0b0 https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... I'm okay with a debugger, but I'm not going the be able to follow where vectorized code is going wrong. Almost stupidly, I told strstr to (f)inish the function's execution, at which point gdb printed the return value: it was a valid pointer! "Odd. Is this run going to succeed?" I continue execution: it crashes. I restart the debugging, and run to the completion of strstr: the pointer is correct, but again, crash. How. The segfaulting address isn't the pointer strstr is returning, either, and nothing modifies foo after the strstr. I single step out of strstr, and immediately print foo. The resulting pointer is wrong. strstr is working fine … but what, the assignment operator is broken? At this point, I'm sure I'm crazy. I disassemble the source, and lo and behold, the disassembly is trivially wrong. Forgive me as I can't amd64 unless I'm looking at it, but the disassembly is something like, call strstr cltq <store foo> We look the odd instruction up; it sign extends eax into rax. That's … not a valid operation on a supposed char pointer … there's not even technically a value in eax at this point. strstr returned a pointer in rax. eax is just the lower 32 bits of that pointer, and sign extending that back into rax makes no sense. It's like the compiler thought the return from strstr was a signed int … and that's where it hits me: C is shooting us in the foot again. > and if no declaration is visible for this identifier, the identifier is implicitly declared exactly as if. in the innermost block containing the function call. the declaration > extern int identifier(); > appeared. The default return for an undeclared function, in C89, is int. (I'm actually not sure what C11 says about this. The C89 rule appears to be gone, but there doesn't seem to be anything in its place, which is bizarre. "Undeclared" does not appear, except for an unrelated footnote.) Slap in the proper include for strstr, recompile. No segfault, disassembly is correct. Upgrade to latest version, disassemble: disassembly is correct. Someone else fixed the bug, in the meantime. Should have just tried upgrading in the first place… (After typing this up: had to dig up the debugging session. It was strchr, not strstr. Potato, potato. Someday … this should be a blog post…)

m-ee 4y ago

I had something similar to some of the things described here, with a dumber cause. Started getting hardfaults in my firmware after a context switch. Thought bad array access, incorrect FPU settings, misbehaving interrupt. All dead ends. Realized that it was only happening when a particular motor was running, then further isolated to when it was only running at higher duty cycles. I had tested the motor in isolation but at low speeds, so I didn't see issues until I tried to run the full application. Turns out the EE had royally screwed up the current sense circuit, and the MCU on the ADC was seeing voltages lower than -1V, well beyond the absolute max ratings for the chip. I guess ST doesn't make guarantees about what happens when you violate those ratings, but a corrupt stack pointer is not what I would have expected.

ndemir 4y ago

This reminded me a problem that I have seen almost 17-18 years ago. There was a file that we were trying to download but it was just being interrupted during download process. When we tried to download the file on another computer, it was fine. Long story short; we changed the ethernet card of that computer and it started downloading file. I don't remember all the details; but probably something wrong witj ethernet card driver.

jacooper 4y ago

Debugging that must've been a PITA for sure.

cybrox 4y ago
If you want to experience this in a slightly more controlled way, I'd recommend you give the game "Turing complete" a spin. It lets you build your own turing complete processor, and define a simple assembly language, starting from NAND gates and you can create your own arbitrarily wild edge cases for specific opcode combinations.
- macintux 4y ago
  https://turingcomplete.game/ https://turingcomplete.game/ to save other curious parties like myself the search.

markus_zhang 4y ago

Sounds fun, wish I had the talent to do some serious debugging.

Izkata 4y ago

> Our code and compilers are constantly changing, and the problem disappeared as suddenly as it appeared ... only to happen again 2 years later in a completely unrelated executable. It does not encourage me how much this sounds like the short story "Coding Machines". The original post even happened right about 2 years after the short story was posted, then in that comment reoccured after another 2 years. https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/

moffkalast 4y ago
Damn, what a story. Though in reality it definitely would've been some guy on the compiler dev team adding it before publishing binaries. I wonder if you could set it up to inject a halt into compiled code that runs if some conditions are met, crashing most of the word's infrastructure on a predetermined date.
twiss 4y ago
Great story. It reminded me of Ken Thompson's Reflections on Trusting Trust, I assume it was inspired by that: https://cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Reflect...
inputError 4y ago
[dead]
armchairhacker 4y ago
I have to say this is a really good story. I think whoever wrote it really understood computers and SE culture, it’s extremely realistic except for the metal switch part and AI. (Or…maybe not except?)
account-5 4y ago
Excellent story, mildly terrified!
highspeedbus 4y ago
That was a great read, thanks.

ericbarrett 4y ago

This is great. Reminds me of a crash I saw early in my career. It was a null-pointer exception, except it occurred right after confirming the address was non-null. This was on a single core with a non-preemptible kernel. So the processor just took the wrong branch! There was simply no other explanation.

jrpelkonen 4y ago
Interesting, how did you fix it? Negate the comparison with an appropriate comment?
- ericbarrett 4y ago
  There was no fix! It was a one-off hardware error.
JoeAltmaier 4y ago
Got to read the assembly to really know what happened. E.g. if the architecture has pointers with non-address bits (modes or segments or whatever) and those bits were set yet the rest of the address was 'null', and the check was for 'all bits zero' then you could conceivably get that situation.
mcculley 4y ago
What hardware/platform was this? I worked on AIX on POWER a long time ago and it had to map the zero page read-only just to support speculative execution of dereferencing the NULL pointer, if I remember right. If you were on a platform that did this wrong, speculative execution could have been dereferencing the NULL pointer.
- ericbarrett 4y ago
  It was a DEC Alpha chip, over two decades ago. (Or MIPS?) Pretty sure there was no speculative execution. In any case, the senior engineer I was working with would have known that—getting to "it just did the wrong thing" was our final and least satisfying conclusion!

dekhn 4y ago

logicchop 4y ago
I suspect that windows still has a subtle FP restoration bug. We do large scale validation of floating point data and occasionally get ever so subtly different results.
- tremon 4y ago
  Given that you say "subtly", have you ruled out rounding/precision errors? I wouldn't be surprised if some processors would play fast-and-loose with the number of significant bits they really honour.
- dekhn 4y ago
  I would be interested in hear the use cases for large scale validation of floating point data. I used to work with processors that occasionally corrupted operations due to hardware manufacturing defects and these kinds of problems are exceptionally hard to debug, so I'm curious what techniques are used. In our case, we built programs that ran enormous numbers of semi-random programs on the accelerator and compared it to reliable results computed offline. About 1 in 1000 chips would - reproducibly - fail certain operations. Identifying this helped solve problems many of our researchers reported on specific accelerator clusters- they would get a Nan in their gradients which would kill training, and it was almost always explainable by a single processor (out of ~thousands) occasionally corrupting a float.
- kevingadd 4y ago
  This is a known thing that happens with bad drivers, they can mess with user-mode FP flags. For a while (IIRC) Cisco's VPN software was corrupting FP state and causing Firefox to hang/crash, for example. At a previous company we had to run a dll in our web servers (provided by the payment processing company) for PCI compliance reasons, and we later discovered it was messing with our FP flags and as a result serialization code was producing invalid floats. That was a fun one.

dekhn 4y ago

One of the best bugs I've seen had a description fairly similar to this. Hot routine run at scale (floating point math for ads ML training) fails at a rate about 0.000000001. Turned out to be a very obscure bug in the context switching code in the linux kernel, the FP registers weren't being restored properly. Another one, the debugging was aided by the fact the developers ensured that everything was accessed through const pointers, so it wasn't their code corrupting their memory.

sidewndr46 4y ago
It has been a while, but a switch to kernel mode followed by a switch back to the same user mode process doesn't actually mess with FP registers. The idea being, the kernel should not be using those anyways. Also minor point: a const pointer is a pointer which always points at the same address. You can still change what is pointed at. You probably meant "a pointer to const"
- dekhn 4y ago
  Not a switch back to the same process- context switching during normal process switching. People have been using the term "const pointer" to refer to "a pointer to const" for 20+ years (as long as I've been doing C++), although that's probably more out of laziness than incorrectness. Certainly the language definition didn't do anybody favors.
  - sidewndr46 4y ago
    OK, that's interesting. Have any details on how FP wasn't being restored?
lisper 4y ago
I had one of these back in the 90s that turned out to be a compiler bug. It was code that ran a mobile robot with an arm. Exact same code running on a Sun workstation never failed, but running on an embedded system running vxWorks crashed intermittently, but only when the arm was moving. Entire heap was corrupted, so by the time the crash occurred there was no hope of getting a stack trace or any hint of what went wrong upstream. Turned out to be two mis-ordered instructions that accessed a value on the stack after the stack pointer had been popped. On vxWorks, interrupts used the same stack as the currently running process, so if an interrupt occurred exactly between these two instructions it would clobber that value, and chaos ensued. Took a full year to figure it out. Good times.