10 ms·
Multiple OS Vendors Release Security Patches After Misinterpreting Intel Docs
- darkerside 9y agoI'm sure the initial reaction here is going to be lamentation about the state of documentation. People will correctly point out that, if multiple entities misread the documentation, it just have been unclear. And they are right. But that doesn't make this Intel's fault alone. Clear or unclear, the documentation described behavior that was understood at the Intel organization, and the shipped product worked as described. Where was the security testing at the OS level? Why can't there be automated test suites that catch unauthorized access issues before ship (if not before merge commit)? If your vendor delivers an insecure product and you don't discover it, how much blame do you share?
- sillysaurus3 9y agoWhy can't there be automated test suites that catch unauthorized access issues before ship (if not before merge commit)? Usually the search space is too large.
- rocqua 9y agoIsn't that what fuzzing is for?
- amluto 9y agoI’d be truly amazed if a fuzzer could have caught this one. You need to invoke debug syscalls with the right parameters and the do a magic two-to-three instruction sequence.
- learningto 9y agoIt's what formal verification is for!
- hedora 9y agoConcolic testing would probably catch it, but only if the person that implemented the hardware model for the theorem prover understood the Intel documentation, which seems unlikely. Basic fuzzing probably wouldn’t catch this; as the other comments point out, the search space is probably too large, and the set of vulnerable executions is probably too small for an undirected random search.
- throwaway2048 9y agoYour idea is akin to searching the space of unknown unknowns, By definition someone can not even begin to quantify the space of what you don't know you don't know.
- empath75 9y agoIt’s as simple as determining if a given program will halt.
- dboreham 9y agoYou forgot to include the sarcasm flag. I suppose "tag" these days, right?
- simion314 9y agoIf you can't trust the CPU documentation how can you test that addition works? even if you could test all possible combinations of terms to add and verify the results there may be a hidden flag somewhere that when flipped will change how addition works.
- Tobba_ 9y agoOn that subject, I'm curious whether there is any CPU out there that sets the overflow flag incorrectly when computing (-1) - n when n is the most negative number (which negates to itself, so implementing subtraction by simply negating the RHS and adding will screw up the flags).
- Vogtinator 9y agoThe ARM ARM documents sub(left, right, no carry) as add(left, ~right, carry set), which is also the most straightforward implementation if you have to account for carry anyway.
- hedora 9y agoThis mentality is why I like OpenBSD. If they can’t sign off on the security/correctness of a pile of code, they delete it, even if it means losing functionality. In this case, they simply didn’t invoke the incomprehensible instruction of doom.
- erric 9y agoWow, the article shows that many vendors mis-read the docs: Apple, Microsoft, FreeBSD, Red Hat, Ubuntu, SUSE Linux, and other Linux distros...as well as VMware and Xen. This is going to be a busy day!
- tpaksoy 9y agoMaybe the docs were written badly? If everyone makes the same mistake, then there might be something wrong in the source.
- paol 9y agoAt that point can it really be attributed to "mis-reading" the docs? If every single independent implementor understood it the same way, the docs were wrong.
- Tobba_ 9y agoTo be fair, Intel docs are so consistently gibberish that it might as well be classified a separate language (similar to english, but only a quarter the information density). In this case it seems they just didn't properly specify a piece of insane behaviour though. Hell, I'd consider it an outright CPU bug if I'm reading this right. Seemingly there's a "feature" where loading SS causes interrupts to be delayed until after the next instruction, even if the next instruction disables interrupts - so you can cause an interrupt to fire on the first instruction of the handler (where it should be impossible).
- ddtaylor 9y agoIs there a PoC exploit source?
- amluto 9y agohttps://lkml.kernel.org/r/67e08b69817171da8026e0eb3af0214b06b4d74f.1525800455.git.luto@kernel.org/67e08b69817171da8026e0eb3af0214b06b4d74f.1525800455.git.luto@kernel.org https://lkml.kernel.org/r/67e08b69817171da8026e0eb3af0214b06...
- ams6110 9y agoInteresting that some of the BSDs (Dragonfly, FreeBSD) are listed as Affected and others (NetBSD and OpenBSD) are listed as Not Affected.
- mmillin 9y agoThere isn't one BSD kernel like there is one Linux kernel, there are several different ones which might borrow from each other, but are developed independently. A comparison can be found on Wikipedia: https://en.wikipedia.org/wiki/Comparison_of_operating_system_kernels https://en.wikipedia.org/wiki/Comparison_of_operating_system...
- barkingcat 9y agoDragonfly descends from FreeBSD so it makes sense that it's affected. The other BSD's have different kernels and vastly different development histories as well.
- amluto 9y agoSome BSDs never allowed debug register writes in the first place, so they were immune.
- lallysingh 9y agoIt's been so long I had to look it up: SS is the stack segment.
- 0x0 9y agoAnd furthermore, the current stack address is determined by the combination of two registers: ss for the segment and rsp/esp/sp for the stack pointer within the segment. I guess the strange behavior around modifying ss comes from the fact that you need to also modify sp immediately afterwards, because otherwise you are running with a wild stack address pointing to random memory. You also can't modify sp before modifying ss because then you are also running with a wild stack, and an interrupt could come in at any time and push things onto random memory.
- twunde 9y agoI'm super happy to see that the BSDs were contacted (with the exception of HardenedBSD).
- sattoshi 9y agoHardenedBSD would just receive it downstream from OpenBSD, wouldn't it? It's like contacting the Linux Mint group while contacting Ubuntu is sufficient.
- nbsd4life 9y agoHardenedBSD is based on FreeBSD, not OpenBSD.
- twunde 9y agoHardenedBSD is a separate OS forked from FreeBSD with its own kernel development. While the BSDs may share some code, they're essentially all different OS'. There is no upstream like Linux has. As a side-note after Spectre/Meltdown, Shawn Webb complained in a NYCBUG thread about getting not being able to get access to these embargoed vulnerabilities.
- mrpippy 9y agoDefinitely, it appears this is thanks to the work/awareness of Nate Warfield at MSRC https://twitter.com/dk_effect/status/993903701419999232 https://twitter.com/dk_effect/status/993903701419999232
- jgtrosh 9y ago> Fixing the bug and having synchronized patches out by yesterday was an industry-wide effort, one that deserves praises, compared to the jumbled Meltdown and Spectre patching process. Is this a fair comparison? I feel like the patching techniques must have been easier to develop than for Meltdown/Spectre. Furthermore, if this affected the same kind of people in this community, maybe this time around benefitted from the communication channels of the previous exercises. Maybe this isn't a comparison to try and badmouth the previous iteration, and instead just tried to show a general improvement in the industry—I just find it a bit unfair.
- gregkh 9y agoThis is one of the first times that I know of that the Linux kernel and Windows kernel developers discussed a security issue together directly. So while the fix was much simpler than Meltdown/Spectre was (Linux was fixed with a patch that was written in 2015) overall, the communication between different OS kernel developers right now is very good. And yes, it is all due to the horrible Meltdown/Spectre problem and how that was handled. We were not allowed to work together for that problem, and we do not want to that to happen again.
- pritambaral 9y ago> We were not allowed to work together for that problem How do you mean?
- SmellyGeekBoy 9y agoIt was covered by NDA.
- LeifCarrotson 9y agoAt first I thought this was an error in implementing new docs related to Meltdown/Spectre. But the original research paper: https://everdox.net/popss.pdf https://everdox.net/popss.pdf the researcher wrote: > Somewhere around the release of the 8086, Intel decided to add a special caveat to instructions loading the SS register...where loading SS with [`pop ss` or `move ss`] would force the processor to disable external interrupts, NMIs, and pending debug exceptions. So it's a really, really old piece of documentation, dating from around 1980. To call it a 'misinterpretation' rather than a vulnerability is extremely generous, given that most Intel engineers spent entire careers in the presence of code vulnerable to this 'misinterpretation' without calling the OS vendors out on their error.
- ianopolous 9y agoOn this particular thing I think the Intel docs are clear. This is where I implemented the same in JPC: https://github.com/ianopolous/JPC/blob/master/src/org/jpc/emulator/execution/decoder/Disassembler.java#L351 https://github.com/ianopolous/JPC/blob/master/src/org/jpc/em...
- an_account_name 9y agoThey might be clear today. Were they always? There are literally decades of timeline to examine here.
- ianopolous 9y agoThey were clear in 2006 when I was reading them. All the emulator implementors were very aware of it. Many old operating systems won't boot without the delay.
- caf 9y agoThe interrupt shadow itself is clear. The specific implication of that for a mov ss ; syscall pair with a hardware breakpoint set on the first instruction is a lot more subtle.
- feikname 9y ago"Both Peterson and the CERT/CC team blamed the "unclear and perhaps even incomplete documentation" Yet the article's title makes it seems like it was the OS implementors faults instead of Intel's. I wonder for what reasons the website tries to shift/soften Intel's fault on this? Seems like there's a trend in news articles to have incoherent titles in relation to content lately, it's really annoying...
- hyperman1 9y agoI'm trying to understand this one, even if most of my ASM knowledge is from the 8086 era. My guess: * When an interrupt, debug exception, ... occurs, the CPU pushes stuff on the stack as part of the task switch. * The stack is managed by 2 registers: SS and (e/r)SP. To change your stack, you have to change both registers. If an interrupt happens and you've changed only 1, stuff gets pushed on an invalid stack and you're toast. * To fix this, the CPU has a wild card: When you change SS, you get exactly 1 instruction that will not be interrupted. The idea is you use that instruction to change (e/r)SP and make the stack valid again. If there is a need for an interrupt, it will be delayed for 1 instruction. * Now this being a security problem, what would happen if you use this second instruction to switch to kernel mode ? It turns out the delayed interrupt happens before the first kernel mode instruction, but in the kernel. * And you can trigger the right kind of interrupt with debug exceptions and single stepping. * And if you do this, the kernel tells the debugger not about the debugged program but about the kernel. Oops. So to fix this, I suppose the kernel checks the debug exception info from the CPU, and if it is debugging the kernel it fixes things up so you go back 1 instruction.
- codedokode 9y ago> To fix this, the CPU has a wild card: When you change SS, you get exactly 1 instruction that will not be interrupted. The idea is you use that instruction to change (e/r)SP and make the stack valid again. If there is a need for an interrupt, it will be delayed for 1 instruction. I wonder, why could not they make a single instruction to change both SS and SP?
- vardump 9y agoThey probably could, but it would have had a large silicon cost for 1980 without performance benefit. That and ISA incompatibility.
- codedokode 9y agoAdding an interrupt delay is also not free and requires changes on hardware level. But if it was done in 80s then of course it was a safe hack because there was no kernel and user mode isolation at that time.
- ysleepy 9y agoThe x86 ISA and its implementations are now in the spotlight of the whole security research community. There is probably a lot more to come since it accumulated a lot of cruft in the name of backwards compatibility. I hope we learn a lot, and take the time to record the experience, for coming platforms like RISC-V and others. Why is there no big CAVEATs document from intel detailing weird quirks. I strongly assume the intel arch engineers are well aware of many of those counter-intuitive behaviours in their products.
- adrianratnapala 9y ago> I strongly assume the intel arch engineers are well aware of many of those counter-intuitive behaviours in their products. But it is likely just kind of distributed, organic knowledge that is hard to condense into a single document. Writing and maintaining such a thing would be a significant project, and (I am speculating here) not the kind of thing that significantly burnishes anyone's performance review. That said, the whole community of assembly-hackers has even broader knowledge of the topic, and could start such a document out in the open. And Intel engineers might likely contribute their own two cents. (Unless lawyers forbid it).
- ysleepy 9y agoI stumbled on a blogpost by bunnie huang which describes a liability angle, which I found to be plausible: https://www.bunniestudios.com/blog/?p=5127 https://www.bunniestudios.com/blog/?p=5127 - worth a read.
- Tobba_ 9y agoAs far as I understand, what's happening is: * There's an old feature which causes POP SS/MOV SS instructions to delay all interrupts until the next instruction has executed, to safely allow changing both SS and SP without an interrupt firing inbetween on a bad stack. * If such an instruction itself causes an interrupt (by triggering a memory breakpoint through the debug registers), it is delayed (as intended). * The delayed interrupt will fire after the second instruction even if the second instruction disabled interrupts. * By means of the above, a MOV SS instruction triggering a #DB followed by an INT n instruction will cause the #DB exception to fire before the first instruction of the interrupt handler, even though this should be impossible (as entering the handlers sets IF=0, disabling interrupts). * The OS #DB handler assumes GS has been fixed up by the previous interrupt handler, which in now under user control.
- dwighttk 9y agoThe vulnerability notes[1] say Apple patched this on May 8, but my last security update was May 3 and I don't currently show any available updates... I wonder if the May 3 patch fixed this, or if my computer might not be affected. [1]https://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD+Reference=631579&SearchOrder=4 https://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD...
- amluto 9y agoThe May 3 patch fixed it. The nature of the fix was such that Linux and Mac OS were able to patch it early without revealing much. In fact, the Linux fix was a patch I wrote in 2015 (for unrelated reasons) and just never got around to upstreaming.
- jey 9y agoLink to referenced patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d8ba61ba58c88d5207c1ba2f7d9a2280e7d03be9 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- tmd83 9y agoCan someone explain the risk factor? It cannot be remotely exploited or though browser if I'm reading it right. But a malicious program with user level access can get kernel access. So exposure to malware running on a limited account can get higher access?
- ComodoHacker 9y agoArchived version: http://archive.is/DxUwA http://archive.is/DxUwA
- std_throwaway 9y agoIs this also affecting code in 64 bit mode?
- dfox 9y agoAs the code in the paper is written for amd64 I would assume that it is affecting it. In the other hand it is somewhat surprising, because loading anything into SS is mostly pointless operation.
- std_throwaway 9y agoThis doesn't even make sense. Why would they keep a behavior that could be considered a bug when creating a new instruction set?
- deleted 9y ago[deleted]
- hyperman1 9y agoI wonder what happens if you execute multiple POP SS instructions. In fact, you could set up a 64K v86 mode segment containing only copies of the POP SS instruction. jmp far into it. When IP reaches the last instruction it wraps around and starts again. Will it ever be interrupted by anything? If the stack usage bothers it, just do MOV SS,AX
- Someone 9y agoIntel thought of that, but doesn’t seem to give us an answer. https://software.intel.com/sites/default/files/managed/7c/f1/253668-sdm-vol-3a.pdf https://software.intel.com/sites/default/files/managed/7c/f1..., page 6-8: ”If a sequence of consecutive instructions each loads the SS register (using MOV or POP), only the first is guaranteed to inhibit or suppress events in this way.” So, we still don’t know. For all I know, it may even depend on the exact cpu used or cpu state.
- dooglius 9y agoTested it: you get an interrupt after only skipping one instruction
- hyperman1 9y agoThanks, both dooglius and Someone. One learns something everyday.
- _bxg1 9y agoI felt a great disturbance in the Force, as if thousands of voices suddenly cried out, "Oops..."
- emily-c 9y agoA bit tangentially related, but I've always wondered why the syscall instruction doesn't use the TSS for stack switching like int does. I guess it does give you more flexibility to load rsp from gs during a cpl 3 -> cpl 0 transition rather than consulting the TSS to switch it automatically. Can anyone weigh in on this?
- floatboth 9y agoWas illumos not affected?