10 ms·

The x86 architecture is the weirdo, part 2

175 points by zdw 5y ago

gpderetta 5y ago
It is interesting that the stores to node are seen as dead. Surely the assignment of the node address to fs:[0] should equivalent of assigning it a global variable, so any state change in node can be observed by any function.
- foota 5y ago
  I believe this is because the intermediate state can't be safely observed by another thread, since there's no locking, so the C++ memory model allows removing them.
  - gpderetta 5y ago
    In the article example itself, f1() and f2() can observe the state. Nothing to do with threads.
    - Someone 5y ago
      I don’t think so. It is global state, but it lives outside the abstract C machine, in its implementation. The compiler knows f1 and f2 live in the abstract C machine and cannot access it. I think its like the difference between errno, a global inside the abstract C machine, and __foo, which a compiler might create, and, if it did so, could assume to be completely under its control.
      - gpderetta 5y ago
        Well, sure we are deep into implementation details, so it is reasonable for the implementation to treat stores to it specially. But the rest of the article goes into details on how treating it specially causes "real pain in the neck" because the optimizer wants to optimize them away. It seems to me that if the store was not treated specially, the issue wouldn't exist in the first place. So either: exception handling in 32 bit mode is sort of an hack and the exception handling code generation doesn't actually expose the store the store through fs to the rest of the optimizer or, more likely, things are much much more complicated than described in the article.
- rayiner 5y ago
  Why? Fs:[0] works out to [FsBase] which is just another store through a pointer.
  - gpderetta 5y ago
    sure, but that's just a global memory location which any function can read from (I think it is part of Windows thread control block, but do not quote me on that). Probably stores through fs are pragmatically not considered globals stores as they would hinder other optimizations.
pm215 5y ago
I have a theory that CPU architectures implicitly have something I call the "architectural weirdness budget", which is the extent to which you can get away with design decisions in the architecture or ABI that differ from the existing established consensus. Any time you do something a bit odd that means that existing software has to do awkward things or might not be no-changes portable to your new architecture, you're spending a bit of your weirdness budget. Some examples: + HPPA's upwards-growing stack + SPARC register windows + PowerPC's slightly oddball varargs ABI + CHERI capabilities are massively spending weirdness budget + these days, being big-endian is spending weirdness budget Doing a few odd things is fine, but if you blow your weirdness budget entirely then your new architecture is in danger of failing because you've cumulatively made adoption of it too expensive for the existing software ecosystem. So you need to be careful that you're spending it on things that really matter; if something's not a big deal, do it the same way everybody else does. (Certainly some of the things in the list above are deliberate choices for things that are, or at least seemed at the time, important. But nobody tried to do all of them at once!) Of course if you're a pre-existing dominant architecture like x86 then you get a lot more freedom to do weird things :-)
- KptMarchewa 5y ago
  What does ARM and RISC-V spend their weirdness budget on?
  - qikInNdOutReply 5y ago
    wasnt there also the long jump limitation weirdness that forced it to create little hopping points, were the instructions were stored to hop to the next point, all of that to get to some memory whos pointer it could not fit into loadInstructions Register? Sorry, its been a while, but i found the idea, to generate those instruction isles into the code quite weird.
  - jmgao 5y ago
    ARM spends it on pc being wrong: when you mov from it, it's off by 8 in ARM mode, and off by 4 in Thumb mode, and if you're using it for pc-relative loads (and stores, if you're crazy), it uses a word aligned value. edit: oh, and softfloat vs softfp vs hardfp vs vfp edit: oh, and how they have two incompatible assembly language dialects that are mostly the same, but in non-trivial code, incompatible
_8j50 5y ago
For the unaware, SEH can be used by malware to evade detection: https://marcoramilli.com/2011/11/28/pop-pop-ret-seh-exploiting-process/ https://marcoramilli.com/2011/11/28/pop-pop-ret-seh-exploiti... , also, X64 uses a table based approach in the PE header: https://www.osronline.com/article.cfm%5earticle=469.htm https://www.osronline.com/article.cfm%5earticle=469.htm
rpasquay 5y ago
I did not understand why on x86 Windows/MSVC cannot implement exceptions the same way as it was done for other architectures. I read that on x64 a different approach is used ("table based" - sounds to me like the "unwind code" approach mentioned in the article). Is this not possible no x86 for some reason ?
- nikbackm 5y ago
  The answer seems to be in the follow-up comments to the post.
- thedeadfish 5y ago
  Table based exceptions only work if everyone plays ball. Its not practical to introduce them after the fact because they will not work with any existing code including OS callbacks. AMD64 can use them because they were there from the beginning.
  - gpderetta 5y ago
    On linux they are also used for x86-32. The previous sjlj ABI was broken about 20 years ago when the Itanium ABI was introduced.
- deleted 5y ago
  [deleted]
thedeadfish 5y ago
Am I the only one who prefers the way x86 did exceptions? I really don't like that with table based exceptions, every function must include bloated unwind tables, and is also forced to conform to a very restrictive set of code generation rules. It also makes writing and assembly and dynamic code generation difficult.
gumby 5y ago
MS was stuck with a poor exception implementation but what I never see explain is how they got there in the first place. It seems crazy to incur a runtime overhead to support something that is hardly ever used. The happy path is where you need the performance most, and exceptions are by definition the "unhappy path". Michael and I talked about exception design a lot in g++ (I certainly had experience of what to do and not do from CLOS) and there was never a consideration of doing anything that might impact runtime performance. At least now I finally understand why some people, especially in the game industry, want to disable exceptions. As a side point, I'm disappointed by Raymond's (18 yo) claim that "Zero cost exceptions" aren't really zero cost. Everything called "zero cost" in C++ means "doesn't add any runtime cost if you don't use it". He argued against a straw man.
- twoodfin 5y ago
  There’s a comment on Raymond’s post that explains it in a plausible-sounding way: https://devblogs.microsoft.com/oldnewthing/20220418-00/?p=106489#comment-139219 https://devblogs.microsoft.com/oldnewthing/20220418-00/?p=10... Table-based exception-handling metadata requires function prologue/epilogue sequences to have constrained forms that can be described by that metadata. When NT was first designed for x86-32, there was a lot of existing asm code that people wanted to easily port over to NT, and that ported code included pretty much every weird function entry/exit sequence you could think of (and lots more that you wouldn’t imagine anyone would ever think of). Switching to table-based metadata would have required modifying all that code, making porting more difficult. At least, I assume that’s the reasoning – I was deeply involved in the SEH runtime code in the ’90s and ’00s when I was on the VC++ compiler team.
  - pjc50 5y ago
    > SEH runtime Note that in Microsoft compiler land there are two kinds of exception, regular C++ ones and "structured exception handling". https://docs.microsoft.com/en-us/cpp/cpp/try-except-statement?view=msvc-170 https://docs.microsoft.com/en-us/cpp/cpp/try-except-statemen... The latter does exception unwinding in the kernel on runtime faults (segv, division by zero, etc). It does _not_ unwind destructors or deallocate storage. And yes, I know about this because I had to debug it, on WinCE ARM, where we eventually discovered that there were two different MS compilers, one of which generated code that could be unwound and one of which didn't.
AlexandrB 5y ago
I'm bummed that The Old New Thing is now on Microsoft's domain. Old blogs seems to be dying out or being moved behind the walls of Medium/Substack/etc.
- jcranmer 5y ago
  > I'm bummed that The Old New Thing is now on Microsoft's domain. Um, it's always been on MS's domain. It has moved from e.g. msdn.com (I think) to microsoft.com, but that's more about MS periodically moving around where its documentation is hosted because shrug, but it was always in same way officially hosted by MS.
- mschaef 5y ago
  It's always been on one Microsoft domain or another, for almost as long as there've been blogs at all. It's almost 20 years ago now, but when Blogs started their initial ascendency as a useful format, Microsoft had a a number of developers start writing publicly facing personal blogs. Raymond Chen was one of that group, also including Michael Kaplan, Eric Lippert and Larry Osterman, among others. As far as I know, The Old New Thing is the only blog from that wave that's still being written - and it's been written every weekday for essentially that entire time. To give an idea of the scope of the work, he did a compilation of posts in book form sixteen years ago, and it was over 500 pages long. https://www.amazon.com/gp/product/B004YWCL5W/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 https://www.amazon.com/gp/product/B004YWCL5W/ref=dbs_a_def_r... I don't read it daily anymore (hasn't been relevant to my work or professional interests for ten years or more), but I do read it occasionally and continue to be both impressed and grateful for the effort he's put into the blog.
titzer 5y ago
This is one of the many reasons that having exceptional control flow explicit in the compiler IR is a Good Thing(tm). The HotSpot client compiler lacked this information (in the beginning) because it was deemed "too expensive to keep" and the result was that many optimizations had ugly special cases (or were disabled altogether). For the most part, if the compiler has some notion of hot/cold code, then these exceptional edges can just be marked as cold and the same heuristics that apply to, e.g. unlikely branches, will do a good job.
- ajross 5y ago
  Pretty much exactly. They had a situation with external visibility into information in the stack frame and don't seem to have made an effort to tell the compiler. I mean... it's their compiler! It's their ABI, even! They have all the levers and dials, and somehow the fact that their exception counter is being optimized out means that the "architecture" is the weirdo? Yeah. At its root this is a Conway's Law bug. The people responsible for "exceptions" had a senseless wall between them and the people responsible for "stack frames".
  - titzer 5y ago
    Indeed. In Java, on every platform that I know of (including x86 and x86-64), exceptions are zero cost, with no dynamic work needed to enter a try. On Windows they are probably stuck with ABI compatibility. It sounds like the design was flawed from the start and definitely not the architecture's fault.
userbinator 5y ago
That state variable is actually quite useful for decompilation too: http://www.openrce.org/articles/full_view/21 http://www.openrce.org/articles/full_view/21
giomasce 5y ago
That's not an x86 weirdness, it's just a choice by Microsoft. It's Windows-on-x86 weirdness.
- russellbeattie 5y ago
  This is what I thought, but the rest of the discussion is way outside my knowledge, so I can't tell what I'm missing. Can someone explain how is this x86's fault?
  - giomasce 5y ago
    I don't think it is. I don't know if the peculiarities of x86 were considered when Microsoft's engineers decided to do SEH the way it is, but there is no hard reason why they couldn't decide to make it table based.
omoikane 5y ago
Isn't it possible to annotate a variable as not being eligible for dead store elimination with something like "volatile"? I wonder why that particular state was causing trouble for Visual C++, assuming that it's able to handle code that deal with other kinds of asynchronous events (e.g. signal handlers).
asveikau 5y ago
This isn't about x86 architecture but historical design choices that then got frozen into an ABI. When Microsoft implemented SEH on ia32, the current approach of lookup table based exceptions (trading memory for runtime code execution, particularly reducing the latter in a success case) was not yet in vogue. When they ported to other architectures and had a choice to introduce a different ABI, they did.

175 points by zdw 5y ago

It is interesting that the stores to node are seen as dead. Surely the assignment of the node address to fs:[0] should equivalent of assigning it a global variable, so any state change in node can be observed by any function.

foota 5y ago
I believe this is because the intermediate state can't be safely observed by another thread, since there's no locking, so the C++ memory model allows removing them.
- gpderetta 5y ago
  In the article example itself, f1() and f2() can observe the state. Nothing to do with threads.
  - Someone 5y ago
    I don’t think so. It is global state, but it lives outside the abstract C machine, in its implementation. The compiler knows f1 and f2 live in the abstract C machine and cannot access it. I think its like the difference between errno, a global inside the abstract C machine, and __foo, which a compiler might create, and, if it did so, could assume to be completely under its control.
    - gpderetta 5y ago
      Well, sure we are deep into implementation details, so it is reasonable for the implementation to treat stores to it specially. But the rest of the article goes into details on how treating it specially causes "real pain in the neck" because the optimizer wants to optimize them away. It seems to me that if the store was not treated specially, the issue wouldn't exist in the first place. So either: exception handling in 32 bit mode is sort of an hack and the exception handling code generation doesn't actually expose the store the store through fs to the rest of the optimizer or, more likely, things are much much more complicated than described in the article.
rayiner 5y ago
Why? Fs:[0] works out to [FsBase] which is just another store through a pointer.
- gpderetta 5y ago
  sure, but that's just a global memory location which any function can read from (I think it is part of Windows thread control block, but do not quote me on that). Probably stores through fs are pragmatically not considered globals stores as they would hinder other optimizations.

pm215 5y ago

I have a theory that CPU architectures implicitly have something I call the "architectural weirdness budget", which is the extent to which you can get away with design decisions in the architecture or ABI that differ from the existing established consensus. Any time you do something a bit odd that means that existing software has to do awkward things or might not be no-changes portable to your new architecture, you're spending a bit of your weirdness budget. Some examples: + HPPA's upwards-growing stack + SPARC register windows + PowerPC's slightly oddball varargs ABI + CHERI capabilities are massively spending weirdness budget + these days, being big-endian is spending weirdness budget Doing a few odd things is fine, but if you blow your weirdness budget entirely then your new architecture is in danger of failing because you've cumulatively made adoption of it too expensive for the existing software ecosystem. So you need to be careful that you're spending it on things that really matter; if something's not a big deal, do it the same way everybody else does. (Certainly some of the things in the list above are deliberate choices for things that are, or at least seemed at the time, important. But nobody tried to do all of them at once!) Of course if you're a pre-existing dominant architecture like x86 then you get a lot more freedom to do weird things :-)

KptMarchewa 5y ago
What does ARM and RISC-V spend their weirdness budget on?
- qikInNdOutReply 5y ago
  wasnt there also the long jump limitation weirdness that forced it to create little hopping points, were the instructions were stored to hop to the next point, all of that to get to some memory whos pointer it could not fit into loadInstructions Register? Sorry, its been a while, but i found the idea, to generate those instruction isles into the code quite weird.
- jmgao 5y ago
  ARM spends it on pc being wrong: when you mov from it, it's off by 8 in ARM mode, and off by 4 in Thumb mode, and if you're using it for pc-relative loads (and stores, if you're crazy), it uses a word aligned value. edit: oh, and softfloat vs softfp vs hardfp vs vfp edit: oh, and how they have two incompatible assembly language dialects that are mostly the same, but in non-trivial code, incompatible

_8j50 5y ago

For the unaware, SEH can be used by malware to evade detection: https://marcoramilli.com/2011/11/28/pop-pop-ret-seh-exploiting-process/ https://marcoramilli.com/2011/11/28/pop-pop-ret-seh-exploiti... , also, X64 uses a table based approach in the PE header: https://www.osronline.com/article.cfm%5earticle=469.htm https://www.osronline.com/article.cfm%5earticle=469.htm

rpasquay 5y ago

I did not understand why on x86 Windows/MSVC cannot implement exceptions the same way as it was done for other architectures. I read that on x64 a different approach is used ("table based" - sounds to me like the "unwind code" approach mentioned in the article). Is this not possible no x86 for some reason ?

nikbackm 5y ago
The answer seems to be in the follow-up comments to the post.
thedeadfish 5y ago
Table based exceptions only work if everyone plays ball. Its not practical to introduce them after the fact because they will not work with any existing code including OS callbacks. AMD64 can use them because they were there from the beginning.
- gpderetta 5y ago
  On linux they are also used for x86-32. The previous sjlj ABI was broken about 20 years ago when the Itanium ABI was introduced.
deleted 5y ago
[deleted]

thedeadfish 5y ago

Am I the only one who prefers the way x86 did exceptions? I really don't like that with table based exceptions, every function must include bloated unwind tables, and is also forced to conform to a very restrictive set of code generation rules. It also makes writing and assembly and dynamic code generation difficult.

gumby 5y ago

MS was stuck with a poor exception implementation but what I never see explain is how they got there in the first place. It seems crazy to incur a runtime overhead to support something that is hardly ever used. The happy path is where you need the performance most, and exceptions are by definition the "unhappy path". Michael and I talked about exception design a lot in g++ (I certainly had experience of what to do and not do from CLOS) and there was never a consideration of doing anything that might impact runtime performance. At least now I finally understand why some people, especially in the game industry, want to disable exceptions. As a side point, I'm disappointed by Raymond's (18 yo) claim that "Zero cost exceptions" aren't really zero cost. Everything called "zero cost" in C++ means "doesn't add any runtime cost if you don't use it". He argued against a straw man.

twoodfin 5y ago
There’s a comment on Raymond’s post that explains it in a plausible-sounding way: https://devblogs.microsoft.com/oldnewthing/20220418-00/?p=106489#comment-139219 https://devblogs.microsoft.com/oldnewthing/20220418-00/?p=10... Table-based exception-handling metadata requires function prologue/epilogue sequences to have constrained forms that can be described by that metadata. When NT was first designed for x86-32, there was a lot of existing asm code that people wanted to easily port over to NT, and that ported code included pretty much every weird function entry/exit sequence you could think of (and lots more that you wouldn’t imagine anyone would ever think of). Switching to table-based metadata would have required modifying all that code, making porting more difficult. At least, I assume that’s the reasoning – I was deeply involved in the SEH runtime code in the ’90s and ’00s when I was on the VC++ compiler team.
- pjc50 5y ago
  > SEH runtime Note that in Microsoft compiler land there are two kinds of exception, regular C++ ones and "structured exception handling". https://docs.microsoft.com/en-us/cpp/cpp/try-except-statement?view=msvc-170 https://docs.microsoft.com/en-us/cpp/cpp/try-except-statemen... The latter does exception unwinding in the kernel on runtime faults (segv, division by zero, etc). It does _not_ unwind destructors or deallocate storage. And yes, I know about this because I had to debug it, on WinCE ARM, where we eventually discovered that there were two different MS compilers, one of which generated code that could be unwound and one of which didn't.

AlexandrB 5y ago

I'm bummed that The Old New Thing is now on Microsoft's domain. Old blogs seems to be dying out or being moved behind the walls of Medium/Substack/etc.

jcranmer 5y ago
> I'm bummed that The Old New Thing is now on Microsoft's domain. Um, it's always been on MS's domain. It has moved from e.g. msdn.com (I think) to microsoft.com, but that's more about MS periodically moving around where its documentation is hosted because shrug, but it was always in same way officially hosted by MS.
mschaef 5y ago
It's always been on one Microsoft domain or another, for almost as long as there've been blogs at all. It's almost 20 years ago now, but when Blogs started their initial ascendency as a useful format, Microsoft had a a number of developers start writing publicly facing personal blogs. Raymond Chen was one of that group, also including Michael Kaplan, Eric Lippert and Larry Osterman, among others. As far as I know, The Old New Thing is the only blog from that wave that's still being written - and it's been written every weekday for essentially that entire time. To give an idea of the scope of the work, he did a compilation of posts in book form sixteen years ago, and it was over 500 pages long. https://www.amazon.com/gp/product/B004YWCL5W/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 https://www.amazon.com/gp/product/B004YWCL5W/ref=dbs_a_def_r... I don't read it daily anymore (hasn't been relevant to my work or professional interests for ten years or more), but I do read it occasionally and continue to be both impressed and grateful for the effort he's put into the blog.

titzer 5y ago

This is one of the many reasons that having exceptional control flow explicit in the compiler IR is a Good Thing(tm). The HotSpot client compiler lacked this information (in the beginning) because it was deemed "too expensive to keep" and the result was that many optimizations had ugly special cases (or were disabled altogether). For the most part, if the compiler has some notion of hot/cold code, then these exceptional edges can just be marked as cold and the same heuristics that apply to, e.g. unlikely branches, will do a good job.

ajross 5y ago
Pretty much exactly. They had a situation with external visibility into information in the stack frame and don't seem to have made an effort to tell the compiler. I mean... it's their compiler! It's their ABI, even! They have all the levers and dials, and somehow the fact that their exception counter is being optimized out means that the "architecture" is the weirdo? Yeah. At its root this is a Conway's Law bug. The people responsible for "exceptions" had a senseless wall between them and the people responsible for "stack frames".
- titzer 5y ago
  Indeed. In Java, on every platform that I know of (including x86 and x86-64), exceptions are zero cost, with no dynamic work needed to enter a try. On Windows they are probably stuck with ABI compatibility. It sounds like the design was flawed from the start and definitely not the architecture's fault.

userbinator 5y ago

That state variable is actually quite useful for decompilation too: http://www.openrce.org/articles/full_view/21 http://www.openrce.org/articles/full_view/21

giomasce 5y ago

That's not an x86 weirdness, it's just a choice by Microsoft. It's Windows-on-x86 weirdness.

russellbeattie 5y ago
This is what I thought, but the rest of the discussion is way outside my knowledge, so I can't tell what I'm missing. Can someone explain how is this x86's fault?
- giomasce 5y ago
  I don't think it is. I don't know if the peculiarities of x86 were considered when Microsoft's engineers decided to do SEH the way it is, but there is no hard reason why they couldn't decide to make it table based.

omoikane 5y ago

Isn't it possible to annotate a variable as not being eligible for dead store elimination with something like "volatile"? I wonder why that particular state was causing trouble for Visual C++, assuming that it's able to handle code that deal with other kinds of asynchronous events (e.g. signal handlers).

asveikau 5y ago

This isn't about x86 architecture but historical design choices that then got frozen into an ABI. When Microsoft implemented SEH on ia32, the current approach of lookup table based exceptions (trading memory for runtime code execution, particularly reducing the latter in a success case) was not yet in vogue. When they ported to other architectures and had a choice to introduce a different ABI, they did.