11 ms·

Is the x86 Architecture Sustainable?

109 points by AshleysBrain 9y ago

banburyt 9y ago
Yes.
uam 9y ago
Would like to hear what someone with compiler writing experience has to say about this. Is it really that much of a problem?
- simias 9y ago
  I can't really answer that but your question reminded me of https://www.strchr.com/x86_machine_code_statistics https://www.strchr.com/x86_machine_code_statistics Turns out that in practice a tiny number of instructions make up the vast majority of all used upcodes in a typical application (although it's not clear in this article how long the tail is, there could be hundreds of instruction in that "others" slice). My understanding is that there are many legacy instructions in x86 that are basically never used nowadays. They're still a problem for silicon designers because they still have to implement them for compatibility purposes but compiler developers can chose not to use them and instead select a more "risc-y" subset of instructions to do the bulk of the work.
- moyix 9y ago
  I'm not sure compiler writers care that much; they don't have to support every x86 instruction and can restrict themselves to whatever subset produces fast code. Where it really kills you are things that have to analyze x86 binary code (e.g., binary static or dynamic analysis tools). Unless you cover every nook and cranny of x86, chances are you'll eventually encounter some program that uses one of these instructions and your analysis will break. And of course emulators like QEMU face the same problem. This problem becomes especially severe in the context of malware – any instruction you forget to model becomes a way for a malicious program to evade your analysis.
Iv 9y ago
In a world where JIT become more and more common, I could see many legacy features being dropped at one point. Alternatively, I wonder how feasible it would be, for a multi-core system, to keep one legacy core for incompatible processes and faster new-arch cores for running the main beef.
- legulere 9y ago
  The current generation of new programming languages are all compiled to machine code (Rust, Go, Swift)
  - kmicklas 9y ago
    That's an implementation detail. Code written in these languages is generally trivially retargetable.
- Qantourisc 9y ago
  You could also use some form of pre-compiled-non-platform specific system, which you then compile/optimize to the current platform. Ideally we should also be able to decompile and recompile code. But this might be hard for x86 die the complexity.
gcatlin 9y ago
Interesting proposal for a forward compatible instruction set architecture by Agner Fog: http://www.forwardcom.info/ http://www.forwardcom.info/
bryanlarsen 9y ago
AFAICT, infrequently used instructions are implemented with microcode. I would guess that each of these instructions have a cost of a few thousand transistors, on a chip with billions.
- ant6n 9y ago
  But aren't there thousands of those instructions?
lloeki 9y ago
Wasn't there some dead weight dropoff† once with amd64, as a x86 CPU is unable to execute 16-bit code when in 64-bit mode? IIRC that was a notable cause of DOSBox becoming slower when going 64-bit as it had to emulate the 16-bit CPU instead of just running on metal, as well as 64-bit Windows (7?) being unable to run old DOS and Win3x software. With an increasingly rolling-release culture on software I can't wait for X-year support for hardware instruction sets /s. Wait, this is already happening for Android phones, where ARM SoC manufacturers just flip it at OS and phone vendors (Nexus 4, OnePlus X). I just realised Apple made a damn right call getting that under their control on that front. † the metal may still be there though, waiting to be jettisoned as the market for 32-bit OSes goes away.
- phkahler 9y ago
  >> waiting to be jettisoned as the market for 32-bit OSes goes away. I didn't know there still was a market for 32-bit x86 OSes. I suppose legacy systems - SCADA and such.
  - bonzini 9y ago
    There's a market for 16-bit OSed too, though often they run virtualized because the hardware might not even exist anymore.
- en4bz 9y ago
  In long mode you no longer have access to the virtual 8086 mode which is most likely what DOSBox was utilizing.
- legulere 9y ago
  Yep they threw out a bunch of stuff. What I remember: the CS, SS, DS, and ES segment registers, binary coded decimal instructions, protection ring 1 and 2 as only 0 and 3 are used. Probably there is a lot of more things that were removed.
  - wolfgke 9y ago
    > protection ring 1 and 2 as only 0 and 3 are used Are they really dropped in x86-64? I thought they are still supported also in 64 bit mode.
muricula 9y ago
This is entirely anecdotal, but a coworker of mine worked at amd, and he once told me that amd did a study and found that instructions needed for backwards compatibility accounted for less than 1% of CPU die space. Big silicon manufacturers are pretty price sensitive and the decision to break backwards compatibility is a clear cut economic one. Clearly they have chosen not to break old software at the expense of continued architectural complexity and maintenance.
- Symmetry 9y ago
  The problem isn't the hardware devoted to decoding or even executing instructions so much as the overall design and verification complexity. The existence of 286 call gates has implications throughout the memory subsystem and now when you're adding your TSX transactional memory instructions you have to think carefully about how they interact with call gates and whether you could be introducing bugs and make sure you have unit tests to uncover those bugs.
  - muricula 9y ago
    Yes, you're right. The silicon giants decided it was cheaper to fund that testing and verification work than to break backwards compatibility.
    - xorblurb 9y ago
      Except they occasionally miss some nasty bug in that area... This is yet another danger to security.
rwmj 9y ago
Surprised to see RISC-V is not mentioned. They take an interesting approach to extensibility where it's basically envisioned in the ISA from the start. Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future. See "ISA Overview" here: https://content.riscv.org/wp-content/uploads/2016/06/riscv-spec-v2.1.pdf https://content.riscv.org/wp-content/uploads/2016/06/riscv-s...
- wolfgke 9y ago
  > Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future. x86 processors also have cpuid > https://en.wikipedia.org/wiki/CPUID https://en.wikipedia.org/wiki/CPUID to signal which extensions are supported or not. In theory it would also be possible to drop deadweight extensions this way for x86, too. The practice shows that at least for the tasks that PCs are used for, one can hardly drop anything here. Exceptions are - 3Dnow! (is not supported in Bulldozer, Bobcat and Zen anymore; cf. https://en.wikipedia.org/w/index.php?title=3DNow!&oldid=769683777#Processors_supporting_3DNow https://en.wikipedia.org/w/index.php?title=3DNow!&oldid=7696...) - A20 gate (not supported anymore since Haswell: cf. https://en.wikipedia.org/w/index.php?title=A20_line&oldid=781982575#A20_gate https://en.wikipedia.org/w/index.php?title=A20_line&oldid=78...)
- drudru11 9y ago
  Agreed, but then again there just isn't any practical implementation available yet. If there was a raspberry pi like device, I think we would start seeing it mentioned more.
  - microcolonel 9y ago
    I'm guessing you mean workstation/server scale implementations? There is a shipping microcontroller implementation and has been for a while now.
  - rwmj 9y ago
    We should have an RV64G implementation this year with any luck.
amluto 9y ago
Even ignoring CPU die size cost, there's another cost to legacy features: some of those features have register state, and contexts switching it is expensive. Simpler architectures can do crazy things like switching between threads in mere tens of cycles. X86: not so much. The OS needs to deal with DS, ES, FS, and GS, and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow. Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes. Amusingly, x86's privilege mechanism is complicated enough that Intel implemented it wrong in a few CPUs with MPX, resulting in a bizarre erratum. I would love to see x86 add a "clean mode" in which a lot of legacy features are inaccessible to user code. Then kernels could try to run programs in clean mode and, if they don't fault, context switching between clean mode threads would be much faster.
- frozenport 9y ago
  Also AVX512! And what the heck are we going to do when we need to save the AVX1024 register state! And we really going to push 1kb to memory per context switch? (I think this is somewhat solved by trapping when the registers are first invoked, and kernel drivers that use enhanced instruction sets actually need to pass flags before using them)
  - jandrewrogers 9y ago
    This highlights one of the advantages of context-switching in user space. The OS context switch can make few assumptions about the registers that actually need to be preserved once a register set has been used. Many user space context libraries, on the other hand, only preserve the half dozen basic registers per the ABI by default (very fast) and let the programmer optionally preserve the other register sets. In the vast majority of cases, you never need to preserve any additional state at the logical context switch points even when you are using those additional registers in your application.
warbiscuit 9y ago
The post posits that, to handle the burden of legacy instructions, an "obvious technically-appealing approach (is) starting over with a clean-sheet architecture". The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?). Of course, doing something like that is probably nowhere near trivial, the devil's always in the details. But I bet this is already being done at the microcode level. Stepping things up to having a published agreement about which instructions were globally considered "legacy", and guidelines for what their equivalents were, would go a long was towards allowing a general feeling that an ISA was evolving, rather than just accumulating weight upon weight.
- kllrnohj 9y ago
  > The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?). This is what x86 CPUs already do and have been doing for years. It's also why Intel & AMD don't care about "x86 complexity" and it's also why even though people love to claim that a switch to RISC would improve efficiency/performance, so far there's really no evidence to support that. Intel just plops their teeny-tiny (in terms of die space) x86-to-internal-microcode transactor on top of their new cores and calls it a day. As long as that translation layer isn't a bottleneck, which it rarely is, then Intel doesn't care.
  - roca 9y ago
    It's not just about legacy instructions which can be decoded down to some microcode. It's about architectural features like SGX, CET, MPX, TSX, VT --- plus the legacy stuff like segment registers and 286 call gates and virtual 8086 mode and so on and so on --- and how they all interact with each other, and how they increase the complexity of context switching, OS support, and so on.
    - kllrnohj 9y ago
jonstokes 9y ago
Oh for pete's sake, the die size cost for supporting the legacy x86 instructions is a fraction of a percent on a modern multicore CPU. This is because the translation hardware stays relatively fixed (in terms of size and complexity and transistor count), while Moore's Law keeps adding more hardware to each core and more cache, etc.. Take a look at this annotated die shot of an old Pentium 4: https://i.stack.imgur.com/QK4gm.jpg https://i.stack.imgur.com/QK4gm.jpg Up at the top is the microcode memory and microcode sequencer. That's the hardware that's responsible for translating all those old, large legacy instructions. It's not really much space on the P4, and the P4 Northwood is a 55-million transistor CPU. Nowadays, depending on if you're talking about a mobile part or a higher-end desktop or server part, the CPUs have between ~1.5 billion to 2 billion transistors. Again, that microcode ROM just doesn't grow very much as you add instructions, even a ton of instructions. And this is, well, one reason why Intel just keeps adding instructions. It's practically free.
- STRML 9y ago
  That's a great graphic! I see a few more from around 2003 on http://chip-architect.com/ http://chip-architect.com/, do you know of anyone who's doing modern chips (x86 or ARM) this way?
- api 9y ago
  You're right when it comes to die size and transistor budget, but complexity can sometimes have hidden costs. Complexity can limit and constrain the design of other aspects of a system in ways that are nearly impossible to predict and often non-obvious even when you're down in the trenches. Not only can today's features limit tomorrow's design, but because those features give rise to mental models and habits of thinking they can limit future design in an almost unconscious way. The more complex your system becomes the more its existing complexity dominates your thinking to the exclusion of innovative ideas. This process happens so quickly in software that it's obvious there: http://www.retrologic.com/jargon/R/rococo.html http://www.retrologic.com/jargon/R/rococo.html
- Retric 9y ago
  It's not just ancient instructions that add costs, P4's MMX is also outdated etc. Further the core issue is the cascade where A limited B and C is limited by B directly and A indirectly, now continue that though 15 generations and things get complicated.
pdkl95 9y ago
> legacy compatibility features [...] those CPUs won't be as efficient, cheap, reliable or secure as they otherwise could be [...] we can't rely on increasing transistor density to bail us out. Legacy features shouldn't have a large impact on transistor usage, and they should have almost no impact on efficiency (although that may involve design choices). Almost all of the transistors are going to be used to implement the cache, routing between the high level blocks (external IO, cores, shared cache), and routing between the various microarchitecture units. Using Broadwell as an example, the CPU cores[1] only use a relatively small amount of amount of area. However, even that is misleading, because most of the acre within each core is dedicated to the microarchitecture, which implements a far smaller set of micro-ops. Inside each core[2], the only place affected by legacy instructions is the decode stage (mostly in the "4-Way Decode" block in [2], probably less in the "PreDecode" block"), and a larger MicoCode ROM. The legacy instructions might not be as efficient as they used to be in older CPUs; as low-priority instructions they may be implemented with less-convenient micro-ops that were designed for other, faster instructions. The software that uses those instructions was probably designed for a far slower CPU anyway, so it's probably no important if those instructions are a "slow" when the CPU overall is much faster. I agree that the interactions between instructions might increase the difficulty of security, concurrency, or other higher-level concerns, but that will depend on the specific instruction. I'm not sure how large these concerns might be, unfortunately, so I cannot say if it's a serious issue. > so their complexity ends up being "dead weight" This sounds suspiciously like the unfortunate recent practice of labeling "old" things as "bad" or "holding back progress" and thus must be removed. In most cases backwards compatibility is a very important feature that adds very little cost. The low cost is possible because you can usually implement older features in terms of the newer features. This is true in CPU microcode, just like it's true in software. [1] http://i59.tinypic.com/2mrxso9.jpg http://i59.tinypic.com/2mrxso9.jpg [2] https://en.wikichip.org/w/images/thumb/a/a1/broadwell_block_diagram.svg/850px-broadwell_block_diagram.svg.png https://en.wikichip.org/w/images/thumb/a/a1/broadwell_block_...
moyix 9y ago
I think many commenters here are viewing this only in terms of die cost. But the blog post is not about that: it's about the other costs that this complexity introduces! Each new feature interacts with all the others in complex ways, and each new feature requires significant new engineering efforts by anyone who has to interact directly with x86 assembly (I sketched a few of these use cases below [1]). The same author has also written [2] about Intel's CET (their attempt to add control flow integrity (CFI) support at the CPU level): > The tail end of Intel's document is rather terrifying; it tries to enumerate the interactions of their CFI feature with all the various execution modes that Intel currently supports, and leaves me with the impression that they're generally heading over the complexity event horizon. [1] https://news.ycombinator.com/item?id=14560267 https://news.ycombinator.com/item?id=14560267 [2] http://robert.ocallahan.org/2016/06/is-control-flow-integrity-worth.html http://robert.ocallahan.org/2016/06/is-control-flow-integrit...
payne92 9y ago
This will no doubt raise hackles, but instruction set architectures are FAR less relevant than they used to be. Fewer and fewer systems or people care about x86 vs ARM vs RISC-V. C has been the intermediate language of choice for decades now, and things like WebAssembly drive that care even lower. And if you really have to, you can binary translate. x86's real problem is Intel's business model. To compete today, you need processor IP you can integrate with other components on your die or system -- that's why ARM wins outside of the data center. In theory, Intel could sell x86 designs a la ARM, but that would completely break their business. My bet: x86 gets squeezed between the rise of GPU vector supercomputing and ARM general purpose CPUs. Apple is already doing this in-house.
- sqeaky 9y ago
  I mostly agree, but I think history and expectations play into it as well. I can imagine customers that claim they don't care, then some ISV releases some software or game that only runs on one platform, then all of sudden the customer cares. The ISV might not be malicious or in cahoots with anyone, perhaps they just have 2 decades of experience releasing for wintel and haven't reconsidered their approach.
- roca 9y ago
  "Fewer people care about the ISA" is one of Intel's "real problems". Intel having a near-monopoly on the x86 architecture is a lot less valuable than it used to be. On the other hand, while it's true that languages and runtimes provide more portability these days, there's still important handwritten assembly, and the amount of investment in compiler implementations still gives popular ISAs an edge.
heisenbit 9y ago
It is not just the complexity but also the question of economics. In general Intel on the server side is under pressure by specialized chips. Incremental spend will rather go into specialized silicon, SSDs and RAM than into CPU where it delivers more bang for the buck. On the PC side people are looking for battery life - something Intel lags. Both Apple and Microsoft are eying the ARM universe causing Intel to recently fire a patent threat in the direction of Redmond and Qualcom. From a volume perspective Intel is big but ARM with both mobile and IoT behind it has the momentum. The tick-tock has slowed. New generations are becoming harder to manufacture. Moore's law ending will assert itself in economic terms. Complexity is not your friend when the volume is increasing elsewhere and adding value becomes harder.
pcwalton 9y ago
At the risk of sounding like a broken record, I like what ARM did with AArch64. AArch64 is not entirely a clean break with 32-bit ARM, but it eliminated a lot of legacy mistakes and is generally a radical shift in the direction of a clean RISC design. Unlike x86-64, which piled 64-bit support on top of the existing ISA, keeping most of the legacy intact, AArch64 aggressively addressed problems such as conditional execution, PC-as-register, the small register file, LDM/STM complexity, barrel shifter complexity, etc. Most importantly, 32-bit mode is an optional feature not mandated by the ARM architecture, hopefully paving the way to dropping the legacy stuff entirely in the future.
- mikepurvis 9y ago
  ARM doesn't have to support legacy binaries in the same way that the wintel ecosystem does. I don't think it's really comparable.
  - runeks 9y ago
    Is it really a problem to emulate this in software, though (honest question)? As far as I can see, if compability with old software is the only concern, then losing 80% performance isn't really a problem since the new CPU is probably 5x faster anyway. Thus making up for the loss, so the old executable runs as fast on a new CPU, using emulation, as it did running in hardware on the old one. So all we'd really lose is a speedup for legacy software -- and who really needs to play Wolfenstein 3D at 80,000 FPS, as long as it works?
    - dtech 9y ago
      Yes, it is a problem. The new CPU won't be 400% faster, but maybe 10-20%. And initially there won't be any new ISA software, so all software will use the old ISA. Thus the new CPU will be much slower for "all" software (from the users perspective) because of the need for emulation. This causes the new CPU to be a bad deal for users, thus causing developers not to release software with the new ISA. This is probably an important reason why IA-64 failed, but x86-64 succeeded, Itanium CPU's ran x86 software at about half the speed for a much higher price. The ARM solution seems like the least suboptimal solution in this case, as it at least eventually allows support to be dropped, when emulation is no longer a problem. The only large-scale successful transition of an ISA migration without backwards compatibility is when Mac Os X transitioned from PPC to x86. It probably helped there that the Apple ecosystem is a bit special and much more closed and controlled, and that the Intel Core Duo's of the time had much better performance than the IBM G5's.
johnsmith21006 9y ago
The biggest threat to x86 is containers. A new unit of work removes the lock in. Then also things like Android and ChromeOS in addition. The other aspect is cloud providers have the data for the next generation of processors.
- 5ilv3r 9y ago
  Containers must be native to the host architecture.
ChuckMcM 9y ago
It is interesting that Intel finds it in this position. Their extend-extend-extend mantra (briefly interrupted in the Itanium until AMD pushed them back into it :-) has left them with a lot of "junk DNA" in their chip. That isn't horrible, the transistors are small and spreading them out helps avoid hot spots, but as pointed out it's also a waste. But this is perhaps the first time in history that there is a realistic chance of them being replaced by an alternate ISA. The largest markets for "computers" (and by that I mean the ones that people compile on and develop systems on and run data centers on, have largely adopted open source (if begrudgingly) and that means you can get functionally identical software (but not performance identical) on different ISAs. So at that point what keeps Intel alive is their volumes make it so much more cost effective than a bespoke architecture. And then there is ARM which is 'open' in the sense that anyone can get a processor architecture license, and 'high volume' as the basis for all of the compute appliances. But that effort has been slow going[1] and continues to falter on corporate models that want to jump to the level of penetration today x86 has in a heartbeat rather than through measured evolution. [1] https://www.nextplatform.com/2017/02/01/arm-server-chips-challenge-x86-cloud/ https://www.nextplatform.com/2017/02/01/arm-server-chips-cha...
yuhong 9y ago
The fun thing is that if they got rid of things like segmentation (make segment register instructions fault), 90%+ of modern x86 user mode code would still work.
ksec 9y ago
Well there is a rumours that Intel is working on a cleaner and leaner x86. http://wccftech.com/intel-developing-new-x86-uarch-succeed-core-generation/ http://wccftech.com/intel-developing-new-x86-uarch-succeed-c...

109 points by AshleysBrain 9y ago

Yes.

uam 9y ago

Would like to hear what someone with compiler writing experience has to say about this. Is it really that much of a problem?

simias 9y ago
I can't really answer that but your question reminded me of https://www.strchr.com/x86_machine_code_statistics https://www.strchr.com/x86_machine_code_statistics Turns out that in practice a tiny number of instructions make up the vast majority of all used upcodes in a typical application (although it's not clear in this article how long the tail is, there could be hundreds of instruction in that "others" slice). My understanding is that there are many legacy instructions in x86 that are basically never used nowadays. They're still a problem for silicon designers because they still have to implement them for compatibility purposes but compiler developers can chose not to use them and instead select a more "risc-y" subset of instructions to do the bulk of the work.
moyix 9y ago
I'm not sure compiler writers care that much; they don't have to support every x86 instruction and can restrict themselves to whatever subset produces fast code. Where it really kills you are things that have to analyze x86 binary code (e.g., binary static or dynamic analysis tools). Unless you cover every nook and cranny of x86, chances are you'll eventually encounter some program that uses one of these instructions and your analysis will break. And of course emulators like QEMU face the same problem. This problem becomes especially severe in the context of malware – any instruction you forget to model becomes a way for a malicious program to evade your analysis.

Iv 9y ago

In a world where JIT become more and more common, I could see many legacy features being dropped at one point. Alternatively, I wonder how feasible it would be, for a multi-core system, to keep one legacy core for incompatible processes and faster new-arch cores for running the main beef.

legulere 9y ago
The current generation of new programming languages are all compiled to machine code (Rust, Go, Swift)
- kmicklas 9y ago
  That's an implementation detail. Code written in these languages is generally trivially retargetable.
Qantourisc 9y ago
You could also use some form of pre-compiled-non-platform specific system, which you then compile/optimize to the current platform. Ideally we should also be able to decompile and recompile code. But this might be hard for x86 die the complexity.

gcatlin 9y ago

Interesting proposal for a forward compatible instruction set architecture by Agner Fog: http://www.forwardcom.info/ http://www.forwardcom.info/

bryanlarsen 9y ago

AFAICT, infrequently used instructions are implemented with microcode. I would guess that each of these instructions have a cost of a few thousand transistors, on a chip with billions.

ant6n 9y ago
But aren't there thousands of those instructions?

lloeki 9y ago

Wasn't there some dead weight dropoff† once with amd64, as a x86 CPU is unable to execute 16-bit code when in 64-bit mode? IIRC that was a notable cause of DOSBox becoming slower when going 64-bit as it had to emulate the 16-bit CPU instead of just running on metal, as well as 64-bit Windows (7?) being unable to run old DOS and Win3x software. With an increasingly rolling-release culture on software I can't wait for X-year support for hardware instruction sets /s. Wait, this is already happening for Android phones, where ARM SoC manufacturers just flip it at OS and phone vendors (Nexus 4, OnePlus X). I just realised Apple made a damn right call getting that under their control on that front. † the metal may still be there though, waiting to be jettisoned as the market for 32-bit OSes goes away.

phkahler 9y ago
>> waiting to be jettisoned as the market for 32-bit OSes goes away. I didn't know there still was a market for 32-bit x86 OSes. I suppose legacy systems - SCADA and such.
- bonzini 9y ago
  There's a market for 16-bit OSed too, though often they run virtualized because the hardware might not even exist anymore.
en4bz 9y ago
In long mode you no longer have access to the virtual 8086 mode which is most likely what DOSBox was utilizing.
legulere 9y ago
Yep they threw out a bunch of stuff. What I remember: the CS, SS, DS, and ES segment registers, binary coded decimal instructions, protection ring 1 and 2 as only 0 and 3 are used. Probably there is a lot of more things that were removed.
- wolfgke 9y ago
  > protection ring 1 and 2 as only 0 and 3 are used Are they really dropped in x86-64? I thought they are still supported also in 64 bit mode.

muricula 9y ago

This is entirely anecdotal, but a coworker of mine worked at amd, and he once told me that amd did a study and found that instructions needed for backwards compatibility accounted for less than 1% of CPU die space. Big silicon manufacturers are pretty price sensitive and the decision to break backwards compatibility is a clear cut economic one. Clearly they have chosen not to break old software at the expense of continued architectural complexity and maintenance.

Symmetry 9y ago
The problem isn't the hardware devoted to decoding or even executing instructions so much as the overall design and verification complexity. The existence of 286 call gates has implications throughout the memory subsystem and now when you're adding your TSX transactional memory instructions you have to think carefully about how they interact with call gates and whether you could be introducing bugs and make sure you have unit tests to uncover those bugs.
- muricula 9y ago
  Yes, you're right. The silicon giants decided it was cheaper to fund that testing and verification work than to break backwards compatibility.
  - xorblurb 9y ago
    Except they occasionally miss some nasty bug in that area... This is yet another danger to security.

rwmj 9y ago

Surprised to see RISC-V is not mentioned. They take an interesting approach to extensibility where it's basically envisioned in the ISA from the start. Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future. See "ISA Overview" here: https://content.riscv.org/wp-content/uploads/2016/06/riscv-spec-v2.1.pdf https://content.riscv.org/wp-content/uploads/2016/06/riscv-s...

wolfgke 9y ago
> Outside a core subset of features that programs can assume, you have to check each extension you need before you use it, so (in theory) it should be possible to drop deadweight extensions in future. x86 processors also have cpuid > https://en.wikipedia.org/wiki/CPUID https://en.wikipedia.org/wiki/CPUID to signal which extensions are supported or not. In theory it would also be possible to drop deadweight extensions this way for x86, too. The practice shows that at least for the tasks that PCs are used for, one can hardly drop anything here. Exceptions are - 3Dnow! (is not supported in Bulldozer, Bobcat and Zen anymore; cf. https://en.wikipedia.org/w/index.php?title=3DNow!&oldid=769683777#Processors_supporting_3DNow https://en.wikipedia.org/w/index.php?title=3DNow!&oldid=7696...) - A20 gate (not supported anymore since Haswell: cf. https://en.wikipedia.org/w/index.php?title=A20_line&oldid=781982575#A20_gate https://en.wikipedia.org/w/index.php?title=A20_line&oldid=78...)
drudru11 9y ago
Agreed, but then again there just isn't any practical implementation available yet. If there was a raspberry pi like device, I think we would start seeing it mentioned more.
- microcolonel 9y ago
  I'm guessing you mean workstation/server scale implementations? There is a shipping microcontroller implementation and has been for a while now.
- rwmj 9y ago
  We should have an RV64G implementation this year with any luck.

amluto 9y ago

Even ignoring CPU die size cost, there's another cost to legacy features: some of those features have register state, and contexts switching it is expensive. Simpler architectures can do crazy things like switching between threads in mere tens of cycles. X86: not so much. The OS needs to deal with DS, ES, FS, and GS, and the microcode needs to deal with CS, SS, and FLAGS. The segment registers are almost useless these days, but saving and restoring them is very slow. Even x86's model of handling CPL (privilege level) isn't ideal. I've tried to push RISC-V toward a model where user and kernel code have entirely separate address spaces, which would let privilege handling move to the front end and potentially allow pipelining across privilege changes. Amusingly, x86's privilege mechanism is complicated enough that Intel implemented it wrong in a few CPUs with MPX, resulting in a bizarre erratum. I would love to see x86 add a "clean mode" in which a lot of legacy features are inaccessible to user code. Then kernels could try to run programs in clean mode and, if they don't fault, context switching between clean mode threads would be much faster.

frozenport 9y ago
Also AVX512! And what the heck are we going to do when we need to save the AVX1024 register state! And we really going to push 1kb to memory per context switch? (I think this is somewhat solved by trapping when the registers are first invoked, and kernel drivers that use enhanced instruction sets actually need to pass flags before using them)
- jandrewrogers 9y ago
  This highlights one of the advantages of context-switching in user space. The OS context switch can make few assumptions about the registers that actually need to be preserved once a register set has been used. Many user space context libraries, on the other hand, only preserve the half dozen basic registers per the ABI by default (very fast) and let the programmer optionally preserve the other register sets. In the vast majority of cases, you never need to preserve any additional state at the logical context switch points even when you are using those additional registers in your application.

warbiscuit 9y ago

The post posits that, to handle the burden of legacy instructions, an "obvious technically-appealing approach (is) starting over with a clean-sheet architecture". The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?). Of course, doing something like that is probably nowhere near trivial, the devil's always in the details. But I bet this is already being done at the microcode level. Stepping things up to having a published agreement about which instructions were globally considered "legacy", and guidelines for what their equivalents were, would go a long was towards allowing a general feeling that an ISA was evolving, rather than just accumulating weight upon weight.

kllrnohj 9y ago
> The approach that immediately occurred to me would be have a layer that translates the legacy instructions into modern equivalents; without as much concern if they are slower to execute in their new form (they're legacy, after all, right?). This is what x86 CPUs already do and have been doing for years. It's also why Intel & AMD don't care about "x86 complexity" and it's also why even though people love to claim that a switch to RISC would improve efficiency/performance, so far there's really no evidence to support that. Intel just plops their teeny-tiny (in terms of die space) x86-to-internal-microcode transactor on top of their new cores and calls it a day. As long as that translation layer isn't a bottleneck, which it rarely is, then Intel doesn't care.
- roca 9y ago
  It's not just about legacy instructions which can be decoded down to some microcode. It's about architectural features like SGX, CET, MPX, TSX, VT --- plus the legacy stuff like segment registers and 286 call gates and virtual 8086 mode and so on and so on --- and how they all interact with each other, and how they increase the complexity of context switching, OS support, and so on.
  - kllrnohj 9y ago

jonstokes 9y ago

Oh for pete's sake, the die size cost for supporting the legacy x86 instructions is a fraction of a percent on a modern multicore CPU. This is because the translation hardware stays relatively fixed (in terms of size and complexity and transistor count), while Moore's Law keeps adding more hardware to each core and more cache, etc.. Take a look at this annotated die shot of an old Pentium 4: https://i.stack.imgur.com/QK4gm.jpg https://i.stack.imgur.com/QK4gm.jpg Up at the top is the microcode memory and microcode sequencer. That's the hardware that's responsible for translating all those old, large legacy instructions. It's not really much space on the P4, and the P4 Northwood is a 55-million transistor CPU. Nowadays, depending on if you're talking about a mobile part or a higher-end desktop or server part, the CPUs have between ~1.5 billion to 2 billion transistors. Again, that microcode ROM just doesn't grow very much as you add instructions, even a ton of instructions. And this is, well, one reason why Intel just keeps adding instructions. It's practically free.

STRML 9y ago
That's a great graphic! I see a few more from around 2003 on http://chip-architect.com/ http://chip-architect.com/, do you know of anyone who's doing modern chips (x86 or ARM) this way?
api 9y ago
You're right when it comes to die size and transistor budget, but complexity can sometimes have hidden costs. Complexity can limit and constrain the design of other aspects of a system in ways that are nearly impossible to predict and often non-obvious even when you're down in the trenches. Not only can today's features limit tomorrow's design, but because those features give rise to mental models and habits of thinking they can limit future design in an almost unconscious way. The more complex your system becomes the more its existing complexity dominates your thinking to the exclusion of innovative ideas. This process happens so quickly in software that it's obvious there: http://www.retrologic.com/jargon/R/rococo.html http://www.retrologic.com/jargon/R/rococo.html
Retric 9y ago
It's not just ancient instructions that add costs, P4's MMX is also outdated etc. Further the core issue is the cascade where A limited B and C is limited by B directly and A indirectly, now continue that though 15 generations and things get complicated.

pdkl95 9y ago

> legacy compatibility features [...] those CPUs won't be as efficient, cheap, reliable or secure as they otherwise could be [...] we can't rely on increasing transistor density to bail us out. Legacy features shouldn't have a large impact on transistor usage, and they should have almost no impact on efficiency (although that may involve design choices). Almost all of the transistors are going to be used to implement the cache, routing between the high level blocks (external IO, cores, shared cache), and routing between the various microarchitecture units. Using Broadwell as an example, the CPU cores[1] only use a relatively small amount of amount of area. However, even that is misleading, because most of the acre within each core is dedicated to the microarchitecture, which implements a far smaller set of micro-ops. Inside each core[2], the only place affected by legacy instructions is the decode stage (mostly in the "4-Way Decode" block in [2], probably less in the "PreDecode" block"), and a larger MicoCode ROM. The legacy instructions might not be as efficient as they used to be in older CPUs; as low-priority instructions they may be implemented with less-convenient micro-ops that were designed for other, faster instructions. The software that uses those instructions was probably designed for a far slower CPU anyway, so it's probably no important if those instructions are a "slow" when the CPU overall is much faster. I agree that the interactions between instructions might increase the difficulty of security, concurrency, or other higher-level concerns, but that will depend on the specific instruction. I'm not sure how large these concerns might be, unfortunately, so I cannot say if it's a serious issue. > so their complexity ends up being "dead weight" This sounds suspiciously like the unfortunate recent practice of labeling "old" things as "bad" or "holding back progress" and thus must be removed. In most cases backwards compatibility is a very important feature that adds very little cost. The low cost is possible because you can usually implement older features in terms of the newer features. This is true in CPU microcode, just like it's true in software. [1] http://i59.tinypic.com/2mrxso9.jpg http://i59.tinypic.com/2mrxso9.jpg [2] https://en.wikichip.org/w/images/thumb/a/a1/broadwell_block_diagram.svg/850px-broadwell_block_diagram.svg.png https://en.wikichip.org/w/images/thumb/a/a1/broadwell_block_...

moyix 9y ago

I think many commenters here are viewing this only in terms of die cost. But the blog post is not about that: it's about the other costs that this complexity introduces! Each new feature interacts with all the others in complex ways, and each new feature requires significant new engineering efforts by anyone who has to interact directly with x86 assembly (I sketched a few of these use cases below [1]). The same author has also written [2] about Intel's CET (their attempt to add control flow integrity (CFI) support at the CPU level): > The tail end of Intel's document is rather terrifying; it tries to enumerate the interactions of their CFI feature with all the various execution modes that Intel currently supports, and leaves me with the impression that they're generally heading over the complexity event horizon. [1] https://news.ycombinator.com/item?id=14560267 https://news.ycombinator.com/item?id=14560267 [2] http://robert.ocallahan.org/2016/06/is-control-flow-integrity-worth.html http://robert.ocallahan.org/2016/06/is-control-flow-integrit...

payne92 9y ago

This will no doubt raise hackles, but instruction set architectures are FAR less relevant than they used to be. Fewer and fewer systems or people care about x86 vs ARM vs RISC-V. C has been the intermediate language of choice for decades now, and things like WebAssembly drive that care even lower. And if you really have to, you can binary translate. x86's real problem is Intel's business model. To compete today, you need processor IP you can integrate with other components on your die or system -- that's why ARM wins outside of the data center. In theory, Intel could sell x86 designs a la ARM, but that would completely break their business. My bet: x86 gets squeezed between the rise of GPU vector supercomputing and ARM general purpose CPUs. Apple is already doing this in-house.

sqeaky 9y ago
I mostly agree, but I think history and expectations play into it as well. I can imagine customers that claim they don't care, then some ISV releases some software or game that only runs on one platform, then all of sudden the customer cares. The ISV might not be malicious or in cahoots with anyone, perhaps they just have 2 decades of experience releasing for wintel and haven't reconsidered their approach.
roca 9y ago
"Fewer people care about the ISA" is one of Intel's "real problems". Intel having a near-monopoly on the x86 architecture is a lot less valuable than it used to be. On the other hand, while it's true that languages and runtimes provide more portability these days, there's still important handwritten assembly, and the amount of investment in compiler implementations still gives popular ISAs an edge.

heisenbit 9y ago

It is not just the complexity but also the question of economics. In general Intel on the server side is under pressure by specialized chips. Incremental spend will rather go into specialized silicon, SSDs and RAM than into CPU where it delivers more bang for the buck. On the PC side people are looking for battery life - something Intel lags. Both Apple and Microsoft are eying the ARM universe causing Intel to recently fire a patent threat in the direction of Redmond and Qualcom. From a volume perspective Intel is big but ARM with both mobile and IoT behind it has the momentum. The tick-tock has slowed. New generations are becoming harder to manufacture. Moore's law ending will assert itself in economic terms. Complexity is not your friend when the volume is increasing elsewhere and adding value becomes harder.

pcwalton 9y ago

At the risk of sounding like a broken record, I like what ARM did with AArch64. AArch64 is not entirely a clean break with 32-bit ARM, but it eliminated a lot of legacy mistakes and is generally a radical shift in the direction of a clean RISC design. Unlike x86-64, which piled 64-bit support on top of the existing ISA, keeping most of the legacy intact, AArch64 aggressively addressed problems such as conditional execution, PC-as-register, the small register file, LDM/STM complexity, barrel shifter complexity, etc. Most importantly, 32-bit mode is an optional feature not mandated by the ARM architecture, hopefully paving the way to dropping the legacy stuff entirely in the future.

mikepurvis 9y ago
ARM doesn't have to support legacy binaries in the same way that the wintel ecosystem does. I don't think it's really comparable.
- runeks 9y ago
  Is it really a problem to emulate this in software, though (honest question)? As far as I can see, if compability with old software is the only concern, then losing 80% performance isn't really a problem since the new CPU is probably 5x faster anyway. Thus making up for the loss, so the old executable runs as fast on a new CPU, using emulation, as it did running in hardware on the old one. So all we'd really lose is a speedup for legacy software -- and who really needs to play Wolfenstein 3D at 80,000 FPS, as long as it works?
  - dtech 9y ago
    Yes, it is a problem. The new CPU won't be 400% faster, but maybe 10-20%. And initially there won't be any new ISA software, so all software will use the old ISA. Thus the new CPU will be much slower for "all" software (from the users perspective) because of the need for emulation. This causes the new CPU to be a bad deal for users, thus causing developers not to release software with the new ISA. This is probably an important reason why IA-64 failed, but x86-64 succeeded, Itanium CPU's ran x86 software at about half the speed for a much higher price. The ARM solution seems like the least suboptimal solution in this case, as it at least eventually allows support to be dropped, when emulation is no longer a problem. The only large-scale successful transition of an ISA migration without backwards compatibility is when Mac Os X transitioned from PPC to x86. It probably helped there that the Apple ecosystem is a bit special and much more closed and controlled, and that the Intel Core Duo's of the time had much better performance than the IBM G5's.

johnsmith21006 9y ago

The biggest threat to x86 is containers. A new unit of work removes the lock in. Then also things like Android and ChromeOS in addition. The other aspect is cloud providers have the data for the next generation of processors.

5ilv3r 9y ago
Containers must be native to the host architecture.

ChuckMcM 9y ago

It is interesting that Intel finds it in this position. Their extend-extend-extend mantra (briefly interrupted in the Itanium until AMD pushed them back into it :-) has left them with a lot of "junk DNA" in their chip. That isn't horrible, the transistors are small and spreading them out helps avoid hot spots, but as pointed out it's also a waste. But this is perhaps the first time in history that there is a realistic chance of them being replaced by an alternate ISA. The largest markets for "computers" (and by that I mean the ones that people compile on and develop systems on and run data centers on, have largely adopted open source (if begrudgingly) and that means you can get functionally identical software (but not performance identical) on different ISAs. So at that point what keeps Intel alive is their volumes make it so much more cost effective than a bespoke architecture. And then there is ARM which is 'open' in the sense that anyone can get a processor architecture license, and 'high volume' as the basis for all of the compute appliances. But that effort has been slow going[1] and continues to falter on corporate models that want to jump to the level of penetration today x86 has in a heartbeat rather than through measured evolution. [1] https://www.nextplatform.com/2017/02/01/arm-server-chips-challenge-x86-cloud/ https://www.nextplatform.com/2017/02/01/arm-server-chips-cha...

yuhong 9y ago

The fun thing is that if they got rid of things like segmentation (make segment register instructions fault), 90%+ of modern x86 user mode code would still work.

ksec 9y ago

Well there is a rumours that Intel is working on a cleaner and leaner x86. http://wccftech.com/intel-developing-new-x86-uarch-succeed-core-generation/ http://wccftech.com/intel-developing-new-x86-uarch-succeed-c...