4 ms·

x86 prefixes and escape opcodes flowchart

98 points by gaul 8mo ago

snvzz 8mo ago
This is in no small part why x86 code density is awful despite variable size encoding.
- themafia 8mo ago
  Awful compared to what? I've seen benchmarks that go both ways in terms of a "winner" but in terms of overall variance there seems to be very little. There are some cases where ARM64 or RISCV do better and there are some cases where x86_64 does better. I can't see code density being a relevant factor when picking one ISA over another. We've got good compilers now anyways.. outside of power consumption.. the ISA wars are dead.
  - deleted 8mo ago
    [deleted]
  - bell-cot 8mo ago
    Technically, code density still matters - because both L1 cache memory and L1 instruction fetch misses are very expensive. But as you point out, code density gets far less attention in tech circles these days. And higher-level decision makers rightfully focus on higher-level system performance metrics.
  - whobre 8mo ago
    Compared to VAX, of course…
  - adgjlsfhk1 8mo ago
    > outside of power consumption. This is a pretty huge caveat. >90% of cpus are <1W (usb cables, wifi cards, storage controllers etc), and 99% are <10W (phones, lots of laptops)
    - adrian_b 8mo ago
      You are right about the order of magnitude of the percentages, but the CPUs <10 W are in phones, tablets, small SBCs (single-board computers) and only in a small fraction of laptops and mini-PCs. The vast majority of laptops and mini-PCs uses more powerful CPUs. While 15 W CPUs were normal many years ago, when Intel launched the "Ultrabook" campaign, in recent years, due to better cooling, most laptops and mini-PCs use 25 W to 30 W CPUs, and a significant fraction of them (bigger than the fraction of the laptops/mini-PCs that uses <=10 W CPUs) uses CPUs with a TDP between 35 W and 65 W.
      - adgjlsfhk1 8mo ago
        Lots of laptops have ~30W cpus, but that's peak usage. A typical laptop has a ~50WH battery, and a ~5W screen. If it's getting 5hr battery life, that means the CPU is averaging <5W.
        adrian_b 8mo ago
        True, but if you are classifying the CPUs by their idle power consumption, then that is completely unrelated to the ISA implemented by the CPU and even the correlation with the power consumed at maximum load is very weak.
debugnik 8mo ago
This site redirects to HN when it notices HN in the referrer.
- chimpontherun 8mo ago
  open in new tab
  - yellowapple 8mo ago
    That doesn't seem to clear the referrer, at least on Firefox. Gotta go a step further and outright copy/paste the URL into an already-created tab.
    - high_na_euv 8mo ago
      Open in private works
- therein 8mo ago
  Wow, I didn't even notice because I have extensions that strip the referrer header. Excellent.
- st_goliath 8mo ago
  If you have JavaScript enabled, that is. JWZ at least does the redirect on the server side. The following is pulled in from `https://soc.me/assets/js/turnBack.js https://soc.me/assets/js/turnBack.js`: const undesirables = [ "news.ycombinator.com/", // "reddit.com/", // disable temporaily "lobste.rs/" ] ; if (undesirables.find(site => document.referrer.includes(site))) { window.location.replace(document.referrer); } I wonder why Reddit is "temporarily not undesirable".
  - mechazawa 8mo ago
    Why are they undesirable though
  - self_awareness 8mo ago
dagenix 8mo ago
https://archive.ph/ZMsnQ https://archive.ph/ZMsnQ
- korrectional 8mo ago
  captcha farm?
  - ycombinatrix 8mo ago
    No captcha for me, though I do often have captcha problems with the site.
tucnak 8mo ago
I respect the disobedience.
st_goliath 8mo ago
Fun little tidbit: The 0x40-0x4f range used for the REX prefix actually clashes with the single-byte encodings for increment/decrement. When AMD designed the 64 bit extension, they had run out of available single-byte opcodes to use as a prefix and decided to re-use those. The INC/DEC instructions are still available in 64 bit mode, but not in their single-byte encodings.
- TheAdamist 8mo ago
  Which clever code can utilize to determine which mode its running in and branch appropriately depending if the inc/dec were executed or not.
- adgjlsfhk1 8mo ago
  It is still crazy to me that intel though having 2 1 byte instructions for something that could just be coded as a single add with immediate instruction (which they had anyway).
adrian_b 8mo ago
On that page, there is a link to another interesting page on the same site: https://soc.me/interfaces/intels-original-64bit-extensions-for-x86.html https://soc.me/interfaces/intels-original-64bit-extensions-f... where there are links to a couple of patents filed by Intel in 2000, about a 64-bit extension of the x86 ISA, which had been implemented in Pentium 4, but which had been nonetheless disabled and hidden from the users, in order to not compete with Itanium. The page explains the content of the patents. As already mentioned by another poster, at least on Firefox you have to open a tab and then copy this link there, to avoid being identified as an "undesirable" :-)
jxors 8mo ago
This flowchart hides the most awful parts (IMO) of x86 prefixes: some combinations of prefixes are invalid but still parsed and executed, like combining two segment overrides, or placing a legacy prefix after a REX prefix. The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. The LOCK prefix is the only prefix that makes the sane choice to reject invalid combinations, rather than silently accept them. Also, the (E)VEX prefix doesn't behave like the other prefixes: it must be placed last, and can therefore only appear once. All other prefixes can be repeated.
- vardump 8mo ago
  I wonder whether there are some prefixes that cause (some) CPUs to execute the instruction a lot slower.
  - jxors 8mo ago
    This is indeed a thing. I believe in general instructions are executed slower when there are more than 4 legacy prefixes. And there are plenty of other timing differences between different microarchitectures
- peterfirefly 8mo ago
  > The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. This is one of the reasons why the x86 could be extended so much. PAUSE is just REP NOP, for example. Segment prefixes in front of conditional branches were used as static branch prediction hints (which I believe have returned in some newer Intel CPUs). Useful if you want to make a hint on newer CPUs that is harmless on older CPUs. Some prefixes have become part of the encoding for certain SIMD instructions, but that is a different case because those prefixes aren't hints.
  - adrian_b 8mo ago
    Not at all. The correct behavior for allowing future extensions has already been introduced by Intel with 80186, in 1982, which has introduced an invalid instruction exception, to be used for all undefined instruction opcodes. This behavior was unlike 8086/8088, which happily executed any undefined instructions, most of them being aliases to defined instructions. For any opcode where current CPUs generate invalid instruction exceptions, it is very easy to define them in future CPUs to encode useful instructions. Had REP NOP generated exceptions in old CPUs, it would have been still fine for it to become PAUSE in current CPUs. Unfortunately, the designers of Intel CPUs have not always followed their own documentation, so not all invalid opcodes generate the exception, as they should. The non-enforcing of this condition has led to the existence of even commercial programs that are invalid or of compilers that generate officially invalid instructions. It is true that there are a few cases when Intel has exploited the fact that some encodings were equivalent with a NOP on old CPUs, by reusing them for some instruction on new CPUs, where this allowed the execution of a program compiled for new CPUs on old CPUs. However this has been possible only for very few instructions, e.g. for branch direction hints, when not executing them on old CPUs does not change the result of a program. In general the reuse of an opcode for a new instruction, when that opcode does not generate exceptions on old CPUs, is very dangerous, because the execution on old CPUs of a program compiled for new CPUs will have unpredictable consequences, like destroying some property of the user. Your example with PAUSE is also one of the very few examples, besides branch hints, where the execution of a new program on old computers is not dangerous, despite the reassignment of the opcode. Some time ago there was a discussion about a bug in some CPU, but I do not remember in which one, where the bug was triggered when the order of the REP prefix and of the 64-bit REX prefix was invalid, but the invalid order was ignored by the older CPUs instead of generating the appropriate exception, which allowed the execution of invalid programs, which did not have any bad effects on old CPUs, but they triggered the bug on that specific new CPU. The new CPU should have been bug-free, but also the programs that triggered the bug should not have existed, as they should have crashed immediately on any older CPU.
dataflow 8mo ago
There's EVEX2 now. It's hard to keep up...
notepad0x90 8mo ago
hiding this one, i hope others downvote it as well. Dick move by the person hosting this redirecting back to HN. Such incessant levels of pettiness are so irritating.

98 points by gaul 8mo ago

snvzz 8mo ago

This is in no small part why x86 code density is awful despite variable size encoding.

themafia 8mo ago
Awful compared to what? I've seen benchmarks that go both ways in terms of a "winner" but in terms of overall variance there seems to be very little. There are some cases where ARM64 or RISCV do better and there are some cases where x86_64 does better. I can't see code density being a relevant factor when picking one ISA over another. We've got good compilers now anyways.. outside of power consumption.. the ISA wars are dead.
- deleted 8mo ago
  [deleted]
- bell-cot 8mo ago
  Technically, code density still matters - because both L1 cache memory and L1 instruction fetch misses are very expensive. But as you point out, code density gets far less attention in tech circles these days. And higher-level decision makers rightfully focus on higher-level system performance metrics.
- whobre 8mo ago
  Compared to VAX, of course…
- adgjlsfhk1 8mo ago
  > outside of power consumption. This is a pretty huge caveat. >90% of cpus are <1W (usb cables, wifi cards, storage controllers etc), and 99% are <10W (phones, lots of laptops)
  - adrian_b 8mo ago
    You are right about the order of magnitude of the percentages, but the CPUs <10 W are in phones, tablets, small SBCs (single-board computers) and only in a small fraction of laptops and mini-PCs. The vast majority of laptops and mini-PCs uses more powerful CPUs. While 15 W CPUs were normal many years ago, when Intel launched the "Ultrabook" campaign, in recent years, due to better cooling, most laptops and mini-PCs use 25 W to 30 W CPUs, and a significant fraction of them (bigger than the fraction of the laptops/mini-PCs that uses <=10 W CPUs) uses CPUs with a TDP between 35 W and 65 W.
    - adgjlsfhk1 8mo ago
      Lots of laptops have ~30W cpus, but that's peak usage. A typical laptop has a ~50WH battery, and a ~5W screen. If it's getting 5hr battery life, that means the CPU is averaging <5W.
      - adrian_b 8mo ago
        True, but if you are classifying the CPUs by their idle power consumption, then that is completely unrelated to the ISA implemented by the CPU and even the correlation with the power consumed at maximum load is very weak.

debugnik 8mo ago

This site redirects to HN when it notices HN in the referrer.

chimpontherun 8mo ago
open in new tab
- yellowapple 8mo ago
  That doesn't seem to clear the referrer, at least on Firefox. Gotta go a step further and outright copy/paste the URL into an already-created tab.
  - high_na_euv 8mo ago
    Open in private works
therein 8mo ago
Wow, I didn't even notice because I have extensions that strip the referrer header. Excellent.
st_goliath 8mo ago
If you have JavaScript enabled, that is. JWZ at least does the redirect on the server side. The following is pulled in from `https://soc.me/assets/js/turnBack.js https://soc.me/assets/js/turnBack.js`: const undesirables = [ "news.ycombinator.com/", // "reddit.com/", // disable temporaily "lobste.rs/" ] ; if (undesirables.find(site => document.referrer.includes(site))) { window.location.replace(document.referrer); } I wonder why Reddit is "temporarily not undesirable".
- mechazawa 8mo ago
  Why are they undesirable though
- self_awareness 8mo ago

dagenix 8mo ago

https://archive.ph/ZMsnQ https://archive.ph/ZMsnQ

korrectional 8mo ago
captcha farm?
- ycombinatrix 8mo ago
  No captcha for me, though I do often have captcha problems with the site.

tucnak 8mo ago

I respect the disobedience.

st_goliath 8mo ago

Fun little tidbit: The 0x40-0x4f range used for the REX prefix actually clashes with the single-byte encodings for increment/decrement. When AMD designed the 64 bit extension, they had run out of available single-byte opcodes to use as a prefix and decided to re-use those. The INC/DEC instructions are still available in 64 bit mode, but not in their single-byte encodings.

TheAdamist 8mo ago
Which clever code can utilize to determine which mode its running in and branch appropriately depending if the inc/dec were executed or not.
adgjlsfhk1 8mo ago
It is still crazy to me that intel though having 2 1 byte instructions for something that could just be coded as a single add with immediate instruction (which they had anyway).

adrian_b 8mo ago

On that page, there is a link to another interesting page on the same site: https://soc.me/interfaces/intels-original-64bit-extensions-for-x86.html https://soc.me/interfaces/intels-original-64bit-extensions-f... where there are links to a couple of patents filed by Intel in 2000, about a 64-bit extension of the x86 ISA, which had been implemented in Pentium 4, but which had been nonetheless disabled and hidden from the users, in order to not compete with Itanium. The page explains the content of the patents. As already mentioned by another poster, at least on Firefox you have to open a tab and then copy this link there, to avoid being identified as an "undesirable" :-)

jxors 8mo ago

This flowchart hides the most awful parts (IMO) of x86 prefixes: some combinations of prefixes are invalid but still parsed and executed, like combining two segment overrides, or placing a legacy prefix after a REX prefix. The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. The LOCK prefix is the only prefix that makes the sane choice to reject invalid combinations, rather than silently accept them. Also, the (E)VEX prefix doesn't behave like the other prefixes: it must be placed last, and can therefore only appear once. All other prefixes can be repeated.

vardump 8mo ago
I wonder whether there are some prefixes that cause (some) CPUs to execute the instruction a lot slower.
- jxors 8mo ago
  This is indeed a thing. I believe in general instructions are executed slower when there are more than 4 legacy prefixes. And there are plenty of other timing differences between different microarchitectures
peterfirefly 8mo ago
> The CPU also doesn't care if you use prefixes that aren't valid for a specific instruction, for example a REP on a non-repeatable instruction. This is one of the reasons why the x86 could be extended so much. PAUSE is just REP NOP, for example. Segment prefixes in front of conditional branches were used as static branch prediction hints (which I believe have returned in some newer Intel CPUs). Useful if you want to make a hint on newer CPUs that is harmless on older CPUs. Some prefixes have become part of the encoding for certain SIMD instructions, but that is a different case because those prefixes aren't hints.
- adrian_b 8mo ago
  Not at all. The correct behavior for allowing future extensions has already been introduced by Intel with 80186, in 1982, which has introduced an invalid instruction exception, to be used for all undefined instruction opcodes. This behavior was unlike 8086/8088, which happily executed any undefined instructions, most of them being aliases to defined instructions. For any opcode where current CPUs generate invalid instruction exceptions, it is very easy to define them in future CPUs to encode useful instructions. Had REP NOP generated exceptions in old CPUs, it would have been still fine for it to become PAUSE in current CPUs. Unfortunately, the designers of Intel CPUs have not always followed their own documentation, so not all invalid opcodes generate the exception, as they should. The non-enforcing of this condition has led to the existence of even commercial programs that are invalid or of compilers that generate officially invalid instructions. It is true that there are a few cases when Intel has exploited the fact that some encodings were equivalent with a NOP on old CPUs, by reusing them for some instruction on new CPUs, where this allowed the execution of a program compiled for new CPUs on old CPUs. However this has been possible only for very few instructions, e.g. for branch direction hints, when not executing them on old CPUs does not change the result of a program. In general the reuse of an opcode for a new instruction, when that opcode does not generate exceptions on old CPUs, is very dangerous, because the execution on old CPUs of a program compiled for new CPUs will have unpredictable consequences, like destroying some property of the user. Your example with PAUSE is also one of the very few examples, besides branch hints, where the execution of a new program on old computers is not dangerous, despite the reassignment of the opcode. Some time ago there was a discussion about a bug in some CPU, but I do not remember in which one, where the bug was triggered when the order of the REP prefix and of the 64-bit REX prefix was invalid, but the invalid order was ignored by the older CPUs instead of generating the appropriate exception, which allowed the execution of invalid programs, which did not have any bad effects on old CPUs, but they triggered the bug on that specific new CPU. The new CPU should have been bug-free, but also the programs that triggered the bug should not have existed, as they should have crashed immediately on any older CPU.

dataflow 8mo ago

There's EVEX2 now. It's hard to keep up...

notepad0x90 8mo ago

hiding this one, i hope others downvote it as well. Dick move by the person hosting this redirecting back to HN. Such incessant levels of pettiness are so irritating.