7 ms·
Intel X86 Encoder Decoder
- dreamcompiler 10y ago"As new instructions are made public"? WTF does that mean? We're not even allowed to know the full instruction set of an X86 CPU?
- mmastrac 10y agoRead this as "as upcoming instructions in new chips are documented"
- wyldfire 10y agoYes, that's likely what the author meant. But, also, the author might have access to development specs or early samples not publicly announced yet.
- user5994461 10y agoThere is a many thousands pages instructions manual for that, that is freely downloadable from Intel. It's re-posted on HN from time to time.
- lobster_johnson 10y agoSpeaking of which... I'd really love to have a reference manual on one of the early x86 processors. Maybe the 386, since it is one of the iconic ones (or the one I have the most nostalgia for) that also started to get interesting features like protected mode, but still wasn't so complicated as to be a multi-volume, 5,000-page series (maybe?). Back in the 1990s, when I was doing assembly language on the Amiga, Motorola sent me the official 68000 manual for free when I called and asked them. It was a really cool book to have on the shelf and occasionally lead through. I looked at abebooks.com, and there are Intel books, but I wouldn't know exactly which one would be the reference manual. Anyone got an ISBN?
- ChickeNES 10y agoFor the 80386 there are two manuals, a Hardware Reference Manual (ISBN 1-55512-069-5) and a Programmers Reference Manual (ISBN 1-55512-022-9). They come up on eBay from time-to-time, I think I paid $30 for my pair.
- lobster_johnson 10y agoThat's exactly what I was looking for, thanks. Found them on abebooks.com easily. Awesome 1980s neon covers.
- matt_d 10y agoTry Bitsavers (an amazing work in the service of historic preservation): http://www.bitsavers.org/ http://www.bitsavers.org/ Previously discussed here: https://news.ycombinator.com/item?id=10143295 https://news.ycombinator.com/item?id=10143295 Internet Archive collection (the Intel documents part): https://archive.org/details/bitsavers_intel https://archive.org/details/bitsavers_intel Library of Congress reference: https://www.loc.gov/item/lcwa00096459/ https://www.loc.gov/item/lcwa00096459/
- userbinator 10y agoIntel used to give away hardcopy manuals for free too, but stopped doing that around 2010: http://styx.head-crash.de/stuff/intel_manuals.jpg http://styx.head-crash.de/stuff/intel_manuals.jpg Now they still have hardcopy, although it's basically just paying for someone else to print and bind the PDFs for you: http://www.lulu.com/shop/search.ep?contributorId=1030088 http://www.lulu.com/shop/search.ep?contributorId=1030088
- SolarNet 10y agoThis is (partially) meant to be used for chips that don't exist. As future instructions are made public software vendors (like compiler vendors) can use this to be ready when the chips come out day 1.
- Sephr 10y agoDon't expect any management engine/system management mode instructions any time soon/ever.
- derefr 10y agoNo reason to hide them; presumably, as with firmware updates, you have to load an Intel signing key into a few registers along with your IME/SMM opcode to get it to execute.
- amluto 10y agoHuh? SMM is fully documented at the CPU level. QEMU can even emulate it. Not sure what you mean by "management mode".
- Sephr 10y agoSorry, I meant the undocumented MEI and SMM features.
- dbcurtis 10y agoNot allowed yet. Ex-Intel here. It takes years to gestate new instructions. First specs are in controlled documents available to Intel employees only. Eventually, when things are nailed down, preliminary specs are available under an NDA where a VP approves/signs the Intel side of the NDA. Tool vendors will get those specs. Finally, when the chip is announced the previously NDA documents become public. I was a CPU designer at multiple companies, and everyone follows a similar process.
- derefr 10y agoYou're just talking about the design phase of a new ISA extension. I think the parent was under the impression that there were instructions in released processors that were undocumented.
- Senji 10y agoThere are.
- amluto 10y agoThere is at least one Intel processor with a public erratum to an instruction (or other mechanism?) that is not public. It's a timed MWAIT, I think.
- hobo_mark 10y agoAnd that impression would be correct. Furthermore, if you are a big enough customer you can also negotiate for special isa extensions yourself (can't provide citation but heard from people at Intel), then of course you'd be the only one to have the documentation to take advantage of them.
- dreamcompiler 10y agoThat was exactly my impression. I don't have a problem with new instructions in new processors being undocumented at first, but purposely undocumented instructions are troubling and would push me away from Intel for new designs.
- jonjohn84 10y agoLooks handy. Thanks for posting, Dan.
- monocasa 10y agoI wonder if this was released due to Xen's recent x86 instruction emulation bugs. http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-9932 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-9932 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-9383 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-9383
- Sephr 10y agoThose CVEs are private. Could you provide a public source?
- Fnoord 10y agoRelevant Xen Security Advisory https://xenbits.xen.org/xsa/advisory-200.html https://xenbits.xen.org/xsa/advisory-200.html https://xenbits.xen.org/xsa/advisory-195.html https://xenbits.xen.org/xsa/advisory-195.html All XSA https://xenbits.xen.org/xsa/ https://xenbits.xen.org/xsa/
- aseipp 10y agoWhy would the release of this library have anything to do with Xen vulnerabilities?
- rdmsr 10y agoProbably not (but who knows, you could probably use this as part of a fuzzer). Instruction emulation is a superset of instruction decoding. You need to decode and then emulate the behavior. Aside: Instruction emulation is pretty finicky and bug prone. I'm not too familiar with Xen, but KVM has had at least 10 instruction emulation CVEs. There were talks at both KVM Forum and Xen Summit last summer mentioning the sketchiness of instruction emulation.
- deleted 10y ago[deleted]
- majke 10y agoThis is awesome. I thought it's closed source. I wonder how close it is to the hardware. Can I assume that whatever is parsed by XED is going to be parsed in the same way by real CPU's?
- mjcharney 10y agoXED author here... That's one goal certainly. Not claiming it is perfect. I have been working on it a long time and many cycles have been run on it between Pin, Intel SDE and some internal simulators that boot OSes, etc.
- thecompilr 10y agoAbout time they open sourced it. Funny how it is Python according to github :)
- mjcharney 10y agoPython is used to generate C code from tables. Also use python to build/package/test.
- chubot 10y agoWhy is that funny? Python seems like the perfect language for this kind of thing. You can see the logic with minimal noise.
- unwind 10y agoIt's funny since it's not very accurate: the Python is used to generate C code from tables describing instruction encodings (that's my understanding, haven't cloned this and built it). But there is plenty of actual C code in there too, and the resulting "thing" is a C library. Python would not be as suitable as the full implementation language, since C is much more easily accessed from other languages, which is a good property for a library like this.
- derefr 10y agoSeems like this would make one able to write the object-code codegen phase of a compiler at a rather higher level (at least if you're only targeting x86, or are willing to write similar libraries for your other arch targets.) Or, to put it another way: looks like a good library for adding "just a bit of JIT" to an interpreter, without going full LLVM.
- dsfyu404ed 10y agoSomething that is universal is not remarkable. People will not make as much use of an unremarkable opportunity as they will of a remarkable one.
- struct 10y agoHow does this compare to ARM's VIXL? [1] https://github.com/armvixl/vixl https://github.com/armvixl/vixl