7 ms·
x86-64 Assembly Language Programming with Ubuntu (2022)
- kd913 3y agoThank you for this resource. Bookmarked and will hopefully prove useful for debugging crash or interviews.
- corysama 3y agoBesides media codecs and embedded microcontrollers, what are major uses of writing raw assembly language these days? I worked in game engines for quite a while and everyone I know of there sticks to intrinsics.
- spicymaki 3y agoDebugging, Emulation and Compiler development comes to mind.
- moonchild 3y agoMedia codecs are a microcosm of routines that want more care and attention than compilers are yet able to give. Some of RAD's decompression kernels are written in assembly, for instance, and a particular unicode transcoding routine was 20% faster in assembly than in c with intrinsics. An anecdote I heard: it was appreciably faster to run the linux version of postgresql in a translation layer under solaris than to run the native version, because the linux c library's strings functions were written in assembly. A particular binary search routine I rewrote in assembly (from c) was about 4x faster; I would estimate that about half of that was from an improved algorithm, and the other half was from the choice of language. (Of course, somebody else later made a c implementation that was a bit faster, owing to an algorithmic improvement...)
- kragen 3y agoi feel like maybe you told me about this particular routine but i failed to bookmark it, or maybe tag the bookmark appropriately; do you have a link?
- xoranth 3y agoMore prosaically, getting compilers to generate branchless code reliably is difficult, there's no intrinsic for CMOV* and similars, and the builtins that should act as an hint don't work[1]. [1]: https://c.godbolt.org/z/6EWzKe5zd https://c.godbolt.org/z/6EWzKe5zd
- moonchild 3y agoThe default probability of any branch, absent further information, should be 0.5, so I wouldn't expect this to do anything. You should rather direct your ire at llvm, which has __builtin_unpredictable which is explicitly supposed to do this but doesn't.
- xoranth 3y agoYeah, makes sense. That said, (G)GP asked what could be done in raw assembly but not with C + intrinsics. My point is that conditional moves are one of usecases badly supported by compilers, and that (may) require dropping to assembly.
- o11c 3y agoMost people who need assembly are going to want to only write assembly-embedded-in-C, and learn the GCC constraint specifiers. This of course assumes that there aren't sufficient intrinsics (these days, there are a lot of intrinsics exposed! [1]). Note that you can specify registers to be used to store a variable without writing any asm, if for some reason the register allocator is confused by what you're doing. Developing a compiler from scratch is the other significant use for writing it. Of course it is quite common to need to read it. [1]: https://gcc.gnu.org/onlinedocs/gcc/x86-Built-in-Functions.html https://gcc.gnu.org/onlinedocs/gcc/x86-Built-in-Functions.ht... Edit: also useful are https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html and https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables....
- x1sec 3y ago> Of course it is quite common to need to read it This is a notable differentiation - Writing assembly is a different skill to reading it from a disassembly. Reverse engineering, malware analysis etc. does not inherently require you to be able to write asm, although it certainly would help.
- JonChesterfield 3y agoasm goto is really useful in this context. It means you can pick the branch instructions and define the exact control flow graph you want. Intrinsics, normal C, inline asm defining basic blocks and asm goto defining the CFG lets you emit exactly the instructions you want, albeit with somewhat challenging syntax. The pinning local variables to registers thing didn't work in llvm a couple of years ago (which seems consistent with the gcc docs) but does work at the boundaries of inline asm and that's generally enough. I like a pin-register intrinsic, something like `u64 pin(u64, enum reg)` where the compile time constant enum names the register and the semantics are a no-op other than constraining the register allocator, but that doesn't seem to be readily available in gcc/clang. I don't have a good answer to constraining instruction scheduling. On reflection it's all somewhat more horrible than it needs to be, perhaps inline compiler IR is a better idea.
- rickoooooo 3y agoExploit payloads (shellcode)
- cinntaile 3y agoI have always wondered why they are written in assembly. Is it "just" to guarantee the exact shape, size and contents of the payload or are there other reasons?
- dazed_confused 3y agoSize is very important but in the case of memory copying errors, removing null bytes is key to prevent the copy from terminating early. Additionally, you may be creating/modifying a new stack in some cases.
- e12e 3y agoI always like to point at heavy thing - but not sure if it inactive? https://2ton.com.au/HeavyThing/ https://2ton.com.au/HeavyThing/
- anonacct37 3y agocryptography. Things that really need to be constant time and not be "optimized" by the compiler. See the .s files in: https://cs.opensource.google/go/go/+/refs/tags/go1.21.5:src/crypto/aes/ https://cs.opensource.google/go/go/+/refs/tags/go1.21.5:src/... I occasionally see it in compression as well.
- Manuel_D 3y agoOperating systems and hardware drivers are big ones. Compilers do a lot of silent stuff behind the scene that can interfere when bare metal is involved. Example: https://github.com/dddrrreee/cs140e-23win/blob/85b9ae3bd46c7e2edaf69b2b3c3afb741e8f4fa2/notes/observability/BUG.md https://github.com/dddrrreee/cs140e-23win/blob/85b9ae3bd46c7...
- 8372049 3y agoNot exactly what you're asking, but reading and manipulating asm is critical in several parts of infosec: reversing, vuln analysis, sploiting and manipulating, etc. The same goes for somewhat related fields suchs as game hacks, cracking and so on. To some extent this also includes writing raw asm.
- latenightcoding 3y agoI don’t write it but I read it almost every day while working on high performance C/C++ code. Reading the generated assembly almost always gives me hints on how to optimize the code better
- corysama 3y agoGodbolt FTW.
- CoastalCoder 3y agoI used it to write a bare-bones JIT compiler for deep-learning kernels. I would have preferred to emit something like LLVM IR instead, but couldn't because of several constraints.
- kragen 3y agoas you know, a lot of day-to-day use of assembly nowadays is writing compilers and debugging compiled programs, though i also got surprisingly good performance in httpdito from a fork-per-client web server http://canonical.org/~kragen/sw/dev3/httpdito-readme http://canonical.org/~kragen/sw/dev3/httpdito-readme dan bernstein makes the argument that, as computers get faster, we use them on bigger problems, which means that computer performance is increasingly dominated by small inner loops, which is precisely the situation where it becomes more rational to put effort into hand-optimizing your small inner loops than to hack on the compiler to hopefully speed up all parts of the program, just as it was in the 01960s for different reasons a different way to attack that problem in many cases is to write a domain-specific compiler from a domain-specific language to machine code, as thompson's regexp engine did, and as verilog compilers do. but i'm not sure how you speed up a media codec that way bernstein has also written a fair bit of assembly to eliminate timing side-channel leaks from cryptographic code
- ack_complete 3y agoGeneral data compression/decompression can also benefit from asm level tuning, including generic compression such as Huffman/zlib and more specific compression like animation compression. MSVC is also a lot worse at vectorization than GCC or Clang and can more easily be beaten with asm. I would agree that direct asm is very rare these days in game engines outside of third-party libraries. There can be significant gains with tuned asm but some combination of intrinsics and ISPC is usually good enough. But it is far more useful to be able to _read_ assembly, for debugging in an optimized build or analyzing release crashes.
- fragmede 3y agoAnother area that demands performance and can pay for it is databases. Proprietary databases have hand written assembly tuned for the exact processor its running on in its hot paths.
- pjmlp 3y agoBeing able to understand the machine code generated by AOT and JIT compilers, and how it maps to the higher level code.
- _xerces_ 3y agoReverse engineering.
- matheusmoreira 3y agoFreestanding nolibc Linux applications. They don't link in the so called "startfiles" so there's nothing there that will bring your program from its ELF entry point to your actual main function. I wrote assembly code to collect process parameters like arguments, environment and auxiliary vector and pass them to a C function of my choice. Conservative garbage collectors. Scanning the native stack for pointers can be done in C but isn't quite enough since there might be pointers in registers. So I wrote assembly code to spill all the registers onto the stack prior to scanning.
- __turbobrew__ 3y agoWriting raw assembly, maybe not that much. Reading raw assembly on the other hand is very useful.
- _8j50 3y agoShellcode.
- ThinkBeat 3y agoThank you for making this available. I am reading it tomorrow. (well starting it). Been looking for something like this for a while
- grandiego 3y agoSome months ago I was trying to learn a bit on this subject (using Debian); from my notes I made a sort of beginner's "self-tutorial"[1]. [1]: https://americati.com/dat/assembler.html https://americati.com/dat/assembler.html
- yla92 3y agoThis is great. Thanks for sharing!
- serialNumber 3y agoThis is incredible! Putting on my holiday to-do list :)
- kragen 3y ago367-page cc-by-nc-sa textbook; though the license permits modification, no source format seems to be provided, just a pdf. it covers mostly introductory userland programming, though there's a chapter on interrupt handling. on skimming the table of contents i don't see anything about page tables, tlbs, atomics, the amd64 memory consistency model, running before sdram is enabled, iommus, mtrrs, or simd. none of those things are necessary to write a compiler or debug most compiled code, except maybe simd this book also looks like it could be a very solid base for a course that did cover one or more of those additional topics; the course would also include some supplementary material it probably teaches more than everything i know about amd64 assembly, except the most important thing, which is that arm assembly is much better
- snvzz 3y agoOr RISC-V for that matter. (while being an order of magnitude simpler)
- kragen 3y agorisc-v is nicer than amd64, but in between ldm/stm, shifted indexing, postincrement/preincrement, and conditional instructions, arm assembly is just about c shame about the t bit tho
- sylware 3y agoYep, RISC-V is supposed to become an heaven for assembly written programs as it is an ISA standard, namely it will "work" on CPUs from many vendors. The main pitfall while writing assembly is the abuse of a preprocessor. Being dependent on the grotesquely and absurdely complex and massive compilers out there is one thing, but moving that dependency to a complex preprocessor is not that much better. So caution and care about that issue must be kept in mind. I wish all RISC-V CPU vendors to pay the silicium real estate price for 64bits because writting once a 64bits RISC-V code and to be able to "run-ish" it everywhere, from "embeded" to servers passing thru workstation, wow. I am writting currently x86_64 assembly, namely the manual "register-ization" of some code paths is done, ready for an easy RISC-V port. Not to mention RISC-V has double the amount of registers, and some code paths will really benefits from this additional register space (intel plans to follow that route). Ofc, this will be mostly micro-arch agnostic assembly code, as it is the case already for x86_64, with maybe simple or very generic static "optimizations" (cache line, alignment, registerization, leveraging some instruction fusions, etc). Worst case scenario, adapted (not rewritten from scratch) assembly written code paths to fit better on a micro-arch with a runtime switch/installation... if really needed. I guess correct code will become very important, maybe more than fast code ("should" become true for hardware design too, if the performance penalty is not too high). RISC-V will need compiler support though, that for legacy support, and sometimes, humanization of the assembly output of some compiled programs may be usefull. All that depends on the success or not of RISC-V, which for that will need ultra-performant implementations all over the board, micro-archs and best silicium process. RISC-V is not perfect, but a more than good enough modern ISA, and the real risk of fragmentation is 32bits/64bits code paths even though some care was provided to make 32bits<->64bits code adapation easy. Mistakes will be made (micro-arch with critical bugs), so it won't happen overnight.
- JayDustheadz 3y agoHopefully not offtopic: if I'd like to learn/use asm on Apple's ARM Macs, what would be a good start? In very short, I'd be interested in, at some point, improving the performance of the code for the apps that my company creates.
- LeFantome 3y agoThey are just ARM64. Start there. https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms https://developer.apple.com/documentation/xcode/writing-arm6...
- snakey 3y agoSlightly off topic but does anyone have any recommendations for a similar style of book that focuses on ARM and/or RISC assembly programming?
- cinntaile 3y agoThere is this guide [0], the same person has also written a book. But I don't think it's a similar style? [0] https://azeria-labs.com/writing-arm-assembly-part-1/ https://azeria-labs.com/writing-arm-assembly-part-1/
- NelsonMinar 3y agoI liked Stephen Smith's "Programming with 64-Bit ARM Assembly Language". https://www.goodreads.com/book/show/53671067-programming-with-64-bit-arm-assembly-language https://www.goodreads.com/book/show/53671067-programming-wit...
- pjmlp 3y agoThankfully using Intel's syntax.
- vanderZwan 3y agoCan you elaborate on what you mean by that, for those of us with little prior exposure to x86 assembly? (my own ASM experience is limited to using TASM to write some games in Z80 on my TI-83 graphing calculator)
- pjmlp 3y agox86 is one of the few architectures that has more than one official syntax. The Intel syntax, common in the PC world since the MS-DOS days, used across Windows and OS/2 as well. In macro Assemblers, inline Assembly in high level languages, and naturally Intel and other x86 manufacturers CPU manuals. It uses the format, op dest, source Then you have the AT&T syntax, for whatever reason when support was added to UNIX for x86, they chose the format used by other architectures. It is only found on GNU/Linux and BSDs, and follows op source, dest Where op is written with the data size prefix, and addressing modes are somehow more complex. Intel: lea eax, [eax + eax * 4] AT&T: lea (%eax, %eax, 4), %eax
- vanderZwan 3y agoThank you for explaining. Since this book targets Ubuntu I'm assuming Linux supports Intel syntax now, so I guess that this also means that it will be a more "portable skill" across different systems.
- pjmlp 3y agoIt has supported Intel syntax for decades now, and yasm/nasm also exist on Linux and BSDs, doesn't change the fact that it is still quite common. Android is probably the only Linux based system where Intel's syntax is favoured via yasm's inclusion on the NDK.
- fredoralive 3y ago