18 ms·
x86-64 Assembly Language Programming with Ubuntu
- Areading314 8y agoOut of curiosity, what are the main reasons to need to actually write assembler in 2018? Compilers? Games? Genuinely curious
- cheez 8y agoI work on compilers and I hate assembly.
- massysett 8y agoPerhaps wanting to know how the computer works.
- AdmiralAsshat 8y agoEmbedded devices?
- LeonM 8y agoI have never had the need to write assembly on any embedded device. There are C (or other high language) compilers for just about any architecture. For very specific instructions (i.e. interrupt jumps) there is usually available as a compiler alias/feature.
- namibj 8y agoIf you want to turn an AVR/AtMega32 into a spectrum analyzer, you need to massage the x, y and z registers that support implicit incement/decrement adressing. C fails hard for such code.
- currymj 8y agoshellcode, maybe, or really tight inner loops that need to use weird vector instructions.
- namibj 8y agoIt's not the vector instructions, it's the careful scheduling of instructions to spend just enough time manipulating pointers when you want to crunch actual data. All while respecting dependency chains and memory stall times. (Hyperthreading helps a lot with the latter, see Nvidia Maxas (nervana systems now) for details on how flexible number of threads benefit weighing of memory load stall hiding vs. register pressure causing more data shuffling.
- bigato 8y agobetter performance and millions of dollars of monthly savings in cost (in a bank) - not x86 though
- jcranmer 8y agoLow-level system code (particularly when you're doing things like servicing interrupts) needs to be written in assembler. The very early boot phase is even more annoying because you're starting out in one of the 16-bit modes and working your way to changing to x86-64 and virtual memory. Outside of system code, the main use for assembly is trying to maximize performance on hot loops. Your optimized matrix-multiply routine, or media decoding kernel, could well be written in assembly. I've seen a few cases where people do things such as manual calling conventions in assembly as well.
- aaaaaaaaaab 8y ago>you're starting out in one of the 16-bit modes and working your way to changing to x86-64 and virtual memory "ontogeny recapitulates phylogeny" https://en.wikipedia.org/wiki/Recapitulation_theory https://en.wikipedia.org/wiki/Recapitulation_theory
- tmpmov 8y agoThe book reasons that learning assembly also teaches the fundamentals of computer architecture. 1.3.1 Gain a Better Understanding of Architecture Issues 1.3.1 Understanding the Tool Chain 1.3.1 Improve Algorithm Development Skills 1.3.1 Improves Understanding of Functions/Procedures 1.3.1 Gain an Understanding of I/O Buffering 1.3.1 Understand Compiler Scope 1.3.1 Introduction to Multi-processing Concepts 1.3.1 Introduction to Interrupt Processing Concepts The book uses the 1.3.1 heading for each and I'm too lazy to change them. Reasons to (maybe/arguably) write assembly: 1) You're bringing up a new board, your bootloader is partially written but you need some customization for the real-time OS you're using. It can be advantageous to do this in assembly 2) You're dealing with some particularly old hardware and (ab)using it for some commercial purpose Of course, I can imagine that for each you'll have someone obstinately state there's no need to use assembly because of some gcc feature. More than one way to get things done, and most use the tools they're comfortable with.
- deleted 8y ago[deleted]
- s3m4j 8y agoNot necessarily "write" assembler, but understand it, and, as always we are not at the end of history. New dominant CPU architecture could arise, people understanding and writing assembler will always be needed.
- fulafel 8y agoTasks involve reading assembly are more common, and it's hard to understand it very well if you've never written any.
- mbel 8y agoIf you are developing a compiler you will need to understand the output assembly. Learning to program in it is one way to get to know it. There is also plenty of little code snippets around run-time libraries and OS that are written directly in assembly (e.g. crt0.o, crt1.o & friends). Basically: somebody has to provide the infrastructure between assembly and high level languages, so that everyone else can write in HLL. Optimization of games or HPC workloads is also a valid use-case, but today you probably want to use intrinsics in C instead.
- namibj 8y agoNot necessarily regarding the latter. Intrinsics only help with the part where your code doesn't fit the instruction set because the latter can encode operations the former can't natively, and the compiler might not always be able to detect such structures.Als Also, if you use assembler you can take better care about scheduling instructions around loops in a way to get very high throughput, because you might see that k-ary search happens to be faster than binary search because you can hide some of the latency you'd otherwise have. Also some other issues requiring higher-level restructuring once you see that your first implementation stalls the processor too hard. Once you reach GPU cores and other non-OoO cores you need to be careful with memory load stalls, and try to use some staging memory or so and e.g. load the data for the iteration after the next iteration into staging, then fetch it into registers in the next iteration, so it is ready in the one where it will be processed. You need such optimization to reach e.g. high throughput in matrix multiplication, because you need to minimize the instructions that use the ALU but don't do actual result-affecting calculations (and instead just e.g. loop/array index manipulation and such). The ability to be good with this is reduced in OoO machines with register renaming, sadly, and I'd like a core that is fast for sequential code due to low-delay high-clock operation, yet exposes the whole register set to the assembly programmer... One of the recent chip startups seems to target such a processor. Obviously also the Mill, but that's vaporware since forever...
- pjmlp 8y agoSome mainframe and workstation architectures never had direct support for Assembly instructions, only intrinsics on their system languages. Burroughs, Xerox PARC workstations, ETHZ workstations as a couple of examples.
- qubax 8y agoTo learn what programming really is and how systems really work. You can't really understand programming without having built a compiler. Or at the very least having done assembly programming. I don't think you can truly understand any basic programming concept like variable, pointer, reference, etc without learning compilers or assembly. Yes, a pointer "points" to something. But what does that really mean? When you realize that it's all just higher level concepts of specific assembly mechanisms of accessing data, you'll have a eureka moment. Don't see how you do that without digging into compilers/assembly. And ultimately, it'll make you a better overall programmer since compilers/assembly is fundamental to every programming language.
- wilsonnb3 8y ago> You can't really understand programming without having built a compiler. Or at the very least having done assembly programming. I don't think you can truly understand any basic programming concept like variable, pointer, reference, etc without learning compilers or assembly. This is just elitism with a little bit of "back in my day" thrown in. I can assure you that I fully understood the concepts of pointers, variables, and references without knowing anything about assembly language or compilers. The concepts are not that difficult to grasp.
- ejanus 8y agoHmm , this is great. But to understand where variables are placed in the stack and how stack is managed your need to play with Assembly. Pointers are pointers but when you play with using Assembly then your know why some languages exposes only reference and lack pointer arithmetic. It sounds easy but it help you to build compilers or whatever.
- saagarjha 8y agoPointers deliberately hide these details from you by obscuring where things are actually located; presenting a unified memory model. To really understand what's going on you need to know how the stack is laid out, how structures work, and how functions are called.
- 8y ago
- veganjay 8y agoFor me, because it is interesting. It is neat to understand how things work at a lower-level. Knowing assembly also helps out when doing capture the flag challenges, specifically reverse engineering and binary exploitation.
- RandomOpinion 8y agoCompilers, embedded systems and similar low level work, reverse engineering where you don't have the binary (e.g. malware), and that kind of thing.
- TheAceOfHearts 8y agoFirmware and driver development. Not in all cases, but you're very likely to encounter it in those domains.
- jahaja 8y agoMaybe not that much to write it, but it's very valuable to understand the disassembly when debugging.
- LeonM 8y agoTake a look at projects such as pixman [0] But other than highly opimized graphics manipulation I can't really think of any other good application for assembly. [0] http://www.pixman.org/ http://www.pixman.org/
- Arkanosis 8y agoTwo use cases I've had in recent years (technically not 2018, but well…): - optimization of simple functions that are called a huge amount of times (the only viable alternative to assembly being intrinsics that come with their own portability constraints), - just in time compilation. Additionally, knowing assembly can help a lot when debugging weird crashes in code written in higher-level languages. gdb's asm layout is an awesome resource if you know how to use it, but someone who's never used assembly before will probably not even consider using it.
- wilsonnb3 8y agoAssembler is still popular for IBM mainframes. The current version has been around since '92 and is called High Level Assembler. It's popular partially because people have codebases that they started writing in the 70's or 80's in assembler that they maintain to this day because it's cheaper than switching it all over to a new language. Pretty much the same reason that COBOL is still around. z/OS (the OS that runs on IBM mainframes) also exposes a lot of it's functionality through HLASM, so it's far more convenient to use than x86 assembly. For whatever reason, C also never really caught on as ubiquitously as it did in the PC world. Probably because IBM themselves generally used their proprietary PL/S language instead back in the 70's and 80's. https://en.wikipedia.org/wiki/IBM_High_Level_Assembler https://en.wikipedia.org/wiki/IBM_High_Level_Assembler
- pjmlp 8y agoUNIX was to C what the browser is to JavaScript. Naturally it didn't caught on mainframes, already using better system languages. It is also barely used on Unisys ClearPath MCP.
- __sdegutis 8y agoThis is fascinating, I didn't know there was a such thing as a high level assembly language, but IBM High Level Assembler has IF/ELSE/ENDIF, and several types of built in loops. I wonder how similar it is to writing in C. One thing this page doesn't mention is structured data types, I suppose these would still have to be implicit like in other assembly languages.
- wilsonnb3 8y agoThe IF/ELSE stuff is similar to the preprocessor macros people write in C. They basically generate HLASM code on the fly based on certain flags being passed to the program and what not. If you're curious what a simple program ends up looking like, I've got one I wrote that copies the contents of one file into another file up on GitLab. Lots of loading registers and what not. https://gitlab.com/thisisnickwilson/zos/blob/master/IO https://gitlab.com/thisisnickwilson/zos/blob/master/IO
- freecodyx 8y agoreverse-engineering ?
- crunchlibrarian 8y agoReversing the NSA's latest bad idea to figure out how to stop it from taking over your network.
- tyingq 8y agoLots of reasons, but one would be processor specific optimizations and/or extensions for performance reasons. OpenSSL, for example, has a fair amount of ASM in it for various processors to speed things up. See https://software.intel.com/en-us/articles/improving-openssl-performance https://software.intel.com/en-us/articles/improving-openssl-... for some examples of what they specifically did for x86/64.
- beefhash 8y ago1. When you have no choice in languages: Homebrew development for, say, the Nintendo Game Boy happens by and large in assembly language. 2. Optimization: Some critial code sections or cryptographic operations can be massively sped up with hand-crafted assembly[1]. 3. Obfuscation: Sometimes, you have to drop to assembly for obfuscation. This is especially the case if you need to/want to fool the Hex-Rays Decompiler. Applications include malware and digital rights management (some might argue the latter is just a sub-species of malware). This also includes writing your own custom assembly language for a custom virtual ISA for the purpose of code obfuscation. 4. Embedded platforms: Some very obscure microcontrollers may only have support for pure assembly, though at least C89 compilers being available seems to be the norm. 5. Education: It can be very enlightening to understand how things work below your "home" layer of abstraction. Some parts of C seem to be much easier to understand once you have a firm grasp on a common assembly language. 6. Compilers: Self-evident, you kind of have to understand the code you emit. [1] See e.g. this benchmark https://monocypher.org/speed https://monocypher.org/speed -- authenticated encryption on libsodium gets an improvement of over 300% from hand-crafted assembly over the portable C implementation
- saagarjha 8y agoAnother common use case is for writing shellcode and other exploit-related work.
- geertj 8y agoCryptography, not only for speed, but also for ensuring implementations are constant time.
- henryfjordan 8y agoI've seen it used a lot in I/O from micro-controllers where speed/timing is important. The arduino C++ library for instance does some weird things under the hood when reading from an Analog-Digital-Converter (read the voltage at some pin), whereas with assembly you know that you can read in exactly 1 instruction. When timing or speed are important, that level of control is useful.
- e12e 8y agoOne practical "high level" example would be analyzing function/type specialization in Julia and understanding the resulting code that can be easily inspected. See eg (from 2013, but at a glance seem to outline the ideas well): http://blog.leahhanson.us/post/julia/julia-introspects.html http://blog.leahhanson.us/post/julia/julia-introspects.html For Julia - it's fairly easy to work "high up" most of the time - and drop down inspecting the code (and profiling) - and unlike many other languages, even high pref libraries will often be julia all the way down (unlike tcl/perl/ruby/python + c/fortran etc). https://docs.julialang.org/en/stable/manual/profile/ https://docs.julialang.org/en/stable/manual/profile/ https://docs.julialang.org/en/stable/devdocs/reflection/ https://docs.julialang.org/en/stable/devdocs/reflection/ Similar for sbcl (common lisp) or various languages that target/use llvm. And obviously for looking at output of optimizing compilers for c, c++, rust, d and similar "medium to high" level languages (Pascal, Ada, crystal, etc).
- msravi 8y agoKnowing assembly made me a better C programmer. I still remember this tech interview I had with a Seattle based company (no, not MSFT) many years ago. The interviewer was trying to twist and turn all kinds of C-pointer questions, but knowing how the code translated to assembly made the whole thing a breeze. I got the job but didn't take it, and I somewhat regret that.
- new_age_garbage 8y agoI was skeptical but this looks like an amazing resource!
- deleted 8y ago[deleted]
- aruggirello 8y ago> It should be noted that Unicode uses 2 bytes for each character. But you're programming "with Ubuntu", not Windows. IMHO you could safely assume/recommend UTF-8.
- Arkanosis 8y agoJust a reminder BTW that since version 2.0 (1996), Unicode is not an encoding scheme but a character set (I avoid the confusing “charset” word on purpose). Therefore, Unicode does not use any number of bytes: it only assigns code points to characters. Windows used to use the UCS-2 encoding scheme which indeed used 2 bytes for each character, but since Windows 2000, it uses UTF-16 instead, which like UTF-8 uses a variable number of bytes per character.
- _m96l 8y agoIndeed. "Unicode" is an abstract character set, it doesn't "use" any bytes. A specific encoding does.
- jhomedall 8y agoEven with UTF-16 that quote is incorrect, due to surrogate pairs. It's only correct for UCS-2, and even then, only if you take 'characters' to mean 'codepoints', and take 'Unicode' to mean 'a specific Unicode encoding'.
- dsamarin 8y agoI am currently taking this class, and am happily surprised this made it here. The book needs more work, but I still believe it's an great resource. For example, on page 11 it says "Note that when the lower 32-bit eax portion of the 64-bit rax register is set, the upper 32-bits are unaffected." In reality, the high order bits are zeroed to avoid a data dependency. I'm going through the entire book for typo hunting :-) Also I found some issues while discussing Unicode, but the class only requires use of the ASCII character set.
- jcranmer 8y ago> For example, on page 11 it says "Note that when the lower 32-bit eax portion of the 64-bit rax register is set, the upper 32-bits are unaffected." In reality, the high order bits are zeroed to avoid a data dependency. Well, there is one case where the upper 32-bits are not zeroed. It turns out that xor eax, eax is assigned to opcode 0x90, which is better known to most people as NOP. If you want real fun, read up on what happens with AVX registers. Whether or not you leave untouched or zero the upper bits are dependent on if you use VEX encoding or not.
- eboyjr 8y agoNo, in section 3.4.1.1 General-Purpose Registers in 64-Bit Mode of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, it says, "32-bit operands generate a 32-bit result, zero-extended to a 64-bit result in the destination general-purpose register." `xor eax, eax` actually generates 0x31 0xc0, and `xor rax, rax` generates 0x48 0x31 0xc0. 0x90 decodes to xchg eax, eax in all modes except long mode, which has no effect. In long mode, the opcode 0x90 has no effect still but is no longer equal to xchg eax, eax.
- CalChris 8y agoIndeed xchg eax,eax is a nop idiom; there are many. Recent microarchitectures simply ignore nops without executing anything. From the Intel® 64 and IA-32 Architectures Optimization Reference Manual: 16.2.2.6 NOP Idioms NOP instruction is often used for padding or alignment purposes. The Goldmont and later microarchitecture has hardware support for NOP handling by marking the NOP as completed without allocating it into the reservation station. This saves execution resources and bandwidth. Retirement resource is still needed for the eliminated NOP.
- jmsmistral 8y agoThanks for sharing!
- lepasana 8y agoThanks for this!
- emily-c 8y agoThese types of resources are awesome though I've always wished that they at least briefly touched upon the x86 memory model/consistency. Understanding the concept of memory barriers should be considered fundamental.
- pjmlp 8y agoThankfully it uses Intel syntax.
- simias 8y agoIf you have a particular need for learning x86 assembly this is great, however I want to point out that if you just want to learn an assembly for the sake of learning very low level development and understanding how CPUs work at the lowest software level I would not advise picking x86. It's a crufty, messy, overcomplicated, plagued with decades of shifting paradigms in CPU ISA design and still maintained for backward-compatibility's sake. If you value your time and sanity consider learning something smaller and more reasonable such as AVR assembly (the kind of controllers you find on Arduinos). It's a lot smaller and you don't even need an OS, you can truly do everything from scratch. If you want something a little more advanced ARM is an obvious target, it's got all the features you'd expect from a modern CPU (SIMD, floating point etc...) and it's not nearly as crazy as x86 assembly.
- dataflow 8y agoLearning doesn't happen in a vacuum though. People generally try to learn things that will be handy for them. AVR or ARM assembly are far less handy to know than x86, so telling people to ditch x86 and learn those instead kind of misses the point.
- makapuf 8y agoSure x86 is more ubiquitous (or is it really? Many arm cores and embedded AVR are released embedded) but the number of times I've needed asm in x86 is way less than I've had with those embedded platforms where a byte is a byte ... (ditto for cycles) edit:typo
- dataflow 8y ago> Sure x86 is more ubiquitous (or is it really? Many arm cores and embedded AVR are released embedded) I can't make sense of this. Is your logic "there are more ARM CPUs than x86 CPUs => there are more programmers dealing with ARM assembly than with x86 assembly"? > but the number of times I've needed asm in x86 is way less than I've had with those embedded platforms Sure, this is your situation. But are you claiming your situation is typical? In your mind do the majority of programmers who deal with assembly deal with embedded platforms as much as you do?
- markhahn 8y agoWhy Ubuntu? IE why is any of it distro-specific?
- Keyframe 8y agosyscalls for OS maybe, but why Ubuntu... who knows
- harry8 8y agoLikely because it's the most popular among target group of students at time of writing. Choosing a distribution allows one to discuss complier, assembler, gdb, text editor with known versions and known availability reducing admin overhead so students can get on with it. It's not a terrible way to go whatever one thinks of the most popular distribution at any given point in time.
- wybiral 8y agoI recently did a video series on x86 using nasm and GCC. It only covers 32-bit but I think that's a better way to start since of the conventions are simpler (especially when interfacing with C code). https://www.youtube.com/playlist?list=PLmxT2pVYo5LB5EzTPZGfFN0c2GDiSXgQe https://www.youtube.com/playlist?list=PLmxT2pVYo5LB5EzTPZGfF...
- agumonkey 8y agoThe TOC is very tempting. Thanks for the link
- fredsanford 8y agoFor those interested... Up until the early 2000's ish, Randall Hyde used to develop a ton of teaching materials and libraries for intel. You can find most of if here: http://www.plantation-productions.com/Webster/index.html http://www.plantation-productions.com/Webster/index.html