8 ms·
Don't pass structs bigger than 16 bytes on AMD64
- pclmulqdq 3y agoThe cost of argument passing is rarely well-understood, and I'm glad someone wrote this. People routinely pass 24-byte objects by value in places like Google, and the cost of that practice doesn't show up on a profiler because it's spread out on every function.
- elromulous 3y ago"in places like Google". Are you speaking from experience? As a former googler, I can say with certainty that the guidelines for passing any non primitive is pointer or ref. string_view might be the only exception I can think of.
- e____g 3y agoI saw std::function and std::string (e.g. TotW 127, https://abseil.io/tips/117 https://abseil.io/tips/117) being passed by value a lot in newer google3 code. Both are larger than 16 bytes.
- jeffbee 3y agoGoogle builds non-PIE, non-PIC, static, profile-guided, link-time-optimized, and post-link-optimized binaries and probably DGAF about calling conventions.
- pclmulqdq 3y agoI have seen the assembly output of Google code, and I will say that my previous comment still stands.
- vitus 3y agoPassing std::function by value is almost definitely wrong these days with absl::AnyInvocable (if you need to store the type) and absl::FunctionRef (if you don't). Rough analogues in the standard are std::move_only_function (C++23) and std::function_ref (C++26). std::string in the case you cited is only really relevant if you std::move() into it and you would otherwise incur a string copy. Yes, it's bigger than 16 bytes (24 bytes), but that pales in comparison to the alternative. (Taking std::string&& would eliminate the possibility of misuse / accidental copies, but that pattern is generally discouraged by Google's style guide for various reasons.) Also, just because you see a certain pattern an awful lot even at Google doesn't mean that it's best practice -- there are plenty of instances of protobufs being passed by value...
- eco 3y agoThis is done so you can use std::move to take ownership of the allocated memory in these objects rather than do a new allocation. Passing by value rather than rvalue reference let's your function be more flexible at the call site. You can pass an rvalue/move or just make a copy at the call site which means the caller (who actually knows if a copy or a move is more appropriate) gets to control how the memory gets allocated. An unnecessary memory allocation is much more of a performance hit than suboptimal calling convention.
- jstimpfle 3y agoIn that case, the optimal interface should take std::string&& no? But it's awkward.
- andersa 3y agoWouldn't this be very annoying to work with, because now you have to explicitly move or copy the string whenever you want to construct one of these objects?
- nitnelave 3y agoIt's kinda what Rust forces you to do, except that std::move is implied. Anything taken by value is equivalent to taking by && unless the type is explicitly marked as Copy (i.e. it can be trivially copied and the copies are implicit). But yeah, in a c++ codebase, good modern practices are often verbose and clunky.
- ismailmaj 3y agoThe function could accept an universal reference instead of an rvalue reference, this avoids the dance the caller has to do to pass a copy. IMO it's hard to beat pass by value considering both performance and cognitive load.
- mort96 3y agoYeah, making it accept a universal reference would fix it... ...but that requires the argument to be a type from a template D: so you'd have to write: template<typename String = std::string> void dosomething(String &&str) ...and that's not quite right either, since you'd want it to be either an rvalue reference or a const lvalue reference
- leni536 3y agonot trivially copyable types are never passed in registers regardless of size
- jtasdlfj234 3y agoPlease prove me wrong on these points. My current belief and understanding is that: 1. `foo(T)` indicates polymorphic over both `T::T(const T&)` and `T::T(T&&)`. This gives you benefits of both pass-by-move (using `std::move` as needed), or copy. 2. Usage of `foo(T&&)` signals code-smell or an anti-pattern as `foo(T)` should be used instead unless it is perfecting forwarding / universal reference `template <typename T> foo(T&&)`.
- pclmulqdq 3y agoYes. A lot of 24-byte structs at Google are passed by value.
- japanman185 3y agoIndeed they are. I read about it from hackernews.
- ductsurprise 3y ago0o...
- pclmulqdq 3y agoI learned about it when I was at Google working on performance.
- dataflow 3y ago> string_view might be the only exception I can think of. What about std::span? std::optional? etc.
- nemetroid 3y agostd::span should apply, but std::optional is at least as large as the type it wraps.
- fyrn_ 3y agoAlmost always 8 + the size of the type. Since it's a one byte book followed by 7 padding bytes, followed by the type itself.
- vitus 3y agoView types like span, string_view, FunctionRef: absolutely pass by value. They're designed to be cheap to copy, and they provide type erasure which can make your API more flexible. You also usually want to pass smart pointers (unique_ptr, shared_ptr) by value to avoid breaking your ownership model, or pass by raw pointer or reference to the held data. The obvious case that comes to mind where you'd want to write const unique_ptr<T>& is when iterating over a vector<unique_ptr<T>>. Otherwise, pass ptr.get() for an optional param, or *ptr for a required param where you've already checked ptr != nullptr. Wrapper types like optional, variant, StatusOr? Usually pass by reference, unless the pointed-to type is small (<= 8 bytes). One common pattern where I see wide structs being passed by value is the use of option structs (https://abseil.io/tips/173 https://abseil.io/tips/173) that are usually constructed via designated initializers. There is some marginal benefit to being able to std::move() out individual fields but it's not really a big deal either way, as said computation is usually only done once on program initialization.
- deleted 3y ago[deleted]
- medler 3y agoPassing larger objects by value is allowed or even encouraged if it lets you avoid a copy (e.g. if you’re about to move the object)
- loeg 3y agoDoes the C++ ABI also spill 24-byte objects to the stack for each call? I guess I don't expect std::string or std::function parameters to be fast but it's still surprising.
- pclmulqdq 3y agoSpilling big objects up the call stack is often not nearly as bad. Most of the time this happens, the object being spilled is constructed during the function, so the copy on return is actually elided.
- bugbuddy 3y agoThat’s only true if the function takes no argument. Otherwise the copy on return is unavoidable.
- pclmulqdq 3y agoThat's not actually true: https://en.cppreference.com/w/cpp/language/copy_elision https://en.cppreference.com/w/cpp/language/copy_elision
- gpderetta 3y agoI think parent is confused with NRVO typically not being triggered when returning a function argument.
- kukkamario 3y agoIn AMD64 ABI, anything larger (that doesn't fit in register) will be returned by basically output pointer argument. Caller reserves space for the return value and gives pointer to the function which then fills it with the return value. Both sides are easy to optimize and no unnecessary copies are made. Function can directly construct return value to the given address and caller can directly give a pointer to a variable if function return value is used to initialise variable.
- Cloudef 3y agoPass by value / pass by ref is quite a bit of mental overhead as it effectively affects your ABI/API. Zig tries to not force this so as long as you "pass by value", the compiler can actually decide to pass it by reference. It does expose this kind of footgun though https://github.com/ziglang/zig/issues/5973#issuecomment-1330743975 https://github.com/ziglang/zig/issues/5973#issuecomment-1330...
- gnulinux 3y agoOof that's a very nasty bug. Is this still relevant in Zig or is there a workaround in the language? I'm not familiar with Zig, heard some good things about it, but this looks like a showstopper.
- slimsag 3y agostill relevant today, but as someone who writes a lot of Zig code I haven't ever really encountered it in the wild. You definitely could, though, and it'd be wildly confusing. Luckily, the Zig core team has recognized it is an issue and plan to address it before 1.0 :)
- Cloudef 3y agoIt's rare to hit it, but if you do, having it happen silently is not ideal for sure.
- ForkMeOnTinder 3y agoI still think noalias-by-default is the way to fix this. https://github.com/ziglang/zig/issues/1108 https://github.com/ziglang/zig/issues/1108 You get all the benefits of Zig being able to choose the function ABI, but if the optimization would have caused a bug, you'll get an immediate panic at the function entrypoint, instead of silently corrupted data.
- MatthiasPortzel 3y agoIt’s a bit confusing when compared with other programming languages. But the Zig docs’ introduction to structs are pretty clear that they can be pass-by-value or pass-by-reference at the compiler’s discretion. If you need a copy, make a copy; if you need a reference, pass a pointer; if you don’t care, the compiler will pick one.
- PaulHoule 3y ago+1 for pointing out that profiling often fails entirely to find widely distributed overhead, like that built into calling conventions.
- kccqzy 3y agoThe tradeoff here is that if you pass a pointer to that 24-byte object instead, when you actually need to use the object you need to deref that pointer. But nothing guarantees that the pointed object is nearby! You could very well cause a cache miss and wait a long time (like 100 nanoseconds) to fetch that 24-byte object from main memory. If that same object is directly passed it's just on the stack. So it's most likely in the cache.
- Remnant44 3y agoYes, but this concern is orthogonal to calling convention. If the parent function does anything to the object, it will be in cache already. If it does not, but loads the fields to register, the parent will take the cache miss instead of the function. You neither gain or lose anything...
- kccqzy 3y agoIt's usually not about a single parent function and a called function. It's usually a long chain of calls where functions keep passing them around, from the point of construction. Imagine some function first creates that 24-byte object, then pass it around by pointer. Ten function calls later dereferencing that pointer will need access to main memory. Now imagine that first function passes the object by value; ten function calls later perhaps the contents of the object is duplicated on the stack ten times, but there's no cache miss. It's a tradeoff between reducing memory usage and reducing cache misses.
- gavinhoward 3y agoThis makes me super happy that my C array types are only 16 bytes.
- flohofwoe 3y agoThis will still spill on the stack when compiling with MSVC (the cutoff size is 8 bytes there).
- gavinhoward 3y agoOh, thank you for letting me know.
- saagarjha 3y agoWhere do you put the capacity?
- alcover 3y agostruct Array{void *ptr; uint64_t cap;} sizeof(struct Array) //16
- norir 3y agoI suspect the OP expected something like: struct Array{void *ptr; uint64_t len; uint64_t cap;} sizeof(struct Array) //24 At any rate, a simple way to get to 16 bytes is the following: struct Array{void *ptr; uint32_t len; uint32_t cap;} How often does one really need a general array with more than 4B elements?
- alcover 3y agoYou are right. My reply was a bit silly. Now thanks to alignment you could use spare bits in the pointer to afford bigger lengths. struct Array{int64_t ptr:60; uint64_t len:34, cap:34;}
- deleted 3y ago[deleted]
- countWSS 3y agoIts common sense: anything passed in registers(preloaded due spec execution) will outperform stack writes and stack wrangling will outperform heap allocated stuff. That why ugly spaghetti code with tons of globals runs blazingly fast and your elegant recursive functions and tuple/struct/lists arguments are incredibly slow, the first one is much easier to optimize into tight assembly loops.
- japanman185 3y agoBut what about my academic mental exercise. You mean I learned all these tools and techniques for nothing? How will my manager know if I am a top engineer or not if my code is not impenetrable with ridiculous concepts applied irrationally.
- Culonavirus 3y agoHalf of the money in IT right now is flowing from investors hearing "blahblah AI blahblah" and seeing dollar bills, which in turn makes half of the movement in IT happen in shitty python code. shrugs
- johnnyanmac 3y agoDifferent scale, different problem. Those "top engineers" at Google are dealing with memory that won't even fit in a hard drive (let alone memory), so stuff like this may as well be pico-optimizations. The logic makes sense for them. The problem is everyone else copying Google without understanding their own data. Any domain where real time performance is important need to put aside the SWE books and pick up a CPU/GPU architecture reference instead.
- eru 3y agoOf course, that assumes that your spaghetti code implements the same algorithm as your elegant code. If the elegant code is O(n) and the spaghetti code is O(n^2) you might notice the difference. And, of course, there's also maintenance to consider. In some sense, compiler are there exactly to turn our elegant solutions into spaghetti code.
- loeg 3y agoThe issue here is the SysV amd64 ABI. You could also just make your language-internal ABI not be SysV? As long as these aren't exposed to SysV C callers, you can use any calling convention you want. https://llvm.org/docs/LangRef.html#calling-conventions https://llvm.org/docs/LangRef.html#calling-conventions For those curious, the relevant diff in neatlang is: https://github.com/Neat-Lang/neat/commit/f4ba38cefc1e26631a50a57dad17fca076af4b57 https://github.com/Neat-Lang/neat/commit/f4ba38cefc1e26631a5.... It looks much more involved than changing the emitted LLVM calling conventions. Possibly the author wants these types exposed with some deterministic calling convention to C programs.
- userbinator 3y agoOr ABIs in general, really. As any Asm programmer can tell you, this is one of the low-hanging fruits that compilers can easily be beaten at --- don't blindly follow convention, do what makes the most sense in a specific scenario.
- 5- 3y agoindeed. another fun thing that c-like compilers generally don't do is multiple entry points in a function. consider a pair of functions foo and foo0 that differ only in that the former performs an additional action on its argument -- perhaps refcount adjustment or type conversion. you can then do (in a register-based abi like amd64 sysv); foo: do the action ;fallthrough foo0: rest of the function follows and have essentially two functions for the price of one.
- zekica 3y agoWon't tail call optimization mostly mitigate this?
- pertymcpert 3y agoYeah it should.
- deleted 3y ago
- cmovq 3y agoOn Windows with the default cdecl calling convention structs bigger than 8 bytes don’t get passed in registers [1]. [1]: https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170 https://learn.microsoft.com/en-us/cpp/build/x64-calling-conv...
- bugbuddy 3y agoIn the provided example, you can fix it without affecting any caller by changing the parameter types from “struct Vector” to “const struct Vector &” pass-by-reference. A lot C++ code that I have seen where pointer bugs existed needlessly used pointers even when passing by reference could have worked while being easier and safer to use.
- giovannibajo1 3y agoThat will still require the compiler to serialize the three registers to the stack, to be able to pass the pointer to the structure to the callee. It seems like the described benefit is avoiding any serialization from registers to stack, which cannot be avoided with pass-by-reference.
- bugbuddy 3y agoBut that would only be possible if you write out all the struct fields you are accessing into function parameters. If the struct is complex with a lot of fields, then you would end up with a messy function signature. Also, I am sure there is a limit to how many parameters you can have before this optimization stops working.
- loeg 3y ago> But that would only be possible if you write out all the struct fields you are accessing into function parameters. If the struct is complex with a lot of fields, then you would end up with a messy function signature. This is the approach the article's author took.
- fsckboy 3y ago>That will still require the compiler to serialize the three registers to the stack, to be able to pass the pointer to the structure to the callee. why can't it simply pass a pointer to the struct (it's probably already on the stack) without rewriting the struct to the stack? isn't that what a reference is?
- jeffbee 3y agoI classify this kind of article as "just enough knowledge to be a pain in the ass". Even if you compile it separately per the instructions, trying to force the compiler to generate ABI-callable functions, you can still undo this error with LTO. Building this program with LTO is dramatically faster in both modes compared to either mode of the non-LTO program. If your program is performance-sensitive, profile it, peak-optimize it, and only then commit jackassery such as unpacking structs into arguments.
- vlovich123 3y agoThat would be good advice except I’ve yet to see a compiler that can make this kind of stuff visible. First it’s distributed throughout your codebase and I’ve not seen any profiler that can show that impact unless you’re lucky and this becomes a hotspot. This applies to nearly all compiler generated code. Valgrind can measure it (sampling profiler probably can’t) but there’s no tooling to highlight diffuse code gen problems
- bugbuddy 3y agoThis is clearly a hot path and not inlining a hot path call is probably the first thing any engineer worth his salt would attempt to fix. If and how it can be fixed is a different question.
- vlovich123 3y agoJust because something is in the hot path does not mean a profiler will be able to highlight this as a hotspot.
- rwmj 3y agoPlus the guy is talking about the supreme importance of performance, but is using reference counting.
- FeepingCreature 3y ago(Author) The refcounting version is about as slow as the garbage collected D version on the benchmark. Refcounting operations can often be elided if you optimize a bit. And this is not a "supreme importance of performance" microoptimization. This single change moved me from place 23 to place 5. That's a 2x speedup!
- kragen 3y agoit's perfectly fine to pass and return structs bigger than 16 bytes by value on amd64 even with the sysv amd64 abi; it's just slow. but it's often worth it to make your code clearer. not in this case, of course, but as loeg points out, inside your own language you can just use a custom abi, as for example each c++ compiler, golang, ocaml, and sbcl do
- deleted 3y ago[deleted]
- smallstepforman 3y agoWhen I first transitioned to x64, I was concerned about graphics vec3 objects (3xfloat) expanding to sizeof()=16 bytes (instead of 12) that I benchmarked the hell out of my graphics engines. Unsuprisingly, using 16 bytes ended up being faster than 12, due to aligning 8 byte reads (internal and on GPU). So a vec3 silently became a vec4 (even though a vec4 also exists). As always, do global, not local benchmarking.
- jstimpfle 3y agoProbably not a lot faster? And apart from what you're doing with this data, isn't this also going to depend a lot on the CPU? I can see that with 16 bytes, a lot of accesses could be 2x aligned 8 bytes (or 1x16) instead of 3x4 bytes. Some other accesses, maybe not so much. And there is also the issue of increased cache pressure.
- connicpu 3y agoOn the GPU especially, having the data aligned to 16 bytes is huge just due to the nature of what a GPU is optimized for and the way memory access is designed. The speedup on x86_64 isn't as huge, but the aligned load instructions for SSE/AVX registers are still quite a bit faster than the unaligned load instructions. It can really add up when you're processing thousands of vectors every frame. The cache lines will only go so far when a single 3D affine transformation matrix (floatx4x4) already takes up an entire cache line by itself.
- gpderetta 3y agox64 ABIs are also significantly better than x86 ones as well.
- maccard 3y agoIt also has the very nice side effect of aligning with SSE sizes, so you can use \_mm\_load\_ps directly, which makes the code neater and us super easy to vectorise
- flohofwoe 3y agoFWIW, on MSVC the cutoff size is 8 bytes before structs are passed on the stack. It's an ABI detail that shouldn't be relied upon in portable code. ...but also don't stress it too much for functions that are not called frequently - and for frequently called small functions like in the example, just make sure that the compiler can inline the code (for instance via LTO) this will unlock much more useful optimizations beyond just passing args in registers.
- deleted 3y ago[deleted]
- DeathArrow 3y agoAs a .NET developer I always pay attention to passing by value vs passing by reference.
- sebazzz 3y agoThere is extra complexity there depending on the call path. Given the premise of the article, you shouldn’t pass structs larger than two object references, or larger dan a Int64 (long). You already get to that point quickly. But in .NET there are also things like allocations to worry about.
- gumby 3y ago> That means that arrays, in addition to start and end, also need a pointer to the base of the array object, where the reference count is stored. Isn’t the size and layout of the structure known at compile time? Why couldn’t the ref count be found by subtraction rather than by being passed? The standard way to implement this is to use address - 1 for such metadata, but any offset in a contiguous allocation works.
- deleted 3y ago[deleted]
- mcv 3y agoThe article links to a presumably extremely specialised benchmark[0] with some surprising results. To me as a simple dev, at least: Java (JIT) beats C++ and even Scala. What is Julia HO and why is it so ridiculously fast? The speed difference between Python and Pypy is massive. Is there any reason not to use Pypy? Shouldn't it be the standard? [0] https://github.com/jinyus/related_post_gen/ https://github.com/jinyus/related_post_gen/
- enticeing 3y agoLooks like the HO means hand optimized, with special datastructures for this benchmark. see: https://github.com/jinyus/related_post_gen/#user-content-fn-1-1b3cf9e6cf1c5b1a68431c1210d8d6d9 https://github.com/jinyus/related_post_gen/#user-content-fn-...
- unstruktured 3y agopypy and cpython don't mix well with each other so you have to create a whole new universe of packages for pypy. Interop with c libraries also adds complexity to the transition. In addition, vanilla python is finally starting to tackle the worst performance problems (like the GIL getting removed for instance), so they might catch up to pypy eventually. There's also several different JITs installable into cpython systems these days too which is less painful than switching to pypy. Personally, I am saddened pypy didn't replace standard python, but even in my own career I couldn't find a big speed difference in practice because all the heavy lifting utilizes C, C++, or fortran libraries wrapped by thin python apis.
- kortex 3y agoPypy doesn't work with all libraries (as well as certain function calls in std), and since that's one of the biggest draws of python, it makes it a non-starter much of the time.
- jlarocco 3y agoThe rule of thumb I've always heard for C++ is that anything non-primitive should be passed as a reference (or a pointer if you really have to) unless you have a good reason to pass by value (and normally you won't). Partly because of the ABI, partly to avoid copy or move constructors. It's a tedious low level detail, but that's what you have to pay attention to in C++ if you want the best performance. And to be clear, it is just a performance optimization - the struct passing code will work fine, just not as fast.
- tdiff 3y agoHow does such issues look in profiler? Is there a robust method to detect performance regression due to arg sizes?
- JoeAltmaier 3y agoSuch a subtle issue! And I lay the blame on our obsession with abstraction. Once we give up responsibility of the machine architecture to a compiler, we then have to chase issues like this. What gestures can I make to my compiler, to make it use better machine operations? It's maddening. Like a game of charades. Sure abstractions gain us tremendously, are an invaluable aid to quickly conceiving of an application and creating something safe and understandable and even beautiful. But we lost all hope of making it fast, or small, or 'efficient' in any physical way. My favorite example from my lifetime: On the Dell Triton our app programmers had a main loop that included a call using complex types. The entry point had, and I disassembled to see, five constructors and destructors called to coerce the types into something the API needed. It consumed the entirety of the embedded processor hyper-thread. They were unconcerned, as it worked as designed. I added a little casting in the call arguments, which reduced the CPU time to something like 15%.
- epigramx 3y agoStuff like that are mainly the realm of the compiler. Unless you talk chiefly to compiler programmers. Exception is writing special asm code of course or other adjacent things.