4 ms·
In an ideal world, we'd be using an ABI that keeps the code (return addresses) and data (spilled function params/locals) in two separate stacks. Then the profil
by 10000truths 2y ago
In an ideal world, we'd be using an ABI that keeps the code (return addresses) and data (spilled function params/locals) in two separate stacks. Then the profiling would be as straightforward as a memory copy of the code stack.
- felixge 2y agoThat's what hardware shadow stacks in modern intel/arm CPUs can do! It just needs to be exposed to user space and become widely available.
- fweimer 2y agoFedora 40 and later have shadow stack support in userspace. It's currently opt-in with glibc (`export GLIBC_TUNABLES=glibc.cpu.x86_shstk=on` is one way to switch it on I believe). The plan is to make this self-tuning eventually in glibc upstream, once the quirks have been ironed out. It will not work with Go as-is because the Go scheduler will have to be taught to switch the shadow stack along with the regular stack, panic/recover needs to walk the shadow stack. But for binaries that do not use CGO, it would be possible to enable this fairly quickly. Hardware support is already widely available. The SHSTK-specific code paths are well-isolated. You would not lose compatibility with older CPUs or kernels.
- felixge 2y agoThanks for the reply! What does the API for accessing the shadow stack from user space look like? I didn't see anything for it in the kernel docs [1]. I agree about the need for switching the shadow stacks in the Go scheduler. But this would probably require an API that is a bit at odds with the security goals of the kernel feature. I'm not sure I follow your thoughts on CGO and how this would work on older CPUs and kernels. [1] https://docs.kernel.org/next/x86/shstk.html https://docs.kernel.org/next/x86/shstk.html
- fweimer 2y agoYou can get the shadow stack pointer using the RDSSPQ instruction. The kernel documentation shows how the shadow stack size is obtained for the main thread. Threads created explicitly using clone3 have the specified shadow stack size. I think this is sufficient to determine the shadow stack boundaries. Regarding older CPUs, what I wanted to point out is that the code to enable and maintain shadow stacks will not be smeared across the instruction stream (unlike using APX instructions, or direct use of LSE atomics on AArch64). It's possible to execute the shadow stack code only conditionally.
- felixge 2y agoThank you so much, this is very helpful and interesting. I'll try to experiment with this at some point.
- fweimer 2y agoGlad to be of help. Note that it's actually GLIBC_TUNABLES=glibc.cpu.hwcaps=SHSTK to enable it with Fedora 40 glibc (and the program needs to be built with -fcf-protection).
- mmaniac 2y agoWould that not complicate function prologues & epilogues? I suppose hardware support (e.g. dedicated call/ret instructions accessing a second stack pointer) is also desirable if you want to do this sort of thing. I recall that Itanium had something like this.
- Joker_vD 2y agoHow would it complicate the prologue/epilogue? It would still be push RBP mov RBP, RDSP ; DSP stands for "data-stack pointer" sub RDSP, <frame-size> ... ; locals are [RBP-n], args are [RBP+n] add RDSP, <frame-size> pop RBP ret ; pops address from [RCSP], where CSP stands for "call-stack pointer" or even, if you don't want a specific base pointer, like this: sub RDSP, <frame-size> ... ; locals are [RBP+n], args are [RBP+frame_size+n] add RDSP, <frame-size> ret In fact, you can do this even now on x86-64: make a second stack segment, point RBP there, and use RBP as if it were that hypothetical RDSP register. The return addresses naturally go into the RSP-pointed area, as they always did. The only tricky part is when you need to call the code that uses SysV x64 ABI, you'll need to conditionally re-align that RSP-pointed area to be 16-bytes aligned which is a bit annoying (in your own code, having RBP always aligned at 16 bytes is trivial since function calls don't break that alignment). There are two main benefits of this layout, as I see it: first, smashing the return address is now way harder that it normally is; second, "the act of traversing all call frames on the stack and collecting the return addresses that are stored inside of them" is now trivial, since the call stack at RSP is now just a dense array of just the return addresses, no pointer chasing needed.
- jerf 2y agoAnd Go is already using a non-standard ABI and has to take real action to convert back to standard ABIs if you want to call external code. That's part of why cgo calls are expensive, the other major bit relating to having to interact with OS threads differently than native Go code does.
- deleted 2y ago[deleted]
- friendzis 2y agoIsn't that what mainly differentiates Harvard from Von Neumann?
- immibis 2y agoNo. They store code and data in separate address spaces. That isn't the same as storing code addresses and data addresses in separate parts of the data space.
- eru 2y agoYes. And all four combinations of Harvard / von-Neumann X single stack / double stack are potentially viable options. Btw, I think Forth has a separate data stack and return address stack?
- p_l 2y agoYes. You can also freely use the return stack, IIRC, so long as you don't mess up a return itself.