5 ms·

Hooking Go from Rust

145 points by carride 4y ago

zasdffaa 4y ago
> Golang doesn’t use libc on Linux, and instead calls syscalls directly Could someone explain this. I'm not familiar with low level linux stuff. Why would you choose not to use libc, what are the implications?
- aviramha 4y ago
  Hey, One of the writers of the article here. This is a very controversial topic in Go ecosystem, and following all the discussions that led to this decision is very hard. In short, I think main reasons are: 1. libc is considered hazard - security, comfort, runtime. It needs to maintain support for so many flows and setups so it's hard to make things better. For example, having `errno` as a global is quite weird (instead of just returning it, which is a whole discussion of it's own) 2. Golang devs like to re-invent the wheel, in a kind of Apple-ish way - all other are doing it wrong, we're going to do it better. ofc it's debatable whether they're correct with their approach. 3. Given they use Plan 9 system design, doing FFI from Go is very expensive (need to switch stack, save context, etc on each call)
  - infiniteregrets 4y ago
    A relevant discussion: https://news.ycombinator.com/item?id=25997506 https://news.ycombinator.com/item?id=25997506
    - metadat 4y ago
      ^ (Go 1.16 will make system calls through Libc on OpenBSD)
  - tialaramex 4y ago
    > For example, having `errno` as a global is quite weird It was quite weird before we wrote concurrent software. Once you're writing concurrent software it's very weird because as originally documented this single error location is shared by all threads, so it necessarily gets concurrently modified while being read, a Data Race. Your 2022 C library typically defines it as Thread Local instead, so when you call some_c_function() and then check errno, you aren't confused by the fact meanwhile another thread tried to open("nonexistent.file") and so their thread set errno to report that the file doesn't exist. Given that none of this fuss is how the actual system calls work, it's understandable that people might want to sidestep it.
    - aviramha 4y ago
      Thanks for elaborating! I totally agree there's valid reasons to drop libc, though not sure if that's the right way. In most modern languages today you can just return a tuple, not needing to mess with TLS.. whereas old times didn't really like passing results in out parameters (by ref) (which makes sense, it's less ergonomic).
      - throwaway894345 4y ago
        I much prefer the ergonomics of passing out-parameters than TLS. Honestly, TLS has always struck me as a code smell in any language.
  - throwaway894345 4y ago
    > Golang devs like to re-invent the wheel, in a kind of Apple-ish way - all other are doing it wrong, we're going to do it better. ofc it's debatable whether they're correct with their approach. I'm very happy with these tradeoffs. Not only is it a pleasure not to have to deal with libc (truly static binaries on Linux, 2mb docker images, etc), but because interop is tedious/expensive, the ecosystem is largely pure-Go. This means that we rarely (if ever) have to deal with C's abysmal build tooling and (lack of) package managers and stuff like cross compilation is trivial. Everything builds consistently on every system, every time, and irrespective of target platform (provided there are no C dependencies).
    - aviramha 4y ago
      My concern here is that many people are now writing software in Go which can only be used from Go. Yes, you can cgo but as you wrote the interop is tedious/expensive. It’s good we’re rewriting old software but this time in a semi closed garden…
      - throwaway894345 4y ago
        I mean, the garden has always been semi-closed, we've never had a truly open garden, and nothing Go did was going to change that. In other words, what language has ever solved the problem of cross-language-reuse in a way that was so simple that we didn't end up rewriting a huge swath of the ecosystem in every new language? It's not like Java or Rust came along and now we can stop rewriting HTTP libraries in subsequent languages. Easy interop has only ever been an illusion (we forget about the difficulties of integrating distinct build systems and memory management systems or the runtime costs of marshaling data between memory layouts). Rather than paying significant costs to preserve that illusion, Go doubled down in order to realize some pretty considerable gains: easy compilation even on niche distros, even when cross compiling, truly static binaries, etc.
        aviramha 4y ago
        We’re sliding into opinions but using C, Python, Rust in each interop had some cost but I did it and got a lot of benefits of using existing good software. Yes C lacks dev ergonomics, but meson + ninja gave a very neat experience..
- rapidlua 4y ago
  This is mostly due to ease of distribution. The binary is self-contained, no need to ship anything but the binary itself. You can also compile locally and then run it elsewhere that is a completely different flavor of Linux. Quite handy when experimenting. Finally, no dependency on libc enables easy cross-compilation. You might not even have libc installed for the architecture you are targeting!
metadat 4y ago
This is badass, thanks for sharing @carride. In celebration that it's Friday, I'm going to test it out right now. In case the author happens to see this and can respond, how did you figure all of this out? Would really enjoy a deep dive covering your process. This is brilliant. I also wonder if it could be extended to Nim, Zig, or any other less-straightforward hookable languages [than C/C++]. Edit: Apologies, maybe this is not that interesting of a wonderment. I did a little research and both Nim and Zig interface with libc. I'm not yet clear on whether node.js or Deno use libc, if any fellow HNers know please leave a reply! If they don't use libc, they could be interesting targets.
- aviramha 4y ago
  Thanks for the kind words. One of the authors here :) Actually this blog post started from a Twitter thread (that I forgot to link in the post https://twitter.com/Aviramyh/status/1544964265961979905 https://twitter.com/Aviramyh/status/1544964265961979905) - if you have any more questions, feel free to ask. We didn't want to go "too low level" so the common dev would enjoy this :) regarding extending, probably possible - I think it'd be easier with Zig (no weird stacks AFAIK) but we don't need it as they probably just use libc like most languages. P.S would love hearing your thoughts about mirrord
- infiniteregrets 4y ago
  Hey, seems like node uses libuv and in fact we did a blogpost on it! https://metalbear.co/blog/mirrord-internals-hooking-libc-functions-in-rust-and-fixing-bugs/ https://metalbear.co/blog/mirrord-internals-hooking-libc-fun...
- generichuman 4y ago
  > Edit: Apologies, maybe this is not that interesting of a wonderment. I did a little research and both Nim and Zig interface with libc. Zig does not depend on libc on Linux, for example: https://github.com/ziglang/zig/blob/master/lib/std/os/linux.zig#L391 https://github.com/ziglang/zig/blob/master/lib/std/os/linux.... You can choose to link libc though.
scottlamb 4y ago
Isn't there something important missing here? My understanding is that Go's non-standard ABI doesn't guarantee much available stack space or the existence of guard pages. IIUC, this places a standard-ABI-like stack frame on the Go stack for calling into Rust, but what guarantee is there that the Rust/libc code won't overflow the small stack? What happens if it does? AFAICT, the answer are "none" and "memory corruption".
- aviramha 4y ago
  We actually address it in the post - https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-guide-to-the-go-laxy/#why-did-this-goroutine-panic-with-our-hook https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-g... > "Goroutine stack is dynamic, i.e. it is constantly expanding/shrinking depending on the current needs. This means any common code that runs in system stack assumes it can grow as it wishes (until it exceeds max stack size) while actually, it can’t unless using Go APIs for expanding. Our Rust code isn’t aware of it, so it uses parts of the stack that aren’t actually usable and causes stack overflow." tl;dr - yes, you're correct and we replace Go stack with system stack for the duration of the call, just like cgo does.
  - scottlamb 4y ago
    Thanks, I somehow missed that whole section!
arriu 4y ago
This is really cool. Thanks for sharing. Does anyone have any recommendations on how to do something like this but in reverse? Calling a go function from rust?
- aviramha 4y ago
  Thanks! Do you mean for an already compiled Go binary? if you can recompile you can use cgo. If not the following requirements come to mind (there are more probably): 1. Allocate g and m if doesn’t exist in current process. 2. Switch to go stack before call Btw there’s the cgocallback routine that manages C code that calls into go so that’s a good look to see what’s needed.
- IceWreck 4y ago
  If you have the source code for both you should ideally use something like GRPC or another way to communicate. Or maybe cgo if you want to wet your hands into that. The approach in this blog works with compiled binaries.
ithrow 4y ago
why doesn't t Go just uses "syscalls" in macos too?
- aviramha 4y ago
  They tried but macOS doesn’t have any user<>kernel stable API so you have to rely on libsystem to provide it. (It broke very often so they changed it to use libsystem)
- infiniteregrets 4y ago
  > Go used to do raw system calls on macOS, and binaries were occasionally broken by kernel updates. Now Go uses libc on macOS, and binaries are forward compatible with future macOS versions just like any other C/C++/ObjC/swift program. OS X 10.10 (Yosemite) is the current minimum supported version. also found a discussion - https://news.ycombinator.com/item?id=18439100 https://news.ycombinator.com/item?id=18439100 that points to why this might be the case
jgavris 4y ago
How big is your Go codebase versus your Rust codebase? Why not rewrite bits in Rust? This is very cool, but seems like a lot of effort and ongoing maintenance.
- aviramha 4y ago
  It’s not the case actually. We’re working on a dev tool called mirrord that lets you create local processes in context of a remote environment so the ergonomics would be of local setup with the benefits of leveraging real cloud environments. The way we accomplish that is we hook sys calls and then choose what happens locally and what happens remotely.
nedsma 4y ago
This is plain and simply incredible.
pijiskhan 4y ago
This is really cool. One question though, does this still work when the section containing the Syscall/RawSyscall/Syscall6 functions is read only?
- aviramha 4y ago
  I'm pretty sure that frida takes care of it (we haven't seen such scenario in the wild where the hooks fail..)

145 points by carride 4y ago

> Golang doesn’t use libc on Linux, and instead calls syscalls directly Could someone explain this. I'm not familiar with low level linux stuff. Why would you choose not to use libc, what are the implications?

aviramha 4y ago
Hey, One of the writers of the article here. This is a very controversial topic in Go ecosystem, and following all the discussions that led to this decision is very hard. In short, I think main reasons are: 1. libc is considered hazard - security, comfort, runtime. It needs to maintain support for so many flows and setups so it's hard to make things better. For example, having `errno` as a global is quite weird (instead of just returning it, which is a whole discussion of it's own) 2. Golang devs like to re-invent the wheel, in a kind of Apple-ish way - all other are doing it wrong, we're going to do it better. ofc it's debatable whether they're correct with their approach. 3. Given they use Plan 9 system design, doing FFI from Go is very expensive (need to switch stack, save context, etc on each call)
- infiniteregrets 4y ago
  A relevant discussion: https://news.ycombinator.com/item?id=25997506 https://news.ycombinator.com/item?id=25997506
  - metadat 4y ago
    ^ (Go 1.16 will make system calls through Libc on OpenBSD)
- tialaramex 4y ago
  > For example, having `errno` as a global is quite weird It was quite weird before we wrote concurrent software. Once you're writing concurrent software it's very weird because as originally documented this single error location is shared by all threads, so it necessarily gets concurrently modified while being read, a Data Race. Your 2022 C library typically defines it as Thread Local instead, so when you call some_c_function() and then check errno, you aren't confused by the fact meanwhile another thread tried to open("nonexistent.file") and so their thread set errno to report that the file doesn't exist. Given that none of this fuss is how the actual system calls work, it's understandable that people might want to sidestep it.
  - aviramha 4y ago
    Thanks for elaborating! I totally agree there's valid reasons to drop libc, though not sure if that's the right way. In most modern languages today you can just return a tuple, not needing to mess with TLS.. whereas old times didn't really like passing results in out parameters (by ref) (which makes sense, it's less ergonomic).
    - throwaway894345 4y ago
      I much prefer the ergonomics of passing out-parameters than TLS. Honestly, TLS has always struck me as a code smell in any language.
- throwaway894345 4y ago
  > Golang devs like to re-invent the wheel, in a kind of Apple-ish way - all other are doing it wrong, we're going to do it better. ofc it's debatable whether they're correct with their approach. I'm very happy with these tradeoffs. Not only is it a pleasure not to have to deal with libc (truly static binaries on Linux, 2mb docker images, etc), but because interop is tedious/expensive, the ecosystem is largely pure-Go. This means that we rarely (if ever) have to deal with C's abysmal build tooling and (lack of) package managers and stuff like cross compilation is trivial. Everything builds consistently on every system, every time, and irrespective of target platform (provided there are no C dependencies).
  - aviramha 4y ago
    My concern here is that many people are now writing software in Go which can only be used from Go. Yes, you can cgo but as you wrote the interop is tedious/expensive. It’s good we’re rewriting old software but this time in a semi closed garden…
    - throwaway894345 4y ago
      I mean, the garden has always been semi-closed, we've never had a truly open garden, and nothing Go did was going to change that. In other words, what language has ever solved the problem of cross-language-reuse in a way that was so simple that we didn't end up rewriting a huge swath of the ecosystem in every new language? It's not like Java or Rust came along and now we can stop rewriting HTTP libraries in subsequent languages. Easy interop has only ever been an illusion (we forget about the difficulties of integrating distinct build systems and memory management systems or the runtime costs of marshaling data between memory layouts). Rather than paying significant costs to preserve that illusion, Go doubled down in order to realize some pretty considerable gains: easy compilation even on niche distros, even when cross compiling, truly static binaries, etc.
      - aviramha 4y ago
        We’re sliding into opinions but using C, Python, Rust in each interop had some cost but I did it and got a lot of benefits of using existing good software. Yes C lacks dev ergonomics, but meson + ninja gave a very neat experience..
rapidlua 4y ago
This is mostly due to ease of distribution. The binary is self-contained, no need to ship anything but the binary itself. You can also compile locally and then run it elsewhere that is a completely different flavor of Linux. Quite handy when experimenting. Finally, no dependency on libc enables easy cross-compilation. You might not even have libc installed for the architecture you are targeting!

metadat 4y ago

This is badass, thanks for sharing @carride. In celebration that it's Friday, I'm going to test it out right now. In case the author happens to see this and can respond, how did you figure all of this out? Would really enjoy a deep dive covering your process. This is brilliant. I also wonder if it could be extended to Nim, Zig, or any other less-straightforward hookable languages [than C/C++]. Edit: Apologies, maybe this is not that interesting of a wonderment. I did a little research and both Nim and Zig interface with libc. I'm not yet clear on whether node.js or Deno use libc, if any fellow HNers know please leave a reply! If they don't use libc, they could be interesting targets.

aviramha 4y ago
Thanks for the kind words. One of the authors here :) Actually this blog post started from a Twitter thread (that I forgot to link in the post https://twitter.com/Aviramyh/status/1544964265961979905 https://twitter.com/Aviramyh/status/1544964265961979905) - if you have any more questions, feel free to ask. We didn't want to go "too low level" so the common dev would enjoy this :) regarding extending, probably possible - I think it'd be easier with Zig (no weird stacks AFAIK) but we don't need it as they probably just use libc like most languages. P.S would love hearing your thoughts about mirrord
infiniteregrets 4y ago
Hey, seems like node uses libuv and in fact we did a blogpost on it! https://metalbear.co/blog/mirrord-internals-hooking-libc-functions-in-rust-and-fixing-bugs/ https://metalbear.co/blog/mirrord-internals-hooking-libc-fun...
generichuman 4y ago
> Edit: Apologies, maybe this is not that interesting of a wonderment. I did a little research and both Nim and Zig interface with libc. Zig does not depend on libc on Linux, for example: https://github.com/ziglang/zig/blob/master/lib/std/os/linux.zig#L391 https://github.com/ziglang/zig/blob/master/lib/std/os/linux.... You can choose to link libc though.

scottlamb 4y ago

Isn't there something important missing here? My understanding is that Go's non-standard ABI doesn't guarantee much available stack space or the existence of guard pages. IIUC, this places a standard-ABI-like stack frame on the Go stack for calling into Rust, but what guarantee is there that the Rust/libc code won't overflow the small stack? What happens if it does? AFAICT, the answer are "none" and "memory corruption".

aviramha 4y ago
We actually address it in the post - https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-guide-to-the-go-laxy/#why-did-this-goroutine-panic-with-our-hook https://metalbear.co/blog/hooking-go-from-rust-hitchhikers-g... > "Goroutine stack is dynamic, i.e. it is constantly expanding/shrinking depending on the current needs. This means any common code that runs in system stack assumes it can grow as it wishes (until it exceeds max stack size) while actually, it can’t unless using Go APIs for expanding. Our Rust code isn’t aware of it, so it uses parts of the stack that aren’t actually usable and causes stack overflow." tl;dr - yes, you're correct and we replace Go stack with system stack for the duration of the call, just like cgo does.
- scottlamb 4y ago
  Thanks, I somehow missed that whole section!

arriu 4y ago

This is really cool. Thanks for sharing. Does anyone have any recommendations on how to do something like this but in reverse? Calling a go function from rust?

aviramha 4y ago
Thanks! Do you mean for an already compiled Go binary? if you can recompile you can use cgo. If not the following requirements come to mind (there are more probably): 1. Allocate g and m if doesn’t exist in current process. 2. Switch to go stack before call Btw there’s the cgocallback routine that manages C code that calls into go so that’s a good look to see what’s needed.
IceWreck 4y ago
If you have the source code for both you should ideally use something like GRPC or another way to communicate. Or maybe cgo if you want to wet your hands into that. The approach in this blog works with compiled binaries.

ithrow 4y ago

why doesn't t Go just uses "syscalls" in macos too?

aviramha 4y ago
They tried but macOS doesn’t have any user<>kernel stable API so you have to rely on libsystem to provide it. (It broke very often so they changed it to use libsystem)
infiniteregrets 4y ago
> Go used to do raw system calls on macOS, and binaries were occasionally broken by kernel updates. Now Go uses libc on macOS, and binaries are forward compatible with future macOS versions just like any other C/C++/ObjC/swift program. OS X 10.10 (Yosemite) is the current minimum supported version. also found a discussion - https://news.ycombinator.com/item?id=18439100 https://news.ycombinator.com/item?id=18439100 that points to why this might be the case

jgavris 4y ago

How big is your Go codebase versus your Rust codebase? Why not rewrite bits in Rust? This is very cool, but seems like a lot of effort and ongoing maintenance.

aviramha 4y ago
It’s not the case actually. We’re working on a dev tool called mirrord that lets you create local processes in context of a remote environment so the ergonomics would be of local setup with the benefits of leveraging real cloud environments. The way we accomplish that is we hook sys calls and then choose what happens locally and what happens remotely.

nedsma 4y ago

This is plain and simply incredible.

pijiskhan 4y ago

This is really cool. One question though, does this still work when the section containing the Syscall/RawSyscall/Syscall6 functions is read only?

aviramha 4y ago
I'm pretty sure that frida takes care of it (we haven't seen such scenario in the wild where the hooks fail..)