6 ms·

Self-Replace: A Utility For Self Replacing Executables

58 points by asicsp 3y ago

gumby 3y ago
Self-modifying code.
benatkin 3y ago
It has an interface and two implementations, UNIX and Windows.
Hello71 3y ago
the windows implementation seems awfully complex. what's wrong with something like _popen("find \"\" & del " + argv[0])? the problem to solve is "we can't delete ourselves, so we need something else to do it for us", but there are a myriad options to choose from, you don't have to write it in rust. hell, you've already got the dubious sleep(100) in there, you can even write _popen("ping 127.0.0.1 & del " + argv[0]) as suggested by stack overflow?
- paulddraper 3y ago
  Not a Windows programmer, but doesn't it prevent a file from being deleted while it is being used/executed? (At least not without black magic?)
  - ilyt 3y ago
    Stupid trick would be to copy to temporary file and run that file, then update the original now-not-used one. Tho that might be weird enough to trigger AV
- the_mitsuhiko 3y ago
  I tried plenty of approaches on windows, this was the one that seems the most reliable and least problematic. It's also the approach other systems already use.
woodruffw 3y ago
Two things: 1. Replacing the current executable is generally okay on Unix and Unix-likes, but it's not particularly well-defined. Linux handles it gracefully, others may ETXTBSY or similar depending on how the replacement is done. 2. The hack being used on Windows is awful: it involves patching the underlying C runtime, can fail in uncontrolled ways (including in an `atexit` handler), and is fundamentally racy. The state it leaves a failed operation in isn't self-healing, and surfaces confusing filesystem state to the user on failure. If you want to replace a running executable on Windows, either ask the user (and have them affirmatively kill the running program) or do the "cooperative rename" trick[1]. It doesn't require any unsafety, much less patching the runtime underneath you. [1]: https://social.msdn.microsoft.com/Forums/vstudio/en-US/07fb6078-c41d-4829-a62d-f22dd112360c/how-to-replace-exe-file-with-same-updated-exe-file?forum=csharpgeneral https://social.msdn.microsoft.com/Forums/vstudio/en-US/07fb6...
- evanelias 3y ago
  On #1, totally agree, and security systems can add an extra winkle... With code-signed binaries on a Mac, macOS Gatekeeper caches signatures by inode. If the binary is replaced/re-written in a way which retains the old inode, and you try to execute it, the OS will SIGKILL the process immediately. Not fun to debug the first time you encounter it!
- bsnnkv 3y ago
  Thanks for the breakdown of why this is a bad idea. I was considering using this for a Windows Rust project I maintain, but now I know better. Ideally tradeoffs like these should be mentioned clearly in the readme.
- the_mitsuhiko 3y ago
  > it involves patching the underlying C runtime Can you please clarify how this "patches the underlying C runtime"? What this crate is doing is not particularly out of the ordinary. You will find similar code in quite a few self updating windows executables. > or do the "cooperative rename" trick[1] I'm not entirely sure how this is much different? This is pretty much what this is doing. It spawns a copy of itself (with the exe being marked for self deletion) and then deletes the other executable.
lohnjemon 3y ago
When I had written my shell to the point where I could execute simple commands, this was the first thing that came to my mind using it. Execute the shell -> Modify the source code of the shell with vim -> Remake the shell -> Execute the newly built shell with the old shell Just for fun :)
HellsMaddy 3y ago
Is there anything like this for C/C++ applications?
samsquire 3y ago
This is interesting. Google Chrome can patch binaries and the Linux kernel can be patched while running. I would like this capability for general purpose software, hot code reloading. Can do seamless non-breaking changes to a live server without a full restart. Would be useful for Function as a Service and WASM runtimes if you could live patch code running. I generally rely on Kubernetes or CI to do a rolling deploy of software to switch to the new version, but if we could patch a server while running that would be interesting. Would have interesting interactions with socket connections using the old code and connections using the new code and all the complicated states you can get into, if your data structures are different. How do you move data if you added a field to your data structure? Presumably if you added a field, there isn't enough memory allocates in the existing data structures... Your malloc is for the wrong size.
- TeMPOraL 3y ago
  > I would like this capability for general purpose software, hot code reloading. Can do seamless non-breaking changes to a live server without a full restart. This is very old-school style, that fell out of favor for... whatever reason this industry likes to make two steps forward, and then 1.9 steps backwards. It survives in the legacy of Smalltalk and Lisp. To a lesser degree, it's featured in Erlang, and I hear C# these days. > Would have interesting interactions with socket connections using the old code and connections using the new code and all the complicated states you can get into, if your data structures are different. If you make a mental shift - treat the OS as runtime, and processes as functions (and shell as a REPL, and environment variables as dynamic binding...) - then the ideas from Smalltalk and Lisp image-based programming will apply directly. At Kubernetes level, I suppose Erlang's take on connecting BEAM runtimes applies as well. > How do you move data if you added a field to your data structure? Presumably if you added a field, there isn't enough memory allocates in the existing data structures... Your malloc is for the wrong size. If this is C (or Rust, I suppose?), then you can't. You have to architect for that from the start. With garbage collected runtimes... I'm not sure how hard is it with static typing, but with dynamically typed languages it's not bad. If you want to see a system that embraces runtime modification of classes, check Lisp (e.g. Common Lisp) or Smalltalk. For example, in case of Common Lisp, if you want to make a substantial runtime modification to some class definition, you have to supply a "old" -> "new" migration function, which will be applied to all existing instances of the old class definition. Note: this is not a hack, it's a legitimate feature of the language/runtime, and - like everything else - can be operated fully automatically.
funcDropShadow 3y ago
I don't understand why people think this is a good idea to do at all. Yes it makes updates easier. But each software doing that has to run with the privileges to update itself. Which is in my book a big security downside. Separating rights to install, update, and remove software from the right to run and use it should be complete separate. I admit this is not very nice for private Windows users. But this is one of the strength of Linux distributions.
- the_mitsuhiko 3y ago
  > But each software doing that has to run with the privileges to update itself. Which is in my book a big security downside. But it does not have to run with those privileges under normal circumstances. It's easy enough to elevate privileges just for the update step.
- ilyt 3y ago
  Because Windows don't have decent package management system. Linux way of "add a repo, upgrade every package from either system or app-own repo" is strictly superior but windows just doesn't have it (and thanks to that there is multi-billion dollar cottage industry to fill those holes)
  - vladvasiliu 3y ago
    They're starting to get there with winget. It's not as comfortable as an actual linux package manager, but it seems to be on the right track.
agateau 3y ago
A little known fact about executables on Windows is that while it's not possible to remove a running executable, it's possible to rename it. I use this in Clyde [1]: on Windows, when clyde needs to upgrade itself (which means uninstalling vN and installing vN+1) it renames itself from `clyde.exe` to `_clyde.exe`. I leave the old `_clyde.exe` around, but a nicer implementation could remove on the next start. [1]: https://github.com/agateau/clyde https://github.com/agateau/clyde
Dwedit 3y ago
Crazy idea, just make your EXE into a stub that loads your actual program, which is implemented in a DLL. When it's time to update, drop a newer DLL in there that the EXE will load instead. Hopefully you won't have to update the stub very often.
- ilyt 3y ago
  Or "launcher replaces the app, app updates the launcher".
stefanos82 3y ago
How does it relate to KSplice [1], from a technical point of view? The first time I heard about it was when Jessica McKellar [2] explained what it is and what it does, in a video right before KSplice, Inc. was acquired by Oracle. [1] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice [2] https://en.wikipedia.org/wiki/Jessica_McKellar https://en.wikipedia.org/wiki/Jessica_McKellar

58 points by asicsp 3y ago

gumby 3y ago

Self-modifying code.

benatkin 3y ago

It has an interface and two implementations, UNIX and Windows.

Hello71 3y ago

the windows implementation seems awfully complex. what's wrong with something like _popen("find \"\" & del " + argv[0])? the problem to solve is "we can't delete ourselves, so we need something else to do it for us", but there are a myriad options to choose from, you don't have to write it in rust. hell, you've already got the dubious sleep(100) in there, you can even write _popen("ping 127.0.0.1 & del " + argv[0]) as suggested by stack overflow?

paulddraper 3y ago
Not a Windows programmer, but doesn't it prevent a file from being deleted while it is being used/executed? (At least not without black magic?)
- ilyt 3y ago
  Stupid trick would be to copy to temporary file and run that file, then update the original now-not-used one. Tho that might be weird enough to trigger AV
the_mitsuhiko 3y ago
I tried plenty of approaches on windows, this was the one that seems the most reliable and least problematic. It's also the approach other systems already use.

woodruffw 3y ago

Two things: 1. Replacing the current executable is generally okay on Unix and Unix-likes, but it's not particularly well-defined. Linux handles it gracefully, others may ETXTBSY or similar depending on how the replacement is done. 2. The hack being used on Windows is awful: it involves patching the underlying C runtime, can fail in uncontrolled ways (including in an `atexit` handler), and is fundamentally racy. The state it leaves a failed operation in isn't self-healing, and surfaces confusing filesystem state to the user on failure. If you want to replace a running executable on Windows, either ask the user (and have them affirmatively kill the running program) or do the "cooperative rename" trick[1]. It doesn't require any unsafety, much less patching the runtime underneath you. [1]: https://social.msdn.microsoft.com/Forums/vstudio/en-US/07fb6078-c41d-4829-a62d-f22dd112360c/how-to-replace-exe-file-with-same-updated-exe-file?forum=csharpgeneral https://social.msdn.microsoft.com/Forums/vstudio/en-US/07fb6...

evanelias 3y ago
On #1, totally agree, and security systems can add an extra winkle... With code-signed binaries on a Mac, macOS Gatekeeper caches signatures by inode. If the binary is replaced/re-written in a way which retains the old inode, and you try to execute it, the OS will SIGKILL the process immediately. Not fun to debug the first time you encounter it!
bsnnkv 3y ago
Thanks for the breakdown of why this is a bad idea. I was considering using this for a Windows Rust project I maintain, but now I know better. Ideally tradeoffs like these should be mentioned clearly in the readme.
the_mitsuhiko 3y ago
> it involves patching the underlying C runtime Can you please clarify how this "patches the underlying C runtime"? What this crate is doing is not particularly out of the ordinary. You will find similar code in quite a few self updating windows executables. > or do the "cooperative rename" trick[1] I'm not entirely sure how this is much different? This is pretty much what this is doing. It spawns a copy of itself (with the exe being marked for self deletion) and then deletes the other executable.

lohnjemon 3y ago

When I had written my shell to the point where I could execute simple commands, this was the first thing that came to my mind using it. Execute the shell -> Modify the source code of the shell with vim -> Remake the shell -> Execute the newly built shell with the old shell Just for fun :)

HellsMaddy 3y ago

Is there anything like this for C/C++ applications?

samsquire 3y ago

This is interesting. Google Chrome can patch binaries and the Linux kernel can be patched while running. I would like this capability for general purpose software, hot code reloading. Can do seamless non-breaking changes to a live server without a full restart. Would be useful for Function as a Service and WASM runtimes if you could live patch code running. I generally rely on Kubernetes or CI to do a rolling deploy of software to switch to the new version, but if we could patch a server while running that would be interesting. Would have interesting interactions with socket connections using the old code and connections using the new code and all the complicated states you can get into, if your data structures are different. How do you move data if you added a field to your data structure? Presumably if you added a field, there isn't enough memory allocates in the existing data structures... Your malloc is for the wrong size.

TeMPOraL 3y ago
> I would like this capability for general purpose software, hot code reloading. Can do seamless non-breaking changes to a live server without a full restart. This is very old-school style, that fell out of favor for... whatever reason this industry likes to make two steps forward, and then 1.9 steps backwards. It survives in the legacy of Smalltalk and Lisp. To a lesser degree, it's featured in Erlang, and I hear C# these days. > Would have interesting interactions with socket connections using the old code and connections using the new code and all the complicated states you can get into, if your data structures are different. If you make a mental shift - treat the OS as runtime, and processes as functions (and shell as a REPL, and environment variables as dynamic binding...) - then the ideas from Smalltalk and Lisp image-based programming will apply directly. At Kubernetes level, I suppose Erlang's take on connecting BEAM runtimes applies as well. > How do you move data if you added a field to your data structure? Presumably if you added a field, there isn't enough memory allocates in the existing data structures... Your malloc is for the wrong size. If this is C (or Rust, I suppose?), then you can't. You have to architect for that from the start. With garbage collected runtimes... I'm not sure how hard is it with static typing, but with dynamically typed languages it's not bad. If you want to see a system that embraces runtime modification of classes, check Lisp (e.g. Common Lisp) or Smalltalk. For example, in case of Common Lisp, if you want to make a substantial runtime modification to some class definition, you have to supply a "old" -> "new" migration function, which will be applied to all existing instances of the old class definition. Note: this is not a hack, it's a legitimate feature of the language/runtime, and - like everything else - can be operated fully automatically.

funcDropShadow 3y ago

I don't understand why people think this is a good idea to do at all. Yes it makes updates easier. But each software doing that has to run with the privileges to update itself. Which is in my book a big security downside. Separating rights to install, update, and remove software from the right to run and use it should be complete separate. I admit this is not very nice for private Windows users. But this is one of the strength of Linux distributions.

the_mitsuhiko 3y ago
> But each software doing that has to run with the privileges to update itself. Which is in my book a big security downside. But it does not have to run with those privileges under normal circumstances. It's easy enough to elevate privileges just for the update step.
ilyt 3y ago
Because Windows don't have decent package management system. Linux way of "add a repo, upgrade every package from either system or app-own repo" is strictly superior but windows just doesn't have it (and thanks to that there is multi-billion dollar cottage industry to fill those holes)
- vladvasiliu 3y ago
  They're starting to get there with winget. It's not as comfortable as an actual linux package manager, but it seems to be on the right track.

agateau 3y ago

A little known fact about executables on Windows is that while it's not possible to remove a running executable, it's possible to rename it. I use this in Clyde [1]: on Windows, when clyde needs to upgrade itself (which means uninstalling vN and installing vN+1) it renames itself from `clyde.exe` to `_clyde.exe`. I leave the old `_clyde.exe` around, but a nicer implementation could remove on the next start. [1]: https://github.com/agateau/clyde https://github.com/agateau/clyde

Dwedit 3y ago

Crazy idea, just make your EXE into a stub that loads your actual program, which is implemented in a DLL. When it's time to update, drop a newer DLL in there that the EXE will load instead. Hopefully you won't have to update the stub very often.

ilyt 3y ago
Or "launcher replaces the app, app updates the launcher".

stefanos82 3y ago

How does it relate to KSplice [1], from a technical point of view? The first time I heard about it was when Jessica McKellar [2] explained what it is and what it does, in a video right before KSplice, Inc. was acquired by Oracle. [1] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice [2] https://en.wikipedia.org/wiki/Jessica_McKellar https://en.wikipedia.org/wiki/Jessica_McKellar