11 ms·
Unfork()
- friend-monoid 7y agoThe inverse of fork is called join, right?
- tom_mellior 7y agoI've read the quirky FAQ and would now be interested in what this really does. The Readme mentions some demo code that's not in the repository. It also instructs us to run this on cat and enjoy, but... what would we observe and enjoy?
- aloknnikhil 7y ago> Permission to read from or write to another process is governed by a ptrace access mode PTRACE_MODE_ATTACH_REALCREDS check I presume that is necessary for this in addition to belonging to the same UID? > As far as I know process_vm_readv isn't even detectable if the agent process is more privileged than the examinee process—so you're free to manipulate your private copy of the application in the comfort of your own address space. Interesting. This would be really useful in debugging. Many issues don't reproduce except for in specific configurations. Having access to the memory dump of the live process "streamed" to the debugger would be great!
- ktpsns 7y ago> Having access to the memory dump of the live process "streamed" to the debugger would be great! This is also possible with standard debuggers, such as GDB: It can attach to a running process and not only examine the memory, but also debug (stop, pause, skip, ...) the stack trace and control flow. Usage is as simple as gdb -p $(pidof my_running_program)
- aloknnikhil 7y agoRight. But the difference is modifying any memory under GDB will be seen by the process. It's not copy-on-write
- coldtea 7y agoIsn't that a feature for debugging tho?
- aloknnikhil 7y agoAbsolutely. I was arguing for "unforking" to say a patched version of the process and verifying a fix without actually modifying the live process.
- andrewaylett 7y agoYou might also find https://rr-project.org/ https://rr-project.org/ interesting -- it lets you step backwards too.
- fooker 7y agogdb supports reverse debugging too
- Enginerrrd 7y agoInteresting. Without letting storage explode, I can't think of an easy way to do this since computation isn't really reversible.
- geofft 7y ago"a PTRACE_MODE_ATTACH_REALCREDS check" and "same UID" are roughly synonyms. ("Roughly" because the actual check also permits privileged processes and denies some special cases such as processes that have done a setuid. REALCREDS, if I remember right, is in contrast to the check done on certain files in /proc.) It's the same check as ptrace itself, so the intuition is "can I strace or gdb this process."
- tempay 7y agoThe big exception to this is inside containers where it is often disabled by default.
- comex 7y agoOr if you have an LSM such as Yama or SELinux set to deny ptrace globally.
- justincormack 7y agoYama lets you ptrace a child process but not others.
- jwilk 7y agoBy default, yes. You can also disable ptrace() completely. (Grep for "ptrace_scope" in the ptrace(2) man page for details.)
- geofft 7y agoAIUI Docker containers by default deny the ptrace syscall (and presumably process_vm_readv/writev), they don't change the permission check. So /proc/$pid/mem, which uses the same permission check, ought to work. (This also means that you don't want or need CAP_SYS_PTRACE to get gdb/strace working in Docker, that lets you ptrace anything and also coincidentally turns off the syscall filter. Just turn the filter off, that works without privileging the processes in the container.)
- euske 7y agoTotally, off topic, but it's funny to think about what kind of the feature would be if we put "un-" on each syscall: unseek unselect unpipe unsync etc.
- colonwqbang 7y agounselect(2) Description: put thread to sleep as long as there is activity on any fd, wake up only when all fds are inactive. Useful for: Scheduling work to be performed only when server is idle. unsync(2) Description: Select a random file, load it into the buffer cache, and remove it from file system. Useful for: Freeing up some disk space in a pinch. unwait(2) Description: Resurrects the previous child process. Useful for: Implementing the !! operator in bash. unsignalfd(2) Description: Invoke signal handler whenever a given fd activates. Useful for: User space interrupts. unopen(2) Description: Create a file which refers to an open fd. Useful for: Implementing /proc/self/fd functionalit.
- marcosdumay 7y ago> unselect(2) > Useful for: Scheduling work to be performed only when server is idle. Yeah that's nice. > unsignalfd(2) > Useful for: User space interrupts. There is libfam. At least on my system, it doesn't have a manual page. > unopen(2) > Description: Create a file which refers to an open fd. That does sounds useful, and I don't know any library that does it.
- datenwolf 7y agoThe "unopen" syscall actually exists, albeit under a different name: linkat https://linux.die.net/man/2/linkat https://linux.die.net/man/2/linkat
- colonwqbang 7y agoI don't think it's the same thing. Linkat is still starting with a file that exists in the filesystem. In my silly world, unopen() would just take any fd (socket, file, pipe, etc.) and create a file system binding which anyone could open. Kind of like how /proc works on Linux today.
- saagarjha 7y agoI wonder if 64 (well, 48) bits of address space is enough to glom together every process on a normal Linux boot without collisions…
- smallnamespace 7y agoIf you assume each process needs, say, 16MB of contiguous space, then you get 48-24 bits left, which by the birthday paradox implies you can have up to 2^(24/2 = 12) ~= 4k processes before you start colliding about half the time.
- saagarjha 7y ago> If you assume each process needs, say, 16MB of contiguous space Unfortunately I’m not sure that’s a good assumption, due to the stack and heap needing to exist even for statically-linked binaries.
- smallnamespace 7y agoYes, but simply replace with 'average number of allocations * number of processes' and * 'average size of allocation'
- sitkack 7y agoWhitequark is a Wizard. They should team up with Sammy.
- rurban 7y agoWhitequark is better than Sammy in SW. He's also the maintainer of SolveSpace. https://m-labs.hk/software/solvespace/ https://m-labs.hk/software/solvespace/
- saagarjha 7y agoAccording to her Twitter, Whitequark prefers feminine pronouns: https://twitter.com/whitequark https://twitter.com/whitequark
- felipelemos 7y agoBy the parent comment I thought Withequark was a group/team of people. English is also not my native language.
- progval 7y ago"They" can mean either a group of people, or a single person of unknown gender (or neutral gender).
- Yajirobe 7y ago...seriously?
- balnaphone 7y agoFYI, whitequark is a woman.
- ignoramous 7y agoSorry for being an ignorant: Who's Sammy? Surely not: https://news.ycombinator.com/user?id=sammy https://news.ycombinator.com/user?id=sammy ?
- Iv 7y agoIsn't that a bit similar to what debuggers typically do when you ask them to attach to a given process?
- saagarjha 7y agoDebuggers touch other processes from afar. This merges the debuggee into yourself.
- skrebbel 7y agoNo, it merges a copy-on-write clone of the debuggee into yourself. That's quite different and, indeed, you can do similar things with it that a debugger could. If I understand this right,the process being unforked into you won't notice a thing and will happily chime on.
- saagarjha 7y ago> No, it merges a copy-on-write clone of the debugger into yourself. But you are the debugger… > If I understand this right,the process being unforked into you won't notice a thing and will happily chime on. Right, whereas when running an actual debugger you need to deal with signals and making sure you don't touch memory.
- skrebbel 7y ago> But you are the debugger… Ah thanks. My autocomplete didn't like the word "debuggee". Edited!
- ignoramous 7y agoThis should go in the FAQ! Thanks for the concise explainer for folks like me who aren't familiar with the domain.
- apeace 7y ago> Nevertheless, I think that with some effort two allocators or even dynamic linkers could survive together. Famous last words.
- whateveracct 7y agouserfaultfd is an extremely intriguing hammer :)
- fsfod 7y agoThe write protected mode[1], if it ever gets merged could have some interesting uses for GCs. [1] https://lore.kernel.org/patchwork/cover/1033856/ https://lore.kernel.org/patchwork/cover/1033856/
- sanxiyn 7y agoI first learned about userfaultfd's utility to GC from https://medium.com/@MartinCracauer/generational-garbage-collection-write-barriers-write-protection-and-userfaultfd-2-8b0e796b8f7f https://medium.com/@MartinCracauer/generational-garbage-coll...
- whateveracct 7y agoI could imagine it being interesting for VMs and DBs too. Imagine a VM whose memory "looks" like virtual memory but under the hood is transparently persisted between invocations.
- fsfod 7y agoThat's kind of one of the main uses of the API so far by QEMU for live migration of VMs by streaming memory on demand over a network https://wiki.qemu.org/Features/PostCopyLiveMigration https://wiki.qemu.org/Features/PostCopyLiveMigration
- psaux 7y agoSounds like alternatives for Git :) But seriously, need to read more on this.
- woodrowbarlow 7y agoalternative for git? i don't follow. can you elaborate?
- jagrsw 7y agoAnd for something completely different - but in the same vein - a stand-alone 'cd' binary - https://github.com/robertswiecki/extcd https://github.com/robertswiecki/extcd - enjoy!
- 0xcde4c3db 7y agoFor those who aren't familiar with the significance of this, "cd" is a shell builtin because the working directory is per-process state. So while it's perfectly valid to write a program that does a chdir(2) and then exits, it's only changing its own working directory, which is pretty useless.
- Hello71 7y agoisn't this more or less the same as gdb -batch -n -ex 'call chdir("whatever")' -p $$
- vidarh 7y agoThat does seem to be conceptually pretty much what it's doing. Except your version works on more architectures. It's a simple example of ptrace() though.
- CriticalCathed 7y agoThis is a hack. I like it. Refreshing.
- keeganpoppen 7y agoif only "hackernews" had more hacks like this xd
- avodonosov 7y agoDoes it pause the process whose memory it is copying? Freezing the process can affect its correct operation. (Sometimes when I need a memory dump of a production java app, I can't take it because can not afford freezing a production app) Without the freeze, the memory copy we get can be inconsistent.
- aargh_aargh 7y agoI have no idea but the FAQ says it's CoW.
- Liskni_si 7y agoIf I understand it correctly, it's more of a Copy-on-Read/Write, and as opposed to fork, it's only one-sided: read/write is only detected and results in a copy on the unfork side; if the original process changes memory, it doesn't result in a copy as nothing is monitoring this (the userfaultfd only monitors the unfork side).
- avodonosov 7y agoThe ideal approach would be if it turned the original process memory into "copy on write", and created a paused exact copy of that process. This would give a consistent, immutable, snapshot of the target process memory, without freezing for the duration of actual memory copying. One could then take a core dump, java heap dump, or similar, of the paused copy process. I'm curious, why does the tool try to copy the original process memory into the memory of the tool itself, risking a collision? Is it impossible to create a third process - an exact copy of the original process?
- Liskni_si 7y agoThe FAQ says: > all while leaving no ptrace and sending no signal If this is a design goal, I'm afraid it is indeed impossible to take a snapshot of the original process. As far as I know (I researched the status quo 2 years ago when I needed copy-on-write for VM cloning/forking), the only way to make a snapshot of a process' address space is to invoke the clone (fork) system call. If you need to take a snapshot of another process, then you need ptrace. But you're absolutely right that the unfork functionality itself can be implemented more robustly by doing this ptrace/fork trick.
- acoye 7y agoYou could build an entire new array of malware with this :D
- pacman128 7y agoSince it brings two processes together, maybe spoon would be a better name.
- tendencydriven 7y agoI don't know why spoon would be a better name, but I massively approve of it
- greenshackle2 7y agoBecause merging two address spaces together is akin to the act of spooning.
- gjm11 7y agoOr, along the same lines, another four-letter word sharing two of its letters with _fork_. But that might be too distracting.
- deckar01 7y agokrof
- keanebean86 7y agoI was thinking Fnnl (pronounced funnel) But that's probably the name of a startup and would confuse people.
- mmoez 7y agoI know Windows doesn't get too much love here. But we have to admit that Win32 has already this kind of feature since ages: Process access routines such as OpenProcess() [1] coupled with ReadProcessMemory() [2] will do the job in a clean way. Taking a snapshot of other processes is also a basic use case of this family of functions [3]. [1] https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-openprocess https://docs.microsoft.com/en-us/windows/win32/api/processth... [2] https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-readprocessmemory https://docs.microsoft.com/en-us/windows/win32/api/memoryapi... [3] https://docs.microsoft.com/en-us/windows/win32/toolhelp/taking-a-snapshot-and-viewing-processes https://docs.microsoft.com/en-us/windows/win32/toolhelp/taki...
- klodolph 7y agoHow clean is it? Can you simply exec the result? Glibc used to have unexec(), which is fairly old, but it was removed because nobody used it (except Emacs, and there were better solutions to the problem it was solving).
- mmoez 7y ago> How clean is it? It's as clean as any official Win32 API which uses their privilege system to restrict/allow accesses to each and any bit of information on the process state and/or memory. > Can you simply exec the result? This is possible using CreateThread() [1] which creates a remote thread inside another process execution context. [1] https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createthread https://docs.microsoft.com/en-us/windows/win32/api/processth... > Glibc used to have unexec() My understanding is that unexec() was more about making a snapshot of the whole process state to an executable on disk.
- hackworks 7y agoThat is my understanding too. Solaris had a flag for dldump (https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/index.html https://docs.oracle.com/cd/E19455-01/806-0627/6j9vhfmop/inde...). Emacs moved to a portable dumper (maybe inspired from XEmacs)
- koolba 7y ago> How limited is this approach? > A: It's true that meshing address spaces is much harder than copying them. ... [truncated] ... 64-bit systems with ASLR are far more forgiving. Nevertheless, I think that with some effort two allocators or even dynamic linkers could survive together. That is a really cool side effect of ASLR! [1]: https://en.wikipedia.org/wiki/ASLR https://en.wikipedia.org/wiki/ASLR
- khaki54 7y agoSeems like this would be useful for re-attaching to a shell or process you have either disowned or otherwise lost control of. Would be neat to see some common use cases on the FAQ page.
- equalunique 7y agoIs it just me, or is this something that would make Linux even more vulnerable to cyber attacks? What protections are there? Would OpenBSD's pledge prevent something like this?