12 ms·
Unraveling rm: what happens when you run it?
- msla 9y agoFor the curious, here's what dtruss is: https://www.freebsd.org/cgi/man.cgi?query=dtruss&sektion=1&manpath=FreeBSD+8.2-RELEASE https://www.freebsd.org/cgi/man.cgi?query=dtruss&sektion=1&m... The dtruss utility traces system calls and (optionally) userland stack traces for the specified programs.
- bhuga 9y agoWhat a cool way to learn. I've been doing this 20 years and it never occurred to me to just unravel some of the basic tools I use every day and see if it matches my expectations.
- msla 9y agoI particularly like using strace/dtrace/dtruss to show the communications between the user process and the kernel. You can learn some very interesting system calls like that, and see how experienced programmers use them. There's another tool of that variety: ltrace shows calls to functions in dynamically-loaded libraries. It... works, mostly, but it doesn't know things like function type signatures, so it has to guess, and sometimes gets it wrong in weird ways. It also doesn't know defines and enums, so it can't turn numbers back into symbolic constants like strace and its close kin can: https://en.wikipedia.org/wiki/Ltrace https://en.wikipedia.org/wiki/Ltrace
- jwilk 9y agoFor tracing functions, there's also latrace: https://people.redhat.com/jolsa/latrace/index.shtml https://people.redhat.com/jolsa/latrace/index.shtml
- lathiat 9y agoThanks for the link, I've mostly been focussing on perf and bpf related things but TIL about LD_AUDIT http://man7.org/linux/man-pages/man7/rtld-audit.7.html http://man7.org/linux/man-pages/man7/rtld-audit.7.html
- gbrown_ 9y ago> To allow it to process the rm command, I had to make a copy of the executable into a temporary directory and execute that. To be clear this is due to SIP.
- y4mi 9y agoYou should really spell out the full term if the acronym is predominantly used somewhere else. Unless I'm wrong and you're actually talking about the session initiation protocol (voice over ip)... But that makes little sense It's probably System Integrity Protection, right?
- jessemillar 9y agoI believe you are correct. System Integrity Protection is likely what they meant.
- deleted 9y ago[deleted]
- floatboth 9y agoI wonder why did Apple block tracing SIP protected programs? Like, wasn't SIP supposed to protect against their modification on disk, like NetBSD's veriexec? How would DTracing rm be dangerous?
- saagarjha 9y agoBecause then people could trace rm in a way that allowed them to run arbitrary code in that process. It's the same issue you'd have if you attached a debugger to the process or loaded a dynamic library in their address space.
- lathiat 9y agoThat is not true, dtrace cannot modify data and is specifically designed not to do so. It can however be used to leak information or read info out of other processes. I was going to say that for this reason even on linux you can't ptrace processes that are not children of the current process (e.g. you can run something under strace, but not attach to an existing process unless you twiddle a flag or do it as root). Having said that, you CAN modify data with ptrace, unlike dtrace. So that's kindof an aside. In any case the idea is that one process can't hijack another even from the same user for ptrace.
- acbart 9y agoCool "anthropology" method. This triggered my interest to go look at the C source code for linux[1]. Turns out the command actually relies on a few different files, including a "remove.h"[2]. I was surprised I didn't immediately see the call to unlink in the source code, but obviously there's a lot of useful infrastructure to build on. Further investigation that the unlinkat[3] syscall was actually being used. Minorly fascinating :) [1] https://github.com/coreutils/coreutils/blob/master/src/rm.c https://github.com/coreutils/coreutils/blob/master/src/rm.c [2] https://github.com/coreutils/coreutils/blob/master/src/remove.c https://github.com/coreutils/coreutils/blob/master/src/remov... [3] https://linux.die.net/man/2/unlinkat https://linux.die.net/man/2/unlinkat
- zokier 9y ago> This triggered my interest to go look at the C source code for linux[1]. Coreutils is not Linux, and GNU tools are notoriously heavyweight. For contrast, here is Toybox implementation of rm: https://github.com/landley/toybox/blob/master/toys/posix/rm.c https://github.com/landley/toybox/blob/master/toys/posix/rm.... busybox: https://git.busybox.net/busybox/tree/coreutils/rm.c https://git.busybox.net/busybox/tree/coreutils/rm.c https://git.busybox.net/busybox/tree/libbb/remove_file.c https://git.busybox.net/busybox/tree/libbb/remove_file.c And finally openbsd: https://github.com/openbsd/src/blob/master/bin/rm/rm.c https://github.com/openbsd/src/blob/master/bin/rm/rm.c
- masklinn 9y agoAnd the actual code for OSX: https://opensource.apple.com/source/file_cmds/file_cmds-264.50.1/rm/rm.c.auto.html https://opensource.apple.com/source/file_cmds/file_cmds-264....
- dfox 9y agoI was somewhat surprised by the small number of process startup/libc initialization syscalls in the trace in original article. On Linux (Debian unstable in my case) you will get order of magnitude more, because rm is dynamically linked (although it seems that only with libc and nothing else), because of libc startup (did you know that linux has amd64-specific syscall arch_prctl(PRCTL_SET_FS), that does exactly what it sounds like?). And then because core utils rm cares about such things as whether stdin is terminal (probably because -f/-i behavior changes depending on that) and does the actual unlink in somewhat convoluted way that involves fstatat() (called twice, for some reason) and only then unlinkat(). Somewhat notably last thing that rm does is trying to lseek() stdin only to get ESPIPE...
- Hello71 9y ago> The first couple of lines in the trace seem to be pretty clearly related to setting up the sudo part of the command. This is not correct. "sudo dtruss" makes sudo run dtruss, so it is not possible for dtruss to track what sudo is doing. You can see what "sudo" does (which is much more complicated) by running "sudo dtruss sudo". You need sudo twice because sudo is a setuid program, and obviously it would be insecure to allow anybody to trace a setuid program (for example, it has to read the shadow file, so if you could trace it then you could dump all the hashes and crack them on your own time). On a separate note, it's probably a bad idea to go reading Linux man pages and expect those to give you accurate information about Mac system calls.
- jwilk 9y agoI don't think OP wanted to trace sudo. But then I'm not sure what sudo was for... Does dtruss require root privileges?
- cowsandmilk 9y agoif you run without sudo, you get "dtrace: failed to initialize dtrace: DTrace requires additional privileges"
- temprature 9y agoIndeed, surely Mac OS includes its own man pages? man 2 mprotect, man 2 unlink
- WhiteOwlLion 9y agorm -rf / is a favorite command of mine.
- zokier 9y agoOne thing I've been thinking is if there would be value in (obviously non-POSIXy) userland that would mirror more closely the underlying syscalls. I'm not sure how it would exactly look like, and obviously there would be still need for higher level utilities too.
- jwilk 9y agoThere's some precedent for that: POSIX has "link" and "unlink" utilities, which just call the corresponding functions. Most people use higher-level "ln" and "rm".
- jwilk 9y ago> The getpid command has a pretty self-explanatory name, but I wondered what the parameters that were passed to the function were. Does anyone know what's going on here? It looks like all syscalls are shown to have 3 arguments, even when they wouldn't need that many... Except close() which is shown to have only one.
- zwp 9y agoThis appears to be an artifact of dtruss(1). Syscalls that aren't special-cased are assumed to have 3 arguments: https://github.com/joyent/dtruss-osx/blob/c3b23e279187056e76c8ad02057d98b67f760290/dtruss#L237 https://github.com/joyent/dtruss-osx/blob/c3b23e279187056e76... There are special cases for 0..6 arguments (including one for close(2)) so I guess getpid(2) slipped through the cracks.
- cat199 9y agonot sure how much macos retains of this, but bsd derivatives essentially convert userland libc 'syscalls' into to a call to a single lower level 'syscall' function which passes data to/from the kernel using a macro / integer list to determine which actual functionality is desired.. https://www.freebsd.org/cgi/man.cgi?query=syscall&sektion=2 https://www.freebsd.org/cgi/man.cgi?query=syscall&sektion=2
- mark-wagner 9y ago> As it turns out, the getpid function supposedly takes no parameters, so the inclusion of the parameters in the call above perplexed me. This is due to the fact that dtruss is actually a shell script trying to emulate truss with a d script and not quite succeeding. See the section starting with the comment "print 0 arg output" in dtruss if you wish to understand the meaning of the 3 "arguments." getpid() is most likely being called with 0 arguments.
- andrewla 9y agoWhat I found most interesting from working on some FUSE filesystems (and from the git pseudo-filesystem) is that removing a file via unlink is not a file operation at all, but an operation on the parent directory. The only way that the filesystem knows how to find a file (an inode) is by finding it in a directory listing (which is itself a filesystem object). The name itself is the giveaway -- "unlink" because you're removing a hard link to a file. Similarly, access permissions are also properties of the embedding in the directory, rather than the bits of the file itself. This is a place where POSIX and Win32 diverge significantly -- in Win32, permissions and access happen at the file level, which is why Windows is testy about letting you delete a file that is in use, while POSIX doesn't care -- the process accessing the file, once the file is open, maintains a link to the inode, and all the file data is intact, just not findable in the directory where it was initially located. A neat trick here is that you can effectively still access the file (even restore it) if a process has the file open, through the /proc filesystem.
- jwilk 9y ago> access permissions are also properties of the embedding in the directory, rather than the bits of the file itself. No, in POSIX they are properties of the file. You can use fchmod() and fchown() to change mode and ownership via an fd. > A neat trick here is that you can effectively still access the file (even restore it) if a process has the file open, through the /proc filesystem. Yes, you can access them; but I don't believe you can link them. (But I'd love to proven wrong. A few years ago I actually needed to un-unlink a non-regular file that was still open.)
- dfox 9y agoYou certainly cannot un-unlink file by linking the pseudofile in /proc somewhere, as that would involve cross-filesystem hardlinks. But it is probably possible to write simple kernel module that would allow you to do that through some non-standard interface.
- andrewla 9y agoI don't think the cross-filesystem hardlink is the problem, since the link in proc is a symbolic link. I can do this experimentally by creating a symbolic link in /tmp (a different filesystem) to a file in /home, and then creating a hard link (with ln -L) from the symbolic link to another file in /home, and the result is a valid hardlink to the same inode as the original file. This doesn't work through /proc for an unlinked file, but only because the underlying link call requires a path, not an inode. You can create a hard link out of proc if the file has not been deleted, though, without any cross-filesystem problems.
- bsder 9y agoThere is an old story floating on the net where a sysadmin did the fateful recursive "rm" on the root directory and managed to hit CTRL-C before it got too far. He then had to jump through a bunch of hoops and use a bunch of strange commands to restore / because of all the missing binaries. I wish I could find it.
- floatboth 9y agoThere's probably more than one of these posts :) Here's the one I found in my bookmarks: http://lambdaops.com/rm-rf-remains/ http://lambdaops.com/rm-rf-remains/
- cesarb 9y agoI do recall reading that story too. It's not the story you want, but the origin story of the ext3grep tool is interesting too: https://web.archive.org/web/20110529114328/https://carlo17.home.xs4all.nl/howto/undelete_ext3.html https://web.archive.org/web/20110529114328/https://carlo17.h...
- nzmsv 9y agohttps://www.ee.ryerson.ca/~elf/hack/recovery.html https://www.ee.ryerson.ca/~elf/hack/recovery.html
- deathanatos 9y agoI had to do a sort of version of this on a machine on mine once. Its hard disk was failing, and it couldn't I/O half of the system-critical binaries, but it did still have some in disk cache. It doubled as my network's SSH gateway/proxy for when I'm coming from an IPv4-only network, as my home only gets a single IPv4 address. > I had a handy program around (doesn't everybody?) for converting ASCII hex to binary, and the output of /usr/bin/sum tallied with our original binary. I copied busybox's nc onto it (needed for the ssh jump); had to copy to /run as / was unwritable due to the disk failure. Now, scp no longer ran, so "copy" was a Python script to turn the binary into a printf command, which is a shell built-in and can write arbitrary binary. (If you ever get into a jam like this, busybox's utilities are very useful.) > But hang on---how do you set execute permission without /bin/chmod? A few seconds thought (which as usual, lasted a couple of minutes) suggested that we write the binary on top of an already existing binary, owned by me...problem solved. I think umask'ing correctly prior to a printf should work nowadays, no? (IDK about in the author's time.) Thankfully in my case, chmod was in disk cache still.
- Theodores 9y agoIf you delete a file in PHP then you 'unlink(file)' rather than 'rm(file)'. Never knew there was a reason, thought it was a quirk.
- jjuhl 9y agoYou are right, in a way, PHP is a quirk.
- iamrohitbanga 9y agoWhy does rm need to call mprotect or ioctl?
- saagarjha 9y agoIt's called by the dynamic loader when it's loading pages into memory.
- saagarjha 9y ago> Some investigation revealed that csops is a system call that is unique to the Apple operating system and can be used to check the signature that is written into a memory page by the operating system. Close. That's one thing that csops can do, but in this case it's being used to extract the entitlements from the binary (CS_OPS_ENTITLEMENTS_BLOB == 7). > I wasn’t sure what the memory addresses that were referenced in the mprotect call actually corresponded to or what the best way to figure it would be. You could fire it up in LLDB and break on mprotect…
- terminado 9y agoFun fact: according to unsubstantiated UNIX lore, "rm" is NOT short-hand for "remove" but rather, it stands for the initials of the developer that wrote the original implementation, Robert Morris. https://en.wikipedia.org/wiki/Robert_Morris_(cryptographer) https://en.wikipedia.org/wiki/Robert_Morris_(cryptographer) I have no proof of this, but through oral tradition, such a tale has been relayed to me. Believe whatever you will.
- AceJohnny2 9y agoConsidering the naming scheme of other basic unix utilities, I'll chalk this one up to "fun coincidence" rather than actual truth. Alternatively, we could make "Robert"/"Bob" a euphemism for nuking files ;) "Yeah I Bob'd the whole build directory" "Bombed?", "No, Bob'd. Like deleted... never mind"
- pjmorris 9y agoObligatory Bobby Tables reference [0]. [0] https://xkcd.com/327/ https://xkcd.com/327/
- schoen 9y agoSome people in computing have definitely gotten their initials into wide distribution: https://en.wikipedia.org/wiki/Mark_Zbikowski https://en.wikipedia.org/wiki/Mark_Zbikowski https://en.wikipedia.org/wiki/Phil_Katz https://en.wikipedia.org/wiki/Phil_Katz ("PK" is not only the beginning of ZIP files but also of, among other things, ODT, DOCX, and JAR files, which are all in turn implemented as ZIP files.)
- winkywooster 9y agoI was curious if this was true, and asked. The answer I got was http://minnie.tuhs.org/pipermail/tuhs/2018-April/013510.html http://minnie.tuhs.org/pipermail/tuhs/2018-April/013510.html and it appears that this isn't true. Specifically the original man page (http://minnie.tuhs.org/cgi-bin/utree.pl?file=V1/man/man1/rm.1 http://minnie.tuhs.org/cgi-bin/utree.pl?file=V1/man/man1/rm....), dated November, 1971, shows Dennis Ritchie and Ken Thompson as the original authors.
- FRex 9y agoIt's very educational to dig into stuff that everyone takes for granted or even try implement your own version that does similar things (at a smaller scale and with less features, performance and security of course, just for educational purposes, not to serve as a real replacement). E.g. writing your own shell sounds hard but you can do it in an hour (of course it'd be very bare compared to even ash), I once did it during a very casual C class while trying to impress the teacher. He even joked it was self hosting when he has seen the end result (as in - I didn't need other shells anymore and could use vim and gcc from my shell so I could use my shell to work on my shell). I wanted to put that into tutorial at some point but there already exists one[0] (that blog seems quite tinker-y actually, it has implementing own Linux sys call too, which is also surprisingly easy to do and I did it for a class at one point too[1]). Going knee deep into this stuff also completely dispels the magic that language and system runtimes, filesystems, file formats, linkers, shells, standard commands (e.g. ls, I had to reimplement it once as a homework) or whatever have, or even better - it's still magic to most people and you're the wizard now! It's also very accomplishing to do something so unique in an hour or two (although to me due to my C and C++ bias at some point making a pastebin clone in an hour in PHP or Python became an unique experience). Even JIT (which sounds scary due to V8, LuaJIT, etc. being so tightly made and complex) is easy to get into and understand at toy scale[2]. [0] - https://brennan.io/2015/01/16/write-a-shell-in-c/ https://brennan.io/2015/01/16/write-a-shell-in-c/ [1] - https://brennan.io/2016/11/14/kernel-dev-ep3/ https://brennan.io/2016/11/14/kernel-dev-ep3/ [2] - http://blog.reverberate.org/2012/12/hello-jit-world-joy-of-simple-jits.html http://blog.reverberate.org/2012/12/hello-jit-world-joy-of-s...
- rrauenza 9y agoI once had to remove a LOT of files, so was looking how to do it efficiently -- find has a -delete option, so I straced it: $ touch a b c d e f g h /tmp/x $ strace find . -delete execve("/usr/bin/find", ["find", ".", "-delete"], [/* 32 vars */]) = 0 [...] openat(AT_FDCWD, ".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY|O_NOFOLLOW) = 4 [...] fcntl(4, F_DUPFD, 3) = 5 fcntl(5, F_GETFD) = 0 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 getdents(4, /* 10 entries */, 32768) = 240 getdents(4, /* 0 entries */, 32768) = 0 close(4) = 0 fcntl(5, F_DUPFD_CLOEXEC, 0) = 4 unlinkat(5, "d", 0) = 0 unlinkat(5, "e", 0) = 0 unlinkat(5, "f", 0) = 0 unlinkat(5, "b", 0) = 0 unlinkat(5, "c", 0) = 0 unlinkat(5, "h", 0) = 0 unlinkat(5, "a", 0) = 0 unlinkat(5, "g", 0) = 0 [...] exit_group(0) = ? +++ exited with 0 +++ https://linux.die.net/man/2/unlinkat https://linux.die.net/man/2/unlinkat int unlinkat(int dirfd, const char *pathname, int flags); If the pathname given in pathname is relative, then it is interpreted relative to the directory referred to by the file descriptor dirfd (rather than relative to the current working directory of the calling process, as is done by unlink(2) and rmdir(2) for a relative pathname). I would expect the reason for this is in case someone is moving directories around while the find is happening ... Each time find enters a directory it doesn't actually chdir(), so it opens that directory and uses it to anchor the removal. rm -r does something similar.
- cat199 9y agowhich is why building from source is handy, if you want to know what something does, just take a look.