7 ms·

Careful with rm: r rm rm - rm -rf rm -rf ~ rm -rf ~/ rm -rf ~/tmp # where did my files go?

by ptman 8y ago

akavel 8y ago
Right, it's theoretically dangerous, or I'd say practically: "please think when using it"... though I personally couldn't find any other serious problematic example other than typing something clearly dangerous in the first place. And I personally found it immensely useful in the other, non-dangerous cases. So I just took the chance and went with the idea, and then wanted to share it with you, so you might also find it useful, maybe! :) Also... umm... you know, a tool isn't really truly Unixy if you can't hurt yourself with it, right? ;P That said, I'm starting to think — should I include a note about the potential dangers in the readme?... they seemed glaringly obvious to me (at least in case of rm-like commands), but can there be readers/users, to whom they'd not be obvious?... edit: Ok, I tried to add a warning in the Readme, that's something I can do easily, and if it helps someone, worth it...
- skybrian 8y ago
  It seems like a whitelist of safe commands would be a good idea? The user can override with -f if they want to live dangerously. Even rm has a -f flag to override some safety measures, though the defaults aren't very safe.
  - drb91 8y ago
    Even better, put it into a non-privileged user by default with no write permission.
    - akavel 8y ago
      Sounds interesting; but is there any way I could change to a different user if not running as root?... also, even if yes, how do I dynamically create a non-privileged user, or find a pre-existing one?
      - gwn7 8y ago
        I would try creating a non-privileged user "up" during installation and switch to that user under the hood before running the command via `su`. (Though it would complicate installation which is effectively non-existent since there is a single binary; and tightly couple the application with the environment. It's a trade-off anyway.) (Edit: Somebody mentioned the user "nobody" which seems to be a better alternative.) In any case I think that the whitelist approach would work better as it would be nice if the tool knew in advance which commands work properly and which ones don't. That way it could inform its users about the allowed & unallowed commands. Commands with side effects (or "impure" in the FP terminology) don't make much sense anyway to be used within this. The main value is fast iteration, in order to verify the expected output of the pipeline. For me it doesn't make much sense to use it with commands whose main purpose is to modify the filesystem instead of generating / transforming data and writing it to stdout. So the criteria for the commands that would make it into the whitelist may be "not having side effects", "writing to stdout", and optionally "reading from stdin". Fastly iterating side effects is unsafe as well, as has already been pointed out. This can be a fine data analysis tool actually, unfortunately only for command line geeks. The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
        seiferteric 8y ago
        I actually like the user idea better. The problem with a whitelist is that even useful commands can have subtle or little known dangerous modes, like "find . -exec rm {}"
        gwn7 8y ago
        Ah. Good point. Whitelisting commands would have already been a bit painful, and now your comment shows that the parameters need to be whitelisted/blacklisted as well, which would be crazy. In a world in which shell commands respected the UNIX philosophy, "find" wouldn't have a silly option like "exec", and other commands wouldn't mix read / write / pure data transform operations in a single command. But it is what it is. So yeah, protection probably needs to be implemented in the user level, for maximum safety. Maybe an alternative and/or complementary solution would be to profile each inputted command to detect if they are attempting write operations (maybe with "strace" or something like that), and cancel the evaluation of the command in the next iterations and/or show a warning.
- CGamesPlay 8y ago
  Nice tool! Your main use case for destroying files is “xargs rm”, which would be used when receiving a piped argument list. Could you have this command run as user “nobody” by default, and have a flag to run as a real user? Or as others have mentioned, use capabilities on the spawned processes to prevent anything other than standard input/output?
- etatoby 8y ago
  I'm not so versant about the state of permission / capability systems on Linux. But if there is a way for you to drop the capability to modify the filesystem before running the first pipeline, you should definitely do it by default (and provide a switch to override it if somebody knows what they are doing.)
somanyquestions 8y ago
yikes! It's always been surprising to me that their isn't a built in "undo" to `rm` e.g. why doesn't `rm` just move files to /tmp?
- clarry 8y ago
  > e.g. why doesn't `rm` just move files to /tmp? And do what when tmp gets full and services & applications start crashing? When someone implements that undo, someone's going to need to write a tool that really does remove files, really. Will the next guy wonder why really-rm doesn't have undo built in?
  - diegoperini 8y ago
    What about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess).
    - hk__2 8y ago
      > What about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess). Same question as parent: "And do what when tmp gets full and services & applications start crashing?"
      - kungtotte 8y ago
        You could do what is more or less an industry standard when it comes to this and warn the user that there's not enough room to allow an undo when deleting said file(s) and prompt if they want to delete it permanently instead. It's not rocket surgery.
  - jxy 8y ago
    I'm never that nervous when doing `rm -fr` on my Plan9 or FreeBSD or macOS systems, which all have frequent snapshots turned on. We could all just work with WORM drives instead.
hinkley 8y ago
I found a new variation recently. Luckily all my code was checked in. Meant to type: rm -rf node_modules && npm install Question: what's immediately next to the & key on a keyboard? Answer: * Actually typed rm -rf node_modules ** npm install Oops.
- nothrabannosir 8y ago
  If you’re on Mac OS X , try installing the “trash” app (from brew iirc) and retrain yourself to never use bare rm anymore. There’s probably similar tools on Linuxes. Theory is: rm is dangerous and should not be used day to day.
  - mynewtb 8y ago
    Or simply have local, quick, regular and tested backups.
  - dzamo_norton 8y ago
    Using btrfs on Linux I have a job that takes a CoW snapshot of the whole filesystem every 10 minutes just so that I can undo whoopsies. I don't retain all of these snapshots, only the most recent one.
ken 8y ago
This is one of the main reasons that I ultimately decided for my Strukt.app to not be built on top of built-in shell commands. For an interactive tool, it's crucial for each operation to be able to report whether it's idempotent, does writeback, etc. Strukt doesn't do 'rm' yet (and nobody has ever requested it), but it does have other operations that do writeback. Once you add one, automatic updates are disabled, and you have to explicitly run the pipeline each time you want it run. We really need a new cross-platform low-level framework for lego-like operations, like shell commands but updated for the 21st century.
- jodrellblank 8y ago
  PowerShell has a design answer for this - each cmdlet takes parameters, but the environment provides common parameters relating to loging, error handling, storing output in variables, and it provides support for "WhatIf" and transactions. WhatIf instructs the command not to make any changes, but merely to log what it would do. You can "Remove-Item C:\Windows -Force -WhatIf" and it will output "What if: Performing the operation "Remove Directory" on target "C:\windows"" but not delete anything. (implementing support for "whatif" is optional, so you can't casually use it on any given cmdlet without first checking that cmdlet supports it and makes sensible use of the parameter instead of ignoring it, but the idea is there).
  - ken 8y ago
    Yep, that's basically the solution that I came up with, too. I have a "dry-run?" flag planned, if I ever end up with enough writeback operations where it'd be useful. (That's the common Unix shell name for it, though it's not entirely consistent -- which is another problem with using Unix built-ins!) One thing I've noticed is that it's really frustrating (and not at all helpful) to require perfect validity from a shell-like tool. There's just too many times where you want to edit something into something else, without needing to find a perfect path of valid and meaningful syntax at every intermediate step. So if you have a flag on an operation that doesn't understand that flag, the flag will have a (?) badge to indicate that it won't be understood or used, but the pipeline will still run.
- 8y ago
djsumdog 8y ago
That's the first thing I thought and it's mentioned in the readme file itself. I don't think I'd ever want to use this tool. Every keystroke could be an arbitrary command, and I wonder about the performance of commands with a lot of I/O or processing. Conceptually it's really neat and I get it, but practically I find things like parameter completion in Fish much more useful.
deleted 8y ago
[deleted]

Careful with rm: r rm rm - rm -rf rm -rf ~ rm -rf ~/ rm -rf ~/tmp # where did my files go?

by ptman 8y ago

Careful with rm: r rm rm - rm -rf rm -rf ~ rm -rf ~/ rm -rf ~/tmp # where did my files go?

Right, it's theoretically dangerous, or I'd say practically: "please think when using it"... though I personally couldn't find any other serious problematic example other than typing something clearly dangerous in the first place. And I personally found it immensely useful in the other, non-dangerous cases. So I just took the chance and went with the idea, and then wanted to share it with you, so you might also find it useful, maybe! :) Also... umm... you know, a tool isn't really truly Unixy if you can't hurt yourself with it, right? ;P That said, I'm starting to think — should I include a note about the potential dangers in the readme?... they seemed glaringly obvious to me (at least in case of rm-like commands), but can there be readers/users, to whom they'd not be obvious?... edit: Ok, I tried to add a warning in the Readme, that's something I can do easily, and if it helps someone, worth it...

skybrian 8y ago
It seems like a whitelist of safe commands would be a good idea? The user can override with -f if they want to live dangerously. Even rm has a -f flag to override some safety measures, though the defaults aren't very safe.
- drb91 8y ago
  Even better, put it into a non-privileged user by default with no write permission.
  - akavel 8y ago
    Sounds interesting; but is there any way I could change to a different user if not running as root?... also, even if yes, how do I dynamically create a non-privileged user, or find a pre-existing one?
    - gwn7 8y ago
      I would try creating a non-privileged user "up" during installation and switch to that user under the hood before running the command via `su`. (Though it would complicate installation which is effectively non-existent since there is a single binary; and tightly couple the application with the environment. It's a trade-off anyway.) (Edit: Somebody mentioned the user "nobody" which seems to be a better alternative.) In any case I think that the whitelist approach would work better as it would be nice if the tool knew in advance which commands work properly and which ones don't. That way it could inform its users about the allowed & unallowed commands. Commands with side effects (or "impure" in the FP terminology) don't make much sense anyway to be used within this. The main value is fast iteration, in order to verify the expected output of the pipeline. For me it doesn't make much sense to use it with commands whose main purpose is to modify the filesystem instead of generating / transforming data and writing it to stdout. So the criteria for the commands that would make it into the whitelist may be "not having side effects", "writing to stdout", and optionally "reading from stdin". Fastly iterating side effects is unsafe as well, as has already been pointed out. This can be a fine data analysis tool actually, unfortunately only for command line geeks. The experience is actually not so far from analyzing data with VIM interactively by piping it to an external UNIX command and getting the result dataset back into the editor. VIM hackers will know :)
      - seiferteric 8y ago
        I actually like the user idea better. The problem with a whitelist is that even useful commands can have subtle or little known dangerous modes, like "find . -exec rm {}"
        gwn7 8y ago
        Ah. Good point. Whitelisting commands would have already been a bit painful, and now your comment shows that the parameters need to be whitelisted/blacklisted as well, which would be crazy. In a world in which shell commands respected the UNIX philosophy, "find" wouldn't have a silly option like "exec", and other commands wouldn't mix read / write / pure data transform operations in a single command. But it is what it is. So yeah, protection probably needs to be implemented in the user level, for maximum safety. Maybe an alternative and/or complementary solution would be to profile each inputted command to detect if they are attempting write operations (maybe with "strace" or something like that), and cancel the evaluation of the command in the next iterations and/or show a warning.
CGamesPlay 8y ago
Nice tool! Your main use case for destroying files is “xargs rm”, which would be used when receiving a piped argument list. Could you have this command run as user “nobody” by default, and have a flag to run as a real user? Or as others have mentioned, use capabilities on the spawned processes to prevent anything other than standard input/output?
etatoby 8y ago
I'm not so versant about the state of permission / capability systems on Linux. But if there is a way for you to drop the capability to modify the filesystem before running the first pipeline, you should definitely do it by default (and provide a switch to override it if somebody knows what they are doing.)

somanyquestions 8y ago

yikes! It's always been surprising to me that their isn't a built in "undo" to `rm` e.g. why doesn't `rm` just move files to /tmp?

clarry 8y ago
> e.g. why doesn't `rm` just move files to /tmp? And do what when tmp gets full and services & applications start crashing? When someone implements that undo, someone's going to need to write a tool that really does remove files, really. Will the next guy wonder why really-rm doesn't have undo built in?
- diegoperini 8y ago
  What about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess).
  - hk__2 8y ago
    > What about undo that expires in 1 minutes? Gmail website does that for sent emails (for 5 seconds I guess). Same question as parent: "And do what when tmp gets full and services & applications start crashing?"
    - kungtotte 8y ago
      You could do what is more or less an industry standard when it comes to this and warn the user that there's not enough room to allow an undo when deleting said file(s) and prompt if they want to delete it permanently instead. It's not rocket surgery.
- jxy 8y ago
  I'm never that nervous when doing `rm -fr` on my Plan9 or FreeBSD or macOS systems, which all have frequent snapshots turned on. We could all just work with WORM drives instead.

hinkley 8y ago

I found a new variation recently. Luckily all my code was checked in. Meant to type: rm -rf node_modules && npm install Question: what's immediately next to the & key on a keyboard? Answer: * Actually typed rm -rf node_modules ** npm install Oops.

nothrabannosir 8y ago
If you’re on Mac OS X , try installing the “trash” app (from brew iirc) and retrain yourself to never use bare rm anymore. There’s probably similar tools on Linuxes. Theory is: rm is dangerous and should not be used day to day.
- mynewtb 8y ago
  Or simply have local, quick, regular and tested backups.
- dzamo_norton 8y ago
  Using btrfs on Linux I have a job that takes a CoW snapshot of the whole filesystem every 10 minutes just so that I can undo whoopsies. I don't retain all of these snapshots, only the most recent one.

ken 8y ago

This is one of the main reasons that I ultimately decided for my Strukt.app to not be built on top of built-in shell commands. For an interactive tool, it's crucial for each operation to be able to report whether it's idempotent, does writeback, etc. Strukt doesn't do 'rm' yet (and nobody has ever requested it), but it does have other operations that do writeback. Once you add one, automatic updates are disabled, and you have to explicitly run the pipeline each time you want it run. We really need a new cross-platform low-level framework for lego-like operations, like shell commands but updated for the 21st century.

jodrellblank 8y ago
PowerShell has a design answer for this - each cmdlet takes parameters, but the environment provides common parameters relating to loging, error handling, storing output in variables, and it provides support for "WhatIf" and transactions. WhatIf instructs the command not to make any changes, but merely to log what it would do. You can "Remove-Item C:\Windows -Force -WhatIf" and it will output "What if: Performing the operation "Remove Directory" on target "C:\windows"" but not delete anything. (implementing support for "whatif" is optional, so you can't casually use it on any given cmdlet without first checking that cmdlet supports it and makes sensible use of the parameter instead of ignoring it, but the idea is there).
- ken 8y ago
  Yep, that's basically the solution that I came up with, too. I have a "dry-run?" flag planned, if I ever end up with enough writeback operations where it'd be useful. (That's the common Unix shell name for it, though it's not entirely consistent -- which is another problem with using Unix built-ins!) One thing I've noticed is that it's really frustrating (and not at all helpful) to require perfect validity from a shell-like tool. There's just too many times where you want to edit something into something else, without needing to find a perfect path of valid and meaningful syntax at every intermediate step. So if you have a flag on an operation that doesn't understand that flag, the flag will have a (?) badge to indicate that it won't be understood or used, but the pipeline will still run.
8y ago

djsumdog 8y ago

That's the first thing I thought and it's mentioned in the readme file itself. I don't think I'd ever want to use this tool. Every keystroke could be an arbitrary command, and I wonder about the performance of commands with a lot of I/O or processing. Conceptually it's really neat and I get it, but practically I find things like parameter completion in Fish much more useful.

deleted 8y ago

[deleted]