4 ms·

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

wtbob 10y ago
Even better, move stuff to /trash/$(date -I), and have your cleanup script regularly clean out any directory from before a week ago.
namepros 10y ago
This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.
- yorwba 10y ago
  Renaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
  - deadbunny 10y ago
    It will when moving across disks.
  - traviscj 10y ago
    Beware race conditions when writing to /var/trashlist (assuming you mean "a file with one path per line.") Proposed tweaks: symbolic link into /var/trashlist directory, where the name of the symbolic link is "<timestamp>-<random stub>-<original basename>". Timestamp first so we can stop once we hit the first too-recent timestamp, random stub to unique the original base name if two different files In different directories are deleted at the same timestamp, original file name for inspection.
jjirsa 10y ago
And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it
- deadbunny 10y ago
  Cron job with a find command fixes that.
  - jjirsa 10y ago
    Cron job doesn't fix anything. You're at 90%, and you have an app error spewing a few MB of logs per minute. Your on-call engineer /bin/rm's the logs, and instead of going to 30%, you're still at 91%, only the files are gone. Your engineer (rightfully) thinks "maybe the files are still on disk, and there's a filehandle holding the space", so instead of checking /proc to confirm, he bounces the service. Disk stays full, but you've incurred a few seconds of app downtime for no reason, and your engineer is still confused as shit. Your cron job won't kick in for hours? Days? In the mean time, you're still running out of disk, and sooner or later you'll hit 100% and have a real outage. Cron job is a stupid hack. It doesn't solve any problems that aren't better solved a dozen other ways.
    - deadbunny 10y ago
      That engineer should have read the documentation. Failing that a `du -a / | sort -n -r` would immediately show the conclusion you jumped to was wrong. Randomly bouncing services on a production machine is a cowboy move. No documentation, no checklists? That's the source of your problem, not a trash command which moves files rather than deleting them.
      - jjirsa 10y ago
        Changing the meaning of a decades old command is the problem - du / can take hours to run on some systems, and randomly bouncing shit in prod is the type of cowboy shit that happens in a panic (see gitlab live doc). Docs and checklists are fine, but at 2am when the jr is on call, you're asking for problems by making rm == mv
e12e 10y ago
Sounds like what you want is a nilfs2 or similar file-system... http://nilfs.sourceforge.net/en/ http://nilfs.sourceforge.net/en/ And/or a system like etckeeper to help keep a log on top of an fs that doesn't keep one for you: http://etckeeper.branchable.com/ http://etckeeper.branchable.com/
atmosx 10y ago
Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in root@domain.com?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you. The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least minimise it.
- foota 10y ago
  Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?
  - atmosx 10y ago
    The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.
    - throwaway8124 10y ago
      Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....). It lead to a "Very Bad Day". I found out about it after reading the post mortem.
      - lovehashbrowns 10y ago
        One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.
arximboldi 10y ago
This goes on my machines: sudo apt install trash-cli alias rm="echo This is not the command you want to use" This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|
- wmu 10y ago
  Didn't know trash-cli, thanks for sharing.
- pizza234 10y ago
  I used to use the same strategy, but it's important to know that on some configurations, there are issues with the gnome trash specifications/tools: https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1442649 this bug has been open for, uh, 7 years.
riceo100 10y ago
Nice, but in practice I've found that when working with replication on big production DBs I usually don't have the space to hold multiple copies of a data dir that I want to delete, and copying elsewhere takes too long. Not a blocker to using it by any means as at least this adds a natural sanity-checking stop.
dotancohen 10y ago
On some machines I've actually masked `rm` with an echo to remind me not to use it, and I would delete with `rrmm`. That would give me pause to ensure that I really mean to remove what I'm removing, and more importantly, that I didn't type `rm` when I meant `mv` (which I actually have done by accident).

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

wtbob 10y ago

Even better, move stuff to /trash/$(date -I), and have your cleanup script regularly clean out any directory from before a week ago.

namepros 10y ago

This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.

yorwba 10y ago
Renaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
- deadbunny 10y ago
  It will when moving across disks.
- traviscj 10y ago
  Beware race conditions when writing to /var/trashlist (assuming you mean "a file with one path per line.") Proposed tweaks: symbolic link into /var/trashlist directory, where the name of the symbolic link is "<timestamp>-<random stub>-<original basename>". Timestamp first so we can stop once we hit the first too-recent timestamp, random stub to unique the original base name if two different files In different directories are deleted at the same timestamp, original file name for inspection.

jjirsa 10y ago

And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it

deadbunny 10y ago
Cron job with a find command fixes that.
- jjirsa 10y ago
  Cron job doesn't fix anything. You're at 90%, and you have an app error spewing a few MB of logs per minute. Your on-call engineer /bin/rm's the logs, and instead of going to 30%, you're still at 91%, only the files are gone. Your engineer (rightfully) thinks "maybe the files are still on disk, and there's a filehandle holding the space", so instead of checking /proc to confirm, he bounces the service. Disk stays full, but you've incurred a few seconds of app downtime for no reason, and your engineer is still confused as shit. Your cron job won't kick in for hours? Days? In the mean time, you're still running out of disk, and sooner or later you'll hit 100% and have a real outage. Cron job is a stupid hack. It doesn't solve any problems that aren't better solved a dozen other ways.
  - deadbunny 10y ago
    That engineer should have read the documentation. Failing that a `du -a / | sort -n -r` would immediately show the conclusion you jumped to was wrong. Randomly bouncing services on a production machine is a cowboy move. No documentation, no checklists? That's the source of your problem, not a trash command which moves files rather than deleting them.
    - jjirsa 10y ago
      Changing the meaning of a decades old command is the problem - du / can take hours to run on some systems, and randomly bouncing shit in prod is the type of cowboy shit that happens in a panic (see gitlab live doc). Docs and checklists are fine, but at 2am when the jr is on call, you're asking for problems by making rm == mv

e12e 10y ago

Sounds like what you want is a nilfs2 or similar file-system... http://nilfs.sourceforge.net/en/ http://nilfs.sourceforge.net/en/ And/or a system like etckeeper to help keep a log on top of an fs that doesn't keep one for you: http://etckeeper.branchable.com/ http://etckeeper.branchable.com/

atmosx 10y ago

Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in root@domain.com?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you. The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least minimise it.

foota 10y ago
Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?
- atmosx 10y ago
  The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.
  - throwaway8124 10y ago
    Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....). It lead to a "Very Bad Day". I found out about it after reading the post mortem.
    - lovehashbrowns 10y ago
      One day, I almost accidentally shut off our master DB so I could update it. It's funny.. I leave these types of tasks for later in the day because I'm going to be tired and they're easy & quick tasks. But that almost backfired on me that day; I read the hostname a few times before it fully hit me that I wasn't where I was supposed to be.

arximboldi 10y ago

This goes on my machines: sudo apt install trash-cli alias rm="echo This is not the command you want to use" This way I managed to get unlearn my rm trigger-happyness and use trash instead. I had too many incidents already... ':|

wmu 10y ago
Didn't know trash-cli, thanks for sharing.
pizza234 10y ago
I used to use the same strategy, but it's important to know that on some configurations, there are issues with the gnome trash specifications/tools: https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1442649 this bug has been open for, uh, 7 years.

riceo100 10y ago

Nice, but in practice I've found that when working with replication on big production DBs I usually don't have the space to hold multiple copies of a data dir that I want to delete, and copying elsewhere takes too long. Not a blocker to using it by any means as at least this adds a natural sanity-checking stop.

dotancohen 10y ago

On some machines I've actually masked `rm` with an echo to remind me not to use it, and I would delete with `rrmm`. That would give me pause to ensure that I really mean to remove what I'm removing, and more importantly, that I didn't type `rm` when I meant `mv` (which I actually have done by accident).