19 ms·

Man accidentally 'deletes his entire company' with one line of bad code

233 points by bhartzer 11y ago

bhartzer 11y ago
He even deleted the backups. >>the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.
- drzaiusapelord 11y ago
  If you're not doing offsite and cold backups, then you're just asking for trouble. If not crap like this then a fire or a ransomeware infection or a malicious employee, etc.
  - jasonjei 11y ago
    He actually was doing a remote backup (although probably not a cold backup). Unfortunately, he had used mount instead of rsync over ssh, making it vulnerable to the rm -rf command.
    - zyxley 11y ago
      That's not a backup, it's a mirror.
      - awqrre 11y ago
        a mirror is kind of a backup if you don't update it live... but of course you should have other backups that are offline.
      - jasonjei 11y ago
        Are you suggesting that you can't backup with rsync? Because you can do full and incremental backups with rsync. In fact Time Machine on OS X looks like it does backups in this manner...
        cyphar 11y ago
        Just using rsync to make copies isn't a backup. If you use rsnapshot (which stores each copy separately) then you have a backup. Copies are not sufficient if you find out that something broke three weeks ago.
  - satysin 11y ago
    If you are not doing offline and offsite backups you are not doing backups at all.
- deleted 11y ago
  [deleted]
- rlpb 11y ago
  In other words, he didn't have backups. A live copy of a running system available to that running system is not a backup.
  - Houshalter 11y ago
    Well it still prevents against the hard drive failing. It doesn't protect against bad code, which he obviously didn't consider.
    - laumars 11y ago
      RAID parity drives also protect against a drive failing but they're not backups either.
      - Houshalter 11y ago
        If it's a duplicate copy of data intended in case of failure then yes it is a backup. Its not an offsite backup, but many people don't keep their personal backups offsite.
        rlpb 11y ago
        It is so deficient as a backup that I don't think it qualifies to be called a backup. That was my point. Backups are expected to protect against data loss for a number of different failure cases (eg. disk failure, hardware fault leading to slow filesystem corruption, fire/theft, failed upgrade, "undo" for accidental change or deletion). There is a point where something addresses so few of these failure cases that you can't reasonably call it a backup.
        laumars 11y ago
        There's another term for what it is: redundancy. Redundancy is there for fast recovery times (even zero downtime depending on how redundancy is implemented). It's not intended to run as a backup as redundancy devices are live and can fail from many of the same causes that will take your primary devices offline (fire, sysadmin fail, etc) Likewise, if your "backups" are always online then it works better for business continuity than it does as a backup. So realistically it's more of a redundancy share.
sp332 11y ago
This felt a bit unlikely, but what really convinced me it was a troll was a follow-up comment where he said he accidentally switched "if" and "of" in a dd command.
- mturmon 11y ago
  Yes. To wit: "I swapped if and of while doing dd. What to do now?" – Marco Marsala Apr 11 at 7:02
- chris_wot 11y ago
  If he mounted his backup media and wiped it, what makes you think he couldn't cockup the dd command? However, under Linux rm -rf / needs --no-preserve-root to work, right?
  - aroch 11y ago
    Depends on which version of `rm` you have. Newer versions (rm from coreutils 8.xx I believe) use no-preserve-root
    - profmonocle 11y ago
      And the question is tagged centos 7, which is from 2014 - far too recent to not have --no-preserve-root.
      - JdeBP 11y ago
        https://git.centos.org/log/rpms!coreutils.git/refs!heads!c7 https://git.centos.org/log/rpms!coreutils.git/refs!heads!c7 indicates that CentOS 7 has coreutils 8.22 . http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=34e3ea055721ecc72e6f636700e8ba7b58069c65 http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit... shows that GNU coreutils gained that option in 2003.
  - astrodust 11y ago
    It's distribution dependent.
twvisitavisitb 11y ago
Since I didn't find any links in the article, here's the original post: http://serverfault.com/questions/769357/recovering-from-a-rm-rf http://serverfault.com/questions/769357/recovering-from-a-rm...
- kamjam 11y ago
  It's in the 2nd paragraph, linked from "called Server Fault" text.
AdmiralAsshat 11y ago
This question that went unanswered in the replies bears repeating: Any idea why the command actually ran? If $foo and $bar were both undefined, rm -rf / should have errored out with the --no-preserve-root message. The only way I can think of that this would have actually worked on a CentOS7 machine is if $bar evaluated to , so what was run was rm -rf /. As the above notes, I'm pretty sure recent versions of Redhat/CentOS actually protect against this sort of thing. On the offchance you're not running a recent server, however, this could also be avoided by using `set -u` in the bash script, as it would cause undefined variables to error out.
- LukeShu 11y ago
  Before you apply `set -u` to all of your scripts, be aware that an empty array counts as undefined. So if you have arrays for which being empty is valid, be sure to `set +u` right before accessing them (and then `set -u` again after).
  - anderskaseorg 11y ago
    Or write ${arr+"${arr[@]}"}, which is, you know, only slightly more awkward than bash arrays already were to begin with. Really you just want to use a real programming language instead.
- harryf 11y ago
  I believe those variables were not handled by the shell but rather in an Ansible "playbook" - see http://docs.ansible.com/ansible/playbooks_variables.html http://docs.ansible.com/ansible/playbooks_variables.html i.e. the variables where happening in a Jinja template and because undefined, rm -rf {foo}/{bar} was transformed by the template engine into rm -rf /
  - Torgo 11y ago
    the playbook will fail if there are undefined variables, I find the story suspect.
amelius 11y ago
+1 for snapshotting filesystems.
- creshal 11y ago
  Or any backup solution worth its salt, including "RAID1 where you yank out and replace one of the drives every other week".
  - robinson-wall 11y ago
    RAID is not a backup. ... Especially when your RAID is busy rebuilding for N hours every other week.
    - daveguy 11y ago
      Well RAID 1 would be a backup if you yanked out and replaced a drive every week. In that case it would be a weekly snapshot.
      - DDub 11y ago
        Unless you're mirroring across more than 2 drives, you have an AID setup.
        daveguy 11y ago
        You are incorrect. RAID 1 is a mirror setup. There are two drives with exactly the same information. One of the two drives is redundant. RAID 1 does not include striping and only requires 2 drives for redundancy.
        Sanddancer 11y ago
        I think what DDub is getting at is that there is no redundancy for the data received while the disk is mirrored to the new twin. For that, you'd need a mirrored pair plus a drive to yank out as the backup.
spriggan3 11y ago
I don't believe that's the truth, for a second. Of course the independent didn't look at the company in question to see if there was any litigation between this guy and his customers.
- chris_wot 11y ago
  The Independent also didn't look up -r as it stands for recursively remove directories...
  - dave2000 11y ago
    They said: "the r deletes everything within a given directory" which is what it does. Non technical readers aren't going to understand "recursively remove directories".
    - chris_wot 11y ago
      Fair point.
TheCams 11y ago
Sorry if that's a stupid question, but does that mean he was running his script as root?
- smhenderson 11y ago
  He was running with full administrative permissions and thus file access permissions on files were ignored. "root" is the name of the default administrative account on Unix and Unix like systems.
  - TheCams 11y ago
    Thanks :)
ikeboy 11y ago
Reminds me of https://archive.is/9R2j8 https://archive.is/9R2j8 (Original thread has been deleted.)
- kbenson 11y ago
  That just makes me really,really want to know what the original company was and what product they were selling.
  - ikeboy 11y ago
    Some kind of botnet access, probably.
- bpchaps 11y ago
  That's awesome. A company I left a while back recently had two servers accidentally rebooted through sort of automated task (probably puppet). The fine, I'm told, was one billion dollars. Someway, somehow, he still works there. :)
  - ikeboy 11y ago
    Details, company name?
    - bpchaps 11y ago
      It's a very large French bank. I don't know anything except that it was a paired batch processing server and didn't push further questions, honestly. For some reason, I found absolutely nothing in the news about it, but my source of information is credible. It doesn't surprise me for a second that it happened, since, for example, I spent months trying to get these guys to fix their literally useless MQ DR failover scripts, but nothing ever came from it, since, they didn't have anywhere to test. With the way they treat their employees - fucking good riddance. There was a giant mess when Disney forced their NOC to train their replacements, but yet these guys did the exact same thing, plus some, and there was no public awareness during or after it. The best part was their push to move everyone to Montreal. Lower pay, not a guaranteed extension and you're forced to move? Okay. The AMRS CTO actually left about a month after he got the position and took me along with one other person over to a new company. Goldman's head of tech actually just left to go to the same place. Not gonna lie, it sounds incredibly suspicious, especially considering the the kinds of shenanigans that went on there... thankfully I'm no longer working there. It's a very, very strange place in finance.
oluwie 11y ago
Reminds me of the time I accidentally typed in 'crontab -d' instead of 'crontab -e'. Those two letters are eerily too close to eachother.
- laumars 11y ago
  I've managed that one as well. One month into a new job and working on servers with complex applications installed that I wasn't yet familiar with. Thankfully I had `crontab -l` just beforehand, otherwise I'd have been screwed.
  - hrktb 11y ago
    Most admins have a text file somewhere with the contents of the crontab. Specially because these things happen more often than it should.
    - laumars 11y ago
      Indeed. I do generally keep backups of crontabs, not just in case of this kind of scenario but also in case the platform blows up in any unexpected ways. Sadly the company in question didn't. However I have since made it a personal policy to always -l before editing so I have a "backup" in my tmux scroll back (that time I mentioned before, it was pure chance that I had -l)
- wahnfrieden 11y ago
  This is one of the reasons why we have infrastructure as code now, so system changes can be reviewed and tested just like application code, and more types of accidents can be reverted via source control :)
  - djsumdog 11y ago
    In the article the guy is using ansible. He even had off-site backups, but they were mounted before his ansible playbooks ran, wiping them out as well.
- 11y ago
castratikron 11y ago
The bash -e and -u options might have saved him here: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/
- giovannibajo1 11y ago
  This. All my scripts begin with "set -euo pipefail", and my editor linter complains loudly if that line isn't there. I wish distros would migrate to making those settings the default, over the years. Even if it would take a while, I think it would be priceless
redbeard0x0a 11y ago
Too bad he wasn't running Illuminos (OpenSolaris) based servers (or even some Solaris versions) that would have just flat out refused to run rm -rf /
- creshal 11y ago
  Recent-ish versions of GNU rm do the same. However, it only protects against `rm -rf /`, not `rm -rf /*`, as the latter is expanded by the shell.
- neerdowell 11y ago
  It's required in POSIX 1003.1-2013 that rm refuse to remove the root directory[0]. [0] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/rm.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/rm...
  - deleted 11y ago
    [deleted]
- 4ad 11y ago
  It's illumos, not Illuminos.
l0c0b0x 11y ago
...and this is why we include a 'backup technology' question in our technical interviews--where 'offsite backup' must follow with something like "possibly the most important type of backup because..."
- AdmiralAsshat 11y ago
  You know the sad thing is that even this isn't idiot-proof and needs to be qualified. One of my customer's brilliant "cost-saving" measures was to have an offsite backup solution that was basically an rsync script that ran every 15 minutes. So when someone on their end did something catastrophic to their data and it took them an hour to notice, they were incredulous that we couldn't help them restore their data even though it was "backed up offsite!" because their "backup" solution had already caught up and duplicated the broken data.
  - ansible 11y ago
    And that's why if you're using rsync, you ought to be using rsnapshot instead, and have generations of backups so that you are not overwriting your most recent one.
    - SmellyGeekBoy 11y ago
      I find rdiff-backup is great as a drop-in replacement.
  - cyphar 11y ago
    Yeah, a copy is not a backup. It's a mistake too many people make. You need to have historical backups.
ams6110 11y ago
Years ago, I worked in an investment bank and we had a programmer put a batch program into production that executed the following as a shell command: rm -rf foo / It was supposed to be: rm -rf foo/ It didn't run as root, but still managed to wipe out all the business data files. What saved us was that the servers were configured with RAID 1 and before the start of the nightly batch cycle, the mirror was "split" and only one copy mounted. So we just had to restore the missing files from the other half of the mirror to revert to the start of the batch window and rerun the entire night's jobs.
- ArkyBeagle 11y ago
  So in other words, the acceptance protocol for this script was inadequate. In the minicomputer era, it was common for a programmer to be required to run it on this one poor donkey of a machine to make it caught nothing on fire before moving to the big machine.
  - ams6110 11y ago
    It was the 1990s, not that it's an excuse but TDD wasn't a "thing" yet, nor was version control (at least not in that shop). For every change to a program we printed the diff, attached a cover memo, and filed it in a cabinet. Yes, there were test systems and programers were supposed to test all their changes, but they also were the ones who deployed their own changes to production so there were ways for this to happen pretty easily.
- gnarbarian 11y ago
  What happened to the programmer?
  - ams6110 11y ago
    Nothing. She continued to work there until after I moved on.
    - mfoy_ 11y ago
Bahamut 11y ago
This made me think of this quote: https://twitter.com/devops_borat/status/41587168870797312 https://twitter.com/devops_borat/status/41587168870797312 More seriously, this isn't the first I've heard of rm -rf backfiring - one of my friends said at one place he worked at, an IT guy walked out one day & never came back after trying to fix a co-worker's computer. He found out after by investigating on his co-worker's computer that the IT guy must have ran rm -rf while root & wiped out everything.
mgbmtl 11y ago
I'm a bit surprised the newspaper did not validate the source? They're basically quoting a Super User / Stack Exchange thread which was probably a troll. If a hosting company had deleted 1535 client accounts, we would have heard other stories about it from angry clients?
- throwanem 11y ago
  Doubtful. The set of people willing to do business with a thumb-fingered goober like this one is likely disjoint with the set of HN readers.
  - loup-vaillant 11y ago
    There's a specific reason why some called it fake: as adviced by others, Masarla ran the dd command to save the raw content of the disks, in case the recovery process screws up somehow. He inverted the `if` and `of` arguments. You'd expect him to pay attention, after what happens. This doesn't pass the smell test for some. Then again, you'd also expect him to be quite stressed out. That does make that mistake a bit more likely.
    - throwanem 11y ago
      Considering that, if this isn't a troll, he wiped out his entire business with a single command, I think you're giving him too much credit here.
      - loup-vaillant 11y ago
        Possibly, but the dd command does have more affordances than say, ld. `if` and `of` do stand for input file and output file, and it's harder to swap named arguments than positional arguments. He could have swapped arguments like `/dev/sdc` and `/dev/sde` though…
        JdeBP 11y ago
        That is fairly probable, in fact. See that person's earlier question about block device naming. * http://serverfault.com/q/728725 http://serverfault.com/q/728725
ausjke 11y ago
with 'set -u' he could have stayed safe. Bash probably should never treat undefined variables as a valid but empty value, it's so dangerous.
kilroy123 11y ago
How is this even remotely possible..? - No developers have a local copy of code on their machines? - No backups at all? Worse case scenario, couldn't you attempt to retrieve the data from the hard-drive? Though, the database(s) would likely not be retrievable.
- true_religion 11y ago
  He's a hosting company. He deleted his clients code/content.
- npolet 11y ago
  This is what I thought. I would have to go out my way to completely nuke the servers I work on. I'm trying to understand what structure this guys company had if everything can be mistakenly deleted without any chance of recovery. Maybe I misread the article and he runs a niche hosting company that has different requirements, but it seems strange to me to be able to completely remove your online body of work in a matter of minutes.
nickpsecurity 11y ago
What's most epic about this is it's in the UNIX Hater's Handbook. One of its rants was how better-designed systems would warn you if you were going to nuke your whole system. The reason is that a command to wipe the whole system was more likely a mistake than developer or admin's intent. UNIX would do it without blinking. Inherently unsafe programming and scripting combined with tools like that meant lots of UNIX boxes went kaput. And today, over two decades later, a person just accidentally destroyed his entire company with one line without warnings on a UNIX. History repeats when its lessons aren't learned. This problem, like setuid, should've been eliminated by design fairly quickly after it was discovered. http://esr.ibiblio.org/?p=538 http://esr.ibiblio.org/?p=538 EDIT: Added link to ESR's review of UNIX Hater's Handbook which links to UHH itself. Nicely covers what was rant, what was fixed, and what remains true. Linking in case people want to work on the latter plus my sour relationship with UNIX. :)
- vinceguidry 11y ago
  One of the more fascinating aspects of human history is how much effort we're willing to devote to creating varied senses of safety, even if that safety is only an illusion. Now, we can call this a failure of design, but really, people who rely on technology they don't understand can't be saved by good design. Sure, this particular case could be fixed by disallowing the recursive flag on the file system root, but safety is never going to be able to be the primary design concern of any technological system. Imagine if a sword were made safety as first-class concern. You can't design a sword that can be used safely by the untrained. No weapon can be, training with a weapon is a prerequisite for safely using it. Similarly, every technology has to be understood by those using it. If you don't you're just inviting trouble. For a business using technology, the needs are actually fairly straightforward. You need an understanding of what needs to be backed up, and a process for performing the backups. If you've picked the former right, (backing up human-readable information rather than data only readable by software programs that might go away in a crash) then risk is minimized.
  - nickpsecurity 11y ago
    Interesting points. I think systems like Burroughs counter the concept in the a lot of safety can be baked into a system. Here's what they did in 1961: http://www.smecc.org/The%20Architecture%20%20of%20the%20Burroughs%20B-5000.htm http://www.smecc.org/The%20Architecture%20%20of%20the%20Burr... Notice that's good UI design for the time, hardware elimination of worst problems, interface checks on functions, limits of what apps can do to system, and plenty recovery. Systems like NonStop, KeyKOS, OpenVMS, XTS-400, JX, and so on added to these ideas. You can certainly bake a strong foundation of safety into a system while allowing plenty flexibility. So, for example, critical files should be write-protected except for use by specific software involved in updates or administrative action. Many above systems did that. Then, one can use VMS-style, versioned filesystem that leaves originals in there in case a rollback is needed so long as there's free space for that. Such a system handling backups and restores with modern-sized HD's wouldn't have nuked everything. Might have even left everything if using lean setup but can't say for this specific case. "You can't design a sword that can be used safely by the untrained." A sword is designed to do damage. A better example would be a saw that's designed to be constructive but with risk of cutting your hand off. Even that can be designed to minimize risk to user. https://www.youtube.com/watch?v=esnQwVZOrUU https://www.youtube.com/watch?v=esnQwVZOrUU "If you've picked the former right, (backing up human-readable information rather than data only readable by software programs that might go away in a crash) then risk is minimized." That's orthogonal. A machine-readable format just needs a program to read it. The risk is whether the data is actually there in whole or part. This leads to mechanisms like append-only storage or periodic, read-only backups that ensure it's there. Or these clustered, replicated filesystems on machines with RAID arrays that lots of HPC or cloud applications use. Also, multiple, geographical locations for the data. People doing the above with proven protocols/tools rarely loose their data. Then there's this guy.
CPLX 11y ago
The real news here is that the Independent will write a feature story on a successful forum troll. Where were they back in the days of the Fucked Company message board when we could have used their help?
brador 11y ago
Why is the data not recoverable? Maybe things have changed, but rm doesn't zero out the drive. And with the backup that was rm too it should all be recoverable. Or am I missing something?
- __david__ 11y ago
  Most of the data is probably still there on the drive. But the filesystem data that says where it all actually is stored is probably irreplaceably gone. If some of that can be recovered then it should be possible to recreate individual files. Without it someone would have to guestimate where all the files are and then maybe manually piece them together (a single file may be in fragments in different parts of the disk). They'd also have to differentiate old deleted versions of files from the most recent deleted versions. So yeah, it could technically be recovered, but it's going to be a very big chore.
- Qantourisc 11y ago
  Not directly no, but some FS give you a hard time to recover the file structure, which in some cases is a big problem. You could probably recover files, but if the backups aren't stored in a tar/zip/... file, it will be hard to recover both the data and the structure.
callesgg 11y ago
Once i did sudo rm -R . when i was in /var When i discovered what i had done and stooped it /var/www was already gone. Luckily we had backups, but that sure did teach me a lesson about rm. These days i look very carefully before using rm -R and also i type the entire path.
- rm_-rf_slash 11y ago
  Sometimes seeing certain usernames can get you to accidentally write the wrong thing.
  - goda90 11y ago
    Reminds me of the guy who chose the Xbox Live gamer tag 'XBOX TURN OFF'
austinjp 11y ago
Yeah so I typed rm -f * the other day after typing rm -f *~ repeatedly in a few different directories. In the 2 seconds it took me to realise, I lost a lot of data. First time I've made that particular typing slip-up in many years. Thankfully I had backups to restore from. Real heart-sink moment. Sure, there should have been aliases for rm -i and I shouldn't have used -f etc etc etc. But sometimes this stuff is going to happen.
miles 11y ago
"Most users agreed that it was unlikely that Mr Marsala would be able to recover any of the data." Perhaps they are unfamiliar with extundelete? http://extundelete.sourceforge.net http://extundelete.sourceforge.net
chiph 11y ago
Not a Unix admin .. but can you swap the rm command with a different executable that prompts you when the -rf option is specified?
- wutbrodo 11y ago
  You can just alias rm to a script of yours that does just that with like, one extra line of bash. I've done this for a couple of commands where I prefer default behavior that isn't specifiable by flags.
bashinator 11y ago
I would like to point out that requiring `set -u` at the top of all your production bash scripts will prevent this kind of disaster - the script will fail if unassigned variables are referenced.
- cpeterso 11y ago
  If anyone knows bash, it's bashinator! :)
pmlnr 11y ago
Nearly done the same thing once by messing up ordering of flags. Thankfully this was before devops tools were present, so a ctrl-c stopped wiping before it got too deep, but a Friday afternoon dowtime is still bad. Tape/blu-ray disk backups can come really handy in these cases, not being easy to wipe them.
vbezhenar 11y ago
A competent specialist will be able to help that guy. rm -rf / is easily fixable if you won't mess around after it. Backups usually have recognizable format, so it's possible to restore backups and then everything from backups.
thinkmoore 11y ago
As part of my PhD research, I developed a shell scripting language (shill-lang.org, previously on hn: https://news.ycombinator.com/item?id=9328277 https://news.ycombinator.com/item?id=9328277) with features that provide safety belts against this sort of error. From speaking to administrators and developers, we believe these types of errors cause take much more worry and time than they are worth. Now that I'm graduating, we've started the process of refining Shill into a product that we can offer to administrators and developers to make their lives simpler. If this sounds like a tool you wish you had (or if you wish a similar tool existed for your platform of choice), we'd love to hear from you.
nihonde 11y ago
Any script that includes rm -rf followed by variables in a path is an accident waiting to happen. Mounting the backup volumes is just icing on the cake for this extremely incompetent web hosting provider. It made me nervous to type rm -rf in this comment form. Those letters are dark magic.
- smegel 11y ago
  > Mounting the backup volumes That sounds more like an accident waiting to happen than a single line of bad code.
xlm1717 11y ago
One take-away from this is that it's probably better to save your backups somewhere where you can't delete them. Make sure that nothing using rm touches your database backups. Also, try to keep them backed up in multiple places. For example, store backups on a server you own, and on a cloud server, like on S3.
- shadeless 11y ago
  Another way this could have been avoided is if he used "--one-file-system" flag, which wouldn't delete backups as they were mounted on a separate filesystem.
megacity 11y ago
Is there ever a situation where someone would want to rm -rf / ?
- __david__ 11y ago
  Nope (almost never), which is why GNU rm requires the '--no-preserve-root' flag if you actually want to do that for some reason.
  - megacity 11y ago
    What are the almost never situations? I'd be super interested, because I cannot think of one at all :D
    - randyrand 11y ago
      Cops / assailant about to bust down your door and its all you know / have time for?
  - timseal 11y ago
    In the manpage for rm, I see "--no-preserve-root do not treat ‘/’ specially (the default)". For real? The default is to do the worst thing?
    - cyphar 11y ago
      No, the default is to do nothing. The (default) refers to "treat root specially" not the flag.
- dpv 11y ago
  Well, it is certainly fun. I never bothered to count exact numbers, but from my experience, close to two thirds of all people, when presented with root shell and no consequences, will run rm -rf in some way. Humble ones issue "rm -rf /usr" or "rm -rf /lib", others go straight to "/bin/rm -rf /". I've seen one person do "rm -rf /* ", immediately followed by "find / -delete". I'd really like to take a peek on his/her thought process at that moment, looked like the desire of destruction was really strong in that one particular brain ;-) So yeah, while its not particularly useful one, there's indeed a situation where one definitely want to run it. disclaimer: I run SELinux playbox with free root access and session recording, and peeking into what others do is also fun. Edit: rm minus rf slash asterisk formatting.
deleted 11y ago
[deleted]
wilkystyle 11y ago
Did he change his ServerFault username? I see commenters referencing @MarcoMarsala, but the OP's name appears to be bleemboy at the moment[0] [0] http://serverfault.com/questions/769357/recovering-from-a-rm-rf http://serverfault.com/questions/769357/recovering-from-a-rm...
- maus42 11y ago
  Looks like he did, the nick was MarcoMarsala earlier this week.
cha5m 11y ago
I was expecting something much more subtle. I guess the best course of action to prevent this would be to alias rm to a custom script, then parse the arguments to make sure the root directory is never recursively deleted, then calling rm from within your script.
maus42 11y ago
Does this article anything else than paraphrase the Serverfault thread? On the first reading I thought they had contacted the poor fellow to confirm his identity or something, but rereading, it would appear that they didn't: no further information than the original source. https://serverfault.com/questions/769357/recovering-from-a-rm-rf https://serverfault.com/questions/769357/recovering-from-a-r... (The user who asked that question uses now a nick, but had the real-soundish name mentioned in the article when I read that serverfault question first time earlier this week.) edit. ...I really hope there isn't a real Marco Marsala someone pretended to be. Search engine results for that name are not great ATM.
beloch 11y ago
Correct me if I'm wrong, but rm doesn't wipe data out, it just deallocates the disk space devoted to it. If you actually managed to wipe out your entire file system with rm you could likely still recover your data with a recovery tool. This story smells a wee bit fishy to me.
- milkey_mouse 11y ago
  It's just hard to get the filesystem entries for the file back. rm doesn't specifically wipe, you're right, but the filesystem entries are deleted, which means you basically have to grep the disk for bits of the file you want with known contents.
- fabiothebest 11y ago
  of course :) it's a troll
sickpig 11y ago
http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 http://serverfault.com/questions/769357/recovering-from-a-rm... "luckily we recovered almost all data!"
jchrisa 11y ago
I lost the private key to one of my AWS servers after it had had a traffic spike due to blog coverage[1]. It was a toy system so it was using local storage, but then it became sort of popular. Luckily I had a process monitor set up so it managed months of uptime before something happened that I couldn't do anything to fix. [1] http://waxy.org/2008/04/exclusive_google_app_engine_ported_to_amazons_ec2/ http://waxy.org/2008/04/exclusive_google_app_engine_ported_t...
OSButler 11y ago
I once had an incident with a server which triggered notification alerts about a failing httpd service. While I was looking into the issue, the mail service suddenly stopped working, then the database service went down - it was like a slow cascading failure, affecting all services on the server one after the other. I finally noticed the 'rm' command in the process list and asked the client if he ran any custom commands as root on the machine. Turns out he followed the instructions on a website to install some custom software without checking any of the commands and just copied & pasted them into the prompt. He basically managed to "rm -rf" on / and deleted his own server. Luckily recent backups were available, so the damage was rather small, but it was interesting to see someone just pasting & executing commands without knowing what they actually do, especially when logged in as root.
ksenzee 11y ago
He says he's recovered almost all the data. FWIW. https://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 https://serverfault.com/questions/769357/recovering-from-a-r...
noonespecial 11y ago
OR "Man learns the value of backups because somethings things go wrong."
- Raphmedia 11y ago
  From the article: "All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)."
  - cyphar 11y ago
    That's not a backup. If your "backup script" requires mounting the backup on a production machine, then it's barely a copy of your data.
adultSwim 11y ago
Oops
Animats 11y ago
This is what comes from treating undefined variables as empty, rather than as errors. Bad language design in the shell.
Xcelerate 11y ago
Toy Story 2 was almost entirely deleted because of this same problem: http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/#gref http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-...
aleden 11y ago
I accidentally set executable permissions on the following Makefile: https://github.com/samalba/acdcontrol/blob/master/Makefile https://github.com/samalba/acdcontrol/blob/master/Makefile While typing quickly I tab-completed to 'Makefile' and hit enter. Although it was a Makefile, it was executed as a bash script. bash ignored the incorrect syntax and executed line 10: rm -rf $(DIRNAME)/* If make parsed the file, $(DIRNAME) would have been nonempty. But it was empty under bash. --no-preserve-root did not protect against this, because the target of the command was '/*'
- lazzlazzlazz 11y ago
  And here it is.
- scintill76 11y ago
  How/why does this work without "#!" at the beginning of the file? I just tested with fish shell, and I get "Exec format error. The file './x' is marked as an executable but could not be run by the operating system." But, from bash or dash it does execute commands from the file.
  - Tiksi 11y ago
    It'll fall back to sh, the default shell (or whatever it's symlinked to): http://paste.click/QxWUMG http://paste.click/QxWUMG In my case that's bash, debian based systems use dash.
    - cyphar 11y ago
      That shouldn't happen if you look at fs/exec.c (search_binary_handler) there isn't a "fallback to shell" option. And fs/binfmt_script.c doesn't fall back to shell either. Are you sure you don't have some weird binfmt_misc hook enabled?
      - JdeBP 11y ago
        The "it" in what you are replying to is not the kernel; but the shell. * http://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.html#tag_001_009_001_001 http://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.htm... * http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...
jo909 11y ago
While as others already pointed out this story seems a little fishy, it serves well to reflect if something like this could in theory happen to your infrastructure. Do you have your backup servers in the same configuration management software (ansible, puppet, ssh-for-loop etc) as the rest of the servers? One grave error (however unlikely) in your base configuration really can take down everything together in one fell swoop. How "cold" are your backups? If the backup media are not physically disconnected and secured, you can most likely construct a scenario where the above, malware, a hacker or a rouge admin could destroy both the backups and the live data. I will certainly suggest some additional safeguards for our backups.
- ansible 11y ago
  Yep, that's what I hope everyone will be doing... thinking about their own backups and infrastructure. We have backups off-site on disconnected media, so that alone prevents the kind of accident we're talking about. We use btrfs send / receive to send OS images from the primary container host to the backup container host. The snapshots are read-only, so I'm fairly sure I can't just 'rm -rf' them, I'd have to actually 'btrfs subvolume delete foobar' them. I should try that though on one of the test servers...
BinaryIdiot 11y ago
According to the thread they were able to recover almost all of the data so far. So the whole "deletes his entire company" no longer seems accurate. Still pretty crazy.
anotheryou 11y ago
he got it recovered https://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 https://serverfault.com/questions/769357/recovering-from-a-r...
justinlardinois 11y ago
It's weird to me that commenting on a Stack Exchange question could get you quoted on several major news websites.
Schwolop 11y ago
I managed to sudo chown -R {useless_user}:{useless_user} {foo}/ with foo undefined, whilst simultaneously distributing that command with dsh to our entire cluster of 10 machines. This was after testing that everything worked on the development machine. So of course, I retraced my steps to find out what went wrong, and killed the development machine too. The upside is that we knew we had issues, and with everything broken the impetus is on the right people to ensure they're fixed before we get distracted by the next shiny feature. Sometimes, setting your servers on fire is the solution to technical debt.
mofle 11y ago
There are many ways you can safeguard [0] `rm`, but in the end, it's better to just use a tool that move files to the trash [1] instead. [0]: https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md#safeguard-rm https://github.com/sindresorhus/guides/blob/master/how-not-t... [1]: https://github.com/sindresorhus/trash-cli https://github.com/sindresorhus/trash-cli
rurban 11y ago
Shouldn't the proper fix be implying -i on -rf / At least in some fork of the coreutils.
drinchev 11y ago
According to a comment in the ServerFault website, he actually managed to recover the data [1]. He consulted a company for data recovery and they gave him a list with the files that they could manage to save [2]. 1 : http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 http://serverfault.com/questions/769357/recovering-from-a-rm... 2 : http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment971005_769400 http://serverfault.com/questions/769357/recovering-from-a-rm...
blaze33 11y ago
Instead of doing system administration with root, couldn't we have a system user with the same privileges as root except it wouldn't have write access to the files of some users (like your clients) ? So you could still rm -rf / all you want, delete everything but still have /home or /var/www content untouched. We run certain programs with limited privileges to mitigates risks (bugs, exploits, etc.), why shouldn't we also limit the privileges of root to mitigates the risk of buggy system administration ? Obviously having actual backups and testing your code before applying it to production is good practice but I feel like doing system administration with root while having potential bugs in your sysadmin code (as in any other software) leaves the door open to the next catastrophic failure.
fabiothebest 11y ago
It was a troll. it was just a joke and a marketing idea for promoting the book he is writing. Use Google Translate if you don't speak Italian: http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/ http://www.repubblica.it/tecnologia/2016/04/15/news/cancella...
jmiserez 11y ago
Nothing to see here, it was a hoax/troll: https://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question https://meta.serverfault.com/questions/8696/what-to-do-with-...
coldtea 11y ago
>Together, the code deleted everything on the computer, including Mr Masarla’s customers' websites, he wrote. Mr Masarla runs a web hosting company, which looks after the servers and internet connections on which the files for websites are stored. And he has no backups? Including rolling backups in unconnected storage? >Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too. Then the probably probably deserved to die. Sorry for the customers though...
mpdehaan2 11y ago
NOTE: this is a hoax: https://news.ycombinator.com/item?id=11514455 https://news.ycombinator.com/item?id=11514455

233 points by bhartzer 11y ago

He even deleted the backups. >>the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.

drzaiusapelord 11y ago
If you're not doing offsite and cold backups, then you're just asking for trouble. If not crap like this then a fire or a ransomeware infection or a malicious employee, etc.
- jasonjei 11y ago
  He actually was doing a remote backup (although probably not a cold backup). Unfortunately, he had used mount instead of rsync over ssh, making it vulnerable to the rm -rf command.
  - zyxley 11y ago
    That's not a backup, it's a mirror.
    - awqrre 11y ago
      a mirror is kind of a backup if you don't update it live... but of course you should have other backups that are offline.
    - jasonjei 11y ago
      Are you suggesting that you can't backup with rsync? Because you can do full and incremental backups with rsync. In fact Time Machine on OS X looks like it does backups in this manner...
      - cyphar 11y ago
        Just using rsync to make copies isn't a backup. If you use rsnapshot (which stores each copy separately) then you have a backup. Copies are not sufficient if you find out that something broke three weeks ago.
- satysin 11y ago
  If you are not doing offline and offsite backups you are not doing backups at all.
deleted 11y ago
[deleted]
rlpb 11y ago
In other words, he didn't have backups. A live copy of a running system available to that running system is not a backup.
- Houshalter 11y ago
  Well it still prevents against the hard drive failing. It doesn't protect against bad code, which he obviously didn't consider.
  - laumars 11y ago
    RAID parity drives also protect against a drive failing but they're not backups either.
    - Houshalter 11y ago
      If it's a duplicate copy of data intended in case of failure then yes it is a backup. Its not an offsite backup, but many people don't keep their personal backups offsite.
      - rlpb 11y ago
        It is so deficient as a backup that I don't think it qualifies to be called a backup. That was my point. Backups are expected to protect against data loss for a number of different failure cases (eg. disk failure, hardware fault leading to slow filesystem corruption, fire/theft, failed upgrade, "undo" for accidental change or deletion). There is a point where something addresses so few of these failure cases that you can't reasonably call it a backup.
      - laumars 11y ago
        There's another term for what it is: redundancy. Redundancy is there for fast recovery times (even zero downtime depending on how redundancy is implemented). It's not intended to run as a backup as redundancy devices are live and can fail from many of the same causes that will take your primary devices offline (fire, sysadmin fail, etc) Likewise, if your "backups" are always online then it works better for business continuity than it does as a backup. So realistically it's more of a redundancy share.

sp332 11y ago

This felt a bit unlikely, but what really convinced me it was a troll was a follow-up comment where he said he accidentally switched "if" and "of" in a dd command.

mturmon 11y ago
Yes. To wit: "I swapped if and of while doing dd. What to do now?" – Marco Marsala Apr 11 at 7:02
chris_wot 11y ago
If he mounted his backup media and wiped it, what makes you think he couldn't cockup the dd command? However, under Linux rm -rf / needs --no-preserve-root to work, right?
- aroch 11y ago
  Depends on which version of `rm` you have. Newer versions (rm from coreutils 8.xx I believe) use no-preserve-root
  - profmonocle 11y ago
    And the question is tagged centos 7, which is from 2014 - far too recent to not have --no-preserve-root.
    - JdeBP 11y ago
      https://git.centos.org/log/rpms!coreutils.git/refs!heads!c7 https://git.centos.org/log/rpms!coreutils.git/refs!heads!c7 indicates that CentOS 7 has coreutils 8.22 . http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=34e3ea055721ecc72e6f636700e8ba7b58069c65 http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commit... shows that GNU coreutils gained that option in 2003.
- astrodust 11y ago
  It's distribution dependent.

twvisitavisitb 11y ago

Since I didn't find any links in the article, here's the original post: http://serverfault.com/questions/769357/recovering-from-a-rm-rf http://serverfault.com/questions/769357/recovering-from-a-rm...

kamjam 11y ago
It's in the 2nd paragraph, linked from "called Server Fault" text.

AdmiralAsshat 11y ago

This question that went unanswered in the replies bears repeating: Any idea why the command actually ran? If $foo and $bar were both undefined, rm -rf / should have errored out with the --no-preserve-root message. The only way I can think of that this would have actually worked on a CentOS7 machine is if $bar evaluated to , so what was run was rm -rf /. As the above notes, I'm pretty sure recent versions of Redhat/CentOS actually protect against this sort of thing. On the offchance you're not running a recent server, however, this could also be avoided by using `set -u` in the bash script, as it would cause undefined variables to error out.

LukeShu 11y ago
Before you apply `set -u` to all of your scripts, be aware that an empty array counts as undefined. So if you have arrays for which being empty is valid, be sure to `set +u` right before accessing them (and then `set -u` again after).
- anderskaseorg 11y ago
  Or write ${arr+"${arr[@]}"}, which is, you know, only slightly more awkward than bash arrays already were to begin with. Really you just want to use a real programming language instead.
harryf 11y ago
I believe those variables were not handled by the shell but rather in an Ansible "playbook" - see http://docs.ansible.com/ansible/playbooks_variables.html http://docs.ansible.com/ansible/playbooks_variables.html i.e. the variables where happening in a Jinja template and because undefined, rm -rf {foo}/{bar} was transformed by the template engine into rm -rf /
- Torgo 11y ago
  the playbook will fail if there are undefined variables, I find the story suspect.

amelius 11y ago

+1 for snapshotting filesystems.

creshal 11y ago
Or any backup solution worth its salt, including "RAID1 where you yank out and replace one of the drives every other week".
- robinson-wall 11y ago
  RAID is not a backup. ... Especially when your RAID is busy rebuilding for N hours every other week.
  - daveguy 11y ago
    Well RAID 1 would be a backup if you yanked out and replaced a drive every week. In that case it would be a weekly snapshot.
    - DDub 11y ago
      Unless you're mirroring across more than 2 drives, you have an AID setup.
      - daveguy 11y ago
        You are incorrect. RAID 1 is a mirror setup. There are two drives with exactly the same information. One of the two drives is redundant. RAID 1 does not include striping and only requires 2 drives for redundancy.
        Sanddancer 11y ago
        I think what DDub is getting at is that there is no redundancy for the data received while the disk is mirrored to the new twin. For that, you'd need a mirrored pair plus a drive to yank out as the backup.

spriggan3 11y ago

I don't believe that's the truth, for a second. Of course the independent didn't look at the company in question to see if there was any litigation between this guy and his customers.

chris_wot 11y ago
The Independent also didn't look up -r as it stands for recursively remove directories...
- dave2000 11y ago
  They said: "the r deletes everything within a given directory" which is what it does. Non technical readers aren't going to understand "recursively remove directories".
  - chris_wot 11y ago
    Fair point.

TheCams 11y ago

Sorry if that's a stupid question, but does that mean he was running his script as root?

smhenderson 11y ago
He was running with full administrative permissions and thus file access permissions on files were ignored. "root" is the name of the default administrative account on Unix and Unix like systems.
- TheCams 11y ago
  Thanks :)

ikeboy 11y ago

Reminds me of https://archive.is/9R2j8 https://archive.is/9R2j8 (Original thread has been deleted.)

kbenson 11y ago
That just makes me really,really want to know what the original company was and what product they were selling.
- ikeboy 11y ago
  Some kind of botnet access, probably.
bpchaps 11y ago
That's awesome. A company I left a while back recently had two servers accidentally rebooted through sort of automated task (probably puppet). The fine, I'm told, was one billion dollars. Someway, somehow, he still works there. :)
- ikeboy 11y ago
  Details, company name?
  - bpchaps 11y ago
    It's a very large French bank. I don't know anything except that it was a paired batch processing server and didn't push further questions, honestly. For some reason, I found absolutely nothing in the news about it, but my source of information is credible. It doesn't surprise me for a second that it happened, since, for example, I spent months trying to get these guys to fix their literally useless MQ DR failover scripts, but nothing ever came from it, since, they didn't have anywhere to test. With the way they treat their employees - fucking good riddance. There was a giant mess when Disney forced their NOC to train their replacements, but yet these guys did the exact same thing, plus some, and there was no public awareness during or after it. The best part was their push to move everyone to Montreal. Lower pay, not a guaranteed extension and you're forced to move? Okay. The AMRS CTO actually left about a month after he got the position and took me along with one other person over to a new company. Goldman's head of tech actually just left to go to the same place. Not gonna lie, it sounds incredibly suspicious, especially considering the the kinds of shenanigans that went on there... thankfully I'm no longer working there. It's a very, very strange place in finance.

oluwie 11y ago

Reminds me of the time I accidentally typed in 'crontab -d' instead of 'crontab -e'. Those two letters are eerily too close to eachother.

laumars 11y ago
I've managed that one as well. One month into a new job and working on servers with complex applications installed that I wasn't yet familiar with. Thankfully I had `crontab -l` just beforehand, otherwise I'd have been screwed.
- hrktb 11y ago
  Most admins have a text file somewhere with the contents of the crontab. Specially because these things happen more often than it should.
  - laumars 11y ago
    Indeed. I do generally keep backups of crontabs, not just in case of this kind of scenario but also in case the platform blows up in any unexpected ways. Sadly the company in question didn't. However I have since made it a personal policy to always -l before editing so I have a "backup" in my tmux scroll back (that time I mentioned before, it was pure chance that I had -l)
wahnfrieden 11y ago
This is one of the reasons why we have infrastructure as code now, so system changes can be reviewed and tested just like application code, and more types of accidents can be reverted via source control :)
- djsumdog 11y ago
  In the article the guy is using ansible. He even had off-site backups, but they were mounted before his ansible playbooks ran, wiping them out as well.
11y ago

castratikron 11y ago

The bash -e and -u options might have saved him here: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/

giovannibajo1 11y ago
This. All my scripts begin with "set -euo pipefail", and my editor linter complains loudly if that line isn't there. I wish distros would migrate to making those settings the default, over the years. Even if it would take a while, I think it would be priceless

redbeard0x0a 11y ago

Too bad he wasn't running Illuminos (OpenSolaris) based servers (or even some Solaris versions) that would have just flat out refused to run rm -rf /

creshal 11y ago
Recent-ish versions of GNU rm do the same. However, it only protects against `rm -rf /`, not `rm -rf /*`, as the latter is expanded by the shell.
neerdowell 11y ago
It's required in POSIX 1003.1-2013 that rm refuse to remove the root directory[0]. [0] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/rm.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/rm...
- deleted 11y ago
  [deleted]
4ad 11y ago
It's illumos, not Illuminos.

l0c0b0x 11y ago

...and this is why we include a 'backup technology' question in our technical interviews--where 'offsite backup' must follow with something like "possibly the most important type of backup because..."

AdmiralAsshat 11y ago
You know the sad thing is that even this isn't idiot-proof and needs to be qualified. One of my customer's brilliant "cost-saving" measures was to have an offsite backup solution that was basically an rsync script that ran every 15 minutes. So when someone on their end did something catastrophic to their data and it took them an hour to notice, they were incredulous that we couldn't help them restore their data even though it was "backed up offsite!" because their "backup" solution had already caught up and duplicated the broken data.
- ansible 11y ago
  And that's why if you're using rsync, you ought to be using rsnapshot instead, and have generations of backups so that you are not overwriting your most recent one.
  - SmellyGeekBoy 11y ago
    I find rdiff-backup is great as a drop-in replacement.
- cyphar 11y ago
  Yeah, a copy is not a backup. It's a mistake too many people make. You need to have historical backups.

ams6110 11y ago

Years ago, I worked in an investment bank and we had a programmer put a batch program into production that executed the following as a shell command: rm -rf foo / It was supposed to be: rm -rf foo/ It didn't run as root, but still managed to wipe out all the business data files. What saved us was that the servers were configured with RAID 1 and before the start of the nightly batch cycle, the mirror was "split" and only one copy mounted. So we just had to restore the missing files from the other half of the mirror to revert to the start of the batch window and rerun the entire night's jobs.

ArkyBeagle 11y ago
So in other words, the acceptance protocol for this script was inadequate. In the minicomputer era, it was common for a programmer to be required to run it on this one poor donkey of a machine to make it caught nothing on fire before moving to the big machine.
- ams6110 11y ago
  It was the 1990s, not that it's an excuse but TDD wasn't a "thing" yet, nor was version control (at least not in that shop). For every change to a program we printed the diff, attached a cover memo, and filed it in a cabinet. Yes, there were test systems and programers were supposed to test all their changes, but they also were the ones who deployed their own changes to production so there were ways for this to happen pretty easily.
gnarbarian 11y ago
What happened to the programmer?
- ams6110 11y ago
  Nothing. She continued to work there until after I moved on.
  - mfoy_ 11y ago

Bahamut 11y ago

This made me think of this quote: https://twitter.com/devops_borat/status/41587168870797312 https://twitter.com/devops_borat/status/41587168870797312 More seriously, this isn't the first I've heard of rm -rf backfiring - one of my friends said at one place he worked at, an IT guy walked out one day & never came back after trying to fix a co-worker's computer. He found out after by investigating on his co-worker's computer that the IT guy must have ran rm -rf while root & wiped out everything.

mgbmtl 11y ago

I'm a bit surprised the newspaper did not validate the source? They're basically quoting a Super User / Stack Exchange thread which was probably a troll. If a hosting company had deleted 1535 client accounts, we would have heard other stories about it from angry clients?

throwanem 11y ago
Doubtful. The set of people willing to do business with a thumb-fingered goober like this one is likely disjoint with the set of HN readers.
- loup-vaillant 11y ago
  There's a specific reason why some called it fake: as adviced by others, Masarla ran the dd command to save the raw content of the disks, in case the recovery process screws up somehow. He inverted the `if` and `of` arguments. You'd expect him to pay attention, after what happens. This doesn't pass the smell test for some. Then again, you'd also expect him to be quite stressed out. That does make that mistake a bit more likely.
  - throwanem 11y ago
    Considering that, if this isn't a troll, he wiped out his entire business with a single command, I think you're giving him too much credit here.
    - loup-vaillant 11y ago
      Possibly, but the dd command does have more affordances than say, ld. `if` and `of` do stand for input file and output file, and it's harder to swap named arguments than positional arguments. He could have swapped arguments like `/dev/sdc` and `/dev/sde` though…
      - JdeBP 11y ago
        That is fairly probable, in fact. See that person's earlier question about block device naming. * http://serverfault.com/q/728725 http://serverfault.com/q/728725

ausjke 11y ago

with 'set -u' he could have stayed safe. Bash probably should never treat undefined variables as a valid but empty value, it's so dangerous.

kilroy123 11y ago

How is this even remotely possible..? - No developers have a local copy of code on their machines? - No backups at all? Worse case scenario, couldn't you attempt to retrieve the data from the hard-drive? Though, the database(s) would likely not be retrievable.

true_religion 11y ago
He's a hosting company. He deleted his clients code/content.
npolet 11y ago
This is what I thought. I would have to go out my way to completely nuke the servers I work on. I'm trying to understand what structure this guys company had if everything can be mistakenly deleted without any chance of recovery. Maybe I misread the article and he runs a niche hosting company that has different requirements, but it seems strange to me to be able to completely remove your online body of work in a matter of minutes.

nickpsecurity 11y ago

What's most epic about this is it's in the UNIX Hater's Handbook. One of its rants was how better-designed systems would warn you if you were going to nuke your whole system. The reason is that a command to wipe the whole system was more likely a mistake than developer or admin's intent. UNIX would do it without blinking. Inherently unsafe programming and scripting combined with tools like that meant lots of UNIX boxes went kaput. And today, over two decades later, a person just accidentally destroyed his entire company with one line without warnings on a UNIX. History repeats when its lessons aren't learned. This problem, like setuid, should've been eliminated by design fairly quickly after it was discovered. http://esr.ibiblio.org/?p=538 http://esr.ibiblio.org/?p=538 EDIT: Added link to ESR's review of UNIX Hater's Handbook which links to UHH itself. Nicely covers what was rant, what was fixed, and what remains true. Linking in case people want to work on the latter plus my sour relationship with UNIX. :)

vinceguidry 11y ago
One of the more fascinating aspects of human history is how much effort we're willing to devote to creating varied senses of safety, even if that safety is only an illusion. Now, we can call this a failure of design, but really, people who rely on technology they don't understand can't be saved by good design. Sure, this particular case could be fixed by disallowing the recursive flag on the file system root, but safety is never going to be able to be the primary design concern of any technological system. Imagine if a sword were made safety as first-class concern. You can't design a sword that can be used safely by the untrained. No weapon can be, training with a weapon is a prerequisite for safely using it. Similarly, every technology has to be understood by those using it. If you don't you're just inviting trouble. For a business using technology, the needs are actually fairly straightforward. You need an understanding of what needs to be backed up, and a process for performing the backups. If you've picked the former right, (backing up human-readable information rather than data only readable by software programs that might go away in a crash) then risk is minimized.
- nickpsecurity 11y ago
  Interesting points. I think systems like Burroughs counter the concept in the a lot of safety can be baked into a system. Here's what they did in 1961: http://www.smecc.org/The%20Architecture%20%20of%20the%20Burroughs%20B-5000.htm http://www.smecc.org/The%20Architecture%20%20of%20the%20Burr... Notice that's good UI design for the time, hardware elimination of worst problems, interface checks on functions, limits of what apps can do to system, and plenty recovery. Systems like NonStop, KeyKOS, OpenVMS, XTS-400, JX, and so on added to these ideas. You can certainly bake a strong foundation of safety into a system while allowing plenty flexibility. So, for example, critical files should be write-protected except for use by specific software involved in updates or administrative action. Many above systems did that. Then, one can use VMS-style, versioned filesystem that leaves originals in there in case a rollback is needed so long as there's free space for that. Such a system handling backups and restores with modern-sized HD's wouldn't have nuked everything. Might have even left everything if using lean setup but can't say for this specific case. "You can't design a sword that can be used safely by the untrained." A sword is designed to do damage. A better example would be a saw that's designed to be constructive but with risk of cutting your hand off. Even that can be designed to minimize risk to user. https://www.youtube.com/watch?v=esnQwVZOrUU https://www.youtube.com/watch?v=esnQwVZOrUU "If you've picked the former right, (backing up human-readable information rather than data only readable by software programs that might go away in a crash) then risk is minimized." That's orthogonal. A machine-readable format just needs a program to read it. The risk is whether the data is actually there in whole or part. This leads to mechanisms like append-only storage or periodic, read-only backups that ensure it's there. Or these clustered, replicated filesystems on machines with RAID arrays that lots of HPC or cloud applications use. Also, multiple, geographical locations for the data. People doing the above with proven protocols/tools rarely loose their data. Then there's this guy.

CPLX 11y ago

The real news here is that the Independent will write a feature story on a successful forum troll. Where were they back in the days of the Fucked Company message board when we could have used their help?

brador 11y ago

Why is the data not recoverable? Maybe things have changed, but rm doesn't zero out the drive. And with the backup that was rm too it should all be recoverable. Or am I missing something?

__david__ 11y ago
Most of the data is probably still there on the drive. But the filesystem data that says where it all actually is stored is probably irreplaceably gone. If some of that can be recovered then it should be possible to recreate individual files. Without it someone would have to guestimate where all the files are and then maybe manually piece them together (a single file may be in fragments in different parts of the disk). They'd also have to differentiate old deleted versions of files from the most recent deleted versions. So yeah, it could technically be recovered, but it's going to be a very big chore.
Qantourisc 11y ago
Not directly no, but some FS give you a hard time to recover the file structure, which in some cases is a big problem. You could probably recover files, but if the backups aren't stored in a tar/zip/... file, it will be hard to recover both the data and the structure.

callesgg 11y ago

Once i did sudo rm -R . when i was in /var When i discovered what i had done and stooped it /var/www was already gone. Luckily we had backups, but that sure did teach me a lesson about rm. These days i look very carefully before using rm -R and also i type the entire path.

rm_-rf_slash 11y ago
Sometimes seeing certain usernames can get you to accidentally write the wrong thing.
- goda90 11y ago
  Reminds me of the guy who chose the Xbox Live gamer tag 'XBOX TURN OFF'

austinjp 11y ago

Yeah so I typed rm -f * the other day after typing rm -f *~ repeatedly in a few different directories. In the 2 seconds it took me to realise, I lost a lot of data. First time I've made that particular typing slip-up in many years. Thankfully I had backups to restore from. Real heart-sink moment. Sure, there should have been aliases for rm -i and I shouldn't have used -f etc etc etc. But sometimes this stuff is going to happen.

miles 11y ago

"Most users agreed that it was unlikely that Mr Marsala would be able to recover any of the data." Perhaps they are unfamiliar with extundelete? http://extundelete.sourceforge.net http://extundelete.sourceforge.net

chiph 11y ago

Not a Unix admin .. but can you swap the rm command with a different executable that prompts you when the -rf option is specified?

wutbrodo 11y ago
You can just alias rm to a script of yours that does just that with like, one extra line of bash. I've done this for a couple of commands where I prefer default behavior that isn't specifiable by flags.

bashinator 11y ago

I would like to point out that requiring `set -u` at the top of all your production bash scripts will prevent this kind of disaster - the script will fail if unassigned variables are referenced.

cpeterso 11y ago
If anyone knows bash, it's bashinator! :)

pmlnr 11y ago

Nearly done the same thing once by messing up ordering of flags. Thankfully this was before devops tools were present, so a ctrl-c stopped wiping before it got too deep, but a Friday afternoon dowtime is still bad. Tape/blu-ray disk backups can come really handy in these cases, not being easy to wipe them.

vbezhenar 11y ago

A competent specialist will be able to help that guy. rm -rf / is easily fixable if you won't mess around after it. Backups usually have recognizable format, so it's possible to restore backups and then everything from backups.

thinkmoore 11y ago

As part of my PhD research, I developed a shell scripting language (shill-lang.org, previously on hn: https://news.ycombinator.com/item?id=9328277 https://news.ycombinator.com/item?id=9328277) with features that provide safety belts against this sort of error. From speaking to administrators and developers, we believe these types of errors cause take much more worry and time than they are worth. Now that I'm graduating, we've started the process of refining Shill into a product that we can offer to administrators and developers to make their lives simpler. If this sounds like a tool you wish you had (or if you wish a similar tool existed for your platform of choice), we'd love to hear from you.

nihonde 11y ago

Any script that includes rm -rf followed by variables in a path is an accident waiting to happen. Mounting the backup volumes is just icing on the cake for this extremely incompetent web hosting provider. It made me nervous to type rm -rf in this comment form. Those letters are dark magic.

smegel 11y ago
> Mounting the backup volumes That sounds more like an accident waiting to happen than a single line of bad code.

xlm1717 11y ago

One take-away from this is that it's probably better to save your backups somewhere where you can't delete them. Make sure that nothing using rm touches your database backups. Also, try to keep them backed up in multiple places. For example, store backups on a server you own, and on a cloud server, like on S3.

shadeless 11y ago
Another way this could have been avoided is if he used "--one-file-system" flag, which wouldn't delete backups as they were mounted on a separate filesystem.

megacity 11y ago

Is there ever a situation where someone would want to rm -rf / ?

__david__ 11y ago
Nope (almost never), which is why GNU rm requires the '--no-preserve-root' flag if you actually want to do that for some reason.
- megacity 11y ago
  What are the almost never situations? I'd be super interested, because I cannot think of one at all :D
  - randyrand 11y ago
    Cops / assailant about to bust down your door and its all you know / have time for?
- timseal 11y ago
  In the manpage for rm, I see "--no-preserve-root do not treat ‘/’ specially (the default)". For real? The default is to do the worst thing?
  - cyphar 11y ago
    No, the default is to do nothing. The (default) refers to "treat root specially" not the flag.
dpv 11y ago
Well, it is certainly fun. I never bothered to count exact numbers, but from my experience, close to two thirds of all people, when presented with root shell and no consequences, will run rm -rf in some way. Humble ones issue "rm -rf /usr" or "rm -rf /lib", others go straight to "/bin/rm -rf /". I've seen one person do "rm -rf /* ", immediately followed by "find / -delete". I'd really like to take a peek on his/her thought process at that moment, looked like the desire of destruction was really strong in that one particular brain ;-) So yeah, while its not particularly useful one, there's indeed a situation where one definitely want to run it. disclaimer: I run SELinux playbox with free root access and session recording, and peeking into what others do is also fun. Edit: rm minus rf slash asterisk formatting.

deleted 11y ago

[deleted]

wilkystyle 11y ago

Did he change his ServerFault username? I see commenters referencing @MarcoMarsala, but the OP's name appears to be bleemboy at the moment[0] [0] http://serverfault.com/questions/769357/recovering-from-a-rm-rf http://serverfault.com/questions/769357/recovering-from-a-rm...

maus42 11y ago
Looks like he did, the nick was MarcoMarsala earlier this week.

cha5m 11y ago

I was expecting something much more subtle. I guess the best course of action to prevent this would be to alias rm to a custom script, then parse the arguments to make sure the root directory is never recursively deleted, then calling rm from within your script.

maus42 11y ago

Does this article anything else than paraphrase the Serverfault thread? On the first reading I thought they had contacted the poor fellow to confirm his identity or something, but rereading, it would appear that they didn't: no further information than the original source. https://serverfault.com/questions/769357/recovering-from-a-rm-rf https://serverfault.com/questions/769357/recovering-from-a-r... (The user who asked that question uses now a nick, but had the real-soundish name mentioned in the article when I read that serverfault question first time earlier this week.) edit. ...I really hope there isn't a real Marco Marsala someone pretended to be. Search engine results for that name are not great ATM.

beloch 11y ago

Correct me if I'm wrong, but rm doesn't wipe data out, it just deallocates the disk space devoted to it. If you actually managed to wipe out your entire file system with rm you could likely still recover your data with a recovery tool. This story smells a wee bit fishy to me.

milkey_mouse 11y ago
It's just hard to get the filesystem entries for the file back. rm doesn't specifically wipe, you're right, but the filesystem entries are deleted, which means you basically have to grep the disk for bits of the file you want with known contents.
fabiothebest 11y ago
of course :) it's a troll

sickpig 11y ago

http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 http://serverfault.com/questions/769357/recovering-from-a-rm... "luckily we recovered almost all data!"

jchrisa 11y ago

I lost the private key to one of my AWS servers after it had had a traffic spike due to blog coverage[1]. It was a toy system so it was using local storage, but then it became sort of popular. Luckily I had a process monitor set up so it managed months of uptime before something happened that I couldn't do anything to fix. [1] http://waxy.org/2008/04/exclusive_google_app_engine_ported_to_amazons_ec2/ http://waxy.org/2008/04/exclusive_google_app_engine_ported_t...

OSButler 11y ago

I once had an incident with a server which triggered notification alerts about a failing httpd service. While I was looking into the issue, the mail service suddenly stopped working, then the database service went down - it was like a slow cascading failure, affecting all services on the server one after the other. I finally noticed the 'rm' command in the process list and asked the client if he ran any custom commands as root on the machine. Turns out he followed the instructions on a website to install some custom software without checking any of the commands and just copied & pasted them into the prompt. He basically managed to "rm -rf" on / and deleted his own server. Luckily recent backups were available, so the damage was rather small, but it was interesting to see someone just pasting & executing commands without knowing what they actually do, especially when logged in as root.

ksenzee 11y ago

He says he's recovered almost all the data. FWIW. https://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 https://serverfault.com/questions/769357/recovering-from-a-r...

noonespecial 11y ago

OR "Man learns the value of backups because somethings things go wrong."

Raphmedia 11y ago
From the article: "All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)."
- cyphar 11y ago
  That's not a backup. If your "backup script" requires mounting the backup on a production machine, then it's barely a copy of your data.

adultSwim 11y ago

Oops

Animats 11y ago

This is what comes from treating undefined variables as empty, rather than as errors. Bad language design in the shell.

Xcelerate 11y ago

Toy Story 2 was almost entirely deleted because of this same problem: http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/#gref http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-...

aleden 11y ago

I accidentally set executable permissions on the following Makefile: https://github.com/samalba/acdcontrol/blob/master/Makefile https://github.com/samalba/acdcontrol/blob/master/Makefile While typing quickly I tab-completed to 'Makefile' and hit enter. Although it was a Makefile, it was executed as a bash script. bash ignored the incorrect syntax and executed line 10: rm -rf $(DIRNAME)/* If make parsed the file, $(DIRNAME) would have been nonempty. But it was empty under bash. --no-preserve-root did not protect against this, because the target of the command was '/*'

lazzlazzlazz 11y ago
And here it is.
scintill76 11y ago
How/why does this work without "#!" at the beginning of the file? I just tested with fish shell, and I get "Exec format error. The file './x' is marked as an executable but could not be run by the operating system." But, from bash or dash it does execute commands from the file.
- Tiksi 11y ago
  It'll fall back to sh, the default shell (or whatever it's symlinked to): http://paste.click/QxWUMG http://paste.click/QxWUMG In my case that's bash, debian based systems use dash.
  - cyphar 11y ago
    That shouldn't happen if you look at fs/exec.c (search_binary_handler) there isn't a "fallback to shell" option. And fs/binfmt_script.c doesn't fall back to shell either. Are you sure you don't have some weird binfmt_misc hook enabled?
    - JdeBP 11y ago
      The "it" in what you are replying to is not the kernel; but the shell. * http://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.html#tag_001_009_001_001 http://pubs.opengroup.org/onlinepubs/007908799/xcu/chap2.htm... * http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

jo909 11y ago

While as others already pointed out this story seems a little fishy, it serves well to reflect if something like this could in theory happen to your infrastructure. Do you have your backup servers in the same configuration management software (ansible, puppet, ssh-for-loop etc) as the rest of the servers? One grave error (however unlikely) in your base configuration really can take down everything together in one fell swoop. How "cold" are your backups? If the backup media are not physically disconnected and secured, you can most likely construct a scenario where the above, malware, a hacker or a rouge admin could destroy both the backups and the live data. I will certainly suggest some additional safeguards for our backups.

ansible 11y ago
Yep, that's what I hope everyone will be doing... thinking about their own backups and infrastructure. We have backups off-site on disconnected media, so that alone prevents the kind of accident we're talking about. We use btrfs send / receive to send OS images from the primary container host to the backup container host. The snapshots are read-only, so I'm fairly sure I can't just 'rm -rf' them, I'd have to actually 'btrfs subvolume delete foobar' them. I should try that though on one of the test servers...

BinaryIdiot 11y ago

According to the thread they were able to recover almost all of the data so far. So the whole "deletes his entire company" no longer seems accurate. Still pretty crazy.

anotheryou 11y ago

he got it recovered https://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 https://serverfault.com/questions/769357/recovering-from-a-r...

justinlardinois 11y ago

It's weird to me that commenting on a Stack Exchange question could get you quoted on several major news websites.

Schwolop 11y ago

I managed to sudo chown -R {useless_user}:{useless_user} {foo}/ with foo undefined, whilst simultaneously distributing that command with dsh to our entire cluster of 10 machines. This was after testing that everything worked on the development machine. So of course, I retraced my steps to find out what went wrong, and killed the development machine too. The upside is that we knew we had issues, and with everything broken the impetus is on the right people to ensure they're fixed before we get distracted by the next shiny feature. Sometimes, setting your servers on fire is the solution to technical debt.

mofle 11y ago

There are many ways you can safeguard [0] `rm`, but in the end, it's better to just use a tool that move files to the trash [1] instead. [0]: https://github.com/sindresorhus/guides/blob/master/how-not-to-rm-yourself.md#safeguard-rm https://github.com/sindresorhus/guides/blob/master/how-not-t... [1]: https://github.com/sindresorhus/trash-cli https://github.com/sindresorhus/trash-cli

rurban 11y ago

Shouldn't the proper fix be implying -i on -rf / At least in some fork of the coreutils.

drinchev 11y ago

According to a comment in the ServerFault website, he actually managed to recover the data [1]. He consulted a company for data recovery and they gave him a list with the files that they could manage to save [2]. 1 : http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment970897_769400 http://serverfault.com/questions/769357/recovering-from-a-rm... 2 : http://serverfault.com/questions/769357/recovering-from-a-rm-rf#comment971005_769400 http://serverfault.com/questions/769357/recovering-from-a-rm...

blaze33 11y ago

Instead of doing system administration with root, couldn't we have a system user with the same privileges as root except it wouldn't have write access to the files of some users (like your clients) ? So you could still rm -rf / all you want, delete everything but still have /home or /var/www content untouched. We run certain programs with limited privileges to mitigates risks (bugs, exploits, etc.), why shouldn't we also limit the privileges of root to mitigates the risk of buggy system administration ? Obviously having actual backups and testing your code before applying it to production is good practice but I feel like doing system administration with root while having potential bugs in your sysadmin code (as in any other software) leaves the door open to the next catastrophic failure.

fabiothebest 11y ago

It was a troll. it was just a joke and a marketing idea for promoting the book he is writing. Use Google Translate if you don't speak Italian: http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/ http://www.repubblica.it/tecnologia/2016/04/15/news/cancella...

jmiserez 11y ago

Nothing to see here, it was a hoax/troll: https://meta.serverfault.com/questions/8696/what-to-do-with-the-rm-rf-hoax-question https://meta.serverfault.com/questions/8696/what-to-do-with-...

coldtea 11y ago

>Together, the code deleted everything on the computer, including Mr Masarla’s customers' websites, he wrote. Mr Masarla runs a web hosting company, which looks after the servers and internet connections on which the files for websites are stored. And he has no backups? Including rolling backups in unconnected storage? >Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too. Then the probably probably deserved to die. Sorry for the customers though...

mpdehaan2 11y ago

NOTE: this is a hoax: https://news.ycombinator.com/item?id=11514455 https://news.ycombinator.com/item?id=11514455