13 ms·
My very first job - ~25 years ago. Destroyed the production payroll database for a customer with a bug in a shell script. No problem - they had 3 backup tapes
by arethuza 11y ago
My very first job - ~25 years ago.
Destroyed the production payroll database for a customer with a bug in a shell script.
No problem - they had 3 backup tapes.
First tape - read fails.
Second tape - read fails.
Third tape - worked.... (very nervous at this point).
I think most people have an equivalent educational experience at some point in their careers.
Edit: Had a project cancelled for one customer because they lost the database of test results..... 4 months work! Their COO (quite a large company) actually apologised to me in person!
Edit: Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were "you need to restore from tape" and then he disappeared. I was not happy as it was his attempts at "improving" things were the cause of the incident! Wouldn't have been angry if he had admitted he had made a mistake and worked with us to fix it - simply saying "restore from tape" and running away was not a good approach.
- repomies691 11y agoTo put these in perspective, it is not that uncommon for a years worth of work to go to waste, just for different reasons (eg. wrong assumptions, no market for the product, etc). However when you do mistakes, the process of encountering them and fixing them is much more stressful in the moment, and most of us try to avoid them as much as we can. However it might be that trying and making mistakes is more productive in longer run than avoiding risks and taking safe and calculative approach.
- arethuza 11y ago"wrong assumptions, no market for the product, etc" Been there, done that! However, to stick to the subject of databases - I found later in my career that the really alarming events are more to do with having data you shouldn't have rather than not having data you should. NB The only one of these that I'm going to admit to involved a demo application being sent out on CDs to hundreds of thousands of end-users by IBM and sample data sourced from a particularly unpleasant alt.* newsgroup - I was CTO and when an engineer approached me and rather sheepishly told me that some of the same data had gone to IBM I had a very bad moment indeed. However, on investigation it turned out the only thing that did go out on the CD was the single word "sheep".
- jvoorhis 11y agoAt a startup earlier in my career, we generated an email address for every customer which they could use to interact with our system. The CEO wrote a script to pick words at random from the Princeton Wordnet database. One afternoon I thought, wait a minute, what's actually in Wordnet? Needless to say, many apology emails were sent later that day.
- aptwebapps 11y agoActual two-word passphrase from an AOL disc once in my possession: 'cloaca market'.
- cvwright 11y agoIn recent years, I created a homework assignment for a security class where the student's job is to crack a set of password files. First easy, un-salted hashes, then salted hashes, ... etc. To illustrate the badness of using dictionary words as a password, I randomly generated a unique password file for each user by sampling from /usr/share/dict/words. > One afternoon I thought, wait a minute, what's actually in Wordnet? I know exactly how you feel. It took me about 5 minutes to find Shutterstock's list of dirty, naughty, obscene, and otherwise bad words [1], and about 20 more minutes to add a blacklist check to my script. Fortunately none of the students had actually been given a bad word. Whew! [1] https://github.com/shutterstock/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words https://github.com/shutterstock/List-of-Dirty-Naughty-Obscen...
- twic 11y ago> the really alarming events are more to do with having data you shouldn't have rather than not having data you should the really really alarming events are more to do with your users having data they shouldn't!
- bottled_poe 11y agoMaking mistakes is acceptable - catastrophic failure is a different story.
- spacecowboy_lon 11y agoReminds me of an incident at a very large UK company where the outsourced operator manged to destroy all of the payroll tapes. First run fails second run loads backup tape into a malfunctioning tape unit which then trashes the tape - they kept doing this for all of the backups. This was the second month of the new out sourced payroll system replacing an in house one which had run for decades without incident.
- gr3yh47 11y agoI got fired from a (sysadmin) job because a servo in a tape library failed while i was working 14h days on conference that I was told was "my only priority". We didn't need the backups at all, prod was fine, and I found out 2 business days after the servo failed when the conference was over. apparently 2 days of not having backups due to a mechanical failure out of my control was beyond unacceptable.
- KB1JWQ 11y agoAh, the good old "blameful postmortem." Sounds like a toxic culture...
- bigchewy 11y agoin the early 2000s I was a young guy and board observer on a .com that was growing extremely rapidly and had a 9 day complete outage. The venture funds, including the one I was part of, came in screaming, "who's getting fired for this", to which the CEO responded: "are you kidding me? We just spent $1.5M educating this team and you want me to hand them over to our competition?" Every investor and other board member got real quiet, realizing how correct the CEO was. Good CEOS recognize that mistakes happen. The best make sure they retain the people that have learned through those mistakes.
- pkaye 11y agoPerhaps one of the senior guy should have learned from past mistakes but they are still in this situation?
- btilly 11y agoEarly 2000s, 9 day complete outage. Don't know if that was the same one, but happened to me at Rent.com. The story is that a change in a shell script meant that backups were not actually being sent properly to tape. That was OK, there was another online backup copy. But the restore process deleted that for 1 hour each day before it was recreated. The production database died during that hour. We had to take the last good backup (several months earlier) and replay WAL logs to bring it up to date. The sysadmin whose mistake it was offered her resignation, and was turned down by the head of tech because he knew she wouldn't have made it if she had a more reasonable load. The head of tech offered his resignation to the CEO and was turned down because the CEO knew that it was due to incorrect company priorities. The next tech hire was a DBA whose sole task was make sure that we have multiple levels of verified backups. In less than a year we were sold to eBay at a nice price. Part of the reason was that they thought that the way that we handled failure said very positive things about the organization.
- oolongCat 11y agoNow that's terrifying. Glad it worked out for you. My tifu is when I first started working for an actual client, I thought I was a genius running my own git server on my own computer. Also I had the ssh key for my server on the same computer. When my hdd failed, I lost everything. Now the funny thing was, while my server was running I had no way to gain access to the server, the customer had already started using it, and they had inserted around 400-500 records already. Now there I was, locked out of my own server, no source code I was working on for around 3-4 months. Lucky for me, someone online had mentioned about a way to gain access to an aws server if you lost the ssh key. I created a new instance, and mapped the old storage to the new instance. Lucky for me this worked. Also, 2 weeks(ish) before this blunder I had sent a copy of my code to a friend who was helping me with some issues I had. So my email saved me. Now I backup regularly, use github(recent projects gitlab) and I have my ssh keys on 3 separate pen-drives and this http://www.oakalleyit.com/node/4 http://www.oakalleyit.com/node/4 ----------------
- Splines 11y agoIt's always nice to dodge a bullet, and doubly nice to know where, when and how that bullet was dodged.
- pingec 11y agoHow do you scan that huge qr code?
- elbigbad 11y agoIn much the same way you might scan this billboard-sized QR code located at the top of a building. Namely, by moving backward. :p https://upload.wikimedia.org/wikipedia/commons/thumb/3/32/Japan-qr-code-billboard.jpg/800px-Japan-qr-code-billboard.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/3/32/Ja...
- cmurf 11y agoAnd what's the pass fail rate?
- pc86 11y agoOracle is extremely good at coming in, completely hosing everything, then disappearing and sending you a $350/hr invoice for the privilege. Their consultants seem to tread the line between totally incompetent and actively malevolent surprisingly well.
- lmilcin 11y agoDo you have anything to back your opinion or you had just one bad experience with Oracle consultant and now generalize to tens or hundreds of thousands of cases that you have absolutely no knowledge about? On my part I only have very positive experience with Oracle consultants, but this is only my opinion based on just a few interactions.
- coco1989 11y agohttp://www.thenerve.org/usc-spent-millions-for-software-it-already-had/ http://www.thenerve.org/usc-spent-millions-for-software-it-a... http://www.computerworld.com/article/2493658/it-management/the-scariest-software-project-horror-stories-of-2012.html http://www.computerworld.com/article/2493658/it-management/t... http://www.scmagazine.com/peoplesoft-systems-vulnerable-to-tokenchpoken/article/424863/ http://www.scmagazine.com/peoplesoft-systems-vulnerable-to-t...
- pc86 11y agoI worked for a Fortune 500 healthcare company that used both Oracle and IBM consultants extensively. I had exactly two positive experiences with Oracle consultants where they delivered as promised and maybe four or five where they delivered but took longer/went over budget/etc to the point where I feel we lost value in the long run. And maybe two dozen experiences that were as I described initially. So in my experience somewhere between 5-8% will be worth it.
- rosser 11y agoAt my last job, we used to have an Oracle-acquired storage product in a rather critical part of our stack. It was having some performance problems one night, during a peak traffic period. It may even have been Cyber Monday; I don't remember exactly. Our then systems team lead called Oracle support, and was advised by the support rep — a third tier rep, mind you, because we had priority support, and had the call escalated when the first guy couldn't do anything — to run an on-board low-level diagnostic. He confirmed with the phone rep three separate times that running such a check would be safe to do with live, mounted filesystems, and would not adversely impact operations. He was answered in the affirmative each time. He hit enter. The site went down. $.75mm in lost revenue over the next 8-10 hours, unfucking that pretty little mess.
- Scarblac 11y agoIn my first job (at a pretty small company) I needed to debug something. I made a dump of the production database, downloaded it to my dev machine, then deleted the live production database instead of the one on my dev machine by accident (typed the command in the wrong shell). This happened while the CEO was talking to someone external to the company, right beside my desk. Little adrenalin rush there. Anyway I had of course just made a backup, and only one order was placed while I was downloading it that I could restore by hand using the email that was also sent for each order, so no damage done. But I'll never make that mistake again :-)
- arethuza 11y agoI had a developer who worked on my team at my last job make almost that exact mistake - I literally saw the blood drain from his face. Fortunately we also had the DBAs in the same room - so HR database restored without anyone noticing. Poor chap - I think he got quite a scare. Fun and games... :-)
- schlowmo 11y agoI think the "wrong shell incident" is quite common: "Just need to load this dump to the testing database and I'm on my way to weekend...ok done...oh wait, if I look closer, this shell isn't the testing system...hell noooo, just nuked 8 hours of work each from 20 product managers." Literally happend to me when I was a working student at a DAX 30 corporation. Never happend to me since then again.
- eric_h 11y agoIt's common enough that I made sure our AWS boxes put the name of the current environment both in the bash prompt and rails console prompts (production is in bright red, staging is a nice soothing purple :).
- RockyMcNuts 11y agoI actually make a terminal-window-size PNG of 'PRODUCTION' and 'TEST' and set it as the background gif at like 10% alpha in the terminal windows, which also have different background colors.
- roflc0ptic 11y agoA coworker of mine used to say "It's not the backup, it's the restore." Meaning your backup process isn't meaningful unless you have a tested and effective means for recreating your system from that backup. It has stuck with me.
- arethuza 11y agoIt's important to check that not only can you restore the database but you also check that it contains the data you think it contains. Finding out that your backup consists of a perfectly backed up empty database isn't much fun!
- wlll 11y agoOne of the first things I did at my current place was to add a Nagios check "Is the most recent backup size within x bytes of the previous one". It wasn't comprehensive, but was fast to implement and was later replaced with full restores and checks, but nevertheless a few weeks later it caught a perfectly formed tar.gz of a completely empty database due to a bug that would have continued creating empty archives until it was fixed.
- saidajigumi 11y agoThis is a fantastic illustration of testing 101: often it's the dumbest possible checks catch huge errors. Get those in place before overthinking. E.g. a team adjacent to mine years ago had a dev who made a one-character typo in a commit that went to production. Which caused many $MM to incorrectly flow out the door post-haste. The bad transactions were fortunately reversible with some work, but I was floored that there were no automated tests gating these changes. It wasn't a subtle problem. The most basic, boring integration test of "run a set of X transactions, check the expected sum" would have prevented that failure.
- chris_wot 11y agoAll these threads are complete gold mines of years of hard won experience and system administration tips! Fascinating, especially the war stories I'm reading - both hilarious and horrifying in equal measure.
- technofiend 11y agoI first spelunked through UNIX sometime in 1985/1986; my employer decided to sell Altos UNIX systems along with their CP/M offerings. No documentation, just a running system and a password although one person hinted that man man would help. We used it for our accounting system along with Peachtree? It turned out we didn't have good backups because the system's disk drive was dirty and misaligned. I fixed it. The backup scripts couldn't deal with going over one floppy and failed anyway. I learned a lot of very useful things on that system - like when I created an extra copy of etc with an unprintable character in the name (etc^?) I got to learn all about using stty to set your erase character, ls -lq to find files with unprintable names and inodes along with ls -li and find -inum to fix my errant /etc^?. Unfortunately for my accountant I didn't learn how to make a backup span multiple volumes until some time after I also learned a very hard lesson about unix device names and wiped the system drive while trying to mkfs on a new volume to expand the system. He ended up rekeying all our accounting data from printouts and I ended up buying him more than one lunch.
- lavamantis 11y ago"rekeying all our accounting data from printouts" I am cringing.
- arethuza 11y agoIf you think that's bad, ~12 years ago I worked on a project that was an online insurance form. The output from the application form was going to be printed out and then submitted to the existing paper-based workflow for handling applications - which involved scanning the application and then workers typing in the data from the scanned form.
- engi_nerd 11y agoNot more than 2.5 years ago I wrote a script to take the bill of materials from a CAD program and cross-reference it with a flat-file database (read: Excel spreadsheet) that contained information on how 'old' part numbers mapped to 'new' part numbers. Shame the CAD program couldn't just be scripted to do this. Anyway, the final output of this was a bill of materials that said what you'd expect: I need X quantity of Y parts, here are the part numbers for each. This was emailed to a "warehouse" specialist, who then, I found out, printed out my BOM, then compared it to that day's (printed) inventory of parts, came up with what needed to be issued directly and what needed to be ordered, and then he manually entered that into the actual warehouse logistics system. And lest you think I was lazy, I asked to be allowed to script all this out, too, only to be told "there is no way for us to submit anything to our logistics system that way, everything has to be entered manually." This wasn't some no-name company, this was a major aerospace company.
- mikehollinger 11y agoYep. I interned at my high school over the summer back in the late 90's and (aside from ghosting pcs or helping with Windows login stuff) also managed the tapes for our main server. The underling in the it shop, one afternoon, was showing me something and to prove a point about RAID, pulled one of the drives from the servers raid array, then plugged it in again. It was magic! Fast forward a month. I do the same thing when talking to someone. The next day, the sever is offline. I either pulled the wrong drive, the array was degraded, or some actual failure occurred overnight. I'll never know. I did, however, learn that day about how veritas backups work, and to always know wtf I'm doing before showing off. :-)
- mstade 11y agoAt my first professional full-time gig as a software engineer I had to find a way of dumping data from barcode scanners, after some software written in-house was supposed to download all data, flush it to disk, then empty the scanners for re-use. Problem was that the flushing to disk part didn't work (but said it did) so all data was lost. This was for scanning attendees at conferences, so all data on attendees were seemingly lost. It wasn't me losing the data, and I don't blame the engineer that did, but management sure didn't mind doing that. I took the heat for it by standing up to them and saying "there's a bad news/good news situation here" and refocusing their attention. Bad news was obviously the data looked lost, the good news was it's probably not overwritten so I suggested we write a very small piece of firmware that just does a memory dump – the scanners didn't have separate slots for storage, firmware etc, it was all in one place – and hope the memory is ordered such that the firmware comes first, followed by all the attendee data. It worked – the firmware was indeed written first and dumping it all revealed all attendee data in raw form. Thankfully it was more or less all ascii so it wasn't difficult to process the dump after it'd been downloaded. We had to do this with 20+ scanners since the mistake wasn't discovered right away. I believe management later tried to resell this as a "data recovery solution" to other firms using these types of scanners – don't think they had much success though. The engineer who lost the data at first mostly got off the hook with some blame game and a warning. It wasn't a very good working environment, and prior to losing the data the engineer had been pushing serious overtime hours without so much as a thank you because management had promised delivery without involving engineering in the discussions. A few months later that engineer and several others, including myself, quit for other jobs.
- mipmap04 11y agoOh man I had a project similar to this recently. My team got hired to document this fairly large database (800+ tables, 2000+ SPs). We decided to use a product that actually writes the documentation to the DB as extended properties. The client specifically did not want us to use dev for this, so they put us in test. Immediately this raised some warning bells in my mind so I let them know that risks regarding what could happen if they move from dev to test without notifying me (if I'm notified, I can script out the extended properties and reapply them to the test after the migration is complete). We're about 2 months in and 90% complete. One morning, ops decides to migrate to test and doesn't let me know. One of my team members logs into the system and tells me that everything has disappeared. At this point, I'm sweating bullets and I'm pretty sure that I'm going to be in some tough conference calls the next couple of days. It's a fixed fee project with a client that we have a really good relationship with and I don't want to be the one that destroys it. After about 15 minutes of pondering my demise and hoping that they back-up the test environment, I get notified from the ops team that they create back ups weekly and test monthly for all environments. Luckily for me, this was a Monday morning so we lost maybe a few hours of work. That being said, that was a very stressful 15 minutes and taught me how important it is to: 1) Back-up every environment - storage is cheap and labor often isn't. Having to recreate work would have been way more expensive than having the systems in place to save the potential lost work. Additionally, you never know what data might be lost during testing or work done in test that you really wish you could retrieve. 2) Test your DR processes - their recovery was smooth and well executed and we had reasonable assurance it would work.
- fit2rule 11y agoSeen this from a different angle: watched a rookie admin delete the only copy of the soon-to-be-production database, which was the result of a year of data-mangling/massaging before it could be put online for production .. saw him do the delete, nothing I could about it in time. Spent a week recovering the data, a block at a time .. learned enough about the database filesystem to write an 'un-delete' tool, sold it after using it to successfully recover the original project, and have never looked back ..
- lavamantis 11y agoYes I think everyone has had one of those "Wanna get away?" moments. Hopefully they're very early in your career and only ever happen once. Mine was small and I was very Jr so nothing more than my pride got hurt. The worst is when someone else notices it before you. We came in to work on Monday and some reported that they couldn't log into the system. We checked logs and noticed there were timeout errors on some SQL queries. Huh, that's odd, why is there a query "update Users set deletedat=now()" with no "where" clause running in production??? Turns out someone was in a hurry and tried to make a test pass without thinking to much about what the code was doing. Then - push right to prod on a Friday evening!
- matwood 11y agoYou do not have backups until you have tested the restore. Typically, I restore the production backups to the test environment. This way test gets up to date data and the backup restores are tested. I have so many stories of data loss from many years working, that I do nothing without backups. From fat finger deletes to raid card batteries failing to multiple disks in the raid array failing at the same time. Data will be lost at some time with the only recovery method being a backup.
- kbenson 11y ago> Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were "you need to restore from tape" and then he disappeared. Tech: "Err, boss, I just totally broke their entire system" Manager: "That's no problem, you got them to sign that contract, right? We always pad those with more services than they have employees or customers to use, so we can use that to cover any problems." Tech: "..." Manager: "Johnson, I'm not hearing the confirmation I expected." Tech: "Sorry boss, there was an issue they were keen to have me look at and... well..." Manager: "Johnson, you get your ass back here now. Do not pass go. Do not collect $200. The phone will not accurately convey the ass-chewing you are about to receive, and you don't deserve the paltry protection it would allow." <click> Tech looks up to see local tech liason. Tech: "Ehm. You need to restore from backup."
- fallinghawks 11y agoSame deal here, we'd been backing up the servers to tape and dutifully boxing and taking one set offsite. Finally someone deleted something by accident and that's when I discovered you should verify that you can read the tape once in a while. Fortunately, not an important document, but that was just luck.
- Arnt 11y agoYou'd like the system administrator at a medical research facility where I once worked. Every Monday around noon, he deleted a file, a randomly file from a randomly chosen production system, then restored that file from backup. I've always admired that, but never emulated it...
- pg_is_a_butt 11y agoprobably because he's obviously creating a smoke screen to cover up the change he made to the provasic liver test results.
- justinclift 11y agoHeh Heh Heh On one contract about a decade ago, the manager used to proactively screw around with server hardware (eg pull a hdd out of a raid set) when bored, to ensure our recovery processes were always battle tested. Didn't generally cause too many issues... until the one time he pulled a working hdd out of a raid5 set that already had a (real) failed drive, which we were already attempting to recover. It was a production database, but thankfully not a mission critical one :>. From memory, he stopped testing things like that for a while afterwards. ;)
- cvwright 11y agoWow, a real-life chaos monkey! https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- justinclift 11y agoYeah, the similarity hasn't escaped me either. ;)
- greggman 11y agotyped del c:\*.* instead of what I meant dir c:\*.* On my bosses PC right in front of him. Fortunately someone had a copy of norton tools back then (~1987)