5 ms·

I agree, it's the fault of the CTO. To me, the CTO sounds pretty incompetent. The junior engineer did them a favor. This company seems like it is an amateur h

I agree, it's the fault of the CTO. To me, the CTO sounds pretty incompetent. The junior engineer did them a favor. This company seems like it is an amateur hour operation, since data was deleted so easily by an junior engineer.

dheera 9y ago
Agreed. Also, why didn't they have a backup of some sort? The hard drive on the server could have failed and it would have been just as bad. Sounds like an incompetent set of people running the production server.
- yulaow 9y ago
  As a lot of companies I bet they HAVE backup, just never tested if the backup process works. It is absurdly common...
  - gaius 9y ago
    This is trivial tho'. Just setup a regular refresh of the dev env via the backup system. Sure it takes longer because you have to read the tapes back but it's worth it for the peace of mind, and it means that every dev knows how to get at the tapes if they need to.
    - joncrocks 9y ago
      Well yes, but there's a string of WTFs here, lots of 'trivial' stuff that wasn't done!
  - csydas 9y ago
    Most likely something like this. There is probably backup software running but it's either nothing but failed jobs or misconfigured so the backups aren't working correctly.
    - lb1lf 9y ago
      My favourite in that regard is an anecdote shared by a customer; he said one of their techs had discovered (luckily in time) that the long-running automated backup script on a small, but important server wrote the backup volume to... ...RAM.
      - hulahoof 9y ago
        And here I've been wondering how could it get worse than writing to local storage ...
        csydas 9y ago
        I can tell you since I work support for a backup product. Lot of people think that naming a folder on a local drive "Disaster Recovery" counts as having an offsite Disaster Recovery copy of backups. The number of large corporations whose backups are in the hands of such people is frightening.
  - dennisgorelik 9y ago
    From my experience, if I do not test my backups they stop working in about 1 year. So if I do not test backups for over a year then my assumption is that I probably do not have working backups.
- deleted 9y ago
  [deleted]
- h1d 9y ago
  OP says the team is 40+ and CTO just let them all walk on a catwalk.
vvanders 9y ago
Yup, I've heard stories of junior engineers causing millions of dollars worth out outages. In those case the process was drilled into, the control that caused it fixed and the engineer was not given a reprimand. If you have an engineer that goes though that and shows real remorse your going to have someone who's never going to make that mistake(or similar ones) again.
- leesalminen 9y ago
  Agreed. Several years ago as a junior dev I was tasked with adding a new feature- only allowing a user to have 1 active session. So, we added a "roadblock" post auth with 2 actions- log out other sessions and log out this session. Well, the db query for the first action (log out other sessions) was missing a where clause...a user_id! Tickets started pouring in saying users were logged out and didn't know why. Luckily the on-call dev knew there was a recent release and was able to identify the missing where clause and added it within the hour. The feature made it through code review, so the team acknowledged that everyone was at fault. Instead of being reprimanded, we decided to revamp our code review process. I never made that kind of mistake again. To this day, I'm a little paranoid about update/delete queries.
  - redler 9y ago
    We all make this mistake eventually, often in far more spectacular fashion. My lessons learned are 1) Always have a USE statement (or equivalent); 2) Always start UPDATE or DELETE queries by writing them as SELECT; 3) Get in the habit of writing the WHERE clause first; 4) If your SQL authoring environment supports the dangerous and seductive feature where you can select some text in the window and then run only that selected text — beware! and 5) While developing a query to manipulate real data, consider topping the whole thing with BEGIN TRANSACTION (or equivalent), with both COMMIT and ROLLBACK at the end, both commented out (this is the one case where I use the run-selected-area feature: after evaluating results, select either the COMMIT or the ROLLBACK, and run-selected). Not all of these apply to writing queries that will live in an application, and I don't do all these things all the time — but I try to take this stance when approaching writing meaningful SQL.
karmajunkie 9y ago
Yep. I had a junior working for me once a few years ago that made a rather unfortunate error in production which deleted all of several customers' data. I could tell he was on pins and needles when he brought it to me, so I let him off the hook right away and showed him the procedures to fix the issue. He said something about being thankful there was a way to fix the problem, and I just smiled and told him A) it would have been my fault if there hadn't been; and B) he wouldn't have had the access he did without safeguards in place. Then I told him a story about the time I managed to accidentally delete an entire database of quarantined email from a spam appliance I was working on several years earlier. Sadly, my CTO at the time did NOT prepare for that. I lost a whole weekend of sleep in recovering that one from logs, and that was when I learned some good tricks for ensuring recoverability....

dheera 9y ago

Agreed. Also, why didn't they have a backup of some sort? The hard drive on the server could have failed and it would have been just as bad. Sounds like an incompetent set of people running the production server.

yulaow 9y ago
As a lot of companies I bet they HAVE backup, just never tested if the backup process works. It is absurdly common...
- gaius 9y ago
  This is trivial tho'. Just setup a regular refresh of the dev env via the backup system. Sure it takes longer because you have to read the tapes back but it's worth it for the peace of mind, and it means that every dev knows how to get at the tapes if they need to.
  - joncrocks 9y ago
    Well yes, but there's a string of WTFs here, lots of 'trivial' stuff that wasn't done!
- csydas 9y ago
  Most likely something like this. There is probably backup software running but it's either nothing but failed jobs or misconfigured so the backups aren't working correctly.
  - lb1lf 9y ago
    My favourite in that regard is an anecdote shared by a customer; he said one of their techs had discovered (luckily in time) that the long-running automated backup script on a small, but important server wrote the backup volume to... ...RAM.
    - hulahoof 9y ago
      And here I've been wondering how could it get worse than writing to local storage ...
      - csydas 9y ago
        I can tell you since I work support for a backup product. Lot of people think that naming a folder on a local drive "Disaster Recovery" counts as having an offsite Disaster Recovery copy of backups. The number of large corporations whose backups are in the hands of such people is frightening.
- dennisgorelik 9y ago
  From my experience, if I do not test my backups they stop working in about 1 year. So if I do not test backups for over a year then my assumption is that I probably do not have working backups.
deleted 9y ago
[deleted]
h1d 9y ago
OP says the team is 40+ and CTO just let them all walk on a catwalk.

vvanders 9y ago

Yup, I've heard stories of junior engineers causing millions of dollars worth out outages. In those case the process was drilled into, the control that caused it fixed and the engineer was not given a reprimand. If you have an engineer that goes though that and shows real remorse your going to have someone who's never going to make that mistake(or similar ones) again.

leesalminen 9y ago
Agreed. Several years ago as a junior dev I was tasked with adding a new feature- only allowing a user to have 1 active session. So, we added a "roadblock" post auth with 2 actions- log out other sessions and log out this session. Well, the db query for the first action (log out other sessions) was missing a where clause...a user_id! Tickets started pouring in saying users were logged out and didn't know why. Luckily the on-call dev knew there was a recent release and was able to identify the missing where clause and added it within the hour. The feature made it through code review, so the team acknowledged that everyone was at fault. Instead of being reprimanded, we decided to revamp our code review process. I never made that kind of mistake again. To this day, I'm a little paranoid about update/delete queries.
- redler 9y ago
  We all make this mistake eventually, often in far more spectacular fashion. My lessons learned are 1) Always have a USE statement (or equivalent); 2) Always start UPDATE or DELETE queries by writing them as SELECT; 3) Get in the habit of writing the WHERE clause first; 4) If your SQL authoring environment supports the dangerous and seductive feature where you can select some text in the window and then run only that selected text — beware! and 5) While developing a query to manipulate real data, consider topping the whole thing with BEGIN TRANSACTION (or equivalent), with both COMMIT and ROLLBACK at the end, both commented out (this is the one case where I use the run-selected-area feature: after evaluating results, select either the COMMIT or the ROLLBACK, and run-selected). Not all of these apply to writing queries that will live in an application, and I don't do all these things all the time — but I try to take this stance when approaching writing meaningful SQL.

karmajunkie 9y ago

Yep. I had a junior working for me once a few years ago that made a rather unfortunate error in production which deleted all of several customers' data. I could tell he was on pins and needles when he brought it to me, so I let him off the hook right away and showed him the procedures to fix the issue. He said something about being thankful there was a way to fix the problem, and I just smiled and told him A) it would have been my fault if there hadn't been; and B) he wouldn't have had the access he did without safeguards in place. Then I told him a story about the time I managed to accidentally delete an entire database of quarantined email from a spam appliance I was working on several years earlier. Sadly, my CTO at the time did NOT prepare for that. I lost a whole weekend of sleep in recovering that one from logs, and that was when I learned some good tricks for ensuring recoverability....