10 ms·

Sorry, but if a junior dev can blow away your prod database by running a script on his _local_ dev environment while following your documentation, you have no o

by Rezo 9y ago

Sorry, but if a junior dev can blow away your prod database by running a script on his _local_ dev environment while following your documentation, you have no one to blame but yourself. Why is your prod database even reachable from his local env? What does the rest of your security look like? Swiss cheese I bet. The CTO further demonstrates his ineptitude by firing the junior dev. Apparently he never heard the famous IBM story, and will surely live to repeat his mistakes: After an employee made a mistake that cost the company $10 million, he walked into the office of Tom Watson, the C.E.O., expecting to get fired. “Fire you?” Mr. Watson asked. “I just spent $10 million educating you.”

jdietrich 9y ago
If someone on their first day of work can do this much damage, what could a disgruntled veteran do? If Snowden has taught us anything, it's that internal threats are just as dangerous as external threats. This shop sounds like a raging tire fire of negligence.
hashkb 9y ago
He didn't follow the docs exactly. That doesn't matter, though, your first day should be bulletproof and if it's not, it's on the CTO. The buck does not stop with junior engineers on their first day.
- austenallred 9y ago
  > He didn't follow the docs exactly Sure, but having the plaintext credentials for a readily-deletable prod db as an example before you instruct someone to wipe the db doesn't salvage competence very much.
  - macns 9y ago
    I wouldn't be surprised if the actual production db was never properly named and was left with an example name.
- FLUX-YOU 9y ago
  Don't tell Etsy that
champagnepapi 9y ago
I agree, it's the fault of the CTO. To me, the CTO sounds pretty incompetent. The junior engineer did them a favor. This company seems like it is an amateur hour operation, since data was deleted so easily by an junior engineer.
- dheera 9y ago
  Agreed. Also, why didn't they have a backup of some sort? The hard drive on the server could have failed and it would have been just as bad. Sounds like an incompetent set of people running the production server.
  - yulaow 9y ago
    As a lot of companies I bet they HAVE backup, just never tested if the backup process works. It is absurdly common...
    - gaius 9y ago
      This is trivial tho'. Just setup a regular refresh of the dev env via the backup system. Sure it takes longer because you have to read the tapes back but it's worth it for the peace of mind, and it means that every dev knows how to get at the tapes if they need to.
      - joncrocks 9y ago
        Well yes, but there's a string of WTFs here, lots of 'trivial' stuff that wasn't done!
    - csydas 9y ago
      Most likely something like this. There is probably backup software running but it's either nothing but failed jobs or misconfigured so the backups aren't working correctly.
      - lb1lf 9y ago
justbaker 9y ago
"It's your first day, we don't understand security so here's the combination to the safe. Have fun!!"
- cwilkes 9y ago
  "we have a bunch of guns, we aren't sure which ones are loaded, all the safeties are off and we modified them to go off randomly"
  - SkyMarshal 9y ago
    "your first day's task will be to learn how to use them by putting them to the heads of our best revenue-generating sales people and pulling the trigger. don't worry it's safe, we'll check back in with you at the end of the day."
mschwaig 9y ago
He might be inept, but in this instance the CTO is mainly just covering his own ass.
- mikeryan 9y ago
  "Yeah the whole site is buggered, and the backups aren't working - but I fired the Junior developer who did it" Is not how you Cover Your Ass ™.
  - mschwaig 9y ago
    You can be inept WHILE covering your ass. I'm not saying he's a genius.
    - yardie 9y ago
      Blaming the new guy is not covering your ass. Blaming the senior engineer who put the credentials in the document would be covering your ass.
Rezo 9y ago
Here's some simple practical tips you can use to prevent this and other Oh Shit Moments(tm): - Unless you have full time DBAs, do use a managed db like RDS, so you don't have to worry about whether you've setup the backups correctly. Saving a few bucks here is incredibly shortsighted, your database is probably the most valuable asset you have. RDS allows point-in-time restore of your DB instance to any second during your retention period, up to the last five minutes. That will make you sleep better at night. - Separate your prod and dev AWS accounts entirely. It doesn't cost you anything (in fact, you get 2x the AWS free tier benefit, score!), and it's also a big help in monitoring your cloud spend later on. Everyone, including the junior dev, should have full access to the dev environment. Fewer people should have prod access (everything devs may need for day-to-day work like logs should be streamed to some other accessible system, like Splunk or Loggly). Assuming a prod context should always require an additional step for those with access, and the separate AWS account provides that bit of friction. - The prod RDS security group should only allow traffic from white listed security groups also in the prod environment. For those really requiring a connection to the prod DB, it is therefore always a two-step process: local -> prod host -> prod db. But carefully consider why are you even doing this in the first place? If you find yourself doing this often, perhaps you need more internal tooling (like an admin interface, again behind a whitelisting SG). - Use a discovery service for the prod resources. One of the simplest methods is just to setup a Route 53 Private Hosted Zone in the prod account, which takes about a minute. Create an alias entry like "db.prod.private" pointing to the RDS and use that in all configurations. Except for the Route 53 record, the actual address for your DB should not appear anywhere. Even if everything else goes sideways, you've assumed a prod context locally by mistake and you run some tool that is pointed to the prod config, the address doesn't resolve in a local context.
- daxfohl 9y ago
  Would you recommend all these steps even for a single-person freelance job? Or is it overkill?
  - _jal 9y ago
    Depends. Do you make mistakes? I absolutely do. "Wrong terminal", "Wrong database", etc. mistakes are very easy to make in certain contexts. The trick is to find circuit-breakers that work for you. Some of the above is probably overkill for one-person shops. You want some sort of safeguard at the same points, but not necessarily the same type. This doesn't really do it for me, but one person I know uses iTerm configured to change terminal colors depending on machine, EUID, etc. as a way of avoiding mistakes. That works for him. I do tend to place heavier-weight restrictions, because they usually overlap with security and I'm a bit paranoid by nature and prefer explicit rules for these things to looser setups. Also, I don't use RDS. I'd recommend looking at what sort of mistakes you've made in the past and how to adjust your workflow to add circuit breakers where needed. Then, if you need to, supplement that. Except for the advice about backups and PITR. Do that. Also, if you're not, use version control for non-DB assets and config!
_jal 9y ago
Seriously. The CTO in question is the incompetent one. S/he failed: - Access control 101. Seriously, this is pure incompetence. It is the equivalent of having the power cord to the Big Important Money Making Machine snaking across the office and under desks. If you can't be arsed to ensure that even basic measures are taken to avoid accidents, acting surprised when they happen is even more stupid. - Sensible onboarding documentation. Why would prod access information be stuck in the "read this first" doc? - Management 101. You just hired a green dev just out of college who has no idea how things are supposed to work. You just fired him in an incredibly nasty way for making an entirely predictable mistake that came about because of your lack of diligence at your job (see above). Also, I have no idea what your culture looks like, but you just told all your reports that honest mistakes can be fatal and their manager's judgement resembles that of a petulant 14 year-old. - Corporate Communications 101. Hindsight and all that, but it seems inevitable that this would lead to a social media trash fire. Congrats on embarrassing yourself and your company in an impressive way. On the bright side, this will last for about 15 minutes and then maybe three people will remember. Hopefully the folks at your next gig won't be among them. My take away is that anyone involved in this might want to start polishing their resumes. The poor kid and the CTO for obvious reasons, and the rest of the devs, because good lord, that company sounds doomed.
- dkrich 9y ago
  Yeah when I read that my first thought was that the CTO reacted that way because he was in fear of being fired himself. I wouldn't be at all surprised if he wrote that document or approved it himself.
  - sillysaurus3 9y ago
    So at what point are you allowed to fire someone for being incompetent? Blowing away the production database seems to rank pretty high. Note that I'm not talking about the situation in this article. That was a ridiculous situation and they were just asking for trouble. I'm asking about the perception that is becoming more and more common, which is that no matter what mistakes you make you should still be given a free pass regardless of severity. Is it the quantity of mistakes? Severity of mistakes? At what point does the calculus favor firing someone over retaining them?
ajeet_dhaliwal 9y ago
Thanks for Tom Watson quote, I'd never heard it before, it's a good one. Also agree with everything else you just said, this is not the junior devs fault at all.

by Rezo 9y ago

jdietrich 9y ago

If someone on their first day of work can do this much damage, what could a disgruntled veteran do? If Snowden has taught us anything, it's that internal threats are just as dangerous as external threats. This shop sounds like a raging tire fire of negligence.

hashkb 9y ago

He didn't follow the docs exactly. That doesn't matter, though, your first day should be bulletproof and if it's not, it's on the CTO. The buck does not stop with junior engineers on their first day.

austenallred 9y ago
> He didn't follow the docs exactly Sure, but having the plaintext credentials for a readily-deletable prod db as an example before you instruct someone to wipe the db doesn't salvage competence very much.
- macns 9y ago
  I wouldn't be surprised if the actual production db was never properly named and was left with an example name.
FLUX-YOU 9y ago
Don't tell Etsy that

champagnepapi 9y ago

I agree, it's the fault of the CTO. To me, the CTO sounds pretty incompetent. The junior engineer did them a favor. This company seems like it is an amateur hour operation, since data was deleted so easily by an junior engineer.

dheera 9y ago
Agreed. Also, why didn't they have a backup of some sort? The hard drive on the server could have failed and it would have been just as bad. Sounds like an incompetent set of people running the production server.
- yulaow 9y ago
  As a lot of companies I bet they HAVE backup, just never tested if the backup process works. It is absurdly common...
  - gaius 9y ago
    This is trivial tho'. Just setup a regular refresh of the dev env via the backup system. Sure it takes longer because you have to read the tapes back but it's worth it for the peace of mind, and it means that every dev knows how to get at the tapes if they need to.
    - joncrocks 9y ago
      Well yes, but there's a string of WTFs here, lots of 'trivial' stuff that wasn't done!
  - csydas 9y ago
    Most likely something like this. There is probably backup software running but it's either nothing but failed jobs or misconfigured so the backups aren't working correctly.
    - lb1lf 9y ago

justbaker 9y ago

"It's your first day, we don't understand security so here's the combination to the safe. Have fun!!"

cwilkes 9y ago
"we have a bunch of guns, we aren't sure which ones are loaded, all the safeties are off and we modified them to go off randomly"
- SkyMarshal 9y ago
  "your first day's task will be to learn how to use them by putting them to the heads of our best revenue-generating sales people and pulling the trigger. don't worry it's safe, we'll check back in with you at the end of the day."

mschwaig 9y ago

He might be inept, but in this instance the CTO is mainly just covering his own ass.

mikeryan 9y ago
"Yeah the whole site is buggered, and the backups aren't working - but I fired the Junior developer who did it" Is not how you Cover Your Ass ™.
- mschwaig 9y ago
  You can be inept WHILE covering your ass. I'm not saying he's a genius.
  - yardie 9y ago
    Blaming the new guy is not covering your ass. Blaming the senior engineer who put the credentials in the document would be covering your ass.

Rezo 9y ago

Here's some simple practical tips you can use to prevent this and other Oh Shit Moments(tm): - Unless you have full time DBAs, do use a managed db like RDS, so you don't have to worry about whether you've setup the backups correctly. Saving a few bucks here is incredibly shortsighted, your database is probably the most valuable asset you have. RDS allows point-in-time restore of your DB instance to any second during your retention period, up to the last five minutes. That will make you sleep better at night. - Separate your prod and dev AWS accounts entirely. It doesn't cost you anything (in fact, you get 2x the AWS free tier benefit, score!), and it's also a big help in monitoring your cloud spend later on. Everyone, including the junior dev, should have full access to the dev environment. Fewer people should have prod access (everything devs may need for day-to-day work like logs should be streamed to some other accessible system, like Splunk or Loggly). Assuming a prod context should always require an additional step for those with access, and the separate AWS account provides that bit of friction. - The prod RDS security group should only allow traffic from white listed security groups also in the prod environment. For those really requiring a connection to the prod DB, it is therefore always a two-step process: local -> prod host -> prod db. But carefully consider why are you even doing this in the first place? If you find yourself doing this often, perhaps you need more internal tooling (like an admin interface, again behind a whitelisting SG). - Use a discovery service for the prod resources. One of the simplest methods is just to setup a Route 53 Private Hosted Zone in the prod account, which takes about a minute. Create an alias entry like "db.prod.private" pointing to the RDS and use that in all configurations. Except for the Route 53 record, the actual address for your DB should not appear anywhere. Even if everything else goes sideways, you've assumed a prod context locally by mistake and you run some tool that is pointed to the prod config, the address doesn't resolve in a local context.

daxfohl 9y ago
Would you recommend all these steps even for a single-person freelance job? Or is it overkill?
- _jal 9y ago
  Depends. Do you make mistakes? I absolutely do. "Wrong terminal", "Wrong database", etc. mistakes are very easy to make in certain contexts. The trick is to find circuit-breakers that work for you. Some of the above is probably overkill for one-person shops. You want some sort of safeguard at the same points, but not necessarily the same type. This doesn't really do it for me, but one person I know uses iTerm configured to change terminal colors depending on machine, EUID, etc. as a way of avoiding mistakes. That works for him. I do tend to place heavier-weight restrictions, because they usually overlap with security and I'm a bit paranoid by nature and prefer explicit rules for these things to looser setups. Also, I don't use RDS. I'd recommend looking at what sort of mistakes you've made in the past and how to adjust your workflow to add circuit breakers where needed. Then, if you need to, supplement that. Except for the advice about backups and PITR. Do that. Also, if you're not, use version control for non-DB assets and config!

_jal 9y ago

Seriously. The CTO in question is the incompetent one. S/he failed: - Access control 101. Seriously, this is pure incompetence. It is the equivalent of having the power cord to the Big Important Money Making Machine snaking across the office and under desks. If you can't be arsed to ensure that even basic measures are taken to avoid accidents, acting surprised when they happen is even more stupid. - Sensible onboarding documentation. Why would prod access information be stuck in the "read this first" doc? - Management 101. You just hired a green dev just out of college who has no idea how things are supposed to work. You just fired him in an incredibly nasty way for making an entirely predictable mistake that came about because of your lack of diligence at your job (see above). Also, I have no idea what your culture looks like, but you just told all your reports that honest mistakes can be fatal and their manager's judgement resembles that of a petulant 14 year-old. - Corporate Communications 101. Hindsight and all that, but it seems inevitable that this would lead to a social media trash fire. Congrats on embarrassing yourself and your company in an impressive way. On the bright side, this will last for about 15 minutes and then maybe three people will remember. Hopefully the folks at your next gig won't be among them. My take away is that anyone involved in this might want to start polishing their resumes. The poor kid and the CTO for obvious reasons, and the rest of the devs, because good lord, that company sounds doomed.

dkrich 9y ago
Yeah when I read that my first thought was that the CTO reacted that way because he was in fear of being fired himself. I wouldn't be at all surprised if he wrote that document or approved it himself.
- sillysaurus3 9y ago
  So at what point are you allowed to fire someone for being incompetent? Blowing away the production database seems to rank pretty high. Note that I'm not talking about the situation in this article. That was a ridiculous situation and they were just asking for trouble. I'm asking about the perception that is becoming more and more common, which is that no matter what mistakes you make you should still be given a free pass regardless of severity. Is it the quantity of mistakes? Severity of mistakes? At what point does the calculus favor firing someone over retaining them?

ajeet_dhaliwal 9y ago

Thanks for Tom Watson quote, I'd never heard it before, it's a good one. Also agree with everything else you just said, this is not the junior devs fault at all.