5 ms·

RIP the engineer

by ky738 10y ago

RIP the engineer

YorickPeterse 10y ago
Still here, and doing just fine.
- boulos 10y ago
  I enjoy your updated profile.
- sslalready 10y ago
  I love your tag line here on HN! Some 15 yrs ago I was hired as a UNIX administrator at some larger company. Despite being fresh from school I already had plenty of experience from spending the 90's hacking and programming on whatever UNIX system fyodor or rootshell.com had an exploit for. When the DBA was leaving for vaccation they didn't hesitate in letting me take over his daily routines. On this particular summer day, I had a simple job: dump the production database and load the data into the test environment. I had to make sure the dump was finished before 4pm when the daily production run started (which, FWIW, continued well into the evening). This was an Oracle shop so I believe the commands were mostly "exp" and "imp" -- with the caveat that the imp command would need an additional parameter to select the test database instead of the production one that was the default. Yeah, you see where this is going. The prod dump finished in time and shortly before leaving work I started importing the data. Then I sat around for a while before I realized I had forgotten about that additional "use the test environment" parameter -- and now I was importing a several hours old dump into the production database while the daily production run was running. I had to call company execs and explain the catastrophe to them, who in turn had to call in the vendor that sold us the system. Those were some pretty scary hours for a 20 yr old kid. Luckily it was just a matter of aborting the production run, reload the prod dump and then reschedule the production run for the day. The next day I had to start my day at the vendor's place to get some shaming, but also a good piece of advice - "always say destructive things out loud before doing them". Then they continued to tell me stories of people they had worked with who really messed things up, and we all had some good, evil laughs. Mistakes build experience, and hard learned lessons even more so. You now have a pretty good conversation starter to put on your CV. Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. :) I believe both GitLab and the community in general will come out stronger from this incident. Thank you all for being so transparent about it.
  - fiedzia 10y ago
    >Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. Oh absolutely. Problem is that gitlab didn't do that. And they will most likely have to learn many more lessons before become reliable.
- tuyguntn 10y ago
  Probably this fail makes Gitlab more reliable in the future. What doesn't kill you makes you stronger.
AsyncAwait 10y ago
Actually, I'll be surprised if he hasn't received any offers by now. I would perhaps specifically hire him to deal with databases as I am pretty sure he's never going to make this mistake again.
- YorickPeterse 10y ago
  I haven't received any offers so far. I don't intend on leaving GitLab any time soon either.

RIP the engineer

by ky738 10y ago

RIP the engineer

YorickPeterse 10y ago

Still here, and doing just fine.

boulos 10y ago
I enjoy your updated profile.
sslalready 10y ago
I love your tag line here on HN! Some 15 yrs ago I was hired as a UNIX administrator at some larger company. Despite being fresh from school I already had plenty of experience from spending the 90's hacking and programming on whatever UNIX system fyodor or rootshell.com had an exploit for. When the DBA was leaving for vaccation they didn't hesitate in letting me take over his daily routines. On this particular summer day, I had a simple job: dump the production database and load the data into the test environment. I had to make sure the dump was finished before 4pm when the daily production run started (which, FWIW, continued well into the evening). This was an Oracle shop so I believe the commands were mostly "exp" and "imp" -- with the caveat that the imp command would need an additional parameter to select the test database instead of the production one that was the default. Yeah, you see where this is going. The prod dump finished in time and shortly before leaving work I started importing the data. Then I sat around for a while before I realized I had forgotten about that additional "use the test environment" parameter -- and now I was importing a several hours old dump into the production database while the daily production run was running. I had to call company execs and explain the catastrophe to them, who in turn had to call in the vendor that sold us the system. Those were some pretty scary hours for a 20 yr old kid. Luckily it was just a matter of aborting the production run, reload the prod dump and then reschedule the production run for the day. The next day I had to start my day at the vendor's place to get some shaming, but also a good piece of advice - "always say destructive things out loud before doing them". Then they continued to tell me stories of people they had worked with who really messed things up, and we all had some good, evil laughs. Mistakes build experience, and hard learned lessons even more so. You now have a pretty good conversation starter to put on your CV. Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. :) I believe both GitLab and the community in general will come out stronger from this incident. Thank you all for being so transparent about it.
- fiedzia 10y ago
  >Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. Oh absolutely. Problem is that gitlab didn't do that. And they will most likely have to learn many more lessons before become reliable.
tuyguntn 10y ago
Probably this fail makes Gitlab more reliable in the future. What doesn't kill you makes you stronger.

AsyncAwait 10y ago

Actually, I'll be surprised if he hasn't received any offers by now. I would perhaps specifically hire him to deal with databases as I am pretty sure he's never going to make this mistake again.

YorickPeterse 10y ago
I haven't received any offers so far. I don't intend on leaving GitLab any time soon either.