5 ms·
Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave. http://edu.mkrecny.com/thoughts/how-i-fired-myself http://e
by mkrecny 13y ago
Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave.
http://edu.mkrecny.com/thoughts/how-i-fired-myself http://edu.mkrecny.com/thoughts/how-i-fired-myself
"Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.
- thejosh 13y agoI remember reading a while ago and cringing at this so bad, this needs to be a cautionary tale of version control, development/staging/production environments and ... backups.
- mkrecny 13y ago"The Startup CTO's Guide to ..." ?
- thejosh 13y agoall it contains is the word: "backups".
- angersock 13y agoMultiple times, for redundancy. It should probably also contain the word "documentation of system configurations".
- sliverstorm 13y agoBut why would a rockstar-ninja-pirate ever need backups?
- FireBeyond 13y agoBackups are not MVP! First to market!
- gabemart 13y agoI got cold sweats reading that story. It's a shame you had to pay the price for the abysmal operating practices of the company.
- mkrecny 13y agoThanks, but ultimately, leaving when I did was extremely positive. I ended up in NYC in the employ of Techstars which turned out to be a much better place for me.
- highace 13y agoGreat story, I remember reading it the first time around. How they didn't explode sooner from that set up and and managed to get as far as they did was probably the most shocking part of it.
- kstop 13y agoFWIW, whoever built a system that had no redundancy or disaster recovery plan or even properly separated environments was responsible. You were just unlucky.
- paulhauggis 13y agoI did something similar about 10 years ago at my first dev job. I forgot a "WHERE" in an update and destroyed every customer in the live db. The issue was that my manager never caught it in my migration file and it went live. I was lucky that we backed up every day..and didn't really lose a ton of data.
- cousin_it 13y agoI never really understood why databases don't have undo.
- specialist 13y agoI largely avoid UPDATE and DELETE statements. Changes are new rows, time ordered, which supersede prior rows. Like an audit log. Works great. Especially for anything that needs history, e.g. medical data.
- brongondwana 13y agoWrote one of those many years ago, yes - for medical data. It's the right solution to just about every problem actually, and I'm surprised that it isn't more commonly taught.
- GFischer 13y agoYeah, logical deletes ftw :) . One thing I haven't figured out is how to archive old records, I think it's not a problem for modern DBMSs, but we have a problem at work with 15 years' worth of historical data.
- gaius 13y agoThey do: http://www.oracle.com/technetwork/database/features/availability/flashback-overview-082751.html http://www.oracle.com/technetwork/database/features/availabi...
- salvadors 13y ago
- wvenable 13y agoFor some reason this is one of the worst of these I've ever read. I've personally f'd up pretty bad restoring a database to the wrong the site and blowing away 3 months of content. We didn't ever do any offsite backups and I just lucky to have an in with our hosting company and they managed to get everything back but those 3 months. Once bitten, you're always way more cautious but it's easy to get lazy again from time to time.
- federalemployee 13y agoThis is why as a dev I never want anything more than read permission with the production environment.
- MichaelGG 13y agoYou may have been the straw that broke the camel's back, but you shouldn't feel bad about that. I recently had a junior sysadmin learn the hard way about BEGIN TRAN/ROLLBACK/COMMIT. But that's all it was, an embarrassing lesson as he explained to a customer why they'd need to wait a little bit longer. Even a major "DELETE FROM Foo" should just immediately get the DBA switching to read-only mode, and then restoring using the transaction logs. It's annoying, it causes a bit of downtime, but it's not a "Big Deal".
- brongondwana 13y agoYeah, that would be a nasty mess for us too... the change would replicate to all our databases and we'd have to stop the world, restore from the previous day's backups, re-run all the database binlogs up until the point where the disaster happened. It would be a couple of hours of total downtime - not a happy place. But we do have daily backups and binlogs in multiple places, so it wouldn't actually lead to data loss. It really sounds like you were lucky NOT to be there!