4 ms·
There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that'
by lifeisstillgood 14y ago
There are a number of comments that add up to "what steps will you take to ensure this does not happen again" - akin to a incident review. As speculation that's fine, as advice, I don't think it should be listened to.
I am reminded of an long-in-the-tooth sysadmin of my acquaintance who logged in everywhere as root. His theory - "they are my boxes. I screw it up, I fix it."
I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing.
An awful lot of sites at 1.2m views would have outsourced the running and development of the whole thing - there are entreprenuers who say its not even worth our time to code up the MVP. I find this approach sensible from a business point of view, but still it does not sit right with me.
I am supposed to have a nice website with lots of good content to attract inbound marketing - so I tried getting someone on textbroker to write an article for me. It read like a High School essay - no life, no anime. And so I will probably write my own CMS and my own content.
And pg sits there and writes his site in his own language, with his own moderation tools. Apart from the hilarious idea he could find a ten person ruby shop to outsource to, its nice to see someone taking the time to play again. Its why I like to see jgc on here too.
I am not entirely sure those thoughts are joined up (I am procrasting like crazy) but if they come to mean anything its we are playing in pg's sandbox. If the sand leaks it's his sand, and the only company this is mission critical to is YC.
- stcredzero 14y ago> It read like a High School essay - no life, no anime. ... I am not entirely sure those thoughts are joined up (I am procrasting like crazy) Your procrasting like crazy has much anime.
- luser001 14y ago> I eventually realised that typing sudo every time he touched a box was no defence against doing the wrong thing. IIRC, sudo logs all commands to syslog. Which might come in handy. Yes, root commands will be logged by bash to .bash_history, but there are limits of # of commands lots, what happens if you are logged in multiple times into same account etc. Anyway, that's why I like sudo.
- UnoriginalGuy 14y agoPlus security. With root login disabled a remote attacker won't have a known username to attack.
- wpietri 14y agoBeyond the logging, which I love, I use it for the differentiation of states. I'm just a little more attentive when I type "sudo" before something. At work, a relatively young engineer accidentally typed a command meant for a test database server into a production window. There was a big rush to restore from backups, and there was a small amount of data loss. One thing that came out of the retrospective, requested by the engineer in question, was the production hat. Before you opened up a connection to a production machine, you had to put on the large pirate hat. You could only put it back when you had closed the connections. I didn't really need it, but it was a great way for people to learn the necessary caution. It also ended up being a nice exclusive lock on futzing with production, and seeing it in use led to some good discussions that otherwise might not have happened. But the main thing was developing a strong differentiation of states in everybody's heads.
- darkarmani 14y agoAnother cheap solution is colorizing the prompt red for dangerous consoles and green for dev machines. This makes it very easy to notice when you selected the wrong terminal.
- Spoom 14y agoI use a different background color for the terminal in question; in this case, I use a dark red to signify a production system, and a dark blue to signify a development system. I find it quite useful. You can do this through Xterm profiles (Edit -> Profiles, Terminal -> Change Profile) in Linux and OS X, and I'm sure there's a way to do it in Windows / PuTTY.
- lifeisstillgood 14y agoI have tried the red / green console but never a pirate hat. My son now definitely thinks work is like his school :-)
- lmm 14y agoTyping sudo won't save you, but using a higher-level interface will. Everyone I've ever known to change something in the database by hand, everyone at all, even on a hobby project that they know like the back of their hand, has screwed it up sooner or later. At some point the pain tells you you should stop doing that, and you create an admin tool that lets you do what you need to repeatably and safely.
- deleted 14y ago[deleted]
- 3pt14159 14y agoI've never screwed it up on a live database, but I do take about 5 mins, first reviewing the keys, the type, whether or not something can be null, checking to see if critical columns have select count(distinct column_name) having count(distinct column_name) > 1; To make sure that there isn't an underlying uniqueness assumption. Sure I could do it in 10 seconds and save myself 290 seconds (a 97% savings!) but then one day I'd have to scramble like crazy in the middle of the night trying to figure out what I screwed up for hours on end. I'm not saying don't build an admin tool, obviously those are needed for things like banning users, but just get in there and carefully fix the data if something is wrong.
- mafro 14y agoThis. Back in the day when I was in more of an analyst role, I ended up /having/ to hack on the live DB frequently (reasons for this were myriad). 1. Always, always make a backup just before the hack. 2. Write a small set queries like 3pt14159's to check uniqueness and other pertinent properties. 3. Write a SELECT query to show the data you are going to change. 4. Borrow the WHERE clause from 3, and write your UPDATE statement. 5. Run 4, and then run 3 again to see that you successfully fixed it. 6. When it goes wrong, restore the backup from 1 :0
- jonob7 14y agoThis is pretty much exactly how I do it. I still sometimes get that sinking feeling in the stomach that I have screwed something up, usually just after I hit the 'execute' button. And I really don't want to have to take the site down to run the restoration.
- philsnow 14y agos/anime/animus/ ? or is this a new usage of "anime"?
- lifeisstillgood 14y agos/anime/anima - as in soul, vitality (Not so much Jung's inner woman) I think the sentence does read better if it is complaining there are not enough cyberpunk Japanese comics on my site though :-)
- lalc 14y agoHere I was thinking you were referring to the Japanese meme "No ___, No Life!"