5 ms·
I once shut down an algorithmic trading server by hastily typing (in bash): - Ctrl-r for reverse-search through history - typing 'ps' to find the process stat
by vijucat 12y ago
I once shut down an algorithmic trading server by hastily typing (in bash):
- Ctrl-r for reverse-search through history
- typing 'ps' to find the process status utility (of course)
- pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)
I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so.
That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat.
Renamed stopserver.sh to stop_server.sh after that incident :|
P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.
- Beltiras 12y agoRepercussions?
- vijucat 12y agoLuckily, and just that one time, there was nothing but disapproving glares.
- phaemon 12y agoMight be better having something like the following at the top of your script: echo "This will STOP THE SERVER. Are you sure you want to do this?" echo "Type 'yes' to continue:" read response if [[ $response != "yes" ]] then echo "You must type 'yes' to continue. Aborting." exit 1 fi echo "Stopping server ..." It barely takes any time to type 'yes' but it makes you stop and think.
- marcosdumay 12y agoThat'll make it impossible to use in scripts. This remember me the time I tried to fix my "rm -r * .o" with a CLI trash system, instead of doing proper backups. Might be better to start reviewing EVERY command one sends to important servers, and testing them if viable. What probably is the line that vijucat took... that's the line that everybody ends up taking, the only thing that changes is the number of accidents needed.
- euid 12y agoThen you add an optional CLI argument that makes it skip the prompt, and use that version in scripts.
- phaemon 12y agoAs 'euid' said, you can just add a check. If you don't want to add proper option checking (because this is the only option) you can do something like: response=$1 echo "This will STOP THE SERVER. Are you sure you want to do this?" echo "Type 'yes' to continue:" [[ $response ]] || read response if [[ $response != "yes" ]] then echo "You must type 'yes' to continue. Aborting." exit 1 fi echo "Stopping server ..." Which will allow you to run the script with "./scriptname.sh yes" to bypass the check. EDIT: Though, as you say, it is much better not to be running anything like this on a live server anyway. Any change should be part of a deployment procedure that is carefully checked and tested, as well as having a rollback procedure in the event of something horrible happening. Of course, in the real world, you sometimes don't have that luxury and you just have to hope and pray :-)
- vijucat 12y agoYes, this solution would have worked for our situation.
- brazzledazzle 12y agoAm I wrong to assume that you weren't using an init script of some sort because this stopped multiple services, perhaps in a specific order?
- deleted 12y ago[deleted]