6 ms·

"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course th

by loafoe 7y ago

"ifconfig eth0 down" on the production bastion host, instead of on my localhost terminal -- and no hands on in the datacenter which was 160km away. Of course the bastion host was the only one not hooked up to remote power reset services.. and only 2 hours left in the service window.. sinking feeling

dudleypippin 7y ago
Oh, yes, that feeling when you mix up “init 5” and “init 6” on a machine 300 miles away. “Huh, it’s taking longer than usual to reboot...”. 20 years later and I still remember the name of the guy I had to call at two in the morning to go power it back on.
acdha 7y ago
Back in the 1990s, something like that lead to a major local data center having to admit that the “24 hour support” meant 12 hours a day with someone who could answer the phone but didn’t have access to the server room.
pjc50 7y ago
I've done that a couple of times, although in better circumstances; also fun variants like "the new kernel didn't have the right ethernet driver in it". I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.
- codeduck 7y ago
  Oh nooooooo
throwaway8941 7y ago
For tmux users: put something like this in your .tmux.conf on production servers: set -g window-style 'fg=red,bg=black' It will color the text red, hopefully reminding you to be extra careful. Adjust according to your preferences. One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys: C-a M-d rm C-m
- KajMagnus 7y ago
  That's nice — I usually start my commands with `# ` but then there's no tab completion. I'll try `ls`
- leni536 7y ago
  I usually use echo for the same purpose.
- gryfft 7y ago
  In this vein, I set my PS1 to bold red capital letters on bastion hosts and alias sudo="echo 'You're on a jump box moron :p'" I do the tmux color trick too-- color coded by environment for each bastion.
- davedx 7y ago
  We do this for our database connections in Azure Data Studio / the Windows version of it. Really great idea.
- PeterisP 7y ago
  Something like this to obviously distinguish environments is good practice - at one company we implemented scripts for terminal color-coding like this after some downtime caused by a destructive backup restore accidentally being run in the production environment instead of an acceptance testing system, which was in all aspects identical to the production system. And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.
JeanMarcS 7y ago
Or rebooting a container. But you’re not in the container anymore but on the host...

by loafoe 7y ago

dudleypippin 7y ago

Oh, yes, that feeling when you mix up “init 5” and “init 6” on a machine 300 miles away. “Huh, it’s taking longer than usual to reboot...”. 20 years later and I still remember the name of the guy I had to call at two in the morning to go power it back on.

acdha 7y ago

Back in the 1990s, something like that lead to a major local data center having to admit that the “24 hour support” meant 12 hours a day with someone who could answer the phone but didn’t have access to the server room.

pjc50 7y ago

I've done that a couple of times, although in better circumstances; also fun variants like "the new kernel didn't have the right ethernet driver in it". I think the first one in my career was discovering that "killall" does something very different on Solaris from Linux.

codeduck 7y ago
Oh nooooooo

throwaway8941 7y ago

For tmux users: put something like this in your .tmux.conf on production servers: set -g window-style 'fg=red,bg=black' It will color the text red, hopefully reminding you to be extra careful. Adjust according to your preferences. One other "defensive scripting" trick I frequently use is starting any `rm` command with `ls`, double checking its output (or triple checking if it's a recursive one), and then replacing `ls` with `rm`. It barely takes any extra time if you're proficient with emacs-style readline hotkeys: C-a M-d rm C-m

KajMagnus 7y ago
That's nice — I usually start my commands with `# ` but then there's no tab completion. I'll try `ls`
leni536 7y ago
I usually use echo for the same purpose.
gryfft 7y ago
In this vein, I set my PS1 to bold red capital letters on bastion hosts and alias sudo="echo 'You're on a jump box moron :p'" I do the tmux color trick too-- color coded by environment for each bastion.
davedx 7y ago
We do this for our database connections in Azure Data Studio / the Windows version of it. Really great idea.
PeterisP 7y ago
Something like this to obviously distinguish environments is good practice - at one company we implemented scripts for terminal color-coding like this after some downtime caused by a destructive backup restore accidentally being run in the production environment instead of an acceptance testing system, which was in all aspects identical to the production system. And I've seen a solution for more secure environments where physical separation was used with the operator having separate monitors/keyboards, and the "important" system having a different color keyboard and monitor frame.

JeanMarcS 7y ago

Or rebooting a container. But you’re not in the container anymore but on the host...