6 ms·

I used to do remote work in firewalls quite often, and after locking myself out once or twice, I came up with a new habit: before making any changes I would sch

I used to do remote work in firewalls quite often, and after locking myself out once or twice, I came up with a new habit: before making any changes I would schedule a reboot for 5min out which would revert any changes. That way if I locked myself out I could just wait for the reboot and get back in.

paulddraper 3y ago
Kinda like changing display settings in Windows. Changes will revert in 15 seconds....
dang 3y ago
And then if it worked for those 5 min before the reboot, you'd redeploy the change 'for real', without a reboot?
- 2snakes 3y ago
  Yeah, there are different kinds of memory in firewalls. Like a running-config and a startup-config. If you just change the running-config and don't commit to the startup-config, when the reboot takes place it'll pull the config from the (non-modified) startup-config instead, reverting changes.
  - Nifty3929 3y ago
    copy run start!
- Nifty3929 3y ago
  My typical workflow was: - Schedule the reboot - do my changes - Make sure everything was working properly - Go get lunch - Notice a bunch of pages and alarms about a firewall going offline - Rush back to my office - Login to the firewall - Schedule the reboot - Re apply the changes - Test it again - CANCEL THE FING REBOOT THIS TIME - Eat my now cold lunch
  - booi 3y ago
    This used to be my workflow as well. I did make a few improvements though - begin change control at 4:55pm on Friday before Christmas - schedule reboot - paste changes - make sure everything is working properly - leave security key on desk - go to christmas party - firewall goes offline, pages go off - remotely log into firewall with phone - rush back to office to get security key - accidentally type init 1 hanging server - discount datacenter remote hands not picking up the phone - rush to datacenter to power cycle server - :(
prox 3y ago
This is clever, I like it.
- knorker 3y ago
  "commit confirmed" from Juniper routers is much better
  - snuxoll 3y ago
    Mikrotik safe mode gets a 3/5 in comparison - it reverts changes you made if you lose connection to the router, so it does it’s job as an anti-lockout mechanism; but I much prefer the atomic nature of a confit commit on junos still.
knorker 3y ago
Standard practice on Cisco routers, where I've worked, is to do "reload 5" before doing dangerous things. On juniper, it's "commit confirmed".
- comboy 3y ago
  or safe mode on mikrotik
networkchad 3y ago
[dead]
hsbauauvhabzb 3y ago
‘sleep 300 && init 6’ was my go-to, but since then systemd has made firing init 6 unreliable (it won’t trigger a reboot locally if root has an open ssh session, at least on Ubuntu).
stouset 3y ago
I did a similar thing in the early days of my career, but I actually caused an outage as a result. In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW. The script runs, everything looks good. Five minutes later, pagers start going off. Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.

paulddraper 3y ago

Kinda like changing display settings in Windows. Changes will revert in 15 seconds....

dang 3y ago

And then if it worked for those 5 min before the reboot, you'd redeploy the change 'for real', without a reboot?

2snakes 3y ago
Yeah, there are different kinds of memory in firewalls. Like a running-config and a startup-config. If you just change the running-config and don't commit to the startup-config, when the reboot takes place it'll pull the config from the (non-modified) startup-config instead, reverting changes.
- Nifty3929 3y ago
  copy run start!
Nifty3929 3y ago
My typical workflow was: - Schedule the reboot - do my changes - Make sure everything was working properly - Go get lunch - Notice a bunch of pages and alarms about a firewall going offline - Rush back to my office - Login to the firewall - Schedule the reboot - Re apply the changes - Test it again - CANCEL THE FING REBOOT THIS TIME - Eat my now cold lunch
- booi 3y ago
  This used to be my workflow as well. I did make a few improvements though - begin change control at 4:55pm on Friday before Christmas - schedule reboot - paste changes - make sure everything is working properly - leave security key on desk - go to christmas party - firewall goes offline, pages go off - remotely log into firewall with phone - rush back to office to get security key - accidentally type init 1 hanging server - discount datacenter remote hands not picking up the phone - rush to datacenter to power cycle server - :(

prox 3y ago

This is clever, I like it.

knorker 3y ago
"commit confirmed" from Juniper routers is much better
- snuxoll 3y ago
  Mikrotik safe mode gets a 3/5 in comparison - it reverts changes you made if you lose connection to the router, so it does it’s job as an anti-lockout mechanism; but I much prefer the atomic nature of a confit commit on junos still.

knorker 3y ago

Standard practice on Cisco routers, where I've worked, is to do "reload 5" before doing dangerous things. On juniper, it's "commit confirmed".

comboy 3y ago
or safe mode on mikrotik

networkchad 3y ago

[dead]

hsbauauvhabzb 3y ago

‘sleep 300 && init 6’ was my go-to, but since then systemd has made firing init 6 unreliable (it won’t trigger a reboot locally if root has an open ssh session, at least on Ubuntu).

stouset 3y ago

I did a similar thing in the early days of my career, but I actually caused an outage as a result. In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW. The script runs, everything looks good. Five minutes later, pagers start going off. Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.