3 ms·

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot eve

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

cyanydeez 5y ago
engineers will tell you that not everything is reversible even if theres no specific cqpacity issue
foobarian 5y ago
What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fallback working given how distributed the system is, and even more difficult to keep it exercised and tested.
- kelp 5y ago
  This is a key feature of Junos on Juniper devices. It's called 'commit confirmed' and it will apply the config and then auto-rollback if you don't confirm it within a certain amount of time. https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html https://www.juniper.net/documentation/us/en/software/junos/c... But Facebook's network is certainly much more complex and automated than just doing one commit on one device. But I think they do still uses Juniper devices at the edge.
  - bigiain 5y ago
    > But Facebook's network is certainly much more complex and automated than just doing one commit on one device. <snark> Software defined networking. In PHP.
  - allarm 5y ago
    Cisco have the rollback feature as well. It’s not implemented as smoothly as in Juniper though.
- bagels 5y ago
  How long do you wait? How do you know if you waited long enough to see negative effects (think caches, your own and caches of others). Waiting too long with a bad config also costs you.
- pepoluan 5y ago
  Use `iptables-apply` instead. iptables-apply reverts to previous config instead of flushing all the rules.

cyanydeez 5y ago

engineers will tell you that not everything is reversible even if theres no specific cqpacity issue

foobarian 5y ago

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out. I suppose with something like BGP it would be very difficult to get such a fallback working given how distributed the system is, and even more difficult to keep it exercised and tested.

kelp 5y ago
This is a key feature of Junos on Juniper devices. It's called 'commit confirmed' and it will apply the config and then auto-rollback if you don't confirm it within a certain amount of time. https://www.juniper.net/documentation/us/en/software/junos/cli/topics/topic-map/junos-configuration-commit.html https://www.juniper.net/documentation/us/en/software/junos/c... But Facebook's network is certainly much more complex and automated than just doing one commit on one device. But I think they do still uses Juniper devices at the edge.
- bigiain 5y ago
  > But Facebook's network is certainly much more complex and automated than just doing one commit on one device. <snark> Software defined networking. In PHP.
- allarm 5y ago
  Cisco have the rollback feature as well. It’s not implemented as smoothly as in Juniper though.
bagels 5y ago
How long do you wait? How do you know if you waited long enough to see negative effects (think caches, your own and caches of others). Waiting too long with a bad config also costs you.
pepoluan 5y ago
Use `iptables-apply` instead. iptables-apply reverts to previous config instead of flushing all the rules.