3 ms·
> That's fine, right up until the point you're running a command (or a series of commands) that change things on the machine and your connection gets broken. Wh
by lamontcg 4y ago
> That's fine, right up until the point you're running a command (or a series of commands) that change things on the machine and your connection gets broken. When that happens there's a very good chance the command being executed will fail in some fashion. If you're very lucky it won't be a big deal, but depending on what you're doing it could be dangerous.
I spent 5 years at Amazon (back a long time ago) being incredibly aggressive with command line root-trust from the old bastion host they had there, working up to regularly executing ssh commands to all 30,000 servers (like I said it was a long time ago, but that's still a considerable stack) every single day. It is the kind of idea that gives a lot of people the vapours to think about, and I wouldn't recommend it if you had good alternatives. But Amazon never turned into a smoking hole, and I never once saw this kind of problem. I certainly had to deal with network issues, and the scripts I used were pretty robust against problems. Most often servers would be crashed and fail to ping, packets would get lost but retransmissions would succeed, or the kernel would be hung only responding to interrupts and userspace was all deadlocked, but sometimes the servers had some resource starvation issues and it would just be very slow and the command would timeout. Nothing bad every happened. I occupy some weird space though where I'm stupid enough to try these kinds of things and just careful enough to not get bitten by them. I've also been a trained cave diver for over a decade now as well (if you get the training and follow the rules, it is safer than driving on the roads in Mexico).
And really you should always be using configuration management to change the state of servers and should have some kind of agent doing that automatically for you (although I've broken that rule probably a thousand times myself), and you should also ensure that what you're doing is 'idempotent', so that if it fails due to some transient error then you can just keep on hammering on the server and the next time or the time after that or whatever it will fix itself. This is the "self-healing" or "computer immunology" concept of configuration management that Mark Burgess came up with back in 1998 or so. If the process gets interrupted just run the process again. That may require some care taken around intermediate states and recovery from intermediate states, but that is just config management 101.