3 ms·

> Running administrative commands through SSH is just silly. With the tooling we have there's not even proper error reporting (how to tell that a command reache

> Running administrative commands through SSH is just silly. With the tooling we have there's not even proper error reporting (how to tell that a command reached its target server but failed there? you can use heuristics of exit code at best). The exit code of the remote process is exactly what you should use; I'm not seeing what you think is heuristical about it. Either you get back that the remote process failed, in which case you know it failed on the remote, or you get back success, in which case you know it succeeded, or you don't get anything back at all, in which case you just don't know. This isn't particular to exit codes per-se, any network message to a remote machine is going to be subject to the network or remote host possibly dying between sending it and receiving confirmation of whether or not it was received and had any effect. If you're using SSH-the-process, it's going to mux things like network failures into that error code. You can either use a library (in a language a bit more robust than Bash) so that you know you're looking at the exit code of the remote process. If you must use SSH-the-process, you can always output the exit code of the remote process that you're interested in into stderr, in a manner that you can parse it back out on the receiving end. > And if it was planned downtime? "Tough luck"? Again, yes, though I don't see the "Tough Luck" about it; if I'm attempting to run some unknown and arbitrary command on a machine that was planned to be down, then yes, I need to step back and realize that as a human, and make sure that I'm doing the right thing. (Again, it depends on the command; what would be the effects of re-running it, and can you programmatically guarantee that re-running is a safe operation?) > You'd much rather the server be monitored, with other systems being resilient than failing every now and then for random reasons and needing special additional actions to be taken to recover them. Well, yes, I would rather the server be monitored. But monitoring is significantly orthogonal to the system I use to run administrative commands on the machine. Again, it's not so much about being resilient as it is about not taking a bad situation and making it worst. I've run a lot of administrative commands across tons of machines that were not idempotent, and if any failed anywhere, I would need to intervene. (I also usually tested heavily prior to these, to make sure there was a decent chance of not needing to intervene later.) But that still, to me, implies that whether a given command should be retried until it definitely runs is at a layer higher than the tooling used to distribute commands out to host machines, and in the general case, could quite very well require a human.

dozzie 10y ago
> The exit code of the remote process is exactly what you should use; I'm not seeing what you think is heuristical about it. Lucky you if your SSH client never failed to connect. How do you think it reports a network failure? > This isn't particular to exit codes per-se, any network message to a remote machine is going to be subject to the network or remote host possibly dying between sending it and receiving confirmation of whether or not it was received and had any effect. You see, there are tools that allow you to easily tell between network failure and remote command failure. RPC protocols are an example. > If you're using SSH-the-process, it's going to mux things like network failures into that error code. You can either use a library (in a language a bit more robust than Bash) so that you know you're looking at the exit code of the remote process. Now let's get practical: find me such a library. >> And if it was planned downtime? "Tough luck"? > [...] if I'm attempting to run some unknown and arbitrary command on a machine that was planned to be down, then yes, I need to step back and realize that as a human, and make sure that I'm doing the right thing. We were talking about configuring servers. You basically tell me that I would have to pay for "agentlessness" with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command. > (Again, it depends on the command; what would be the effects of re-running it, and can you programmatically guarantee that re-running is a safe operation?) Sorry, I'm lost here: what re-running? The command didn't get the chance to run in the first place. I don't know where did you get the idea that I propose running the command again if it got through to the remote server and failed there.
- deathanatos 10y ago
  > Now let's get practical: find me such a library. Certainly. I'm most familiar with Python, and for that, both Paramiko and asyncssh support returning the exit status of a command, distinct from any network failure. As I mentioned in the above reply, even with command-line ssh (and any other library, really), if you're creative about how you structure the output on stderr, you can encode the exit status there to be able to differentiate it from ssh returning other errors. E.g., if you pipe the entirety of stderr through, say, base64 and then append at the end a newline and the exit status, you can parse out the exit status on the receiving end. (This is admittedly quite a hack: using a library in Python with direct access to the data is much nicer. The ssh binary is mostly a porcelain interface to the SSH protocol; libraries are a better tool for building on top of.) > with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command. Perhaps this is just a difference in our production environments. We strive very hard to have all our machines online at all times. (Excepting short outages for rebooting for patches, etc., of course.) We use VMs, so there's not much point to having it be shut down. > Sorry, I'm lost here: what re-running? My larger point was about non idempotent commands. If your infrastructure retries commands, are you just very careful to not have non-idempotent stuff? Generally, these things are either "run at most once" or "run at least once"; running exactly once is (in general) impossible, as the command might finish successfully, but the host machine crash prior to recording that success. Again, if you can structure the command to be idempotent, you can work around this, but in my experience there always seems to be some patch or database upgrade that this can't be accomplished for.

dozzie 10y ago

> The exit code of the remote process is exactly what you should use; I'm not seeing what you think is heuristical about it. Lucky you if your SSH client never failed to connect. How do you think it reports a network failure? > This isn't particular to exit codes per-se, any network message to a remote machine is going to be subject to the network or remote host possibly dying between sending it and receiving confirmation of whether or not it was received and had any effect. You see, there are tools that allow you to easily tell between network failure and remote command failure. RPC protocols are an example. > If you're using SSH-the-process, it's going to mux things like network failures into that error code. You can either use a library (in a language a bit more robust than Bash) so that you know you're looking at the exit code of the remote process. Now let's get practical: find me such a library. >> And if it was planned downtime? "Tough luck"? > [...] if I'm attempting to run some unknown and arbitrary command on a machine that was planned to be down, then yes, I need to step back and realize that as a human, and make sure that I'm doing the right thing. We were talking about configuring servers. You basically tell me that I would have to pay for "agentlessness" with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command. > (Again, it depends on the command; what would be the effects of re-running it, and can you programmatically guarantee that re-running is a safe operation?) Sorry, I'm lost here: what re-running? The command didn't get the chance to run in the first place. I don't know where did you get the idea that I propose running the command again if it got through to the remote server and failed there.

deathanatos 10y ago
> Now let's get practical: find me such a library. Certainly. I'm most familiar with Python, and for that, both Paramiko and asyncssh support returning the exit status of a command, distinct from any network failure. As I mentioned in the above reply, even with command-line ssh (and any other library, really), if you're creative about how you structure the output on stderr, you can encode the exit status there to be able to differentiate it from ssh returning other errors. E.g., if you pipe the entirety of stderr through, say, base64 and then append at the end a newline and the exit status, you can parse out the exit status on the receiving end. (This is admittedly quite a hack: using a library in Python with direct access to the data is much nicer. The ssh binary is mostly a porcelain interface to the SSH protocol; libraries are a better tool for building on top of.) > with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command. Perhaps this is just a difference in our production environments. We strive very hard to have all our machines online at all times. (Excepting short outages for rebooting for patches, etc., of course.) We use VMs, so there's not much point to having it be shut down. > Sorry, I'm lost here: what re-running? My larger point was about non idempotent commands. If your infrastructure retries commands, are you just very careful to not have non-idempotent stuff? Generally, these things are either "run at most once" or "run at least once"; running exactly once is (in general) impossible, as the command might finish successfully, but the host machine crash prior to recording that success. Again, if you can structure the command to be idempotent, you can work around this, but in my experience there always seems to be some patch or database upgrade that this can't be accomplished for.