4 ms·
And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the mach
by Cthulhu_ 5y ago
And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts.
Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it.
I had a lot more respect for the ops people there than I had for my fellow SWE's.
- raverbashing 5y agoI disagree. There are always bugs that will happen only on the prod machines. Sure, they are rare, but they exist. > just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration Well, that's not risk free neither. That needs a code change and you risk exposing secrets, or logging more than you should. (Though I agree with the general idea of you shouldn't be SSHing if you can do it another way)
- Dylan16807 5y ago> Ops team told him how to do his job - log the configuration - instead of give him access "do your job" is a really rude way to refer to debugging by logging instead of debugging interactively.
- sokoloff 5y agoI suspect this usage is an awkward phrasing of “told him how to accomplish the result he wanted”; even though in common usage it implies a lack of overall competence, here I think it just implied a lack of a specific recipe/technique.
- ubercow13 5y agoDoesn’t seem so, as GP specifically said this caused them to lose respect for their colleagues.
- sokoloff 5y agoThat’s a possible interpretation of their comment, but not the only interpretation and certainly not something they specifically said. (Another interpretation is that the author previously held Ops in higher regard than SWE and that this event did not change that.)
- robertlagrant 5y agoI imagine if overzealous rule enforcement is the cause of respect that the OP would've always had more respect for Ops :D
- foldr 5y agoSide node: Why do we have to talk about these things in such highly charged terms in our profession? It sounds like all that happened here is that one person made a reasonable request, and another person gave a reasonable explanation of why the request couldn't be granted.
- AtlasBarfed 5y agoBecause people establish protective totalitarian control over their corporate turf because budgeting, it become full of dogma (the gods said thou shalt not ssh and we said amen). Dogma results in religious wars
- closeparen 5y agoYou had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?
- rytis 5y agoyup, that sounds strange. if the environment is that advanced, then there shouldn't be a problem of taking the node in question out of the service pool and let the devs loose on it (assuming there's no info that they aren't supposed to see). when they're done, just recycle it.
- theptip 5y agoWhere did they say it was a single node issue?
- tuananh 5y agoi think they mean, in case he mess up a single node, it should still be fine
- theptip 5y agoI see, that makes more sense, thanks for the clarification. I’d still rather not have to debug arbitrary mutations to the env or file system in a production container though. It seems to me that shelling into a prod container/VM is discouraged not because you might cause it to fail, but because you might produce undefined behavior while claiming it is healthy (more like Byzantine faults). For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Debugging this kind of issue is a nightmare. Not to mention that developers can often run arbitrary SQL from a prod shell when the app is backed by a DB.
- yjftsjthsd-h 5y ago> For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Or far worse, not fail.
- xorcist 5y ago> log the configuration That's a change, inherently much more risky than the "cat /.../env" the colleague wanted to do. Also the change might well cause the problem to go away, and now you know nothing instead. The principle is good, but it sounds like it has taken on a life on its own. The ops guy could also have executed the cat command on the spot with the right privileges. Sure, it's gatekeeping, but so is the four eyes principle, and a little of that gatekeeping can be necessary to keep those nines rolling.
- throwaway81523 5y agoIf messing up a single instance is going to take your site down, you are nowhere near seven 9's.
- teitoklien 5y agoSounds weird , logging env vars would be even more risk prone to leaking credentials, than just letting an employee ssh into the machine and see the env vars. Also like others noted here , If one guy sshing into one machine can screw your seven nine uptimes. Then you never had a seven nine uptime (you’re company is probably just gambling on that uptime metric until some component fails)
- StreamBright 5y agoThis is the sort of fanboyism that I usually avoid when working. It is perfectly reasonable to SSH into a node for debugging. In fact I do it very often in a heavily automated infrastructure as code environment.
- flyinghamster 5y agoNot to mention that if you're using Ansible to manage machines, it depends on SSH to do its job.
- StreamBright 5y agoExactly. Any time this topic comes up the people who are arguing for no SSH the ones never operated a production environment successfully and they are busy chasing ghosts.
- brendangregg 5y agoMost issues we hit are solved by the monitoring GUIs (Atlas, FlameCommander, PerfDash, S3 log collection, etc.), but some aren't, and there's a ton more information available from the CLI tools. Imagine losing your five nines because no one can SSH on and try a few ad hoc tools. But yes, CLI can be dangerous, even for observability. E.g., people running strace(1) on production apps and causing outages due to the strace overhead (I wrote a prior post about that). You need to understand the risks and overheads of all tools. It's why I have a "pull no punches" policy when writing eBPF tool man pages [0]: If the overhead can be bad, it should say so clearly. [0] https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRIPTS.md https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRI...
- vidarh 5y agoI know at least one place where they do allow ssh as a last resort, but the VMs that gets ssh'd to get flagged to be recycled automatically. So you can log in and debug, but once you're done, the VM is replaced with a clean one.