6 ms·
I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For cont
by mdaniel 2y ago
I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it
I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again
- namxam 2y agoBut what is the alternative?
- mdaniel 2y agoThere's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-of-settings https://github.com/bottlerocket-os/bottlerocket#description-...> In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box
- candiddevmike 2y agoI built a config management tool, Etcha, that uses short lived JWTs. I extended it to offer a full shell over HTTP using JWTs: https://etcha.dev/docs/guides/shell-access/ https://etcha.dev/docs/guides/shell-access/ It works well and I can "expose" servers using reverse proxies since the entire shell session is over HTTP using SSE.
- g-b-r 2y ago“All JWTs are sent with low expirations (5 seconds) to limit replability” Do you know how many times a few packets can be replayed in 5 seconds?
- candiddevmike 2y agoSure, but this is all happening over HTTPS (Etcha only listens on HTTPS), it's just an added form of protection/expiration.
- artificialLimbs 2y agoI don’t understand why this is more secure than limiting SSH to local network only and doing ‘normal’ ssh hardening.
- candiddevmike 2y agoNone of that is required here? Etcha can be exposed on the Internet with a smaller risk profile than SSH: - Sane, secure defaults - HTTP-based--no fingerprinting, requires the correct path (which can be another secret), plays nicely with reverse proxies and forwarders (no need for jump boxes) - Rate limited by default - Only works with PKI auth - Clients verify/validate HTTPS certificates, no need for SSHFP records.
- ndndjdueej 2y agoIaC, send out logs to Splunk, health checks, slow rollouts, feature flags etc? Allow SSH in non prod environments and reproduce issue there? In prod you are aiming for "not broken" rather than "do whatever I want as admin".
- riddley 2y agoHow do you troubleshoot?
- mdaniel 2y agoIn my world, if a developer needs access to the Node upon which their app is deployed to troubleshoot, that's 100% a bug in their application. I am cognizant that being whole-hog on 12 Factor apps is a journey, but for my money get on the train because "let me just ssh in and edit this one config file" is the road to ruin when no one knows who edited what to set it to what new value. Running $(kubectl edit) allows $(kubectl rollout undo) to put it back, and also shows what was changed from what to what
- yjftsjthsd-h 2y agoHow do you debug the worker itself?
- mdaniel 2y agoSeparate from my sibling comment about AWS SSM, I also believe that if one cannot know that a Node is sick by the metrics or log egress from it, that's a deployment bug. I'm firmly in the "Cattle" camp, and am getting closer and closer to the "Reverse Uptime" camp - made easier by ASG's newfound "Instance Lifespan" setting to make it basically one-click to get onboard that train Even as I type all these answers out, I'm super cognizant that there's not one hammer for all nails, and I am for sure guilty of yanking Nodes out of the ASG in order to figure out what the hell has gone wrong with them, but I try very very hard not to place my Nodes in a precarious situation to begin with so that such extreme troubleshooting becomes a minor severity incident and not Situation Normal
- __turbobrew__ 2y agoIf accidentally nuking a single node while debugging causes issues you have bigger problems. Especially if you are running kubernetes any node should be able to fall off the earth at any time without issues. I agree that you should set a maximum lifetime for a node on the order of a few weeks. I also agree that you shouldn’t be giving randos access to production infra, but and the end of the day there needs to be some people at the company who have the keys to the kingdom because you don’t know what you don’t know and you need to be able to deal with unexpected faults or outages of the telemetry and logging systems. I once bootstrapped an entire datacenter with tens of thousands of nodes from an SSH terminal after an abrupt power failure. It turns out infrastructure has lots of circular dependencies and we had to manually break that dependency.
- ozim 2y agoHow do you handle db. Stuff I work on is write heavy so spawning dozens of app copies doesn’t make sense if I just hog the db with Erie locks.
- mdaniel 2y agoI must resist the urge to write "users can access the DB via the APIs in front of it" :-D But, seriously, Teleport (back before they did a licensing rug-pull) is great at that and no SSH required. I'm super positive there are a bazillion other "don't use ssh as a poor person's VPN" solutions
- zavec 2y agoThis led me to google "teleport license," which sounds like a search from a much more interesting world.
- mdaniel 2y agoTo save others the search: https://github.com/gravitational/teleport/pull/35259 https://github.com/gravitational/teleport/pull/35259 Apache to AGPLv3
- aspenmayer 2y agoYou might be interested in Peter F. Hamilton's Commonwealth Saga sci-fi series, then. Among other tech, it involves the founding of a megacorp that exploits the discovery and monopolization of wormhole technology for profit, causing a rift between the two founders, who each remind me of Steve Jobs and Steve Wozniak in their cooperation and divergence. https://en.wikipedia.org/wiki/Commonwealth_Saga https://en.wikipedia.org/wiki/Commonwealth_Saga
- Hikikomori 2y agoYo, dudes, how’s it hanging?
- 2y ago
- advael 2y agoPrinciple of least privilege trivially prevents updating system packages. Like if you don't want people using apt, don't give people root on your servers?
- ashconnor 2y agoYou can audit if you put something like hoop.dev, Tailscale, Teleport or Boundary in between the client and server. Disclaimer: I work at Hashicorp.
- LtWorf 2y agoBut I avoid hascicorp stuff whenever I can!
- akira2501 2y ago> any ssh access is an antipattern. Not generally. In one particular class of deployments allowing ssh access to root enabled accounts without auditing may be.. but this is an exceptionally narrowed definition. > I hope I never have sshd running on any machine again Sounds great for production and ridiculous for development and testing.
- mdaniel 2y ago> Sounds great for production and ridiculous for development and testing. I believe that "practice how you're going to play" to get devs into the habit of not using a crutch to treat deployments like they are their local machine. The time to anticipate failures is in the thinking time, not "throw it over the wall and we'll think later"
- akira2501 2y agoYour devs should not be managing deployments. Making deployable software and actually worrying about the deployment environment don't need particularly tight coordination. I'd also worry that you're overfitting your software to whatever third party deployment target you've selected.
- blueflow 2y agoEven for immutable OSes, SSH is a great protocol for bidirectionally authenticated data / file transfer.