4 ms·

But what is the alternative?

by namxam 2y ago

mdaniel 2y ago
There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-of-settings https://github.com/bottlerocket-os/bottlerocket#description-...> In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box
- cyberax 2y ago
  Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet https://github.com/Cyberax/gimlet You can use it to tunnel arbitrary traffic inside your VPC.
- _hyn3 2y ago
  [dead]
candiddevmike 2y ago
I built a config management tool, Etcha, that uses short lived JWTs. I extended it to offer a full shell over HTTP using JWTs: https://etcha.dev/docs/guides/shell-access/ https://etcha.dev/docs/guides/shell-access/ It works well and I can "expose" servers using reverse proxies since the entire shell session is over HTTP using SSE.
- g-b-r 2y ago
  “All JWTs are sent with low expirations (5 seconds) to limit replability” Do you know how many times a few packets can be replayed in 5 seconds?
  - candiddevmike 2y ago
    Sure, but this is all happening over HTTPS (Etcha only listens on HTTPS), it's just an added form of protection/expiration.
- artificialLimbs 2y ago
  I don’t understand why this is more secure than limiting SSH to local network only and doing ‘normal’ ssh hardening.
  - candiddevmike 2y ago
    None of that is required here? Etcha can be exposed on the Internet with a smaller risk profile than SSH: - Sane, secure defaults - HTTP-based--no fingerprinting, requires the correct path (which can be another secret), plays nicely with reverse proxies and forwarders (no need for jump boxes) - Rate limited by default - Only works with PKI auth - Clients verify/validate HTTPS certificates, no need for SSHFP records.
ndndjdueej 2y ago
IaC, send out logs to Splunk, health checks, slow rollouts, feature flags etc? Allow SSH in non prod environments and reproduce issue there? In prod you are aiming for "not broken" rather than "do whatever I want as admin".

But what is the alternative?

by namxam 2y ago

But what is the alternative?

There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-of-settings https://github.com/bottlerocket-os/bottlerocket#description-...> In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box

cyberax 2y ago
Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet https://github.com/Cyberax/gimlet You can use it to tunnel arbitrary traffic inside your VPC.
_hyn3 2y ago
[dead]

candiddevmike 2y ago

I built a config management tool, Etcha, that uses short lived JWTs. I extended it to offer a full shell over HTTP using JWTs: https://etcha.dev/docs/guides/shell-access/ https://etcha.dev/docs/guides/shell-access/ It works well and I can "expose" servers using reverse proxies since the entire shell session is over HTTP using SSE.

g-b-r 2y ago
“All JWTs are sent with low expirations (5 seconds) to limit replability” Do you know how many times a few packets can be replayed in 5 seconds?
- candiddevmike 2y ago
  Sure, but this is all happening over HTTPS (Etcha only listens on HTTPS), it's just an added form of protection/expiration.
artificialLimbs 2y ago
I don’t understand why this is more secure than limiting SSH to local network only and doing ‘normal’ ssh hardening.
- candiddevmike 2y ago
  None of that is required here? Etcha can be exposed on the Internet with a smaller risk profile than SSH: - Sane, secure defaults - HTTP-based--no fingerprinting, requires the correct path (which can be another secret), plays nicely with reverse proxies and forwarders (no need for jump boxes) - Rate limited by default - Only works with PKI auth - Clients verify/validate HTTPS certificates, no need for SSHFP records.

ndndjdueej 2y ago

IaC, send out logs to Splunk, health checks, slow rollouts, feature flags etc? Allow SSH in non prod environments and reproduce issue there? In prod you are aiming for "not broken" rather than "do whatever I want as admin".