5 ms·

I Cannot SSH into My Server Anymore (and That's Fine)

116 points by TheWiggles 9mo ago

lawrencegripper 9mo ago
I’ve been down a similar journey with Fedora Core OS and have loved it. The predictability and drop in toil is so nice. https://blog.gripdev.xyz/2024/03/16/in-search-of-a-zero-toil-homelab-with-immutable-linux/ https://blog.gripdev.xyz/2024/03/16/in-search-of-a-zero-toil...
- samhclark 9mo ago
  Yeah, this is closer to what I do, too. I was surprised not to see a Containerfile in the linked github repo in the article (https://github.com/lthms/tinkerbell https://github.com/lthms/tinkerbell) I found working with normal `dnf` and normal config files much easier than dealing with Ignition and Butane. Plus, working with your image in CI/CD instead of locally fixed my ZFS instability. When Fedora kernel updates, but ZFS doesn't support that version yet, now it fails in GitHub Actions and the container is never built, so there's no botched update that my NAS mistakenly picks up.
stryan 9mo ago
Quadlets are a real game changer for this type of small-to-medium scale declarative hosting. I've been pushing for them at work over ugly `docker compose in systemd units` service management and moved my home lab over to using them for everything. The latter is a similar setup to OP except with OpenSUSE MicroOS instead of Fedora CoreOS and I'm not so brave as to destroy and rebuild my VPS's whenever I make a change :) . On the other hand, MicroOS (and I'm assuming FCOS) reboots automatically to apply updates with rollback if needed so combined with podman auto-update you can basically just spin up a box, drop the files on, and let it take care of itself (at least until a container update requires manual intervention). A few things in the article I think might help the author: 1. Podman 4 and newer (which FCOS should definitely have) uses netavark for networking. A lot of older tutorials and articles were written back when Podman used CNI for it's networking and didn't have DNS enabled unless you specifically installed it. I think the default `podman` network is still setup with DNS disabled by default. Either way, you don't have to use a pod if you don't want to anymore, you can just attach both containers to the same network and it should Just Work. 2. You can run the generator manually with "/usr/lib/systemd/system-generators/podman-system-generator --dry-run" to check Quadlet validity and output. Should be faster than daemon-reload'ing all the time or scanning the logs. And as a bit of self-promotion: for anyone who wants to use Quadlets like this but doesn't want to rebuild their server whenever they make a change, I'm created a tool called Materia[0] that can install, remove, template, and update Quadlets and other files from a Git repository. [0] https://github.com/stryan/materia https://github.com/stryan/materia
- plagiarist 9mo ago
  Do you know if it is possible to run a quadlet as an ephemeral systemd-sysuser? That would solve all my current problems.
  - stryan 9mo ago
    Not sure I'm following; you want to create a an emphemeral system account and run a root-less Podman container as it? I don't think that's something supported out of the box but you may be able to jury rig something together by putting the quadlets directly in `/etc/containers/systemd/users/` instead of putting them in a home directory (since I'm assuming this is a systemd-sysuser created account and thus without a home).
    - plagiarist 9mo ago
      Yes, that's it. Have things running isolated by a sysuser as well as in a rootless container. I would be running containers for LAN software (like forgejo) where I'd rather have the data on disk or in a podman volume instead of in a home directory.
amluto 9mo ago
> I’ve later learned that restarting a container that is part of a pod will have the (to me, unexpected) side-effect to restart all the other containers of that pod. Anyone know why this is? Or, for that matter, why Kubernetes seems to work like this too? I have an application for which the natural solution would be to create a pod and then, as needed, create and destroy containers within the pod. (Why? Because I have some network resources that don’t really virtualize, so they can live in one network namespace. No bridges.) But despite containerd and Podman and Kubernetes kind-of-sort-of supporting this, they don’t seem to actually want to work this way. Why not?
- stryan 9mo ago
  Yeah I was a little confused at this line; as far as I can tell you can restart containers that are a part of a Podman pod without restarting the whole pod just fine. I just verified this on one of my MicroOS boxes running Podman v5.7.1 . Podman was changing pretty fast for a while so it could be an older version thing, though I'd assume FCOS is on Podman 5 by now.
- gucci-on-fleek 9mo ago
  > Anyone know why this is? In Podman, a pod is essentially just a single container; each "container" within a pod is just a separate rootfs. So from that perspective, it makes sense, since you can't really restart half of a container. (But I think that it might be possible to restart individual containers within a pod; but if any container within a pod fails, then I think that the whole pod will automatically restart) > Why? Because I have some network resources that don’t really virtualize, so they can live in one network namespace. You can run separate containers in the same network namespace with the "--network" option [0]. You can either start one container with its own automatic netns and then join the other containers to it with "--network=container:<name>", or you can manually create a new netns with "podman network create <name>" and then join all the containers to it with "--network=<name>". [0]: https://docs.podman.io/en/latest/markdown/podman-run.1.html#network-mode-net https://docs.podman.io/en/latest/markdown/podman-run.1.html#...
  - amluto 9mo ago
andrewmcwatters 9mo ago
I concede that this is the state of the art in secure deployments, but I’m from a different age where people remoted into colocated hardware, or at least managed their VPSs without destroying them every update. As a result, I think developers are forgetting filesystem cleanliness because if you end up destroying an entire instance, well it’s clean isn’t it? It also results in people not knowing how to do basic sysadmin work, because everything becomes devops. The bigger problem I have with this, is the logical conclusion is to use “distroless” operating system images with vmlinuz, an init, and the minimal set of binaries and filesystem structure you need for your specific deployment, and rarely do I see anyone actually doing this. Instead, people are using a hodgepodge of containers with significant management overhead, that actually just sit on like Ubuntu or something. Maybe alpine. Or whatever Amazon distribution is used on ec2 now. Or of course, like in this article, Fedora CoreOS. One day, I will work with people who have a network issue and don’t know how to look up ports in use. Maybe that’s already the case, and I don’t know it.
- irishcoffee 9mo ago
  > The bigger problem I have with this, is the logical conclusion is to use “distroless” operating system images with vmlinuz, an init, and the minimal set of binaries and filesystem structure you need for your specific deployment, and rarely do I see anyone actually doing this. In the few jobs I’ve had over 20 years, this is common in the embedded space, usually using yocto. Really powerful, really obnoxious tool chain.
- bitwize 9mo ago
  What you describe is from the "pets" era of server deployment, and we are now deep into the "cattle" era. Train yourself on destroying and redeploying, and building observability into the stack from the outset, rather than managing a server through ssh. Every shop you go to professionally is going to work like this. Eventually, Linux desktops will work like this also, especially with all the work going into systemd to support movable home directories, immutable OS images with modular updates, and so forth.
  - andrewmcwatters 9mo ago
crawshaw 9mo ago
The idea that an "observability stack" is going to replace shell access on a server does not resonate with me at all. The metrics I monitor with prometheus and grafana are useful, vital even, but they are always fighting the last war. What I need are tools for when the unknown happens. The tool that manages all my tools is the shell. It is where I attach a debugger, it is where I install iotop and use it for the first time. It is where I cat out mysterious /proc and /sys values to discover exotic things about cgroups I only learned about 5 minutes prior in obscure system documentation. Take it away and you are left with a server that is resilient against things you have seen before but lacks the tools to deal with the future.
- gear54rus 9mo ago
  Agreed, this sounds like some complicated ass-backwards way to do what k8s already does. If it's too big for you, just use k3s or k0s and you will still benefit from the absolutely massive ecosystem. But instead we go with multiple moving parts all configured independently? CoreOS, Terraform and a dependence on Vultr thing. Lol. Never in a million years I would think it's a good idea to disable SSH access. Like why? Keys and non-standard port already bring China login attempts to like 0 a year.
- ValdikSS 9mo ago
  >What I need are tools for when the unknown happens. There are tools which show what happens per process/thread and inside the kernel. Profiling and tracing. Check Yandex's Perforator, Google Perfetto. Netflix also has one, forgot the name.
- reactordev 9mo ago
  Or… you build a container, that runs exactly what you specify. You print your logs, traces, metrics home so you can capture those stack traces and error messages so you can fix it and make another container to deploy. You’ll never attach a debugger in production. Not going to happen. Shell into what? Your container died when it errored out and was restarted as a fresh state. Any “Sherlock Holmes” work would be met with a clean room. We have 10,000 nodes in the cluster - which one are you going to ssh into to find your container to attach a shell to it to somehow attach a debugger?
gucci-on-fleek 9mo ago
Fedora IoT [0] is a nice intermediate solution. Despite its name, it's really good for servers, since it's essentially just the Fedora Atomic Desktops (Silverblue/Kinoite) without any of the desktop stuff. It gets you atomic updates, a container-centric workflow, and easy rollbacks; but it's otherwise a regular server, so you can install RPMs, ssh into it, create user accounts, and similar. This is what I do for my personal server, and I'm really happy with it. [0]: https://fedoraproject.org/iot/ https://fedoraproject.org/iot/
dorfsmay 9mo ago
Perfect timing for me, I've just been spending my side-project time in the last few weeks on building the smallest possible VMs with different glibc distros exactly for this, running podman containers, and comparing results.
starttoaster 9mo ago
So it's AWS Fargate with a different name? That's cool for cloud hosted stuff. But if you're on prem, or manage your own VPS' then you need SSH access.
- npodbielski 9mo ago
  I bought last year mobo with IPMI so in theory I could have forgot about SSH and just inspect startup logs if it would fail to start. Though I must say I am not brave enough and my family uses it so I prefer to have jest one broken service instead of enire machine. But it is possible.
  - denkmoon 9mo ago
    Except you've replaced something good with something worse. IPMI really isn't an improvement over having SSH to the system. It definitely has more security holes.
    - npodbielski 9mo ago
      Can you SSH into broken grub? Can you change BIOS settings? Also giving access like that outside your home network is not a good idea. So security issues does not matter that much.
      - denkmoon 9mo ago
        I didn't say not to use IPMI, I said it's not a security improvement over SSH. For exactly the reason you point out, giving access outside your home. Nothing wrong with exposing SSH provided you take it seriously and know what you're doing. Nobody in their right mind would ever put IPMI on anything but a protected isolated network.
yigalirani 9mo ago
real programmers can ssh to their servers
- libHacker 9mo ago
  It's true. There's no reason to disable ssh. If you need it, it's there. If not, just don't use it.
npodbielski 9mo ago
That is looks interesting. An idea to configure server on run via symtemd would probably mean that migrating from machine to machine would be very easy. It always meant for me at least two days of carefull planning, copying od files testing and fixes because I always forgot about some obscure config changes I did somewhere, like adding DNS entry somewhere or disabling default SMTP on debian.
skeptic_ai 9mo ago
You can try docker compose with Watch tower. Then you just deploy a new branch: dev, prod. On server side counterparty you fetch updates on git, if anybody change, it will run docker compose, which will build your image and put it live. Worked well for me a few years. Problems: when you have issues you need to look into pertainer logs to see why it failed. That’s one big problem, if prefer something like Jenkins to build it instead. And if you have more groups of docker compose, you just put another sh script to do this piling on the main infrastructure git repo, which on git change will spawn new git watchers
hahahahhaah 9mo ago
It is not just fine, it is best practice. SSH is 2020s version of 2010s driving to the data centre.
gerdesj 9mo ago
I'm glad you have a stack that works for you. The great thing is we have choice and it was not always so. I suggest that you be careful of the DevOps way. Sometimes a "pet" is the way to go, especially if you only have one. If you have a thundering herd then you'll be hand rolling your own nonsense with the best of the cloudy cowboys and have a out of service sign that says "they did it" for when the lights wink out! I also notice that the word security does not grace your blog posting. That is a sure sign of the DevOps Way 8) You might look into the sysadmin way. Its boring, to be sure: all that fussing over security and the like! You could look into VPNs for access to your gear. An IPSEC, OpenVPN or Wireguard seems to keep most baddies away simply because it is a lot of effort to even engage with one. There are a huge number of ways that a VPN is configured. Then you have ssh, which can be very securely configured (or not). You can also use firewalls and I'm sure you do. If you have a static IP at home then simply filter for that. Make use of allow/deny lists - there are loads for firewalls of all sorts. Dumping remote shell access is not useful.
VladVladikoff 9mo ago
This is right up there with people who are happy to not have root access on their phones.
Zopieux 9mo ago
"tech blogger not reinvent NixOS but badly" challenge (impossible)
deathanatos 9mo ago
> [Kubernetes…] Managed clusters could make things easier, but they aren’t cheap. That would defeat the initial motivation behind retiring moana. Kubernetes is perfectly fine running on a "cluster" with a single node / this seems to be under the misconception that k8s requires >1 node. It doesn't, though obviously a single or two node cluster will not maintain a majority if a node goes down. For self-hosting, though, that might be perfectly acceptable. (My own self-hosted server was a single-node k8s cluster.)
k_bx 9mo ago
I am also slowly preparing myself to the world where there is no SSH into the server machine. I am following what's happening around IncusOS. Already sold on Incus for my containers, it does make sense on a paper: safe auto-updates, no manual key management, all you need is managed via API in a cluster (usually).
- mvdwoord 9mo ago
  Can you tell me a bit more on how you use Incus.. is it just personal use, or otherwise? What type of workloads do you run on it, and how is your networking setup / experience?
  - k_bx 9mo ago
    I use it for professional use, running production services which don't require 99.999 availability and have relatively low traffic, basically some internal dashboards and tools. I develop my programs as deb/systemd packages and deploy in "fat" ubuntu/debian incus containers. I have a cluster with three machines where one is used for build-containers and second for production containers. I am looking forward having time to have ZFS streaming incremental backup of the containers. For big servers I use Proxmox, which is great, but Incus (and IncusOS) feel a bit more futuristic, where Proxmox is more bullet-proof enterprise solution.
lynx97 9mo ago
Stockholm syndrome
jo-m 9mo ago
I have a somewhat similar setup on the application layer (rootless podman, quadlet), but it's NixOS, and there still is SSH ;) https://github.com/jo-m/fluffy https://github.com/jo-m/fluffy
ThePowerOfFuet 9mo ago
>Now that tinkerbell is up and running, I cannot even SSH into it. In fact, nothing can. What's the point? Disable password auth, key only, and leave it be until the day you need a shell. Then nobody but you can SSH into it.

116 points by TheWiggles 9mo ago

I’ve been down a similar journey with Fedora Core OS and have loved it. The predictability and drop in toil is so nice. https://blog.gripdev.xyz/2024/03/16/in-search-of-a-zero-toil-homelab-with-immutable-linux/ https://blog.gripdev.xyz/2024/03/16/in-search-of-a-zero-toil...

samhclark 9mo ago
Yeah, this is closer to what I do, too. I was surprised not to see a Containerfile in the linked github repo in the article (https://github.com/lthms/tinkerbell https://github.com/lthms/tinkerbell) I found working with normal `dnf` and normal config files much easier than dealing with Ignition and Butane. Plus, working with your image in CI/CD instead of locally fixed my ZFS instability. When Fedora kernel updates, but ZFS doesn't support that version yet, now it fails in GitHub Actions and the container is never built, so there's no botched update that my NAS mistakenly picks up.

stryan 9mo ago

Quadlets are a real game changer for this type of small-to-medium scale declarative hosting. I've been pushing for them at work over ugly `docker compose in systemd units` service management and moved my home lab over to using them for everything. The latter is a similar setup to OP except with OpenSUSE MicroOS instead of Fedora CoreOS and I'm not so brave as to destroy and rebuild my VPS's whenever I make a change :) . On the other hand, MicroOS (and I'm assuming FCOS) reboots automatically to apply updates with rollback if needed so combined with podman auto-update you can basically just spin up a box, drop the files on, and let it take care of itself (at least until a container update requires manual intervention). A few things in the article I think might help the author: 1. Podman 4 and newer (which FCOS should definitely have) uses netavark for networking. A lot of older tutorials and articles were written back when Podman used CNI for it's networking and didn't have DNS enabled unless you specifically installed it. I think the default `podman` network is still setup with DNS disabled by default. Either way, you don't have to use a pod if you don't want to anymore, you can just attach both containers to the same network and it should Just Work. 2. You can run the generator manually with "/usr/lib/systemd/system-generators/podman-system-generator --dry-run" to check Quadlet validity and output. Should be faster than daemon-reload'ing all the time or scanning the logs. And as a bit of self-promotion: for anyone who wants to use Quadlets like this but doesn't want to rebuild their server whenever they make a change, I'm created a tool called Materia[0] that can install, remove, template, and update Quadlets and other files from a Git repository. [0] https://github.com/stryan/materia https://github.com/stryan/materia

plagiarist 9mo ago
Do you know if it is possible to run a quadlet as an ephemeral systemd-sysuser? That would solve all my current problems.
- stryan 9mo ago
  Not sure I'm following; you want to create a an emphemeral system account and run a root-less Podman container as it? I don't think that's something supported out of the box but you may be able to jury rig something together by putting the quadlets directly in `/etc/containers/systemd/users/` instead of putting them in a home directory (since I'm assuming this is a systemd-sysuser created account and thus without a home).
  - plagiarist 9mo ago
    Yes, that's it. Have things running isolated by a sysuser as well as in a rootless container. I would be running containers for LAN software (like forgejo) where I'd rather have the data on disk or in a podman volume instead of in a home directory.

amluto 9mo ago

> I’ve later learned that restarting a container that is part of a pod will have the (to me, unexpected) side-effect to restart all the other containers of that pod. Anyone know why this is? Or, for that matter, why Kubernetes seems to work like this too? I have an application for which the natural solution would be to create a pod and then, as needed, create and destroy containers within the pod. (Why? Because I have some network resources that don’t really virtualize, so they can live in one network namespace. No bridges.) But despite containerd and Podman and Kubernetes kind-of-sort-of supporting this, they don’t seem to actually want to work this way. Why not?

stryan 9mo ago
Yeah I was a little confused at this line; as far as I can tell you can restart containers that are a part of a Podman pod without restarting the whole pod just fine. I just verified this on one of my MicroOS boxes running Podman v5.7.1 . Podman was changing pretty fast for a while so it could be an older version thing, though I'd assume FCOS is on Podman 5 by now.
gucci-on-fleek 9mo ago
> Anyone know why this is? In Podman, a pod is essentially just a single container; each "container" within a pod is just a separate rootfs. So from that perspective, it makes sense, since you can't really restart half of a container. (But I think that it might be possible to restart individual containers within a pod; but if any container within a pod fails, then I think that the whole pod will automatically restart) > Why? Because I have some network resources that don’t really virtualize, so they can live in one network namespace. You can run separate containers in the same network namespace with the "--network" option [0]. You can either start one container with its own automatic netns and then join the other containers to it with "--network=container:<name>", or you can manually create a new netns with "podman network create <name>" and then join all the containers to it with "--network=<name>". [0]: https://docs.podman.io/en/latest/markdown/podman-run.1.html#network-mode-net https://docs.podman.io/en/latest/markdown/podman-run.1.html#...
- amluto 9mo ago

andrewmcwatters 9mo ago

I concede that this is the state of the art in secure deployments, but I’m from a different age where people remoted into colocated hardware, or at least managed their VPSs without destroying them every update. As a result, I think developers are forgetting filesystem cleanliness because if you end up destroying an entire instance, well it’s clean isn’t it? It also results in people not knowing how to do basic sysadmin work, because everything becomes devops. The bigger problem I have with this, is the logical conclusion is to use “distroless” operating system images with vmlinuz, an init, and the minimal set of binaries and filesystem structure you need for your specific deployment, and rarely do I see anyone actually doing this. Instead, people are using a hodgepodge of containers with significant management overhead, that actually just sit on like Ubuntu or something. Maybe alpine. Or whatever Amazon distribution is used on ec2 now. Or of course, like in this article, Fedora CoreOS. One day, I will work with people who have a network issue and don’t know how to look up ports in use. Maybe that’s already the case, and I don’t know it.

irishcoffee 9mo ago
> The bigger problem I have with this, is the logical conclusion is to use “distroless” operating system images with vmlinuz, an init, and the minimal set of binaries and filesystem structure you need for your specific deployment, and rarely do I see anyone actually doing this. In the few jobs I’ve had over 20 years, this is common in the embedded space, usually using yocto. Really powerful, really obnoxious tool chain.
bitwize 9mo ago
What you describe is from the "pets" era of server deployment, and we are now deep into the "cattle" era. Train yourself on destroying and redeploying, and building observability into the stack from the outset, rather than managing a server through ssh. Every shop you go to professionally is going to work like this. Eventually, Linux desktops will work like this also, especially with all the work going into systemd to support movable home directories, immutable OS images with modular updates, and so forth.
- andrewmcwatters 9mo ago

crawshaw 9mo ago

The idea that an "observability stack" is going to replace shell access on a server does not resonate with me at all. The metrics I monitor with prometheus and grafana are useful, vital even, but they are always fighting the last war. What I need are tools for when the unknown happens. The tool that manages all my tools is the shell. It is where I attach a debugger, it is where I install iotop and use it for the first time. It is where I cat out mysterious /proc and /sys values to discover exotic things about cgroups I only learned about 5 minutes prior in obscure system documentation. Take it away and you are left with a server that is resilient against things you have seen before but lacks the tools to deal with the future.

gear54rus 9mo ago
Agreed, this sounds like some complicated ass-backwards way to do what k8s already does. If it's too big for you, just use k3s or k0s and you will still benefit from the absolutely massive ecosystem. But instead we go with multiple moving parts all configured independently? CoreOS, Terraform and a dependence on Vultr thing. Lol. Never in a million years I would think it's a good idea to disable SSH access. Like why? Keys and non-standard port already bring China login attempts to like 0 a year.
ValdikSS 9mo ago
>What I need are tools for when the unknown happens. There are tools which show what happens per process/thread and inside the kernel. Profiling and tracing. Check Yandex's Perforator, Google Perfetto. Netflix also has one, forgot the name.
reactordev 9mo ago
Or… you build a container, that runs exactly what you specify. You print your logs, traces, metrics home so you can capture those stack traces and error messages so you can fix it and make another container to deploy. You’ll never attach a debugger in production. Not going to happen. Shell into what? Your container died when it errored out and was restarted as a fresh state. Any “Sherlock Holmes” work would be met with a clean room. We have 10,000 nodes in the cluster - which one are you going to ssh into to find your container to attach a shell to it to somehow attach a debugger?

gucci-on-fleek 9mo ago

Fedora IoT [0] is a nice intermediate solution. Despite its name, it's really good for servers, since it's essentially just the Fedora Atomic Desktops (Silverblue/Kinoite) without any of the desktop stuff. It gets you atomic updates, a container-centric workflow, and easy rollbacks; but it's otherwise a regular server, so you can install RPMs, ssh into it, create user accounts, and similar. This is what I do for my personal server, and I'm really happy with it. [0]: https://fedoraproject.org/iot/ https://fedoraproject.org/iot/

dorfsmay 9mo ago

Perfect timing for me, I've just been spending my side-project time in the last few weeks on building the smallest possible VMs with different glibc distros exactly for this, running podman containers, and comparing results.

starttoaster 9mo ago

So it's AWS Fargate with a different name? That's cool for cloud hosted stuff. But if you're on prem, or manage your own VPS' then you need SSH access.

npodbielski 9mo ago
I bought last year mobo with IPMI so in theory I could have forgot about SSH and just inspect startup logs if it would fail to start. Though I must say I am not brave enough and my family uses it so I prefer to have jest one broken service instead of enire machine. But it is possible.
- denkmoon 9mo ago
  Except you've replaced something good with something worse. IPMI really isn't an improvement over having SSH to the system. It definitely has more security holes.
  - npodbielski 9mo ago
    Can you SSH into broken grub? Can you change BIOS settings? Also giving access like that outside your home network is not a good idea. So security issues does not matter that much.
    - denkmoon 9mo ago
      I didn't say not to use IPMI, I said it's not a security improvement over SSH. For exactly the reason you point out, giving access outside your home. Nothing wrong with exposing SSH provided you take it seriously and know what you're doing. Nobody in their right mind would ever put IPMI on anything but a protected isolated network.

yigalirani 9mo ago

real programmers can ssh to their servers

libHacker 9mo ago
It's true. There's no reason to disable ssh. If you need it, it's there. If not, just don't use it.

npodbielski 9mo ago

That is looks interesting. An idea to configure server on run via symtemd would probably mean that migrating from machine to machine would be very easy. It always meant for me at least two days of carefull planning, copying od files testing and fixes because I always forgot about some obscure config changes I did somewhere, like adding DNS entry somewhere or disabling default SMTP on debian.

skeptic_ai 9mo ago

You can try docker compose with Watch tower. Then you just deploy a new branch: dev, prod. On server side counterparty you fetch updates on git, if anybody change, it will run docker compose, which will build your image and put it live. Worked well for me a few years. Problems: when you have issues you need to look into pertainer logs to see why it failed. That’s one big problem, if prefer something like Jenkins to build it instead. And if you have more groups of docker compose, you just put another sh script to do this piling on the main infrastructure git repo, which on git change will spawn new git watchers

hahahahhaah 9mo ago

It is not just fine, it is best practice. SSH is 2020s version of 2010s driving to the data centre.

gerdesj 9mo ago

I'm glad you have a stack that works for you. The great thing is we have choice and it was not always so. I suggest that you be careful of the DevOps way. Sometimes a "pet" is the way to go, especially if you only have one. If you have a thundering herd then you'll be hand rolling your own nonsense with the best of the cloudy cowboys and have a out of service sign that says "they did it" for when the lights wink out! I also notice that the word security does not grace your blog posting. That is a sure sign of the DevOps Way 8) You might look into the sysadmin way. Its boring, to be sure: all that fussing over security and the like! You could look into VPNs for access to your gear. An IPSEC, OpenVPN or Wireguard seems to keep most baddies away simply because it is a lot of effort to even engage with one. There are a huge number of ways that a VPN is configured. Then you have ssh, which can be very securely configured (or not). You can also use firewalls and I'm sure you do. If you have a static IP at home then simply filter for that. Make use of allow/deny lists - there are loads for firewalls of all sorts. Dumping remote shell access is not useful.

VladVladikoff 9mo ago

This is right up there with people who are happy to not have root access on their phones.

Zopieux 9mo ago

"tech blogger not reinvent NixOS but badly" challenge (impossible)

deathanatos 9mo ago

> [Kubernetes…] Managed clusters could make things easier, but they aren’t cheap. That would defeat the initial motivation behind retiring moana. Kubernetes is perfectly fine running on a "cluster" with a single node / this seems to be under the misconception that k8s requires >1 node. It doesn't, though obviously a single or two node cluster will not maintain a majority if a node goes down. For self-hosting, though, that might be perfectly acceptable. (My own self-hosted server was a single-node k8s cluster.)

k_bx 9mo ago

I am also slowly preparing myself to the world where there is no SSH into the server machine. I am following what's happening around IncusOS. Already sold on Incus for my containers, it does make sense on a paper: safe auto-updates, no manual key management, all you need is managed via API in a cluster (usually).

mvdwoord 9mo ago
Can you tell me a bit more on how you use Incus.. is it just personal use, or otherwise? What type of workloads do you run on it, and how is your networking setup / experience?
- k_bx 9mo ago
  I use it for professional use, running production services which don't require 99.999 availability and have relatively low traffic, basically some internal dashboards and tools. I develop my programs as deb/systemd packages and deploy in "fat" ubuntu/debian incus containers. I have a cluster with three machines where one is used for build-containers and second for production containers. I am looking forward having time to have ZFS streaming incremental backup of the containers. For big servers I use Proxmox, which is great, but Incus (and IncusOS) feel a bit more futuristic, where Proxmox is more bullet-proof enterprise solution.

lynx97 9mo ago

Stockholm syndrome

jo-m 9mo ago

I have a somewhat similar setup on the application layer (rootless podman, quadlet), but it's NixOS, and there still is SSH ;) https://github.com/jo-m/fluffy https://github.com/jo-m/fluffy

ThePowerOfFuet 9mo ago

>Now that tinkerbell is up and running, I cannot even SSH into it. In fact, nothing can. What's the point? Disable password auth, key only, and leave it be until the day you need a shell. Then nobody but you can SSH into it.