6 ms·

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear,

by hnarn 3y ago

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists to make scaling and redundancy easier, but it’s only easier if you fully understand the implications of every configuration that you make. Complexity causes incidents, so my mantra will always be that if you propose to introduce complexity, have a justification ready for why its inherent risks are outweighed by the benefits. If you’re deploying your side projects in an infrastructure this complex, I would strongly suggest taking a step back and questioning if the same benefits really couldn’t be achieved in a simpler way.

CommanderData 3y ago
If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.
- aequitas 3y ago
  I have setups where the container orchestration was done completely by Terraform or Puppet. Docker Compose also comes a long way if you are not running a FAANG webscale service. If you manage your VM's basics with configuration management and infrastructure as code anyways, it's easy to bolt containers onto that and treat them like any other installable piece of code. Kubernetes does bring a nice abstraction if you have DevOps teams and separate Ops for your bare metal or Cloud.
- hnarn 3y ago
  It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question. If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to resolve the issues arising from that architecture. But I don't think it's fair to start a general discussion about architecture from a point of "Step 0: The application is made to run in Kubernetes". By the way, is the statement that "most applications today are containerized" really true? I would claim that you can only count applications into this category that are published as "containerized by default", and to me that certainly doesn't seem true. I've encountered a few, but they're exceedingly rare. If you also count "applications that are available as a container", then sure, but then the statement should be that "most applications today *can* be containerized", or they are "available" in containerized form. Whether something is "more complicated than a single container" depends not only on the application, but how you choose to deploy it. If you take a hypothetical web application running nginx for PHP with a Postgres backend, you can deploy both of those as containers, but you certainly don't have to. You could skip using containers entirely, or you could deploy the nginx component as unmanaged containers as if they were an RPM/DEB-package, while not involving containers for your database at all. Yes, it requires more configuration for all the things that are no longer "magic", but with static infrastructure the work you put in comes back to you as "simplicity": the moving parts are easier to understand, which means problems are easier to troubleshoot. Yes, you will lose "auto-healing", but you also lose "auto-nuking", and so on. I want to be clear again that I'm not saying Kubernetes is objectively bad, but I am saying that I think a lot of people using it have not considered whether it's a suitable solution for their current problems (never-mind imaginary, future problems). To me, it's like comparing a lawn mower to the space shuttle, and then trying to motivate building the latter by saying that a lawn mower will never be able to get into space: if you're running a web site that is looking at a few hundred thousand users, you're still (probably) not "going to space". Fundamentally, I think the over-use of Kubernetes is nothing more than an example of premature optimization. The idea goes something like: "If we become the most successful /thing/ in the world, the infrastructure is already made in such a way that we can scale" -- but there are no decisions when it comes to infrastructure where you get the pros but can escape the cons, and those cons for Kubernetes specifically seem to rarely be discussed out of fear of sounding like a Luddite.
  - hdjjhhvvhga 3y ago
    You have omitted the crux of the issue: the whole point of GitOps is that you can always roll your deployments back, to any particular commit if needed. The very fact that GitOps was used against itself means it was set up incorrectly.
    - hnarn 3y ago
      I never even mentioned gitops, and furthermore (to my point), gitops principles do not require kubernetes.
- selfmodruntime 3y ago
  compose? docker swarm?
- Gud 3y ago
  Reduce your abstractions and reduce your dependencies for a more stable environment. It will give you greater control while you at the same time don’t need to relearn everything every 5 years. Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? This fatal scenario could have been easily avoided with a basic rsync cronjob to rsync.net or another $10 virtual machine.
  - diarrhea 3y ago
    > Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? I am very interested in this upcoming space. I had set up devbox.io for local development, and it even offers the concept of services. Like a docker compose for local development, just directly on the host. Nasty and hairy stuff like Python is still best shoved into a container to be done with it, but even there we are seeing a move to venv by default (Debian 12+), which should solve a lot of issues, making *nix more viable again. So I guess the main distinction so far is that containers are more performant, at less isolation than VMs, and containers have a gigantic ecosystem of build tools. It's easy to build a OCI image. Everyone and their mother has put together a Dockerfile by now. Not so with VMs. That might change if nix catches on and we get proper tools to fully describe VMs a priori. I reckon technology is already good enough to make VM startups performant enough (Firecracker, ...)?
    - fhaldridge7 3y ago
      I tried cloud-hypervisor recently and it booted the VM in 70ms, but I have no idea how to deploy this. We already have a deployment workflow for OCI images, doing this for VMs would mean a big refactoring
    - Gud 3y ago
      I believe this gigantic ecosystem of build tools is there because the solution itself is needlessly complex. I also believe using python for web development is a mistake.
- isitmadeofglass 3y ago
  [dead]
- hdjjhhvvhga 3y ago
  > If the application is containerised, like most are these days, how do you propose running these on said VM? As far as I can tell, this kind of load can be perfectly served by a €40 bare metal Hetzner instance (together with a Minecraft server and what not) so even a simple Docker Compose would do.
nucleardog 3y ago
> Remember, Kubernetes exists to make scaling and redundancy easier, but it’s only easier if you fully understand the implications of every configuration that you make. It exists to make it easier. Did anyone ask if it’s something you needed to be doing in the first place? For many, I think using kubernetes is like buying something you don’t need at the store because it’s on sale and thinking it’s saving you money. I think we all have a desire to build out infinitely scalable and 100% reliable solutions because _obviously_ the site can’t go offline. But did you ever actually ask what would happen if it did? I worked with one company that, about 8 years into their existence, botched a data migration and went down for an entire month. Customers called CS every day begging them to fix it, they refunded everyone’s fees for the month, and… not one customer cancelled. That company is still around, still growing strong, and still growing. I’m not suggesting going to the other extreme but… if your site goes down for a few hours overnight until you wake up and fix it, for most projects it’s not going to be the end of things. Hell, GitLab is still around! (On that note… Take _durability_ seriously. Have backups. rclone everything you have into b2 or something. Even if it takes you a few days to restore the entire service, at least it will come back. If you lose everyone’s data _then_ you’re probably done.)
JohnMakin 3y ago
It's hard to blame kubernetes as the issue here. The way EKS is implemented, it's basically EC2's running a bunch of containers with a convenient control plane and network interface abstracted over them. If they had used EC2's/VM's, they could have still run into the same issue by blowing away a host's local volume they were never actually backing up. I have administrated social media websites (not this software, mainly discourse) and have seen this exact thing happen. Luckily, the backups worked.

by hnarn 3y ago

CommanderData 3y ago

If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

aequitas 3y ago
I have setups where the container orchestration was done completely by Terraform or Puppet. Docker Compose also comes a long way if you are not running a FAANG webscale service. If you manage your VM's basics with configuration management and infrastructure as code anyways, it's easy to bolt containers onto that and treat them like any other installable piece of code. Kubernetes does bring a nice abstraction if you have DevOps teams and separate Ops for your bare metal or Cloud.
hnarn 3y ago
It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question. If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to resolve the issues arising from that architecture. But I don't think it's fair to start a general discussion about architecture from a point of "Step 0: The application is made to run in Kubernetes". By the way, is the statement that "most applications today are containerized" really true? I would claim that you can only count applications into this category that are published as "containerized by default", and to me that certainly doesn't seem true. I've encountered a few, but they're exceedingly rare. If you also count "applications that are available as a container", then sure, but then the statement should be that "most applications today *can* be containerized", or they are "available" in containerized form. Whether something is "more complicated than a single container" depends not only on the application, but how you choose to deploy it. If you take a hypothetical web application running nginx for PHP with a Postgres backend, you can deploy both of those as containers, but you certainly don't have to. You could skip using containers entirely, or you could deploy the nginx component as unmanaged containers as if they were an RPM/DEB-package, while not involving containers for your database at all. Yes, it requires more configuration for all the things that are no longer "magic", but with static infrastructure the work you put in comes back to you as "simplicity": the moving parts are easier to understand, which means problems are easier to troubleshoot. Yes, you will lose "auto-healing", but you also lose "auto-nuking", and so on. I want to be clear again that I'm not saying Kubernetes is objectively bad, but I am saying that I think a lot of people using it have not considered whether it's a suitable solution for their current problems (never-mind imaginary, future problems). To me, it's like comparing a lawn mower to the space shuttle, and then trying to motivate building the latter by saying that a lawn mower will never be able to get into space: if you're running a web site that is looking at a few hundred thousand users, you're still (probably) not "going to space". Fundamentally, I think the over-use of Kubernetes is nothing more than an example of premature optimization. The idea goes something like: "If we become the most successful /thing/ in the world, the infrastructure is already made in such a way that we can scale" -- but there are no decisions when it comes to infrastructure where you get the pros but can escape the cons, and those cons for Kubernetes specifically seem to rarely be discussed out of fear of sounding like a Luddite.
- hdjjhhvvhga 3y ago
  You have omitted the crux of the issue: the whole point of GitOps is that you can always roll your deployments back, to any particular commit if needed. The very fact that GitOps was used against itself means it was set up incorrectly.
  - hnarn 3y ago
    I never even mentioned gitops, and furthermore (to my point), gitops principles do not require kubernetes.
selfmodruntime 3y ago
compose? docker swarm?
Gud 3y ago
Reduce your abstractions and reduce your dependencies for a more stable environment. It will give you greater control while you at the same time don’t need to relearn everything every 5 years. Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? This fatal scenario could have been easily avoided with a basic rsync cronjob to rsync.net or another $10 virtual machine.
- diarrhea 3y ago
  > Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? I am very interested in this upcoming space. I had set up devbox.io for local development, and it even offers the concept of services. Like a docker compose for local development, just directly on the host. Nasty and hairy stuff like Python is still best shoved into a container to be done with it, but even there we are seeing a move to venv by default (Debian 12+), which should solve a lot of issues, making *nix more viable again. So I guess the main distinction so far is that containers are more performant, at less isolation than VMs, and containers have a gigantic ecosystem of build tools. It's easy to build a OCI image. Everyone and their mother has put together a Dockerfile by now. Not so with VMs. That might change if nix catches on and we get proper tools to fully describe VMs a priori. I reckon technology is already good enough to make VM startups performant enough (Firecracker, ...)?
  - fhaldridge7 3y ago
    I tried cloud-hypervisor recently and it booted the VM in 70ms, but I have no idea how to deploy this. We already have a deployment workflow for OCI images, doing this for VMs would mean a big refactoring
  - Gud 3y ago
    I believe this gigantic ecosystem of build tools is there because the solution itself is needlessly complex. I also believe using python for web development is a mistake.
isitmadeofglass 3y ago
[dead]
hdjjhhvvhga 3y ago
> If the application is containerised, like most are these days, how do you propose running these on said VM? As far as I can tell, this kind of load can be perfectly served by a €40 bare metal Hetzner instance (together with a Minecraft server and what not) so even a simple Docker Compose would do.

nucleardog 3y ago

> Remember, Kubernetes exists to make scaling and redundancy easier, but it’s only easier if you fully understand the implications of every configuration that you make. It exists to make it easier. Did anyone ask if it’s something you needed to be doing in the first place? For many, I think using kubernetes is like buying something you don’t need at the store because it’s on sale and thinking it’s saving you money. I think we all have a desire to build out infinitely scalable and 100% reliable solutions because _obviously_ the site can’t go offline. But did you ever actually ask what would happen if it did? I worked with one company that, about 8 years into their existence, botched a data migration and went down for an entire month. Customers called CS every day begging them to fix it, they refunded everyone’s fees for the month, and… not one customer cancelled. That company is still around, still growing strong, and still growing. I’m not suggesting going to the other extreme but… if your site goes down for a few hours overnight until you wake up and fix it, for most projects it’s not going to be the end of things. Hell, GitLab is still around! (On that note… Take _durability_ seriously. Have backups. rclone everything you have into b2 or something. Even if it takes you a few days to restore the entire service, at least it will come back. If you lose everyone’s data _then_ you’re probably done.)

JohnMakin 3y ago

It's hard to blame kubernetes as the issue here. The way EKS is implemented, it's basically EC2's running a bunch of containers with a convenient control plane and network interface abstracted over them. If they had used EC2's/VM's, they could have still run into the same issue by blowing away a host's local volume they were never actually backing up. I have administrated social media websites (not this software, mainly discourse) and have seen this exact thing happen. Luckily, the backups worked.