7 ms·

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multipl

by Keycap 6y ago

Why would you think it would go away? Its literaly the best option currently for what it is doing. It has an unprecedented support behind it as well. Multiply Vendors support it through a certificate k8s managed service. It solves really problems out ouf the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair. It allows for simple IaaC. Is it young? Yes. Do we need more people with more expierence? yes. Is this a problem? No.

smt88 6y ago
"out of the box" is a huge stretch. So is "simple". > Is it young? Yes. Do we need more people with more expierence? yes. Of course this is a problem! It's costly (in time, money, and security) to pay your team to ramp up on Kubernetes. The question is, what is the actual benefit? 99% of companies don't need it at all.
- Keycap 6y ago
  The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all. Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them. It is a ton easier to allow someone only access to one namespace, limited ingress domains (which get provisioned automatically) and allow them to only run non root containersl The question is not what the actual benefit is (its clear and i mentioned it in my paragraphs before), the question is will kubernetes ever become so lightweight, stable, easy to use so that its feasable for normal people to run it on a 3-5 node cluster in small companies. And please lets ignore all those small companies where people log into their VMs by hand and maintaining them by using snapshots and cloning existing VMs. Those small companies might and should just migrate to cloud managed services completly.
  - smt88 6y ago
    > Its costly and risky to run 100 VMs... A tiny fraction of companies need to run 100+ VMs. It takes a lot of traffic to require that scale, unless you're just throwing money at infra to avoid optimizing your code (which can be cost effective). A large chunk of companies should just be on something like SquareSpace or Shopify, another large chunk should be using off-the-shelf services like Azure's App Containers or Amazon's Elastic Beanstalk/Lightsail/whatever else they have now, and some internal services can easily run on a single VM with no orchestration at all. All that stuff is really expensive at scale, but unless you're a large company, your engineering time is going to cost way more than just using whatever managed services your cloud offers. > ...maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. This is not the only alternative to using Kubernetes. You can automate all of this without Kubernetes, or you can use a cloud provider's managed Kubernetes service. A lot of companies get by just fine with Heroku. If you assume people are using practices from 2005 (or that cloud providers don't already provide a layer of abstraction on top of Kubernetes), of course Kubernetes looks better.
    - Keycap 6y ago
      I did mention myself that small companies should just use cloud services right? I did not say that everyone should migrate to kubernetes as far as i rmemeber but kubernetes to me is not a fad and it fixes real issues companies have.
- louwrentius 6y ago
  I agree so much with your sentiment. This is what will get lost on hn: 99.99% of companies don't need FAANG-level capabilities. I bet that 80% of people moving to K8s just follow hype and don't even benefit in any netto positive way. And it's probably 99% but I'm just too afraid to be wrong.
thu2111 6y ago
I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it: https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/920#shard-performance https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/920... Kubernetes looks like a massive LOSS in the case of this service. The perceived scaling benefits didn't materialise at all, and their costs more than doubled. They also spent a lot of engineering time optimising startup time of this job: 7 PRs and complex writeups/testing were required. Just so they could try to auto-scale a job to reduce the hit of the hugely multiplied base costs. If not for Kubernetes that eng time might have been spent adding features to the product instead. It solves really problems out out the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair Most of these problems are created by Kubernetes, like "ingress" which is a term only Kubernetes and low level networking uses, "cert management" which often isn't required in more traditional setups beyond provisioning web servers with the SSL files, "health checks" are a feature of any load balancer and the whole point of VMs is to avoid the need to repair hardware. Finally the demand auto-scaling as seen in this case is (a) possible without Kubernetes and (b) not actually working well anyway. Frankly this set of bugs, writeups, PRs etc is quite scary. I worked with Borg at Google and saw how it was both a force multiplier but also a huge timesink in many ways. It made some complex things simple but also made some simple things absurdly complex. At one point I had a task to just serve a static website on Borg, it turned into a three month nightmare because the infrastructure was so fragile and Google-specific. A bunch of Linux VMs running Apache would have been far faster to set up and run.
- jarv 6y ago
  > Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it: Note this is just one service out of many and has very low latency requirements, and due to our slow pod start times in this case it didn't make sense to auto-scale. We are auto-scaling for other services in K8s. > says their cost went from $290/month when running in VMs to $700/month These numbers represent a snapshot in time where we are overprovisioning, for this service for the migration. You are correct that cost benefit here remains to be seen for this service. We are doing things like isolating services into separate node pools which may not allow us to be as efficient. Safer and faster deploys was a huge win for us, for this service and others. This of course is compared to our existing configuration using VMs and Chef to manage them. disclaimer: blog post author
pjmlp 6y ago
30+ years of experience seeing fads come and go in this fashion driven industry.
- Keycap 6y ago
  Do you have any argument for it or is it a generic statement independent of anything? Because there have been other 'fads' still being strong today. Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself. mesos, nomad, docker swarm and co. Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them. Cloud native is here to stay; It will affect and already affects tools, applications and architecture.
  - secondcoming 6y ago
    > Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them. I want a VM because I want my dev environment to exactly replicate the production environment, down to the kernel. It also means that if I leave my employer I can just delete my VM on my personal machine. We've also not had any issues with maintaining production VMs, our release pipeline packages up our code into a debian package and them creates a VM image from that. This is then deployed across all regions. If you need to roll back just deploy the previous VM image version. GCP handles killing old instances and starting new instances for you.
    - Keycap 6y ago
      I did not talk about your specific use case. Feel free to be happy with your setup. Also i'm not able to determine if your setup would still be much better (faster, easier to maintain etc.) if you would set completly on containers instead of a VM as i'm not aware of your workload at all.
    - auspex 6y ago
      Having the application run the same in dev as in production is one of the main selling features of a containerized environment.
selfhoster11 6y ago
Technologies don't last long. Another new tech comes out that does the job better or more cheaply, and the old tech either dies or competes. I think that K8s is much too complicated for medium-scale workloads, so as soon as something makes it easier than K8s to scale and manage workloads at that scale, K8s will be out of that space.
- Keycap 6y ago
  And thats why you have to stay on your toes and migrate away if it starts to make sense. It doesn't make much sense to migrate to k8s if you don't have an issue. And it doesn't make sense to wait for the next thing to happen when you have an issue you need to fix now.
rossmohax 6y ago
Kubernetes hype looks, feels and smells very similar to what was happening with OpenStack. Lavish conferences, every vendor out there announces "support for Kubernetes" in their products, evangelists and many many burnt engineers.

by Keycap 6y ago

smt88 6y ago

"out of the box" is a huge stretch. So is "simple". > Is it young? Yes. Do we need more people with more expierence? yes. Of course this is a problem! It's costly (in time, money, and security) to pay your team to ramp up on Kubernetes. The question is, what is the actual benefit? 99% of companies don't need it at all.

Keycap 6y ago
The features i mentioned, are solved 'out of the box'. I'm not seeing a huge stretch here at all. Its costly and risky to run 100 VMs, maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. It is cost ineffective to have security audits on 100 VMs, maintaining access to them, auditing whats happening on them. It is a ton easier to allow someone only access to one namespace, limited ingress domains (which get provisioned automatically) and allow them to only run non root containersl The question is not what the actual benefit is (its clear and i mentioned it in my paragraphs before), the question is will kubernetes ever become so lightweight, stable, easy to use so that its feasable for normal people to run it on a 3-5 node cluster in small companies. And please lets ignore all those small companies where people log into their VMs by hand and maintaining them by using snapshots and cloning existing VMs. Those small companies might and should just migrate to cloud managed services completly.
- smt88 6y ago
  > Its costly and risky to run 100 VMs... A tiny fraction of companies need to run 100+ VMs. It takes a lot of traffic to require that scale, unless you're just throwing money at infra to avoid optimizing your code (which can be cost effective). A large chunk of companies should just be on something like SquareSpace or Shopify, another large chunk should be using off-the-shelf services like Azure's App Containers or Amazon's Elastic Beanstalk/Lightsail/whatever else they have now, and some internal services can easily run on a single VM with no orchestration at all. All that stuff is really expensive at scale, but unless you're a large company, your engineering time is going to cost way more than just using whatever managed services your cloud offers. > ...maintaining them and keeping them up to date, monitoring them and knowing when they are no longer needed. This is not the only alternative to using Kubernetes. You can automate all of this without Kubernetes, or you can use a cloud provider's managed Kubernetes service. A lot of companies get by just fine with Heroku. If you assume people are using practices from 2005 (or that cloud providers don't already provide a layer of abstraction on top of Kubernetes), of course Kubernetes looks better.
  - Keycap 6y ago
    I did mention myself that small companies should just use cloud services right? I did not say that everyone should migrate to kubernetes as far as i rmemeber but kubernetes to me is not a fad and it fixes real issues companies have.
louwrentius 6y ago
I agree so much with your sentiment. This is what will get lost on hn: 99.99% of companies don't need FAANG-level capabilities. I bet that 80% of people moving to K8s just follow hype and don't even benefit in any netto positive way. And it's probably 99% but I'm just too afraid to be wrong.

thu2111 6y ago

I dunno, because of writeups like GitLabs? We're lucky GitLab is so transparent. It lets us see that their claims don't appear to always match the actual reality. Here's an example. Blog post says: "After transitioning each service, we enjoyed many benefits of using Kubernetes in production, including much faster and safer deploys of the application, scaling, and more efficient resource allocation" Wrong. Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it: https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/920#shard-performance https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/920... Kubernetes looks like a massive LOSS in the case of this service. The perceived scaling benefits didn't materialise at all, and their costs more than doubled. They also spent a lot of engineering time optimising startup time of this job: 7 PRs and complex writeups/testing were required. Just so they could try to auto-scale a job to reduce the hit of the hugely multiplied base costs. If not for Kubernetes that eng time might have been spent adding features to the product instead. It solves really problems out out the box like load balancing, ingress, cert management, autoscaling, health checks, autorepair Most of these problems are created by Kubernetes, like "ingress" which is a term only Kubernetes and low level networking uses, "cert management" which often isn't required in more traditional setups beyond provisioning web servers with the SSL files, "health checks" are a feature of any load balancer and the whole point of VMs is to avoid the need to repair hardware. Finally the demand auto-scaling as seen in this case is (a) possible without Kubernetes and (b) not actually working well anyway. Frankly this set of bugs, writeups, PRs etc is quite scary. I worked with Borg at Google and saw how it was both a force multiplier but also a huge timesink in many ways. It made some complex things simple but also made some simple things absurdly complex. At one point I had a task to just serve a static website on Borg, it turned into a three month nightmare because the infrastructure was so fragile and Google-specific. A bunch of Linux VMs running Apache would have been far faster to set up and run.

jarv 6y ago
> Actual analysis by the engineers of one of their migrated jobs (urgent-other sidekiq shard) says their cost went from $290/month when running in VMs to $700/month when running in Kubernetes. They tried to use auto-scaling but failed completely and ended up disabling it: Note this is just one service out of many and has very low latency requirements, and due to our slow pod start times in this case it didn't make sense to auto-scale. We are auto-scaling for other services in K8s. > says their cost went from $290/month when running in VMs to $700/month These numbers represent a snapshot in time where we are overprovisioning, for this service for the migration. You are correct that cost benefit here remains to be seen for this service. We are doing things like isolating services into separate node pools which may not allow us to be as efficient. Safer and faster deploys was a huge win for us, for this service and others. This of course is compared to our existing configuration using VMs and Chef to manage them. disclaimer: blog post author

pjmlp 6y ago

30+ years of experience seeing fads come and go in this fashion driven industry.

Keycap 6y ago
Do you have any argument for it or is it a generic statement independent of anything? Because there have been other 'fads' still being strong today. Kubernetes solves real problems which have been hard for a long time. Its the first thing you, as an infrastructure team, want to have to be able to provide your teams a manageable environment for yourself. mesos, nomad, docker swarm and co. Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them. Cloud native is here to stay; It will affect and already affects tools, applications and architecture.
- secondcoming 6y ago
  > Developers don't want a VM and you can't manage and maintain VMs if someone else is doing something with them. I want a VM because I want my dev environment to exactly replicate the production environment, down to the kernel. It also means that if I leave my employer I can just delete my VM on my personal machine. We've also not had any issues with maintaining production VMs, our release pipeline packages up our code into a debian package and them creates a VM image from that. This is then deployed across all regions. If you need to roll back just deploy the previous VM image version. GCP handles killing old instances and starting new instances for you.
  - Keycap 6y ago
    I did not talk about your specific use case. Feel free to be happy with your setup. Also i'm not able to determine if your setup would still be much better (faster, easier to maintain etc.) if you would set completly on containers instead of a VM as i'm not aware of your workload at all.
  - auspex 6y ago
    Having the application run the same in dev as in production is one of the main selling features of a containerized environment.

selfhoster11 6y ago

Technologies don't last long. Another new tech comes out that does the job better or more cheaply, and the old tech either dies or competes. I think that K8s is much too complicated for medium-scale workloads, so as soon as something makes it easier than K8s to scale and manage workloads at that scale, K8s will be out of that space.

Keycap 6y ago
And thats why you have to stay on your toes and migrate away if it starts to make sense. It doesn't make much sense to migrate to k8s if you don't have an issue. And it doesn't make sense to wait for the next thing to happen when you have an issue you need to fix now.

rossmohax 6y ago

Kubernetes hype looks, feels and smells very similar to what was happening with OpenStack. Lavish conferences, every vendor out there announces "support for Kubernetes" in their products, evangelists and many many burnt engineers.