25 ms·

Ask HN: Have You Left Kubernetes?

• colordrops 4y ago
  I went from Kubernetes to Ansible to Nix for my personal installation. Kubernetes was too complicated, and Ansible too brittle.
• thdespou 4y ago
  Too soon. Everyone is still milking it. Ask again in 10 years time.
• mars16 4y ago
  not in definitive, it was more a "see you later". The company didn't have enough resources (people) to maintain the cluster and we decided to use ECS while we were still maturing as a team.
• justinsaccount 4y ago
  My guess is most people are hitting the Trough of Disillusionment https://en.wikipedia.org/wiki/Gartner_hype_cycle https://en.wikipedia.org/wiki/Gartner_hype_cycle
• TobbenTM 4y ago
  Kind of? For my new projects nowadays, I'm pushing mainly serverless approaches using AWS Lambdas (behind API Gateways for stuff that needs to be reachable by HTTP). I think this shifts the complexity from managing Kubernetes and its accompanying ten-thousand-yaml-files to infrastructure-as-code and the complexities of dealing with AWS. And I happen to prefer the latter, even though it's not infinitely better by any margin. For the few things that needs to be always-online, or 3rd party self-hosted apps, I'm still on Kubernetes, or pure Docker if possible.
  • nova22033 4y ago
    infrastructure-as-code and the complexities of dealing with AWS So cloudformation YAML? or CDK?
    • stoobs 4y ago
      Or terraform/terragrunt, Pulumi or one of the other options out there
    • mdm12 4y ago
      Not the OP, but I have had success with CDK. The main advantages for me have been discoverability with respect to resource properties, along with proper, higher-level abstractions pertaining to AWS infrastructure. https://aws.amazon.com/blogs/devops/leverage-l2-constructs-to-reduce-the-complexity-of-your-aws-cdk-application/ https://aws.amazon.com/blogs/devops/leverage-l2-constructs-t...
    • TobbenTM 4y ago
      Terraform only for me, have not experienced anything better.
  • roflyear 4y ago
    How do you get over vendor lock-in, and the issue where it is often difficult to debug these solutions? Also how is the cost of Lambda? I know for AWS the logic apps have a really high cost (but function apps seem to be reasonable)
    • TobbenTM 4y ago
      Vendor lock-in is always omni-present, even if you are using Kubernetes. The only difference is that the vendor lock-in you get when you run Kubernetes is mainly with the infrastructure layer hosting the code, instead of the code entrypoints. Not to mention any managed services you want to use, which also will lock you in. So I don't do anything extreme to avoid vendor lock-in, other than making my code general enough to only have a small surface area for the lambda entry point. As a practical example, all of my APIs hosted on Lambdas are ordinary ASP.Net apps that would work identically if hosted in Docker containers. Pricing so far is one of the biggest benefits of doing serverless approaches. I'm down to paying a couple of dollars per month for something that I'd pay tenfold for if doing Kubernetes. Both the monetary sum of only paying for what you're using, and also not having to worry about cluster maintenance, scaling and management is a godsend.
• paulgb 4y ago
  We did. Our use case is spinning up containers on demand to user actions, giving them ephemeral, internet-routable hostnames, and shutting them down when all inbound connections have dropped. Because users are waiting to interact with these containers, we found the start times with Kubernetes too slow and its architecture to be a bad fit. We ended up writing our own control plane that uses NATS as a message bus. We are in the process of open sourcing it here: https://github.com/drifting-in-space/spawner https://github.com/drifting-in-space/spawner
  • cbanek 4y ago
    > we found the start times with Kubernetes too slow Just curious if you could elaborate here? I work with k8s on docker, and we're also going to be spinning up ephemeral containers (and most of the other things you say) with jupyter notebooks. We're all in on k8s, but since you might be ahead of me, just wondering what hurdles you have faced? Our big problem was fetching containers took too long since we have kitchen sink containers that are like 10 GB (!) each. They seem to spin up pretty fast though if the image is already pulled. I've worked on a service that lives in the k8s cluster to pull images to make sure they are fresh (https://github.com/lsst-sqre/cachemachine https://github.com/lsst-sqre/cachemachine) but curious if you are talking about that or the networking? From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser?
    • paulgb 4y ago
      Jupyter notebooks are actually a use case we think about a lot, you can try a live demo with a Jupyter notebook here: https://jamsocket.com/tmpenv/ https://jamsocket.com/tmpenv/ It wasn't really one thing with Kubernetes that was slow, but that the more we tried to optimize it the less of core Kubernetes we were using and so the less value we were getting for the complexity tax we were paying. The image pulling you mention is a good example of that; having pre-pulled images is a big factor, but we have too many images to push every image to every node, instead we'd like the scheduler to be aware of which node has which image. We could do that with node affinity, but what we'd end up building would be more work than if we wrote our own scheduler to support it from day one. > From what it looks like in your repo it might be that you need to do session timing (like ms) response time from a browser? Our goal is subsecond container starts. We're not there yet, and might not get there with Docker, but we have a POC that is there with WebAssembly-based workloads. Too bad those are rare :) (By the way, I'm always happy to chat about this stuff, my email is in my profile)
• zrail 4y ago
  Never started. My work infrastructure is Elastic Beanstalk and my personal infra is either hand-managed containers or Dokku. Previous gig maintained an internal abstraction on top of Kubernetes but it wasn't something I ever had to mess with.
• jokethrowaway 4y ago
  Still going strong in clients works, big companies. For personal projects I roll with just docker. I'm bothered by the minimal requirements of k8s, I want to deploy on 5$ machines
  • chemicalnovae 4y ago
    Take a look at k3s[0] for lighter k8s. [0] https://rancher.com/docs/k3s/latest/en/ https://rancher.com/docs/k3s/latest/en/
    • imWildCat 4y ago
      K3s is awesome! I used it to host several small projects in cheap virtual machines. The setup is very straightforward. I guess we just need better editing support of YAMLs. Thanks for the recommendation!
  • edmcnulty101 4y ago
    do you run docker swarm or anything? or just containers on the virtual machine? I would love to deploy with docker and no other orchestration tools.
• besus 4y ago
  Sort of..... I went from on-metal K8s clusters, which were a complete PITA and required a full team to manage, to using EKS which has been everything K8s should be... easy peasy.
  • troelsSteegin 4y ago
    EKS... https://aws.amazon.com/eks/getting-started/ https://aws.amazon.com/eks/getting-started/
• sidcool 4y ago
  Not really. It is working well for us, but it wasn't as easy to begin with. Storage, Networking & Debugging are the biggest challenges.
• oneplane 4y ago
  We have had a few teams try, but as soon as you go beyond "I want to run some code for a bit", nobody really has anything for you. Instead of trying to re-invent the wheel (service discovery, mutual TLS, cross-provider capabilities) successfully, it went downhill quite fast and they moved back. (this was mostly due to cost as other services can get expensive really quickly, and because of the lack of broadly available knowledge for the custom stuff they had to build) If a team were to start with no legacy and no complexity and there isn't going to be multi-team/multi-owner/shared-services I could see them using something else. But that applies to anything.
  • deleted 4y ago
    [deleted]
• dewlinedew2 4y ago
  Sure we are replacing Kops with EKS now..
• substation13 4y ago
  To those who have used K8s extensively... 1. Is it really so complicated? 2. Is that complexity incidental or essential? 3. Could we get away with a simpler set of abstractions for 90% of applications?
  • lapser 4y ago
    1. I used to use it at an old company in the early days of K8s. We ran our own setup, as EKS and AKS didn't exist. GCP did, but we were on AWS. It really is very complicated, however, with EKS, GCP, and AKS, it makes it a lot easier. Note, for users it's a lot simpler than the alternatives. Sure it's no heroku, but it vastly makes things easier compared to running on AWS, GCP, Azure, or worse, on bare metal. 2. Essential. K8s solves a problem that's quite complex. You can't really solve it in a simple manner. 3. Probably, but that 10% will require the additional abstractions and complications anyway, and it'll be easier to manage one system rather than 2.
  • canucklady 4y ago
    k8s is actually very simple as a user. It's complicated to operate it yourself without EKS< GKE, etc. But from an end user perspective you write some delcarative manifests, they get put into an event bus, and then the state of the world is reconciled with your manifests. Easy peasy.
    • api 4y ago
      > It's complicated to operate it yourself without EKS< GKE, etc. ... and we're now fully back to the mainframe era with the people in white coats who "run" the computer. The cloud truly is mainframe 2.0.
  • mixxit 4y ago
    imagine trying to create an infinitely scalable system that is bound by aws account limits
    • threeseed 4y ago
      a) AWS account limits are very flexible if you have spend enough money with them. b) Kubernetes clusters can span multiple accounts, clouds etc.
• zomglings 4y ago
  Not only have we left Kubernetes, we left Docker. Replaced with Linux servers and SSH. Have done a lot of work with k8s in the past. Not the right tool for my startup.
  • louwrentius 4y ago
    Why did you come to this conclusion and how are Linux servers a better fit?
    • zomglings 4y ago
      Came to this conclusion for teams of our current size based on years of experience and experimentation (never at the expense of the business). They are a better fit because they are much easier to manage and it's much easier to debug issues when something goes wrong. We are a small team and we hope to stay that way. But our operational responsibilities are growing significantly. The extra cognitive overhead of working with technologies like kubernetes would prevent us from scaling up our effort the way that we want to. It's hard to answer your question in detail outside of a very long essay.
  • k8sToGo 4y ago
    Are you using at least Nomad or something?
    • zomglings 4y ago
      Nope, literally SSH and bash scripts. We are fully open source (except for our security/operational code): https://github.com/bugout-dev https://github.com/bugout-dev
  • cpach 4y ago
    Interesting! Feel free to elaborate. What does your CI/CD/deployment pipeline look like? Do you use something like Ansible, Puppet, Chef, Salt etc?
    • 4y ago
• seibelj 4y ago
  Managing k8s requires a strong DevOps team. A strong DevOps engineer is a solid software engineer plus a specialty. These people are rare and expensive and extremely hard to attract to startups. Therefore you either can’t find anyone or more likely you hire less good DevOps engineers. The solution is to not use k8s as a startup. The less a DevOps engineer can shoot themselves in the foot the better.
• lkurusa 4y ago
  We use Nomad from Hashicorp, it's super simple. Never liked the complexity K8s brings along.
  • dimitrios1 4y ago
    I never liked the cost Hashicorp products bring along.
    • OJFord 4y ago
      What, zero? (GP didn't say anything about using Hashicorp-managed products, they're open source and beer-free to use. Another comment says Hashicorp's platform doesn't even offer Nomad (yet?) anyway.)
      • pentium166 4y ago
        If you need features offered by the self-managed Enterprise version of a Hashicorp product, I've heard the price tag is something like low six figures per product.
      • dimitrios1 4y ago
        Using terraform without TFE is something I would never recommend to any large org. Been there, done that.
      • wlonkly 4y ago
        Hashicorp is very inflexible about support plans -- either you go all in on their Enterprise product, or you're self supported. By the time you've licensed Nomad, Consul and Vault -- because they interact and you will find Nomad support ends where Consul support begins, and so on -- it is a LOT of money.
    • lkurusa 4y ago
      Hmm. Please could you explain further? I'm genuinely curious what costs you associate with Hashicorp products.
• Bayart 4y ago
  Nope, I like k8s. What I don't like is people trying to be overly smart with it and leaving a configuration hell of templates, weird network configurations and broken certs behind them. For my personal workloads it's all basic containers with a reverse proxy, though.
• varispeed 4y ago
  I really wish Rancher didn't abandon Rancher 1.6 and moved to k8s. This was a perfect solution for a small business and bare metal. I am trying to move on k3s but it is just too complex to run anything and there is still not solved problem of exposing services to internet. What I want is to declare I want this service to be under this domain and this IP - so for that you still need to configure your load balancer (bare metal) manually, setup certificates etc. I am writing a tool to automate this, but it's been a pain.
  • toteno 4y ago
    > What I want is to declare I want this service to be under this domain and this IP - so for that you still need to configure your load balancer (bare metal) manually, setup certificates etc. I am writing a tool to automate this, but it's been a pain. After initial setup you can do it quite easily. Exposing a service on selected domain is several lines in Ingress and adding certificates is several more. Example: https://cert-manager.io/docs/tutorials/acme/nginx-ingress/#step-7---deploy-a-tls-ingress-resource https://cert-manager.io/docs/tutorials/acme/nginx-ingress/#s...
    • varispeed 4y ago
      So this is not going to work for several reasons. One being that on bare metal you don't have a cloud provider, so there is no load balancer it can talk to. Second - it will setup a hostname and a certificate on the ingress, but there is no way to contact it from outside world. The domain still needs A record pointing at the server and in the cluster that may be a local IP or a set of IPs. What I have in mind is an external server that is not being a part of the cluster that bears the role of load balancer. It will contact the cluster and look for services and then setup up a reverse proxy based on their declared hostname, then setup certificates and update DNS records at DNS provider. As far as I know something like this does not exist. Maybe Traefik has such a capability, but their documentation is so complex I have no idea.
      • toteno 4y ago
        Actually I'm using it on bare metal and it works. Initial setup wasn't very hard but I think it could be more intuitive. Overall I think documentation for self-hosting kubernetes sometimes a bit incomplete. Yes, I need to add A records with IPs for each domain, but that's one time setup. I did it manually, but you can automate it [1] (depends on what you use for DNS provider but you can extend it to support your provider or maybe there is another existing solution). I'm not sure that one server in front of the cluster is more reliable than using all cluster nodes for load balancing. I guess that in automated solutions like [1] cluster's node could be automatically deleted from DNS if it went down. My setup is not so big so I don't have real need for load balancing, but it seems possible with existing solutions. [1] https://github.com/kubernetes-sigs/external-dns https://github.com/kubernetes-sigs/external-dns
• superice 4y ago
  Yes! My startup of 5 people did. We started out with a managed Kubernetes cluster on DigitalOcean, but there were a number of reasons that caused us to not be very comfortable with that setup. - Taking random .yml configs from The InternetTM to install an Nginx Ingress with automatic LetsEncrypt certs felt not-exactly-great. It's no better than piping curl to bash, except the potential impact is not that your computer is dead, but the entirety of prod goes down. - Because of this, upgrades of Kubernetes are a pain. The DigitalOcean admin panel will complain about problems in 'our' configs, that aren't actually OUR configs. We don't know how to fix that, or if ignoring the warnings and upgrading will break our production apps. - Upgrades of Kubernetes itself aren't actually zero downtime, and we couldn't figure out how to do that (even after investing a significant amount of research time). - We were using only a tiny subset of the functionality in Kubernetes. Specifically we wanted high-availability application servers (2+ pods in parallel) with zero-downtime deployments, connecting to a DO managed PostgreSQL instance, with a webserver that does SSL-termination in front of it. - Setting up deployments from a GitLab CI/CD pipeline was pretty hard, and it turned out the functionality for managing a Kubernetes cluster from GitLab was not really done with our use case in mind (I think?). - It would be bad enough if DigitalOcean shit the bed, but the biggest problem was that we couldn't reliably recognize if something was a problem caused by us, or by DO. Try explaining that one to your customers. Summarizing: it was just too complex and fragile, even once you wrap your head around what the hell a Pod, a Deployment, an Ingress and Ingress Controller, and all of the other Kubernetes lingo actually means. I suspect you need a dedicated infra person who knows their stuff to make this work, so it could very well make sense for larger companies, but for our situation it was overkill. We were not intellectually in control of this setup, and I do not feel comfortable running production workloads (systems used by 20k high-school students, mission-critical applications used by logistical companies) on something we couldn't quite grasp. We went to a much simpler setup on Fly.io, and have been happy since. It's a shame they seem to be too young of a company to really be super reliable, but I suspect this is only a matter of time. In terms of feature set, it's all we need.
  • dimitar 4y ago
    Well all of those issues are fixable, but I think it is a totally valid reason not to use k8s if you don't have a dedicated infra person/team.
• leetrout 4y ago
  My current company uses it and that is not a bad choice. I prefer nomad which I have used since 2016. I have spent the past couple weeks working with kustomize since I do not like helm and while it gets the job done I think Tanka would be better. We are on GKE which makes things a lot easier and I personally would not choose to run my own cluster. (disclaimer: i worked at hashi for 4 months in 2020 but not related to nomad)
• davidkuennen 4y ago
  No. It's working so well for us. I love it.
• mothsonasloth 4y ago
  At work; no. Personal; I use docker-compose on VMs
• deleted 4y ago
  [deleted]
• 3np 4y ago
  Nomad + Consul(with Consul Connect) + Vault. With Terraform obv. We don't really have a use-case for Boundary but it looks pretty neat as well if you do. Was on k8s for years and I don't miss it one bit. While there definitely is some complexity once you get serious and set everything up properly with raft, federation, Connect, CAs, proxies, ACLs, proper secrets lifecycles... I find it's worth it. With the current assumptions that HC will keep improving and existing bugs and edge-cases will be ironed out.
• mixxit 4y ago
  we have used k8s for about 4 years and are now slowly moving back from k8s to fargate creating a scalable system is complicated within aws account limits all we really want is to shove docker containers behind a load balancer and not worry about having to manage yet another system
  • mrweasel 4y ago
    Fargate just sort of registerede as a “hosted Kubernetes” in my mind, guess not.
    • wlonkly 4y ago
      It can be "hosted serverless Kubernetes" (in the EKS flavour), or "hosted serverless Amazon ECS" (in that flavour).
  • throwaway093483 4y ago
    Fargate runs with EKS or ECS. Does this mean you dropped EKS for ECS?
  • mh- 4y ago
    I'm curious what AWS "account limits" you ran into. I've very rarely come across a quota/limit that wasn't increasable upon request in AWS.
• dijit 4y ago
  Went to nomad, which is working better for my workloads. There's still use-cases where k8s wins; but nomad handles state a bit better and is easier to reason about from scratch.
  • bluehatbrit 4y ago
    I really like the look of nomad and want to give it a go. The two things holding me back are: 1) I don't really want to manage the installation but there aren't any(?) cloud hosts for nomad that I can see. 2) It doesn't seem as widely used so community support seems thin. There aren't many blog posts about good patterns with it etc, and I'd worry that we'd get stuck and end up reverting back to k8s.
    • dradtke 4y ago
      HCP might be what you want, but it doesn't support Nomad yet, and unfortunately it's not clear when it will. https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374 https://discuss.hashicorp.com/t/status-of-hcp-nomad/33374
    • rahen 4y ago
      There is no installation needed with Nomad, it's a standalone binary. Just fire it in a small Debian (or Alma) instance on EC2 or GCE and you're done. That should solve point 1. Point 2 is debatable. Lots of people nowadays put Kubernetes on their resume but that doesn't mean they are great architects or technicians, yet a good part of running production on Kubernetes is doing it right. You'll see much fewer people with Nomad on their resume, but on the other hand you know they're not here for the buzz, they're usually more experienced and know what they're talking about.
  • riadsila 4y ago
    Koyeb also moved off Kubernetes and went with Nomad. We started with Kubernetes, thinking it was the right abstraction layer for us to build our platform, but then quickly ran into major limitations. The big ones: as others have mentioned in this thread, its complexity; security (we wanted to explore using Firecracker on Kubernetes, but it was very experimental at that time); we were not interested in keeping up with its release cycles; global and multi-zone deployments was not as straightforward as we needed; and the overhead (10-25% of RAM) was a cost we were not willing to take (we are around 100MB with our new architecture). We wrote about our decision to switch here: https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-kubernetes-to-nomad-firecracker-and-kuma https://www.koyeb.com/blog/the-koyeb-serverless-engine-from-...
• thdxr 4y ago
  Yes after using it for 5 or so years. Now building fully with serverless
  • ralusek 4y ago
    Do you have your full server code run in a cloud function, and let the function handle routing based on inputs? Or do you have a function per behavior?
• roschdal 4y ago
  Kubernetes is horribly complex.
  • osigurdson 4y ago
    Can you be more specific?
    • standyro 4y ago
      It's often very difficult for developers to test applications in development because of the complexity of setting up `minikube` or `k3s`, so if your small to medium sized company doesn't have a dedicated devops or QA team, it can be too much overhead. I've worked with talented engineers who struggle getting up to speed with networking in k8s because there are a litany of tools, best practices, and terms to learn (nodeport, loadbalancer, and various ingress controllers) and it's heavily dependent on which cloud provider you're using (i.e. GKS , AWS, or Azure) so that adds weeks to dev time. This is a major problem if the team isn't well versed with devops tools (which is often the case at smaller companies) and can lead to lots of issues pushing a lot of work to a devops resource (or team) which requires setting up a separate dev/staging cluster. I think the preferred alternative for companies that struggle with this overhead is to use a slightly more expensive managed service, especially if you're just developing a typical MVC/MV* app.
  • Adiqq 4y ago
    Kubernetes is not really complex at what it tries to achieve, but there might be some scenarios, where you need more tailored tool for the job. Would I use k8s just for static websites or single API? No. Would I use k8s, for rarely updated solution, where low costs are #1 priority? No. Would I use k8s for complex microservice architecture with a long list of ever-growing implicit/explicit requirements and a lot of moving parts? Definitely, because now you just need to either use some built-in k8s feature and/or use CNCF eco-system to supply you with almost anything you need. Kubernetes gives standardization, it's good in enterprise, where high complexity and poor communication is normal. It covers a lot of typical requirements for applications and you can learn a lot about solution just by looking at k8s cluster. However it's a time sink, if you really want to learn more about k8s and CNCF eco-system.
• synthe102 4y ago
  In my lab at home I ditched everything and I know run k8s on my servers with everything managed using ArgoCD and GitOps, best config I had by far. At work, we're currently trying to migrate our stack to k8s. Why ? Because our startup is getting bigger and bigger and our current platform sucks, but our products are becoming a lot more complex as time goes. We benched a few platforms and landed on EKS + ArgoCD + Vault. Works really well.
• 0x202020 4y ago
  I switched jobs recently and became the defacto DevOps person so have been able to deploy mostly how I want. I’ve used kubernetes at multiple jobs, side projects and at home but for a cost and time constrained startup we are leveraging ECS/Lambda/Batch/Cloudfront. B2C application, mostly low traffic with nearly no traffic off hours. Occasionally we’ll get a big rush, 2 to 3 orders of magnitude more traffic than usual, from a marketing push and haven’t ran into any issues yet. I still run KEDA at home for managing plex, home assistant, some game servers and other of my own projects. But being the only one who is using the cluster is a different use case than getting RBAC, ingress and management set up correctly for a production cluster IMO. I’ve never had the sole responsibility or permission over a cluster before, so it was a daunting step I decided not to take for my own sake
• UtahJazz777 4y ago
  If you are using AWS, no one even really needs Kubernetes there. It goes without saying that you can handle pretty much any task and any load with AWS.
  • oneplane 4y ago
    Depends on your workload. AWS is pretty slow and constrained on ECS-EC2 and Fargate. AppRunner is even more limited and more expensive (to the point where we could hire a full time 6 FTE team to run on-prem K8S). AWS has EKS for a reason.
  • llama052 4y ago
    Good luck managing a labyrinth of virtual machines in a cattle friendly way like a Kubernetes cluster can without a bucket of other tooling to invest in.
• stormbrew 4y ago
  I tried using k8s for a personal projects cluster a while back and found it very frustrating to use in a whole bunch of ways, whether managed or not. I ended up just using straight docker swarm and it works fine for that level of need, especially combined with something like portainer. Much simpler and easy to understand what's going on. Obviously it's not a very useful solution on its own beyond a certain scaling point but it probably meets most small use case needs. But it doesn't get brought up as an option very often because docker basically FUDed themselves by having two things called swarm and then loudly killing the older one making everyone think it no longer exists.
  • piaste 4y ago
    +1 for Swarm. It's not for hyperscalers and it's got a limited feature set compared to k8s, but it's simple enough that you can really learn how it works and how to make it do what you want even if it's only a small part of your job. If you just want redundant services, zero-downtime upgrades, and either manual-only scaling or very restricted autoscaling, Swarm is likely sufficient. Main downside? Not available as a managed service, at least from major providers. Then again, if you're OK with managed services, you would probably prefer either a fully-managed PaaS (Heroku, Azure Web Apps, etc.) or a managed k8s.
    • bigjoes 4y ago
      This is kinda the mental-cage I am in right now: For some small amount of containers (300 at most, almost all webserver-like), I would like to have some basic high availability and scheduling on a few nodes. K8S, K3S and even Nomad feels overkill, I tried all of them. Swarm on the other hand is so easy so setup and get running, its seems like the perfect solution. The only thing stopping me is the stigma of Swarm being dead, which is not even the case right now (there is still support but no new features / communication). I feel like starting with swarm right now would be perfectly fine but using a technology which likely may be declared official dead in about 1-2 years, just some how feels wrong. This is my own mental-cage-issue here, right?
      • stormbrew 4y ago
• SantiagoElf 4y ago
  Never used it. Didn't pass my BS test. I am glad that people are moving on to something that will exhaust their creative juices on something ... pointless instead of focusing on delivering value for their customers :) More people using the brand new tech - less competition :)
• derefr 4y ago
  We started with a full-stack k8s approach (on GKE); left (switching to plain GCE VMs); then came back much more conservatively, just using GKE for the stateless business-layer while keeping stateful components on dedicated VMs. Much lower total maintenance burden. (Hard-won bit of experience: k8s + Redis really don't like each-other if Redis 1. is configured to load from disk, and 2. your memory limit for the Redis container is somewhat-tightly bounded. At least from the k8s controller's perspective, Redis apparently uses ~400% of its steady-state memory while reading the AOF tail of an RDB file — getting the container stuck in an OOM-kill loop until you come along and temporarily de-bound its memory.) However, we're considering switching back to k8s for stateful components, with a different approach: allocating single-node node-pools with taints that map 1:1 to each stateful component, effectively making these more like "k8s-managed VMs" than "k8s-managed containers." The point would be to get away from the need to manage the VMs ourselves, giving them over to GKE, while still retaining the assumptions of VM isolation (e.g. not having/needing memory limits, because the single pod is the only tenant of the VM anyway.)
  • jupp0r 4y ago
    Thanks for your insights! One question regarding the approach to hosting stateful components you describe last: what’s the difference between doing this as single node pools vs pod constraints like anti affinity?
    • nijave 4y ago
      It would make node pool operations like version upgrades more predictable since you'd know for sure which apps are running a given node pool It can also make monitoring resource usage a little easier since you can just monitor at the node level
  • paraiuspau 4y ago
    I am wondering, did you ever look at tuning MALLOC_ARENA_MAX? This sort of constant consumption of memory is fairly well aligned to the default tuning of MALLOC_ARENA_MAX, which is 8 * nproc. We've just tuned it for a java-based app which was also stuck in OOMkill hell, and this has completely resolved the situation (MALLOC_ARENA_MAX=2).
• dbingham 4y ago
  Nope, moving to Kubernetes and happily staying there. For context, I ran a DevOps team for the last 4 years that managed two products on AWS - one on EKS and one on ECS. I was mostly managing by the time we had k8s in our stack, so I didn't get to interact with it much directly, but I know infrastructure generally (and I know ECS inside and out, unfortunately). For that infrastructure, we had a whole team managing the ECS deployment. We managed the EKS infrastructure with the equivalent of one DevOp's time or less for years. It was only when it started scaling to millions of users that we needed to give it more time and attention. Both infrastructures (ECS and EKS) were pretty complex with multiple services that needed their own configuration and handling. I left that company a few months back to try to build my own thing and I just finished building out the alpha infrastructure for it on Kubernetes. I can now safely say, as an infrastructure engineer, Kubernetes is an absolute joy to work with compared to lesser abstractions. At least, when someone else is managing the control plane for you. It has exactly the right abstractions, with the right defaults, and it behaves basically exactly as it should. Yes, it's complicated. Yes, there are a lot of moving pieces. Yes, there are hard problems. That's just the reality of software infrastructure. That's not kubernetes, those are just the problems of infrastructure. There's a whole set of problems kubernetes is working to solve in addition to those. Remove kubernetes and you still have those problems, but then you also have the whole set of problems Kubernetes solves as well. I think what's really happening with this whole "Kubernetes is too complicated thing" is that a lot of teams expect to be able to use it like Heroku. That's not what it is. Or they try to build out infrastructures with javascript/php/python/etc engineers. You wouldn't try to have a team of front end engineers build your rest backend. It's not reasonable to expect javascript engineers to know how to build and operate an infrastructure - at least not with out dedicating themselves to learning the tooling and space full time for a while. Think of it from the perspective of a frontend engineer learning Python and Django to build out a rest backend, and then multiply the complexity by 4. That's just infrastructure regardless of what you're using. If you just need to run and scale a container fast and simple, with maybe a single database - then sure the PaaS providers might fit your needs for a while. But eventually the trade off is going to be cost and limitations. You'll eventually need a piece of infrastructure they don't provide. TL;DR Kubernetes isn't the problem here. Infrastructure work is just plain complicated. If you want multiple services, high availability, reliability, scalability, security, and performance, it's just complicated and hard. Don't short change it. Dedicate someone to learning it or hire someone who knows it.
• kmac_ 4y ago
  I've moved to a company that doesn't use Kubernetes at the moment (and that's a 100% calculated and rational decision). What I see, is that a lot of effort is put to provide functionalities that Kubernetes brings. In case of running a bunch of services, when you wish to do that in a stable and secure way, Kubernetes cuts down running costs. It covers so much cross cutting concerns that reimplementation of those capabilities is not possible unless you have heavy $$$ to spend.
  • mountainriver 4y ago
    Same, we are on ecs and there is a lot of reinventing the wheel
    • tootie 4y ago
      Genuinely curious what things k8s solves that you are reinventing. I run ECS and find that using microservices and their managed offerings (ie RDS, SQS) we don't any complicated topography to do complex work.
      • mountainriver 4y ago
        1. Creating confined development environments containing multiple services. With k8s we can spin up a cluster locally, install all the dependencies and develop on it. 2. Remote development. With k8s we can develop right out of the cluster, ECS has no comparative. 3. Installing OSS software. K8s has loads of supported packages for OSS tooling.
  • spmurrayzzz 4y ago
    I think you're right to point out how much ground k8s covers and to replace every vertical that it integrates could be challenging/costly. But k8s is not a zero-cost abstraction, so I think the calculus here is often more nuanced. In the case of my org, we optimized for the features we thought were valuable and amortized that effort over time. Notably this was early in k8s history (2014/2015), but the fruits of those efforts have aged well so far (8 years or so). Small code footprint to cover service discovery, cert provisioning, release orchestration, and configuration management. The whole devops stack is less than 3k SLOC. Service ecosystem is ~150 distributed systems, roughly about 5 million SLOC, running on just over 1k servers on AWS. I think if the aim is not to completely replace what k8s does, but to cherry pick the features that give you some pareto distribution of value, sometimes its worth it to build in-house. Nothing wrong of course with going with k8s for many orgs, but in our case we didn't have to reinvent the whole wheel to live without it.
• ghostoftiber 4y ago
  WellYesButNo.gif The two philosophies at megacorp here seem to be "I built it from the ground up to target X service" where X service is usually amazon serverless or something, and "I built it in docker containers but I don't know about the cloud". The former is a conscious decision, and we (Architecture) have a serious, sit down discussion with them about what it actually means to be fully cloud native for that particular service. This discussion ranges from cost analysis, to things like "is your application actually build correctly to do this", to "you're not going to have access to onprem resources if you do this", even asking them simply "why". A lot of the time when the teams realize they're going to be on the hook for the cost alone they back out, and a lot of teams try to do it because "we don't understand K8s". Well, it doesn't get much better in Cloud Run either folks because you're trading K8s yaml for terraform or cloudformation. Where it has been successful is for teams which own APIs which only get called once a month, or very low traffic APIs. I hate to say it boils down to cost, but a lot of the time it really does boil down to cost. Additionally we've seen a weird boomerang effect as clouds offer K8s clusters which are simply priced per pod rather than per worker node (like GKE Autopilot). A lot of teams which straddled the middle of "low traffic but not low enough to really migrate" have found they're quite happy in GKE Autopilot. They use autoscalers to provide surge protection, but they just use Autopilot with 1 or 2 pods running and it keeps the costs down. That also means we can migrate them to beefier clusters in a heartbeat if they get the Hug of Death or something from HN. ;) The second use case I discussed gets railroaded into our K8s clusters we built ourselves because we can typically get them to use our templates which provide ingresses and service meshes and the developers don't have to think about it too much, and the Devops team is comfortable with the technologies. While it means that there's a bit of "rubber stamping" and potential waste, it's allowed us to use K8s and the nice features it provides without having to invest too much in thinking about it for an individual application.
• lumost 4y ago
  I've never gotten too deep with K8s. It always came across as incredibly complex to maintain with limited managed service support. Whenever I spoke to engineers pushing it, the problems it solved didn't resonate with me as someone whose spent the last 10 years running hundreds of services across thousands of servers. These days I'm a huge fan of CDK and Pipelines style deployments. I prefer to treat my compute layer as a swappable component which I'll change as and when I need to. I tend to lean towards serverless offerings which take care of the internal scaling details if I can while still giving me a traditional "instance", and if I can't then I'll go for the next best managed offering. I've yet to see an example where internal tooling doesn't become a mess over time, and K8S requires a ton of work to keep things sensible.
  • nailer 4y ago
    Yep CDK and/or Pulumi. It’s very easy to map your own custom concepts and logic to your cloud provider, rather than making a cloud provider on top of the cloud provider you already pay for.
• fasteddie31003 4y ago
  My opinion on Kubernetes is that it's great orchestration software that is trying to fix poor underlying application architecture issues. The biggest underlaying software application architecture issue today is the idea of a single responsibility worker. Why do applications have one worker only working on messages from one queue? This architecture looks great on a whiteboard, but has issues around spikes in traffic, uses tons of unused server resources, and requires lots of custom software plumbing. The solution is a generic worker that does work from any queue. It is such an obvious fix to lots of the scaling issue that large software applications face today. I'm personally only using Tempral.io, which is a generic worker orchestrator, from now on when making large distributed applications.
  • throwawaymaths 4y ago
    I think this is due to "needing teams to be independent" which for some orgs is a real thing (Conway's law) but there are small teams trying to build microservices, which, IMO, is a bad idea.
    • fasteddie31003 4y ago
      I don't think the two are mutually exclusive
• alexswensen 4y ago
  I am not sure so much that Kubernetes itself is an issue, as far as the technology. I'm personally a fan of serverless/lambda style functions, but my understanding is that many of those can run on Kubernetes under the hood. Same goes for heroku/digital ocean app services. Even elastic beanstalk. If you are large enough that you need to manage your own k8s cluster, that is one thing, but I would encourage you to look at your needs from a usage and compute perspective long before you start solutionizing with trendy technologies.
• andybak 4y ago
  I'm particulary interested in a variant of this question. My company has clients who usually have very simple requirements. A Python/Django app server and a database. Sometimes there will be another background service or two (memcached or equivalent etc). The most complex site we had was the above but with some Postgres replication clients. We use docker and docker-compose. We've used ansible in the past as well as fabric and other simple solutions. We've had a couple of devs try and convince us that we should be using Kubernetes and I counter with "it's overkill for what we need". Am I wrong?
  • KronisLV 4y ago
    > We've had a couple of devs try and convince us that we should be using Kubernetes and I counter with "it's overkill for what we need". Am I wrong? You're not necessarily wrong, as long as Docker Compose isn't incompatible with what you're trying to do - e.g. if you'd need overlay networking across multiple nodes, or scheduling things across them in one go, then Docker Compose might not be the best fit and you might instead be better served by looking in the direction of Nomad or even Docker Swarm, though the future there is unclear - maintenance mode project, but very similar to Docker Compose and comes out of the box with any Docker install. Either way, Kubernetes might indeed be overkill for simple setups, unless you're using just a subset of its functionality and are running lightweight clusters, like K3s or K0s. I guess some might be pushing it because it's basically become the industry standard, at least in some capacity, in some places, or maybe people just want to put it on their CVs.
  • adra 4y ago
    Back in time, k8s was a glorified docker swarm and swarm was largely compose spread over multiple servers, so if you deploy everything on a single computer and don't have requirements to care about redundancy/failover and all that, then k8s is almost certainly overkill.
  • AtNightWeCode 4y ago
    I would first do calculation of what it would cost to host it on vanilla cloud services. They are often cheaper than what people think, if you include work hours needed.
• kristianpaul 4y ago
  I haven’t tried it yet, the docs looks promising but if I can’t run it on baremetal with official supported docs I won’t bother
• scottLobster 4y ago
  My company (and sector for that matter) is typically 10 years behind the mainstream, so we're just transitioning a giant legacy monolith to kubernetes/micro-services now (literally installing Longhorn today). To be honest, even with the technical overhead it'll probably solve a lot of problems for us from a workflow perspective. We've (the engineers) been arguing for more component-level testing for years (as opposed to the all-up E2E testing we're required do now, which typically turns into component-level testing anyway), and containerizing everything is a good excuse to push it into reality. It'll also make deployments a lot easier (just roll back to X image if there's a problem). Right now we have tens of thousands of lines of hand-written deployment scripts that manage everything and have to be maintained, and intimate knowledge of how they work is often limited to who wrote it (many of whom are no longer with the company), and if there's a problem you have to do surgery on the environment. Kubernetes will give us a unified deployment architecture with problems you can google.
  • miscaccount 4y ago
    From my experience, making downgrade of a single component will not be easier with K8 unless you design for it. Also from my experience, people will start complaining as soon as the new deployment with k8 start failing and they have to fix it. But its a good opportunity to make the transition to more stable architecture. My suggestion is to take it slow and do changes one system at a time. Start with stateless application with less risky deployment and as you learn move others.
• deleted 4y ago
  [deleted]
• iasay 4y ago
  Not yet. We are still deluding ourselves that the 3x cost increment and insane complexity increase we can barely manage to keep spinning is actually a business benefit. Note: this isn't everyone's end game but I suspect it's realistic for a lot of people. I would like to go back to cleanly divided, architected IaaS and ansible. It was fast, extremely reliable, cheaper to run, had a much lower cognitive load and a million less footguns. What's more important possibly is not everything can be wedged into containers cleanly despite the promises.
  • nimbius 4y ago
    you might consider migrating to systemd controlled rootless dockerless podman. helm even has a plugin for podman.
    • iasay 4y ago
      I wouldn't bother. I'd just consolidate our product out of microservices and run more small clusters of monoliths all started from systemd.
    • robbintt 4y ago
      Do you know of some writeups for this? I am like halfway there but just mess with podman+systemd on the weekend.
  • krmboya 4y ago
    Also a big fan of sticking to ansible and plain VMs, at least for most cases I've encountered. To me, a VM in the cloud already feels like a container and you can use the cloud provider's APIs to scale up and down virtual instances as needed
    • bob1029 4y ago
      > To me, a VM in the cloud already feels like a container This is the mental abstraction I've been operating with for over a decade now. All of our products are monolithic binaries that can be installed on bare-ass windows or linux machines. For all intents & purposes, basic AWS/Azure/et. al. VM hosting is our containerization strategy. We just pushed the tricky bits down into our software. 95% of our pain is resolved by using a modern .NET stack and leaning hard on their Self-Contained Deployment model. Our software has zero external dependencies at deploy time, so there isn't much to orchestrate. Anything that talks to a 3rd party system is managed purely via configuration in our software.
• jdoss 4y ago
  Yep! Well, kinda... I still use it at work but for any of my personal stuff at home or for my side projects I use Fedora CoreOS [1] with Butane YAML [2] which I template with Jinja2. Being able to define a VM with Butane and launch it quickly is pretty great. Nothing I am running requires the benefits that Kubernetes can bring to my workloads and the reduced complexity is a breath of fresh air. I am slowly moving towards using Hashicorp's Nomad running on Fedora CoreOS using the Podman and QEMU drivers. I rolled out a Nomad at work for internal projects and it let's me get things done quickly without living in a total YAML hellscape. 1: https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/ https://docs.fedoraproject.org/en-US/fedora-coreos/getting-s... 2: https://coreos.github.io/butane/examples/ https://coreos.github.io/butane/examples/
• benfrancom 4y ago
  I migrated a company from k8s to ECS/Fargate in 2019. Kubernetes is very flexible, but I opted for simplicity. The result of the migration was that there is little underlying infrastructure to maintain, and ongoing operational costs were lowered by 50% year over year. The CTO and I liked the setup so much, we started converting another large client of theirs. I followed up with them at the beginning of 2022 to see how things were going, and they still love it. There is so little maintenance, and now they have more time to focus on what they do best–Software! Other options on the horizon that I'm testing include utilizing AWS Copilot with ECS/Fargate, and/or Copilot with Amazon App Runner.
  • cies 4y ago
    Chose Fargate over K8 too. I made the call, so no need for migrations :)
  • mr337 4y ago
    I have settled on the ECS camp as well. Took a run at Kubernetes and was blown away by the complexity. With ECS/Fargate I don't spend any time on it. It just works for our setup. I still wonder from time to time if I am missing something not going Kubernetes.
    • adra 4y ago
      Are you big enough to need terraform? If the answer is yes, you may have a good justification to move to kubernetes migrate tf->k8s with lots of benefits for the app teams (if they care). If you're just yolo setup your cloud in AWS web console and you're fine with that, then you may not see much lift. A good reason to use declarative (often infrastructure as code) approach to deployments is that it improves bus factor and the ability to hire people who can pick up and maintain the infrastructure.
      • bernf 4y ago
        AWS CDK exists and IMO is way better than terraform if you're on AWS. So much so that terraform is making their own variant to be more CDK like.
• vincentdm 4y ago
  We adopted it in 2017 and got rid of it in 2021. It introduced a lot of complexity, while still leaving a lot of issues up to us to figure out. E.g. deployment strategies. Also: our main reason to adopt Kubernetes was to stay cloud-agnostic, but we soon realized that this is as unrealistic as writing a complex app's SQL in a vendor-independent way. Instead, we decided to embrace our cloud (AWS) by using their CDK tooling and leveraging their features as much as possible. If we ever need to switch to another cloud we will bear the cost then, but for now it is clearly YAGNI.
• jialutu 4y ago
  No, and I wouldn't, since I absolutely love it. I've put our entire build pipeline and everything into one single cluster at the moment, and been finding it incredibly straight-forward and easy to build our CI/CD pipelines using it. Do I recommend Kubernetes to other people/companies though? Absolutely not! The learning curve is incredibly steep, and it really does take investment into understanding how it works. But to anyone who is looking to use Kubernetes, I highly recommend https://helm.sh https://helm.sh since it actually makes templating deployments significantly easier.
• BilalBudhani 4y ago
  Luckily I never got picked up K8s, I found Docker Swarm simple and easy to use.
  • macspoofing 4y ago
    It's a little sad about Docker Swarm because for a certain critical mass of nodes, Docker Swarm is actually quite nice because of lower overhead and complexity. The problem is that Docker Swarm seems to be a dead project now.
• causi 4y ago
  I only ever see Kubernetes mentioned on HN in two contexts: "Kubernetes was ruining my business" and "Kubernetes saved my business".
  • Cyberdog 4y ago
    Well, "Kubernetes didn't really have much effect on my business" doesn't really make for an interesting or memorable post.
• hestefisk 4y ago
  Yes. I manage hosting for a sizeable online business. I thought it would be more efficient to run our containers on Kubernetes, but it’s very complex. Now I run all production workloads on FreeBSD jails managed with iocage. It works very well for our needs and with less overhead.
• stickfigure 4y ago
  Our story involves moving onto k8s, then moving off it. We run most of our app on Google App Engine explicitly to avoid devops work. However, we have a stateless-but-memory-hungry image manipulation service that was just too expensive on GAE. We migrated that service to k8s on Digital Ocean. It was a disaster. I mean, it worked, but suddenly we were spending a lot of time learning k8s and fussing with k8s and it slowed down feature development. K8s is a time sink. So we migrated the service to Digital Ocean App Platform and velocity returned to normal. I'm not wholly thrilled with DO App Platform. It has some maturity issues, and while it's cheaper than GAE, RAM is still more expensive than Elastic Beanstalk (which charges you more or less the EC2 VM cost). So we'll probably move it there someday.
  • neurostimulant 4y ago
    If RAM cost is an issue, why not renting dedicated servers? You can rent a dedicated server with 256gb RAM for less than $400/mo on various low cost providers such as Hetzner and OVH.
    • stickfigure 4y ago
      We don't need anywhere near that much RAM per instance, and we have somewhat bursty traffic. Cloud is a decent fit. I'm willing to pay for the convenience, but that doesn't mean I won't cost-optimize.
• dzikimarian 4y ago
  No. It's really good. We have about 100 devs in multiple teams. Kubernetes provides great level of standardization and transparency - completely different experience than VMs, where admin team had too much ability to cut corners and build technology debt. People would riot if they had to go back to these days. A few warnings: * It takes some resources. Maybe can be mitigated with k3s or similar, but I don't have first hand knowledge here. * It requires some time to learn and configure properly. If your entire team is 3 people and you are on limited budget, probably not a good idea. * Adopt some tools (helm?), standardize deployments, where possible. Bare k8s is bit too much for daily work. * Read good practices and don't try to be smarter, at least until you really know what you are doing. Limit misconfiguration may really burn you at least convenient moment.
• geek_at 4y ago
  I switched back to Docker swarm manages by portainer. So much easier and all I really needed is one node taking over when another dies
• andyshi 4y ago
  The second question is my question as well. Because K8s has failed to deliver as a true "platform". i.e Application teams still have to care about infrastructure. That statement is true even for the "managed K8s" by cloud providers. But what's the alternative? We are stuck.
  • auspex 4y ago
    Fargate
• owlbynight 4y ago
  I've been dragging me feet on implementing k8s, suspecting that its complexity would eventually be reduced by its evolution. And then a couple weeks ago, I was tasked with standing up a new Ansible AWX server, which now done via a k8s operator. It was an exquisitely painful experience. This is potentially a bad example because I'm pretty sure IBM's plan with AWX is now to make me suffer, but through that entire process, k8s just felt like extreme overkill. I'm pretty sure that's going to be the last time I use k8s. I know it makes sense for some use cases, but I it just doesn't feel intuitive in any way. And although it may seem more efficient, I absolutely dread having to troubleshoot any problems down the road. I'm probably not the target audience, but thought I'd leave a comment for fun anyway.
  • volume 4y ago
    Can you share more about your awx-operator and/or k78s pains? I was tinkering with it this weekend and I'd like to compare notes. I got the basic install via kustomize working just now, because I got stuck with the helm-based method.
• vlunkr 4y ago
  We switched to it in the last year. It's been good for us, but it really depends on your use case. We deploy lots of different applications with different scaling requirements, so it's a great fit. If you're deploying one app without any unusual deployment requirements, it's probably overkill. Certainly there is a learning curve, but once you've got it down it's easy to throw new things into the mix. People talk about it being incredibly complex, and honestly I don't see it. Yeah there's a layer of jargon you have to dive into, but it all makes sense once you start building something with it. By far the most complex pieces for us are the integration points with AWS (we're using EKS.) The examples/docs available are just not that great.
• 0xbadcafebee 4y ago
  I would love to. But what I hate about K8s is how you can't not use it. It's like Jenkins. A total piece of shit, slow, buggy, insecure, maintenance headache, expensive to maintain, never works the way you want without a ton of work, lots of footguns, bad practice is the default. But try explaining to management how you don't want to use Jenkins and they'll just come back with "but it's free" and "everyone uses it" and "no vendor lock-in". They don't understand that they're asking you to become a Ferrari mechanic when you really need a Ford F-350 pick-up.
• woopwoop24 4y ago
  what i see as a consultant and i am with k8s since v1.2 that companies try to get from 0 to 100 and then wonder why the fail. so you are going from java 1.7 on jboss with a tightly coupled monolith to microservices on k8s with docker. All the nice things k8s can provide goes hand in hand with the ability to work together. cert provisioning, network infra, storage, db offers etc.. there is so much k8s needs to succeed and all the teams have to work hand in hand which is what the companies brutally underestimate. i would stick no matter the company size on IAAS + $Deploymenttool (ansible or so) and docker and then get comfortable with and only then, when everything works as intended make the switch to k8s.
• kokizzu2 4y ago
  http://kokizzu.blogspot.com/2021/08/you-dont-need-kubernetes.html http://kokizzu.blogspot.com/2021/08/you-dont-need-kubernetes...
• unixhero 4y ago
  Any love for LXC here?
  • p_l 4y ago
    LXC is nice, but different scope.
    • unixhero 4y ago
      Okay, good point.
• didip 4y ago
  Why would I leave Kubernetes? It’s the best thing since Microwave and sure as heck way better than Mesos.
• davewritescode 4y ago
  No, in fact we've gone running towards it after some initial success, especially when combined with ArgoCD for CD and Istio as a service mesh. My company has a lot of experience with running applications on VMs and Amazon's ECS. Our VM automation ultimately became expensive to maintain and ECS had its own set of issues I could probably fill up a blog post with. From the Operations side, Kubernetes is scary. It's easy to screw things up and you can definitely run into problems. I understand why folks who work mostly on that side of the house are put off by the complexity of Kubernetes. However, from the application side of things, our developers have been THRILLED with Kubernetes. For most developers my company provides a nice paved road experience with minimal customization required. For advanced use cases, we allow developers to use the Kubernetes API (along ArgoCD + GateKeeper policies) as a break glass type of approach. Istio gives the infra team the ability to easily move services between clusters and make policy changes easily. It also allows us to make use of Knative, although I think the Istio requirement is no longer there. That said, you should be using managed Kubernetes wherever possible and not running your own clusters. That's where trouble lurks.
  • therealdrag0 4y ago
    Same story for us. We’ve been moving towards k8s and it’s been great for app devs. We ran in plain VMs for a decade and it was a good time to switch at 2k employees, maybe 500 devs?
  • clutchdude 4y ago
    ArgoCD was our missing lynch pin for getting workloads migrated over and supported. It makes it that much easier to actually use the cluster rather than mess with endless configuration tooling. Is it the best engineered tool? Probably not. But it's the one that works best for us.
  • nazka 4y ago
    I’m curious, do you use Vault, Datadog, or some Falco maybe? What is the rest of your Infra stack?
• tomphoolery 4y ago
  Kinda? We use Cloud Run because for our workloads GKE was a lot more expensive. So far, it's been great. I wouldn't say I've "left" Kubernetes, since from what I understand Cloud Run implements the Knative standard, which is itself built on Kubernetes. But much like it was predicted early on, I think Kubernetes is best used as a means of building an infrastructure platform, not an infrastructure platform in and of itself. You certainly can cobble all this stuff together and build a nice system, but it takes a lot of work, and there's probably a hosting company out there which already does something similar enough that you can adopt. With this approach to hosting and deployment, I think Kubernetes' main advantage is that it opens the door to new kinds of infrastructure businesses, not that it makes hosting a website any easier.
  • steren 4y ago
    Note that Cloud Run is not built on Kubernetes, but on Borg. It implements the Knative Serving API spec, mainly for portability reason with Knative and Kubernertes. Source: I'm the Cloud Run PM and we have commmunicated about that publicly in the past.
    • seabrookmx 4y ago
      TIL! Do you have any Google docs or blog posts that talk about this? I always wondered why you need a Serverless VPC connector for "vanilla" Cloud Run (or you have to use Cloud Run on GKE) to access VPC resources, but I suppose this answers that question.
  • seabrookmx 4y ago
    +1 for Cloud Run. I've tried many of the serverless platforms and maybe it's the types of applications I work on, but I've found most of their limitations (short runtime, limited access to resources on your private network) basically make them useless. The more self-hosted types that don't have these limitations lose out on many of the benefits or are leaky abstractions on k8s. Cloud Run has all the benefits I want: extremely easy deployment and scaling, as well as the ability to scale to zero if you need it (though generally you don't), while still being able to run basically whatever workload I want. My current employer is mostly a Python shop but we recently deployed a little .NET core service on Cloud Run and it's been awesome.
• CommanderData 4y ago
  No. I do feel like k8s will be superceded with vendor specific offerings though. Even with a CKA I think there's just too much overhead with what is fundamentally /usually compute at scale.
• rekrsiv 4y ago
  I've yet to encounter a non-smelly k8s deployment that was started before everyone knew how it works or why it works. On the other hand, once everyone on the team has experience building such a system from scratch, then deploying k8s and using it somehow becomes straightforward. It's almost as if we need to learn how a tool works before being able to use it effectively. Anyways, what we (actually didn't) replace it with: - Don't let your devs learn about k8s on the job. - Let them run side-projects on your internal cluster. - Give them a small allowance to run their stuff on your network and learn how to do that safely. - Give your devs time to code review each other's internally-hosted side-projects-that-use-k8s. - Reap the benefits of a team that has learnt the ins and out of k8s without messing up your products.
  • deleted 4y ago
    [deleted]
  • meldyr 4y ago
    What do you mean with side projects? Are they paid? If you want your Devs to learn kubernetes you should pay them for doing it. If you can't, hire a contractor with the expertise you need.
    • lc9er 4y ago
      Running a side project on company infrastructure seems like a disaster waiting to happen for both parties.
      • rekrsiv 4y ago
        It's true that running side projects on a company cluster in an environment where no one is quite sure how to use the tool properly is a disaster waiting to happen. Fortunately, k8s can act as a very secure sandbox when it's configured properly, so you'll know how to mitigate such a disaster once your company has trained its engineers on how to use the tool effectively.
    • dan_quixote 4y ago
      I took it to mean "sandbox"
• kodah 4y ago
  I use remote Docker contexts with my home cloud now instead of Kubernetes. I'm actually looking at moving to Docker Swarm. As a single dev mostly satisfying my own needs, it's pretty much all I need. Happy to answer questions.
  • tommica 4y ago
    I've also looked at docker swarm mode, but as a noob in these things, its the persistent database that makes me nervous - how do you do it? And do you have any tips or tricks?
    • kodah 4y ago
      I keep it simple by scheduling services to particular nodes: https://docs.docker.com/engine/swarm/services/#control-service-placement https://docs.docker.com/engine/swarm/services/#control-servi...
• mvaliente2001 4y ago
  No. In my current company k8s + helm + istio + argocd + good support from SRE/infra team has made things pleasurable. Seniors introduce the complexities of the system to juniors in a controlled paced way. One thing I would change is to replace terraform for crossplane.
• sebiw 4y ago
  Solo creator. Never went there. Don't miss anything. :-)
• g105b 4y ago
  Personally, I never joined. I have tried, I really have. I spent so long trying to move my business over, because on paper it's a developer's dream. Everything in config files. Perfection! But in reality, I think I developed an allergic reaction to complexity and hype. I took some metrics; things like recording the time taken, steps taken and happiness generated from my current build/release stages, then comparing to k8s. In conclusion, struggling to learn k8s forced me to find joy in the simplicity - knowing that one day (that will never come), I can just hire someone to do this... "It's only a problem when it's a problem". For now, I have a lovely bash script that is triggered on Github releases (using Actions), which uses doctl to do the following: 1) Create a new server from my baseline image 2) Run the setup steps as defined in the Dockerfile, although it doesn't use docker (it just makes sense to keep the configuration I used to have) 3) Copy the built-and-tested version of the repository to the new server 4) Run any post deployment scripts, like database migrations, whatever 5) Move the reserved IP to the new server It takes about a minute from me clicking "new release" in Github to seeing the changes hit production. If there's a problem, I move the reserved IP back. Load balancers, database clusters, etc... they're all set up manually because "it's only a problem when it's a problem". Kuberneeties only ever generated problems for me.
• llama052 4y ago
  Hell no, I remember managing hundreds of virtual machines in datacenters & cloud, using Ansible and a myriad of other tooling. It's nice when you're at a small scale and you don't have a lot of people making changes, but over time as it grows the pain grows with it unless you've enforced a consistent cattle model. The longer VMs live with custom changes/code and updates over time the more brittle they can become. Part of the cattle model is so that you can recreate/rebuild when changing code so things stay consistent. The drift from infrastructure as code can be scary otherwise. With the cattle model you need to have pipelines in place to build new VM images for infrastructure updates (packer etc), have multiple APIs to hit (easier in cloud) to upload images and serve them in a non damaging way. (HA deployments/rollouts/dealing with load balancers) It's certainly a non-trivial amount of work. With Kubernetes, a lot of this tooling comes out of the box. You've got autoscaling, load balancing, health-checks, limits/requests, failure mitigation, service mesh options. On top of that it's served in a strict semi-consistent way. Good luck replicating that with virtual machines without a lot of tooling and effort. If you can learn the Kubernetes tooling it can do a lot for you. However I agree that not all setups need it, a lot of times small setups never grow and that's ok a few virtual machines aren't that big of a deal. We still use virtual machines for workloads that aren't container friendly, and to be honest these days I abhor it, even with pipelines in place.
  • Already__Taken 4y ago
    > The longer VMs live with custom changes/code and updates over time the more brittle they can become. Honestly kubernetes is not harder than dealing with this. It's keeping you back in the land of default google-able problems longer as weird tweaks and unique configs aren't piling up to make esoteric issues.
• jFriedensreich 4y ago
  we migrated all erlang, golang and nodejs code to edge workers on cloudflare and simply did not need to run containers anymore so there was also no need for k8. i would say this reduced the operational complexity by 2 orders of magnitude.
  • kentonv 4y ago
    Do you mean that you got erlang and golang code running in Wasm under Workers, or did you convert the logic to JavaScript?
    • jFriedensreich 4y ago
      No we rewrote all golang and erlang code in javascript. The erlang and golang runtime and concurrency primitives do not really matter for most use cases when you have unlimited edge functions running wherever users are. On top of that hiring is just so much simpler when you have a single language everywhere.
• ancieque 4y ago
  Story of one of the projects I am involved in: We came from Ansible managed deployments of vanilla docker with nginx as single node ingress with another load balancer on top of that. Worked fine, but HA for containers that are only allowed to exist once in the stack was one thing that caused us headaches. Then, we had a workshop for Rancher RKE. Looked promising at the start, but operating it became a headache as we didn't have enough people in the project team to maintain it. Certificates expiring was an issue and the fact that you actually kinda had to baby-sit the cluster was a turn off. We killed the switch to kubernetes and moved back to Ansible + nginX + docker. In the meantime we were toying around with Docker Swarm for smaller scale deployments and inhouse infrastructure. We didn't find anything to not like and are currently moving into that direction. How we do things in Swarm: 1. Monitoring using an updated Swarmprom stack (https://github.com/neuroforgede/swarmsible/tree/master/environments/test/test-swarm/stacks/02_monitoring https://github.com/neuroforgede/swarmsible/tree/master/envir...) 2. Graphical Insights into the Cluster / Debugging -> Portainer 3. Ingress: Treafik together with tecnativa/docker-socket-proxy so that traefik does not have to run on the managers 4. Container Autoscaling: did not need it yet for our internal installations as well as our customer deployments on bare metal, but we would go for a solution based on prometheus metrics, similar to https://github.com/UnclePhil/ascaler https://github.com/UnclePhil/ascaler 5. Hardware Autoscaling: We would build a custom script for this based on prometheus that automatically orders servers of Hetzner using their hcloud-cli 6. Volumes: Hetzner Cloud Plugin, see https://github.com/costela/docker-volume-hetzner https://github.com/costela/docker-volume-hetzner - Looking forward to CSI support though. 7. Load Balancer + SSL: in front of the Swarm using our Cloud Provider Reasons that we would dabble in k8s again: 1. A lot of projects are k8s only (see OpenFaaS for example) 2. Finer grained control for User permissions 3. Service Mesh to introduce service accounts without requiring to go through a custom proxy
• julienchastang 4y ago
  I've been a K8s user for some time, but it does drive me bat shit crazy. My main beef with it is I often cannot discern the logic of how things work. For the developer platforms and systems I enjoy working with, you are presented with primitive axioms that you can then bootstrap your knowledge upon to derive more complex ideas (e.g., any decent programming language, or OS). K8s does not work that way -- at least as far as I can tell. A priori knowledge gains you nothing. When I run into a problem on K8s, I copy/paste the error into a search engine and I am presented with a 200 message long GitHub issue with users presenting their various solutions (how does this command relate to my original problem, who knows?), some work, but most of the time, they don't and you are left in a bigger hole than when you started. I end up tearing the whole things down and starting over, most of the time. That last comment is the biggest "code smell" for me with K8s. When it is easier just to nuke the thing and begin again, there is a problem.
  • p_l 4y ago
    I'll put blame on bad documentation and tutorials becoming the norm for k8s versus what was common early on, because k8s is very much about building more complex ideas from primitive axioms. The whole resource model is built around simple ideas being used to build more complex ideas. Wish there was some better docs out there, not sure if I could handle writing one from scratch :/
• mikestaszel 4y ago
  Approaching the 1 year mark of running my team's various data pipelines on k8s. Keeping up to date with k8s/EKS version lifecycles has been more work than expected. No plans to stop using it anytime soon.
• deleted 4y ago
  [deleted]
• standyro 4y ago
  Yes, the last two companies I've worked for migrated to AWS Fargate and Serverless/Lambda. There are some advantages to using k8s when you have large stateless applications that need to scale, but it requires well thought out patterns for things like caching and involvement with the dev team from the beginning. Most small to medium size companies get no real benefit from Kubernetes as it introduces a lot of devops overhead and rot (examples: the companies that chose to use Skaffold instead of Helm, or customized deploy scripts that don't make sense to developers and aren't integrated into a CI/CD pipeline)
• sdevonoes 4y ago
  - Terraform to spin up VMs in the cloud (e.g., "give me a Ubuntu machine with 4GB of ram") - Ansible to provision such VMs - Docker to start/stop containers on such VMs It feels like a breeze of fresh air!
• unity1001 4y ago
  Hell no. Im a big convert. It solves a !lot! of problems.
• majodev 4y ago
  No and we are happily using it within our overcommitted cluster (combination of shared and dedicated nodepools). We are a small team of 5 infrastructure engineers and previously managed 200+ libvirt VMs running on bare-metal HA hypervisors in a GlusterFS storage pool (software agency, different customer application services). We started to migrate to GKE in 2017 and finished within a year or so. I know many associate k8s with a yaml mess, but this is actually our most favourite part of it. We are able to describe a whole customer project in this format and it's not something we have to maintain in-house (Ansible). As long as you don't try to be smart (templating/helm, operator dependance), it works out pretty well, prefer plain manifests and extend that with you own validation scripts. Nevertheless, if you have no 24/7 operations, stay the hell away from bare-metal - go managed.
• exabrial 4y ago
  We never started because we realized how silly it was. A clustering solution is a great thesis, but nobody has "done it right" yet. K8s was over-designed from the get-go and missed what actually causes issues in scaling. We feel the same about Docker. People have no idea what's running when they download a docker image. Stuff can be buried deep deep within an operating system image. Security should be simple, transparent, and minimal so it can be reviewed easily. Reviewing a docker image is impossible. I'm convinced the correct place for isolation is systemd. This guy wrote a great starter for hardening the crap out of your services: https://docs.arbitrary.ch/security/systemd.html https://docs.arbitrary.ch/security/systemd.html Systemd offers a bridge too with nspawn if you're not ready to undertake ultra minimal hardening of services. Scaling is a "sexy" problem to have though, and software "engineers" love to think that their SAAS product with 100 users is going to take Google scale workloads; thusly what could be done in a LAMP stack on a single DO server, is inflated into a fantasy that will never come to fruition.
  • wlonkly 4y ago
    > People have no idea what's running when they download a docker image. This is true of software packages and especially of third-party libraries; supply chain attacks are supply chain attacks. But similarly, supply chain controls are supply chain controls, and using Docker does not mean running someone else's container. (For example, we build our own hardened base images, and on those we install our own services, and the result is precisely as trusted as building our own hardened AMI and installing our services on that.)
• efxhoy 4y ago
  We run a few largish rails apps. Main rails containers, nginx, sidekiq and almost everything in ECS. RDS for postgres. Elasticache redis. Opensearch for ES. We tried moving to k8s but found the current setup simpler. Our terraform state is a huge blob though and if we started from scratch i think we would have defaulted to EKS for more things.
• rcarmo 4y ago
  I hacked together https://github.com/piku https://github.com/piku and run all my personal projects using it. Haven’t looked back in years, although I do use k8s at work.
• ParadisoShlee 4y ago
  Yes, Kube is a mistake for many. I'm a consultant who specialises in Kube, but actually spends most of my time decommissioning it and migrating to other solutions.
• fierro 4y ago
  a lot of these "just use bare VMs" arguments don't address how you do service discovery, load balancing, networking, rolling deployments, etc. A lot of the benefit my team gets from K8s is from these abstractions, not anything to do with "scale" per se
• motbus3 4y ago
  in our case we integrated k8s in our dev pipeline and all devs need to do to release a app in merge pr to main branch. it all scales and works fine. there is one or two problems around but not enough for me to consider it does not work.
• horsawlarway 4y ago
  No. Nor do we plan to. Honestly - I even use it personally for my self-hosted stuff at this point. The learning curve is... steep. But once you come out the other side, it's a great tool.
• sweaver 4y ago
  This: https://github.com/pluralsh/plural https://github.com/pluralsh/plural