12 ms·

The Future of Kubernetes

147 points by kiyanwang 5y ago

KronisLV 5y ago
I've actually written about why i'd personally look towards Docker Swarm for smaller projects and setups in my blog post "Docker Swarm over Kubernetes": https://blog.kronis.dev/articles/docker-swarm-over-kubernetes https://blog.kronis.dev/articles/docker-swarm-over-kubernete... (though it is a pretty brief stream of consciousness) Either that or Hashicorp Nomad, because it is also a pretty nice solution and seems to overall have a brighter future than Swarm does, even if the HCL which it uses is a little bit weird and needs porting over from Docker Compose files. Admittedly, in the last year, i've also seen Kubernetes be tolerable for on-prem deployments in the form of K3s, which is a lovely project and wastes way less memory and CPU for smaller clusters: https://k3s.io/ https://k3s.io/ I know that many out there won't necessarily have to or won't want to run it on-prem and instead will just run whatever Kubernetes solution will be offered to them by their cloud vendor, which is also a valid point, but not one that i have to deal with. Instead, i often find myself with a VM that has about 8 GB of RAM and needs to run everything from the actual cluster, to PostgreSQL, RabbitMQ, Redis, a few Java apps and who knows what else inside of a development environment, because clearly that was possible without containers so also should be possible with them. There, running something like Rancher or even a K8s cluster that's made from scratch (or initialized with kubeadm or whatever) would be insanity. And having a shared cluster for the whole company might also necessitate people to just manage it, which isn't in the cards. Also, in my personal hybrid cloud (homelab + cloud VPSes), running Kubernetes would be comparatively expensive and wasteful, since Docker Swarm still has a lower overhead and Swarm provides me with most of the functionality that i might want, oftentimes in the form of regular containers, e.g. Caddy/Nginx/httpd web server as an ingress with some configuration files. Actually, anyone who has ever tried to get the K3s Traefik ingress working with a custom wildcard certificate will probably understand why i might actually prefer to do that, since in K3s you'll need the following for such a setup to work: - a ConfigMap for Traefik, knowledge about the structure of the ConfigMap (tls.stores.default.defaultCertificate) - a TLSSecret for storing the actual certificate/key - a TLSStore (which i also needed to actually use the secret, spec.defaultCertificate.secretName) - a HelmChartConfig for Traefik to load the ConfigMap with the mounted secrets and config All of those weren't sufficiently documented because apparently the popular usecase was to use Let's Encrypt with either cert-manager or another solution altogether. All of it took digging through GitHub to find out and trial and error to configure it all. Versus just reading a page for a web server that i want to run and dropping some values in a config file.
pjmlp 5y ago
Basically, Kubernetes as Cloud OS regardless of what kind of hypervisor powers the whole stack. The irony of UNIX having won, only to be replaced by System 360 roadmap.
- hestefisk 5y ago
  Not sure I understand your second sentence. Are you equating Kubernetes’ roadmap to that of System 360? FWIW, I am not a big fan of Kubernetes. It’s an incredibly complex beast. I much prefer the simplicity of *BSD over Kubernetes any day.
  - pjmlp 5y ago
    Kind of, Kubernetes is just another way of doing mainframe workloads, whose main concepts all trace back to System 360. 20 years ago I was deploying EAR files into application servers without caring which OS they would run on. Nowadays I package managed runtimes into containers, that I don't care if they run directly on top of a type 1 hypervisor, VM or whatever. Basically what mainframe OS virtualization looks like nowadays and traces its roots back to System 360.
    - hestefisk 5y ago
      Good point. Kubernetes actually reminds me a lot of J2EE around version 2.0. Towering abstractions and complexity.
- wereHamster 5y ago
  Do you see System 360 and Kubernetes as parallel technologies?
  - pjmlp 5y ago
    I see Kubernetes + type 1 hypervisors replacing UNIX, thus achieving the same kind of OS virtualization done on IBM mainframes that trace their roots back to System 360.
jitl 5y ago
It’s funny to see this pitched as the “future” when we were doing most of these shenanigans during Airbnb’s Kubernetes rollout in 2018. The only thing that’s futuristic from the perspective of 2018 Airbnb infra is Knative. Knative sounds like a good idea — especially for control-plane services or devtools where scaling to zero is something that would actually happen. The commentary around PersistedVolumes is true (don’t write user data as json pls) but simple. We had a few use-cases for big persisted volumes or even host volumes managed by a daemonset, where object store performance didn’t cut it. One was indexes for a search service. You can’t push these 64gb hunks of data into S3 without paying a ton, much better to use a file system. Same thing with translation string data, a PV with a local SQLite database or other indexed read-optimized format is 100x faster and 100x less expensive than “object storage”. We did enforce that the PV was always read-only for the “main” container in a pod, so you needed to add a sidecar or daemonset if you wanted to sync data there.
l0rn 5y ago
Mh... As far as I understand this article is promoting the fact to factor in kubernetes or the cloud platform directly into the orchestrator system (i.e. k8s). I think this won't happen. I think k8s was successful because it allows to run unix-like applications with little adoption effort of the application itself.
leetrout 5y ago
""" Generally, the Kubernetes we know today will disappear, and developers will not care """ I truly cannot wait. I love the premise but hate the complexity. Nomad stole my mindshare in 2016 and I would still choose it over k8s today for small applications. Even choosing it for managing VMs in some cases. (Disclaimer: worked at hashi but not on nomad nor did i use it while there).
ThePhysicist 5y ago
A difference between the Linux movement and the Kubernetes ecosystem is that most of the open-source tooling developed for K8s is built by venture-funded startups (e.g. Istio or Envoy). And the endgame of these companies is almost always an open-core product that is functional but in practice just broken enough to not be usable without buying premium support or the enterprise edition. So I'm a bit skeptical about the long-term viability of the platform. Personally I think we'll see new paradigms emerge that will mostly obsolete containers and complex orchestration platforms, you can e.g. already see some pretty interesting ideas around WASM. Before Kubernetes OpenStack was the big thing, but now it's mostly used as a substrate to run Kubernetes on.
- rdli 5y ago
  Istio was started by Google & IBM, while Envoy was created by Matt Klein at Lyft. One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. As a note, technically Google & Lyft are VC-funded, so that’s true, although I wouldn’t put Google in the startup category, and Lyft’s business model is unrelated to Envoy.
  - ThePhysicist 5y ago
    I was under the impression that Istio and Envoy are significantly driven by Tetrate (https://www.tetrate.io/ https://www.tetrate.io/), which in my understanding was started by the Istio creators and sells enterprise editions of the two solutions [1]. So yeah it's open source, I can fork it and that's nice, but let's be honest Tetrate aims to monetize these products. Not saying this can't work and be great for the community at the same time, but I'm at least a bit skeptical. 1: https://www.businesswire.com/news/home/20210310005295/en/Tetrate-Started-by-Istio-Founders-Raises-40-Million-to-Help-Enterprises-With-Cloud-Native-Application-Networking-Platform https://www.businesswire.com/news/home/20210310005295/en/Tet...
    - bassarid 5y ago
      I've been running Istio for years and have never heard of Tetrate. Tetrate might be started by people who originally worked on Istio, but Istio itself is primarily driven by Google. Istio development is primarily driven by its steering committee [0][1], which is led by Google. If you look at the contributors [2], 6 of the top 10 are Google employees (perhaps more - not everyone lists their employer on GitHub). They have an open design process, posting design documents and similar to their Google Drive [3]. The team is likewise active at community events, where they solicit input that drives what they work on. [0]: https://istio.io/latest/blog/2020/steering-changes/ https://istio.io/latest/blog/2020/steering-changes/ [1]: https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqcmdWXDuCaPC0kPS21BuAfCL8/edit?usp=sharing https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqc... [2]: https://github.com/istio/istio/graphs/contributors https://github.com/istio/istio/graphs/contributors [3]: https://istio.io/latest/get-involved/ https://istio.io/latest/get-involved/
tkiolp4 5y ago
K8s will be superseded by a simpler yet more powerful tool sooner rather than later. Just like it happened with: - Corba, MRI -> http apis - XML, Soap -> json - Flash -> html5 - Perforce, svn, mercurial -> git Etc. We can’t think of anything better than k8s right now, but that’s just how it works. The moment that new tech appears, we all will be asking ourselves “how could we possibly use k8s?”
- hutrdvnj 5y ago
  This is not always the case. Sometimes we use a technology for decades and just upgrade its version. Like HTML, HTTP, GCC, ... the list goes on. It's very possible that we still use k8s as major container orchestrator in a decade, just in a higher version.
  - xg15 5y ago
    I think a better way to think about it is in separating API and implementation or wire format and semantics. In terms of the wire-format and the low-level concepts, HTTP2 and 3 are completely different from HTTP1.x. it's not an extension like HTTP1.1 was, it's a complete reinvention. However, what they did was to reuse the high-level semantics and the "API" of HTTP1, so as a web developer you can pretty much pretend you're still using HTTP1 and let the browser and web server do the rest. This makes transition significantly easier. I could imagine something similar with K9s. Maybe some other product will come along that supports (a subset of) the k9s API but does something different under the hood. I believe, k9s itself did so with Docker: You can still define k9s images using Dockerfiles, even though no piece of Docker is actually involved in the process.
- radicalbyte 5y ago
  I find this amusing considering that the JSON community are slowly re-inventing XML with their versions of XSL, XSLT etc.
  - tkiolp4 5y ago
    JSON is already 20 years old. I guess things can’t stay the way they are forever (and 20 years in the IT world is a lot).
livinginfear 5y ago
I've managed to avoid working with Kubernetes thus far in my career. I admit I don't have a great understanding of it. I can't think of a single technology that seems to unanimously stir up such bitter feelings in the developer community. Not even node.js, or webpack is as contentious as k8s.
- leetrout 5y ago
  You are both missing nothing and missing everything. Haha. Try to hold out for the next generation! It will surely arrive soon? Do you use any container orchestrators today?
- jitl 5y ago
  Systemd stirs up similar angry because it was built for similar reasons to become this god-management-layer that’s useful for getting shit done but also hugely more complex (…in implementation at least) than the status quo it replaced. I worked on Kubernetes/containers migration at Airbnb, and I wouldn’t recommend a Kubernetes investment for a company with <300 engineers, who can dedicate 10+ people to dealing with it. At Notion we are quite happy with Amazon ECS.
  - Matl 5y ago
    The comparison to systemd is often made, but imo very unfair. systemd may have a complex implementation but it's usage is rather simple and consistent across distros so in that way it definitely simplifies on what came before it, which was an init system simple in its implementation but finicky and inconsistent in usage. It's kind of like I'd much rather have a programming language that has a complex compiler but it takes care of so many details that its usage (programming) is simple - because I am not the one who has to deal with the compiler complexity - rather than making the jobs of the compiler devs easier, (why, if that's their primary job?), but making my everyday job programming in such language harder?
  - __jem 5y ago
    I don't understand why people always recommend ECS over EKS for smaller shops. EKS is kind of creaky, but we haven't really had any major problems with it, and the tooling around the k8s ecosystem is sooooo much bigger and better in terms of developer experience. We felt like we were constantly reinventing the wheel with ECS, and while I fully acknowledge that there's both less surface area and blast radius for things to go wrong with ECS, I'm not sure it's THAT much more stable. ECS is pretty half-baked too. Almost all of my problems are with EKS specifically, and if we compare ECS to GKE, I think the argument is even less strong.
c0l0 5y ago
I am worried that the supreme arrogance of abstraction-builders and -apologists that this article's vernacular, too, emanates will be the cause of a final collapse of the tech and IT sector. Or maybe I even wish for it to happen, and fast. Everything gets SO FRICKIN' COMPLEX these days, even the very simple things. On the flip side, interesting things that used to be engineering problems is relegated to being a "cloud cost optimization" problem. You can just tell your HorizontalPodAutoscaler via some ungodly YAML incantation to deploy your clumsy server a thousand time in parallel across the rent-seekers' vast data centers. People write blogs on how you can host your blog using Serverless Edge-Cloud-Worker-Nodes with a WASM-transpiled SQLite-over-HTTP and whatnot. This article's author fantasizes and fetishizes interconnected lanscapes of OpenID-Connected workloads that are, and I QUOTE, "stateful applications that are stateless". Because "This means data should be handled externally to the application using other abstractions than filesystems, e.g., in databases or object stores." - yeah, fine and dandy, but which platforms of the future should these services be hosted on? And who will develop and shepherd and maintain and fix them?
- leetrout 5y ago
  And for something as "simple" as S3 how much is it gonna cost to get your data back out when you need to leave?
- eof 5y ago
  The complexity of the systems (mostly!) reflects the reality of the needs. Thousands of services working in orchestration in an environment that is, by definition, hostile; is non trivial. Little parts of a system being over engineered may be the fault of an engineers ego somewhere, but the massive complexity of distributed systems isn’t some ego trip or frivolous fiddling with shiny new tech. It’s just big and hard for mere mortals. It’s hard to say when k8s actually decreases complexity, but I think most will get there before ~100 software engineers do.
  - p_l 5y ago
    My current record is one week with 1 infrastructure person trying to run small containerized application on Azure. This is specific to Azure (I could get it running on GCP no problem, AWS would be a bit more complex), but it reflects, IMO, how k8s is simplifying operations because I could do it easily with k8s on all of those platforms (and that includes setting and managing the cluster). I lost track on how many hours (more like man-months, mythical or not) k8s has saved so far in my work, whether it was in team of 2 or team of 15 supporting multiple other teams.
deleted 5y ago
[deleted]
gtirloni 5y ago
TL;DR; - Use OpenID tokens instead of secrets - Use Istio instead of ingress - Use Knative
- codeduck 5y ago
  > Use Istio instead of ingress Sow the wind, reap the whirlwind.
politelemon 5y ago
> Linux is still ubiquitous, and part of our stack, but few developers care much about it because we have since added a few abstractions on top. It's the same that will happen to the traditional Kubernetes we know today. I think the tone of the article writes this as a positive, but I do not see this as a good thing. Looking at the ecosystem of k8s today, all that has happened is the same concepts that existed in OSes have been reinvented. This future abstraction sounds like yet another migration, but I wasn't able to tell what the net benefit was aside from addressing some intellectual or academic concerns (x should not y). I'll definitely try to keep my mind open, but, to borrow a sentence pattern often used in the post, more abstractions should not be seen as a better thing.
mkl95 5y ago
I have been learning Kubernetes on the side for months. My biggest pet peeve is that it's riddled with problems that were solved by much older technologies or simply did not happen before. Debugging some CrashLoopBackOff error that could be happening for dozens of obscure reasons is not my idea of fun.
rickspencer3 5y ago
I appreciated the article, but it read more like iterating on a few pain point in Kubernetes than envisioning a very different future. Where I work, we use Kubernetes very very heavily and some of the article reflects some of my experience, in the sense that any challenge we have, the community response is that we should add MORE complexity. Using secrets is complex? Rearchitect your whole app to use OIDC tokens is the answer (for an example from the article). I think that Kubernetes was designed, released, and funded, to attack the AWS hegemony. As such, it was designed to provide a cloud abstraction layer for developers to target instead of the AWS API(s). As a result, it is designed for an organization to run a single cluster with many relatively trivial workloads (enter Helm for packaging "off the shelf" apps). However, it turned out for some of us, the need is to run the same non-trivial application in multiple regions of multiple Cloud Providers. Here, Kubernetes really falls short, because Cloud Service Providers don't seem particularly motivated to make Kubernetes-based applications more portable, and Kubernetes remains a very leaky abstraction in terms of versions, storage, and networking. To me, the future of Kubernetes looks like the community rallying around a single "distro" of Kubernetes that is reasonably consistent across all major cloud providers, but also makes basic assumptions such as how the CI/CD pipeline, observability and SLOs, self-managed packaging, etc... Here, I think the Linux analogy holds, in the sense that different Linux distros emerged for different sets of users, and communities were able to rallying around solution sets and provide support to each other.
- p_l 5y ago
  What do you think of mixing things like crossplane (and other "operators" wrapping provider-specific bits) together with KubeFedV2?
- hodgesrm 5y ago
  We use Kubernetes for our cloud database platform, which we are extending to new environments. It's been quite successful. We were able to port from AWS to GCP in a matter of weeks. We don't try to share tenants within a single Kubernetes cluster. With that experience in mind, the storage section of the article is a hand-wave. Kubernetes is a major platform for high performance databases themselves. Saying that Kubernetes developers should "use databases" is not a very plausible answer in this case. Object storage is not a panacea either. It does not exist equally in all locations and building caches to make it work efficiently in low-latency apps is a non-trivial exercise. (Like years of implementation work by PhDs if you want something good.) Locally attached NVMe SSD is still the winner for a lot of applications. It needs to be first class in Kubernetes and it needs to be easy to allocate pods that offer it. Applications can deal with resilience through app-level replication. Databases have been doing it for years.
kissgyorgy 5y ago
Saying that "few developers care much about Linux" is like saying "few people care about breathing". Dude, it's everywhere and you can't do anything without Linux nowadays (not even game development anymore, because it is more popular on the Desktop).
- sidlls 5y ago
  Linux is not popular enough on desktop to matter for game development or basically any other consumer software.
  - kissgyorgy 5y ago
    Sure, but there is Steam Deck: https://www.pcworld.com/article/394895/what-is-valve-proton-steam-deck-games-software-explained.html https://www.pcworld.com/article/394895/what-is-valve-proton-...
  - LoveGracePeace 5y ago
    You're wrong, game developers in recent years have started caring a lot more about Linux: https://en.wikipedia.org/wiki/List_of_game_engines https://en.wikipedia.org/wiki/List_of_game_engines Half the game engines on that list target Linux, not 5%, not 15%, half.
    - pjmlp 5y ago
      They care so much that Android games seldom land on GNU/Linux.
      - LoveGracePeace 5y ago
        Hmm, I guess, but I never play Android games, I loathe a touch interface. Keyboard and mouse are the only logical user interfaces to a machine, ever since 1968 with Engelbart.
  - dijit 5y ago
    I work in gamedev, in 2014 when I started, the gameservers were windows. Thanks to rent seeking by Microsoft (the cloud providers got rekt by license cost for Windows) and Googles strong push for Stadia (forcing game engines to support linux, which usually shared some amount of code with the game server) things have changed. Now; most things being made are on Linux. (the actual development is still mostly happening on Windows Desktops though)
magicalhippo 5y ago
Our main product is essential a CRUD desktop application assisted by a lot of background services, mainly various integrations. We're in the process of charting out a path to a cloud-based offering, driven by customer requests. We're operating in a niche so we'll never have millions of concurrent users. We have ~1000 customers, and number of users per customer follows a power law. Our largest customer by far currently has a ~100GB database and ~100 concurrent users. Since we're relatively small (currently ~25 employees) we don't have the resources to "constantly" be changing our stack, and so I fear vendor lock-in. At least Kubernetes seems to be something that'll be around for a while, and it's available from several cloud providers. So while it seems overkill it would appear it has that going for it. However the complexity does scare me a bit. It's a giant leap from our current "dumb" setup of a database and core service on a plain server. Reading recommendations or helpful suggestions would be appreciated. Are there better options out there for small fish like us?
- eof 5y ago
  I’d say you don’t need k8s. You just need a good terraform module to set things up for quickly on multiple providers. K8s is worth all the work cus then you can scale magically. You don’t need to scale magically. You just need to be able to easily deploy. In fact if you think scale will never be an issue you could just run it on hardware, seems like everything would easily fit in a single server.
  - p_l 5y ago
    And with terraform, you're going to be writing pretty much from scratch with every provider out there, as outside of very basics of infrastructure the offerings differ wildly, and even if all you need is a VM with dumb load balancer you have to write ~5x$CLOUD_PROVIDERS resources and abstract it around. Personally k8s is least interesting in scaling, because you could easily scale with other options if you had money to burn
    - jeltz 5y ago
      The same is true for Kubernetes too from my experience. Only the very basics of it works cross provider.
j_m_b 5y ago
Would be interesting to see what Facebook came up with in this space. Not a fan of their "service" but I love their libs.
- sciurus 5y ago
  https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin...
throwaway984393 5y ago
Kubernetes has always had poor design decisions that have rarely ever been improved, because the designers care more about keeping people using their system than making it simpler. The way forward is not to keep adding abstractions to get around these design problems. The way forward is to fix the design. We need something simpler than K8s. Something that does not need a million abstractions, yet accomplishes the basic goals that we want: efficiency, scalability, security, reliability, reproducibility, immutability, etc. I think the future we should be driving for is a distributed operating system. A real operating system whose primitives run distributed applications. K8s already reproduces features that exist in the kernel, and the kernel already does a lot of the heavy lifting of networking and containerization. What's left is to add userland and kernel features so apps can make a syscall or library call to tell the operating system how they'd like to scale, or deploy, or be traced, etc. This would simplify without adding abstractions. It would also be controlled by the open source world, rather than by corporations, making an incentive to finally get rid of crappy design patterns and make the right design be top priority. Those of you who have used Mosix clusters already have an idea how simple and effective this can be. SSI clusters didn't take off because they were focused on unmodified applications, and sharing memory and threads across systems is hard. But by requiring applications to design themselves for a distributed OS, we can instead allow apps to run independently yet communicate seamlessly across systems as if they were just on one.
- dijit 5y ago
  You're probably looking for Nomad or Mesos honestly. Kubernetes, as designed, is a framework and a rather leaky one in my not so humble and subjective opinion. it accomplishes a pretty lofty goal (bin-packing, deployment, eventual reconcilation of service and generalising the network/storage so that you don't care about underlying infra) but it does so at a rather heavy cost of complexity and because you can swap out underlying pieces: if you're not on a cloud provider then you're probably going to spend the majority of your life maintaining it. But, you probably want Nomad, if you're thinking about kubernetes seriously.
  - Hallucinaut 5y ago
    > if you're not on a cloud provider... and lord help you if you are but choose to try to use the same approach across environments and eskew the markup on something like EKS. I've been thumping my head against trying to have a rather trivial HA service working on both a Hyper-V and EC2 cluster equally for weeks and it seems near impossible for reasons no leading lights in the space quite articulates.
notepalf 5y ago
k8s is very powerfull tough complex speccialy if you manage the cluster yourself. Docker swarm is a simpler alternative that provides all the features required by the majority of use cases.
kissgyorgy 5y ago
> This means we are designing for a waste of 30%. Another reason for using conservative target utilizations is that the HPA often works with a response time of a minute or more. This is one of the many things that's really wrong with Kubernetes. A big selling point of it is binpacking; not wasting resources, but it can't even achieve that. You are just overcomplicating your infrastructure without the big win of almost-perfect resource utilization.
me_me_mu_mu 5y ago
I hope k8s and related needless complexities go away for the masses. I would rather focus on building a product and clicking a couple buttons that allow millions to use it (managed solutions). Oh no it costs 10k, let me spend 1m+ on hiring people to play with configuration files. Idk why companies are pushing for k8s at points when they could rather hire 10 engineers/tech team to build better and more features than fiddling around with y’all files and adding so many feedback loops through new “processes”. Even when things work it takes a long time to A) get there organizationally and B) it still doesn’t work lol
- ferdowsi 5y ago
  That company that hires 10 engineers to create a bespoke orchestration system will inevitably end up re-inventing a buggy, poorly specified implementation of Kubernetes. Kubernetes gives a standardized set of abstractions and concepts that means hiring and development is easier, not harder.
  - sublimefire 5y ago
    Not really, they will just end up using managed services in AWS. Kubernetes exists (in the eyes of the corp) only as a tool that avoids vendor lock-in.
    - cjalmeida 5y ago
      Cloud “value added services” ie, all but elastic VM, networking and storage, are typically very poor compared to whatever you can find in k8s just one helm install away.
- marcinzm 5y ago
  That already exists and is called Heroku. It works until it doesn't and then you need a bunch of engineers to work around the limitations.
hodgesrm 5y ago
I tripped over this early in the article and had to rush back to write a comment. > Interestingly, Linux was the platform upon which we built everything a decade or more ago. Linux is still ubiquitous, and part of our stack, but few developers care much about it You definitely care about it when something breaks in a container and you have to figure out what's going on. Abstractions are fine until something doesn't work.
- pjmlp 5y ago
  That is when I open a cloud support ticket.
bazhova 5y ago
The problem is that these products are way too still-in-development. You want to use knative, istio, kubeflow and OIDC auth? Get ready to open some pull requests and start following Gitlab tickets you have no control over. I say this as someone who has rolled a full knative, istio, kubeflow stack in multiple clouds and on prem. Good luck making it bank-grade security compliant... The ecosystem just isn't there yet. There is so much pain involved still.
mt42or 5y ago
Lot of theories disconnected from the reality.
mt42or 5y ago
Istio is a big Kubernetes failure story. So much complicated although it could have been directly integrated to Kubernetes.
deleted 5y ago
[deleted]

147 points by kiyanwang 5y ago

I've actually written about why i'd personally look towards Docker Swarm for smaller projects and setups in my blog post "Docker Swarm over Kubernetes": https://blog.kronis.dev/articles/docker-swarm-over-kubernetes https://blog.kronis.dev/articles/docker-swarm-over-kubernete... (though it is a pretty brief stream of consciousness) Either that or Hashicorp Nomad, because it is also a pretty nice solution and seems to overall have a brighter future than Swarm does, even if the HCL which it uses is a little bit weird and needs porting over from Docker Compose files. Admittedly, in the last year, i've also seen Kubernetes be tolerable for on-prem deployments in the form of K3s, which is a lovely project and wastes way less memory and CPU for smaller clusters: https://k3s.io/ https://k3s.io/ I know that many out there won't necessarily have to or won't want to run it on-prem and instead will just run whatever Kubernetes solution will be offered to them by their cloud vendor, which is also a valid point, but not one that i have to deal with. Instead, i often find myself with a VM that has about 8 GB of RAM and needs to run everything from the actual cluster, to PostgreSQL, RabbitMQ, Redis, a few Java apps and who knows what else inside of a development environment, because clearly that was possible without containers so also should be possible with them. There, running something like Rancher or even a K8s cluster that's made from scratch (or initialized with kubeadm or whatever) would be insanity. And having a shared cluster for the whole company might also necessitate people to just manage it, which isn't in the cards. Also, in my personal hybrid cloud (homelab + cloud VPSes), running Kubernetes would be comparatively expensive and wasteful, since Docker Swarm still has a lower overhead and Swarm provides me with most of the functionality that i might want, oftentimes in the form of regular containers, e.g. Caddy/Nginx/httpd web server as an ingress with some configuration files. Actually, anyone who has ever tried to get the K3s Traefik ingress working with a custom wildcard certificate will probably understand why i might actually prefer to do that, since in K3s you'll need the following for such a setup to work: - a ConfigMap for Traefik, knowledge about the structure of the ConfigMap (tls.stores.default.defaultCertificate) - a TLSSecret for storing the actual certificate/key - a TLSStore (which i also needed to actually use the secret, spec.defaultCertificate.secretName) - a HelmChartConfig for Traefik to load the ConfigMap with the mounted secrets and config All of those weren't sufficiently documented because apparently the popular usecase was to use Let's Encrypt with either cert-manager or another solution altogether. All of it took digging through GitHub to find out and trial and error to configure it all. Versus just reading a page for a web server that i want to run and dropping some values in a config file.

pjmlp 5y ago

Basically, Kubernetes as Cloud OS regardless of what kind of hypervisor powers the whole stack. The irony of UNIX having won, only to be replaced by System 360 roadmap.

hestefisk 5y ago
Not sure I understand your second sentence. Are you equating Kubernetes’ roadmap to that of System 360? FWIW, I am not a big fan of Kubernetes. It’s an incredibly complex beast. I much prefer the simplicity of *BSD over Kubernetes any day.
- pjmlp 5y ago
  Kind of, Kubernetes is just another way of doing mainframe workloads, whose main concepts all trace back to System 360. 20 years ago I was deploying EAR files into application servers without caring which OS they would run on. Nowadays I package managed runtimes into containers, that I don't care if they run directly on top of a type 1 hypervisor, VM or whatever. Basically what mainframe OS virtualization looks like nowadays and traces its roots back to System 360.
  - hestefisk 5y ago
    Good point. Kubernetes actually reminds me a lot of J2EE around version 2.0. Towering abstractions and complexity.
wereHamster 5y ago
Do you see System 360 and Kubernetes as parallel technologies?
- pjmlp 5y ago
  I see Kubernetes + type 1 hypervisors replacing UNIX, thus achieving the same kind of OS virtualization done on IBM mainframes that trace their roots back to System 360.

jitl 5y ago

It’s funny to see this pitched as the “future” when we were doing most of these shenanigans during Airbnb’s Kubernetes rollout in 2018. The only thing that’s futuristic from the perspective of 2018 Airbnb infra is Knative. Knative sounds like a good idea — especially for control-plane services or devtools where scaling to zero is something that would actually happen. The commentary around PersistedVolumes is true (don’t write user data as json pls) but simple. We had a few use-cases for big persisted volumes or even host volumes managed by a daemonset, where object store performance didn’t cut it. One was indexes for a search service. You can’t push these 64gb hunks of data into S3 without paying a ton, much better to use a file system. Same thing with translation string data, a PV with a local SQLite database or other indexed read-optimized format is 100x faster and 100x less expensive than “object storage”. We did enforce that the PV was always read-only for the “main” container in a pod, so you needed to add a sidecar or daemonset if you wanted to sync data there.

l0rn 5y ago

Mh... As far as I understand this article is promoting the fact to factor in kubernetes or the cloud platform directly into the orchestrator system (i.e. k8s). I think this won't happen. I think k8s was successful because it allows to run unix-like applications with little adoption effort of the application itself.

leetrout 5y ago

""" Generally, the Kubernetes we know today will disappear, and developers will not care """ I truly cannot wait. I love the premise but hate the complexity. Nomad stole my mindshare in 2016 and I would still choose it over k8s today for small applications. Even choosing it for managing VMs in some cases. (Disclaimer: worked at hashi but not on nomad nor did i use it while there).

ThePhysicist 5y ago

A difference between the Linux movement and the Kubernetes ecosystem is that most of the open-source tooling developed for K8s is built by venture-funded startups (e.g. Istio or Envoy). And the endgame of these companies is almost always an open-core product that is functional but in practice just broken enough to not be usable without buying premium support or the enterprise edition. So I'm a bit skeptical about the long-term viability of the platform. Personally I think we'll see new paradigms emerge that will mostly obsolete containers and complex orchestration platforms, you can e.g. already see some pretty interesting ideas around WASM. Before Kubernetes OpenStack was the big thing, but now it's mostly used as a substrate to run Kubernetes on.

rdli 5y ago
Istio was started by Google & IBM, while Envoy was created by Matt Klein at Lyft. One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. As a note, technically Google & Lyft are VC-funded, so that’s true, although I wouldn’t put Google in the startup category, and Lyft’s business model is unrelated to Envoy.
- ThePhysicist 5y ago
  I was under the impression that Istio and Envoy are significantly driven by Tetrate (https://www.tetrate.io/ https://www.tetrate.io/), which in my understanding was started by the Istio creators and sells enterprise editions of the two solutions [1]. So yeah it's open source, I can fork it and that's nice, but let's be honest Tetrate aims to monetize these products. Not saying this can't work and be great for the community at the same time, but I'm at least a bit skeptical. 1: https://www.businesswire.com/news/home/20210310005295/en/Tetrate-Started-by-Istio-Founders-Raises-40-Million-to-Help-Enterprises-With-Cloud-Native-Application-Networking-Platform https://www.businesswire.com/news/home/20210310005295/en/Tet...
  - bassarid 5y ago
    I've been running Istio for years and have never heard of Tetrate. Tetrate might be started by people who originally worked on Istio, but Istio itself is primarily driven by Google. Istio development is primarily driven by its steering committee [0][1], which is led by Google. If you look at the contributors [2], 6 of the top 10 are Google employees (perhaps more - not everyone lists their employer on GitHub). They have an open design process, posting design documents and similar to their Google Drive [3]. The team is likewise active at community events, where they solicit input that drives what they work on. [0]: https://istio.io/latest/blog/2020/steering-changes/ https://istio.io/latest/blog/2020/steering-changes/ [1]: https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqcmdWXDuCaPC0kPS21BuAfCL8/edit?usp=sharing https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqc... [2]: https://github.com/istio/istio/graphs/contributors https://github.com/istio/istio/graphs/contributors [3]: https://istio.io/latest/get-involved/ https://istio.io/latest/get-involved/

tkiolp4 5y ago

K8s will be superseded by a simpler yet more powerful tool sooner rather than later. Just like it happened with: - Corba, MRI -> http apis - XML, Soap -> json - Flash -> html5 - Perforce, svn, mercurial -> git Etc. We can’t think of anything better than k8s right now, but that’s just how it works. The moment that new tech appears, we all will be asking ourselves “how could we possibly use k8s?”

hutrdvnj 5y ago
This is not always the case. Sometimes we use a technology for decades and just upgrade its version. Like HTML, HTTP, GCC, ... the list goes on. It's very possible that we still use k8s as major container orchestrator in a decade, just in a higher version.
- xg15 5y ago
  I think a better way to think about it is in separating API and implementation or wire format and semantics. In terms of the wire-format and the low-level concepts, HTTP2 and 3 are completely different from HTTP1.x. it's not an extension like HTTP1.1 was, it's a complete reinvention. However, what they did was to reuse the high-level semantics and the "API" of HTTP1, so as a web developer you can pretty much pretend you're still using HTTP1 and let the browser and web server do the rest. This makes transition significantly easier. I could imagine something similar with K9s. Maybe some other product will come along that supports (a subset of) the k9s API but does something different under the hood. I believe, k9s itself did so with Docker: You can still define k9s images using Dockerfiles, even though no piece of Docker is actually involved in the process.
radicalbyte 5y ago
I find this amusing considering that the JSON community are slowly re-inventing XML with their versions of XSL, XSLT etc.
- tkiolp4 5y ago
  JSON is already 20 years old. I guess things can’t stay the way they are forever (and 20 years in the IT world is a lot).

livinginfear 5y ago

I've managed to avoid working with Kubernetes thus far in my career. I admit I don't have a great understanding of it. I can't think of a single technology that seems to unanimously stir up such bitter feelings in the developer community. Not even node.js, or webpack is as contentious as k8s.

leetrout 5y ago
You are both missing nothing and missing everything. Haha. Try to hold out for the next generation! It will surely arrive soon? Do you use any container orchestrators today?
jitl 5y ago
Systemd stirs up similar angry because it was built for similar reasons to become this god-management-layer that’s useful for getting shit done but also hugely more complex (…in implementation at least) than the status quo it replaced. I worked on Kubernetes/containers migration at Airbnb, and I wouldn’t recommend a Kubernetes investment for a company with <300 engineers, who can dedicate 10+ people to dealing with it. At Notion we are quite happy with Amazon ECS.
- Matl 5y ago
  The comparison to systemd is often made, but imo very unfair. systemd may have a complex implementation but it's usage is rather simple and consistent across distros so in that way it definitely simplifies on what came before it, which was an init system simple in its implementation but finicky and inconsistent in usage. It's kind of like I'd much rather have a programming language that has a complex compiler but it takes care of so many details that its usage (programming) is simple - because I am not the one who has to deal with the compiler complexity - rather than making the jobs of the compiler devs easier, (why, if that's their primary job?), but making my everyday job programming in such language harder?
- __jem 5y ago
  I don't understand why people always recommend ECS over EKS for smaller shops. EKS is kind of creaky, but we haven't really had any major problems with it, and the tooling around the k8s ecosystem is sooooo much bigger and better in terms of developer experience. We felt like we were constantly reinventing the wheel with ECS, and while I fully acknowledge that there's both less surface area and blast radius for things to go wrong with ECS, I'm not sure it's THAT much more stable. ECS is pretty half-baked too. Almost all of my problems are with EKS specifically, and if we compare ECS to GKE, I think the argument is even less strong.

c0l0 5y ago

I am worried that the supreme arrogance of abstraction-builders and -apologists that this article's vernacular, too, emanates will be the cause of a final collapse of the tech and IT sector. Or maybe I even wish for it to happen, and fast. Everything gets SO FRICKIN' COMPLEX these days, even the very simple things. On the flip side, interesting things that used to be engineering problems is relegated to being a "cloud cost optimization" problem. You can just tell your HorizontalPodAutoscaler via some ungodly YAML incantation to deploy your clumsy server a thousand time in parallel across the rent-seekers' vast data centers. People write blogs on how you can host your blog using Serverless Edge-Cloud-Worker-Nodes with a WASM-transpiled SQLite-over-HTTP and whatnot. This article's author fantasizes and fetishizes interconnected lanscapes of OpenID-Connected workloads that are, and I QUOTE, "stateful applications that are stateless". Because "This means data should be handled externally to the application using other abstractions than filesystems, e.g., in databases or object stores." - yeah, fine and dandy, but which platforms of the future should these services be hosted on? And who will develop and shepherd and maintain and fix them?

leetrout 5y ago
And for something as "simple" as S3 how much is it gonna cost to get your data back out when you need to leave?
eof 5y ago
The complexity of the systems (mostly!) reflects the reality of the needs. Thousands of services working in orchestration in an environment that is, by definition, hostile; is non trivial. Little parts of a system being over engineered may be the fault of an engineers ego somewhere, but the massive complexity of distributed systems isn’t some ego trip or frivolous fiddling with shiny new tech. It’s just big and hard for mere mortals. It’s hard to say when k8s actually decreases complexity, but I think most will get there before ~100 software engineers do.
- p_l 5y ago
  My current record is one week with 1 infrastructure person trying to run small containerized application on Azure. This is specific to Azure (I could get it running on GCP no problem, AWS would be a bit more complex), but it reflects, IMO, how k8s is simplifying operations because I could do it easily with k8s on all of those platforms (and that includes setting and managing the cluster). I lost track on how many hours (more like man-months, mythical or not) k8s has saved so far in my work, whether it was in team of 2 or team of 15 supporting multiple other teams.

deleted 5y ago

[deleted]

gtirloni 5y ago

TL;DR; - Use OpenID tokens instead of secrets - Use Istio instead of ingress - Use Knative

codeduck 5y ago
> Use Istio instead of ingress Sow the wind, reap the whirlwind.

politelemon 5y ago

> Linux is still ubiquitous, and part of our stack, but few developers care much about it because we have since added a few abstractions on top. It's the same that will happen to the traditional Kubernetes we know today. I think the tone of the article writes this as a positive, but I do not see this as a good thing. Looking at the ecosystem of k8s today, all that has happened is the same concepts that existed in OSes have been reinvented. This future abstraction sounds like yet another migration, but I wasn't able to tell what the net benefit was aside from addressing some intellectual or academic concerns (x should not y). I'll definitely try to keep my mind open, but, to borrow a sentence pattern often used in the post, more abstractions should not be seen as a better thing.

mkl95 5y ago

I have been learning Kubernetes on the side for months. My biggest pet peeve is that it's riddled with problems that were solved by much older technologies or simply did not happen before. Debugging some CrashLoopBackOff error that could be happening for dozens of obscure reasons is not my idea of fun.

rickspencer3 5y ago

I appreciated the article, but it read more like iterating on a few pain point in Kubernetes than envisioning a very different future. Where I work, we use Kubernetes very very heavily and some of the article reflects some of my experience, in the sense that any challenge we have, the community response is that we should add MORE complexity. Using secrets is complex? Rearchitect your whole app to use OIDC tokens is the answer (for an example from the article). I think that Kubernetes was designed, released, and funded, to attack the AWS hegemony. As such, it was designed to provide a cloud abstraction layer for developers to target instead of the AWS API(s). As a result, it is designed for an organization to run a single cluster with many relatively trivial workloads (enter Helm for packaging "off the shelf" apps). However, it turned out for some of us, the need is to run the same non-trivial application in multiple regions of multiple Cloud Providers. Here, Kubernetes really falls short, because Cloud Service Providers don't seem particularly motivated to make Kubernetes-based applications more portable, and Kubernetes remains a very leaky abstraction in terms of versions, storage, and networking. To me, the future of Kubernetes looks like the community rallying around a single "distro" of Kubernetes that is reasonably consistent across all major cloud providers, but also makes basic assumptions such as how the CI/CD pipeline, observability and SLOs, self-managed packaging, etc... Here, I think the Linux analogy holds, in the sense that different Linux distros emerged for different sets of users, and communities were able to rallying around solution sets and provide support to each other.

p_l 5y ago
What do you think of mixing things like crossplane (and other "operators" wrapping provider-specific bits) together with KubeFedV2?
hodgesrm 5y ago
We use Kubernetes for our cloud database platform, which we are extending to new environments. It's been quite successful. We were able to port from AWS to GCP in a matter of weeks. We don't try to share tenants within a single Kubernetes cluster. With that experience in mind, the storage section of the article is a hand-wave. Kubernetes is a major platform for high performance databases themselves. Saying that Kubernetes developers should "use databases" is not a very plausible answer in this case. Object storage is not a panacea either. It does not exist equally in all locations and building caches to make it work efficiently in low-latency apps is a non-trivial exercise. (Like years of implementation work by PhDs if you want something good.) Locally attached NVMe SSD is still the winner for a lot of applications. It needs to be first class in Kubernetes and it needs to be easy to allocate pods that offer it. Applications can deal with resilience through app-level replication. Databases have been doing it for years.

kissgyorgy 5y ago

Saying that "few developers care much about Linux" is like saying "few people care about breathing". Dude, it's everywhere and you can't do anything without Linux nowadays (not even game development anymore, because it is more popular on the Desktop).

sidlls 5y ago
Linux is not popular enough on desktop to matter for game development or basically any other consumer software.
- kissgyorgy 5y ago
  Sure, but there is Steam Deck: https://www.pcworld.com/article/394895/what-is-valve-proton-steam-deck-games-software-explained.html https://www.pcworld.com/article/394895/what-is-valve-proton-...
- LoveGracePeace 5y ago
  You're wrong, game developers in recent years have started caring a lot more about Linux: https://en.wikipedia.org/wiki/List_of_game_engines https://en.wikipedia.org/wiki/List_of_game_engines Half the game engines on that list target Linux, not 5%, not 15%, half.
  - pjmlp 5y ago
    They care so much that Android games seldom land on GNU/Linux.
    - LoveGracePeace 5y ago
      Hmm, I guess, but I never play Android games, I loathe a touch interface. Keyboard and mouse are the only logical user interfaces to a machine, ever since 1968 with Engelbart.
- dijit 5y ago
  I work in gamedev, in 2014 when I started, the gameservers were windows. Thanks to rent seeking by Microsoft (the cloud providers got rekt by license cost for Windows) and Googles strong push for Stadia (forcing game engines to support linux, which usually shared some amount of code with the game server) things have changed. Now; most things being made are on Linux. (the actual development is still mostly happening on Windows Desktops though)

magicalhippo 5y ago

Our main product is essential a CRUD desktop application assisted by a lot of background services, mainly various integrations. We're in the process of charting out a path to a cloud-based offering, driven by customer requests. We're operating in a niche so we'll never have millions of concurrent users. We have ~1000 customers, and number of users per customer follows a power law. Our largest customer by far currently has a ~100GB database and ~100 concurrent users. Since we're relatively small (currently ~25 employees) we don't have the resources to "constantly" be changing our stack, and so I fear vendor lock-in. At least Kubernetes seems to be something that'll be around for a while, and it's available from several cloud providers. So while it seems overkill it would appear it has that going for it. However the complexity does scare me a bit. It's a giant leap from our current "dumb" setup of a database and core service on a plain server. Reading recommendations or helpful suggestions would be appreciated. Are there better options out there for small fish like us?

eof 5y ago
I’d say you don’t need k8s. You just need a good terraform module to set things up for quickly on multiple providers. K8s is worth all the work cus then you can scale magically. You don’t need to scale magically. You just need to be able to easily deploy. In fact if you think scale will never be an issue you could just run it on hardware, seems like everything would easily fit in a single server.
- p_l 5y ago
  And with terraform, you're going to be writing pretty much from scratch with every provider out there, as outside of very basics of infrastructure the offerings differ wildly, and even if all you need is a VM with dumb load balancer you have to write ~5x$CLOUD_PROVIDERS resources and abstract it around. Personally k8s is least interesting in scaling, because you could easily scale with other options if you had money to burn
  - jeltz 5y ago
    The same is true for Kubernetes too from my experience. Only the very basics of it works cross provider.

j_m_b 5y ago

Would be interesting to see what Facebook came up with in this space. Not a fan of their "service" but I love their libs.

sciurus 5y ago
https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin...

throwaway984393 5y ago

Kubernetes has always had poor design decisions that have rarely ever been improved, because the designers care more about keeping people using their system than making it simpler. The way forward is not to keep adding abstractions to get around these design problems. The way forward is to fix the design. We need something simpler than K8s. Something that does not need a million abstractions, yet accomplishes the basic goals that we want: efficiency, scalability, security, reliability, reproducibility, immutability, etc. I think the future we should be driving for is a distributed operating system. A real operating system whose primitives run distributed applications. K8s already reproduces features that exist in the kernel, and the kernel already does a lot of the heavy lifting of networking and containerization. What's left is to add userland and kernel features so apps can make a syscall or library call to tell the operating system how they'd like to scale, or deploy, or be traced, etc. This would simplify without adding abstractions. It would also be controlled by the open source world, rather than by corporations, making an incentive to finally get rid of crappy design patterns and make the right design be top priority. Those of you who have used Mosix clusters already have an idea how simple and effective this can be. SSI clusters didn't take off because they were focused on unmodified applications, and sharing memory and threads across systems is hard. But by requiring applications to design themselves for a distributed OS, we can instead allow apps to run independently yet communicate seamlessly across systems as if they were just on one.

dijit 5y ago
You're probably looking for Nomad or Mesos honestly. Kubernetes, as designed, is a framework and a rather leaky one in my not so humble and subjective opinion. it accomplishes a pretty lofty goal (bin-packing, deployment, eventual reconcilation of service and generalising the network/storage so that you don't care about underlying infra) but it does so at a rather heavy cost of complexity and because you can swap out underlying pieces: if you're not on a cloud provider then you're probably going to spend the majority of your life maintaining it. But, you probably want Nomad, if you're thinking about kubernetes seriously.
- Hallucinaut 5y ago
  > if you're not on a cloud provider... and lord help you if you are but choose to try to use the same approach across environments and eskew the markup on something like EKS. I've been thumping my head against trying to have a rather trivial HA service working on both a Hyper-V and EC2 cluster equally for weeks and it seems near impossible for reasons no leading lights in the space quite articulates.

notepalf 5y ago

k8s is very powerfull tough complex speccialy if you manage the cluster yourself. Docker swarm is a simpler alternative that provides all the features required by the majority of use cases.

kissgyorgy 5y ago

> This means we are designing for a waste of 30%. Another reason for using conservative target utilizations is that the HPA often works with a response time of a minute or more. This is one of the many things that's really wrong with Kubernetes. A big selling point of it is binpacking; not wasting resources, but it can't even achieve that. You are just overcomplicating your infrastructure without the big win of almost-perfect resource utilization.

me_me_mu_mu 5y ago

I hope k8s and related needless complexities go away for the masses. I would rather focus on building a product and clicking a couple buttons that allow millions to use it (managed solutions). Oh no it costs 10k, let me spend 1m+ on hiring people to play with configuration files. Idk why companies are pushing for k8s at points when they could rather hire 10 engineers/tech team to build better and more features than fiddling around with y’all files and adding so many feedback loops through new “processes”. Even when things work it takes a long time to A) get there organizationally and B) it still doesn’t work lol

ferdowsi 5y ago
That company that hires 10 engineers to create a bespoke orchestration system will inevitably end up re-inventing a buggy, poorly specified implementation of Kubernetes. Kubernetes gives a standardized set of abstractions and concepts that means hiring and development is easier, not harder.
- sublimefire 5y ago
  Not really, they will just end up using managed services in AWS. Kubernetes exists (in the eyes of the corp) only as a tool that avoids vendor lock-in.
  - cjalmeida 5y ago
    Cloud “value added services” ie, all but elastic VM, networking and storage, are typically very poor compared to whatever you can find in k8s just one helm install away.
marcinzm 5y ago
That already exists and is called Heroku. It works until it doesn't and then you need a bunch of engineers to work around the limitations.

hodgesrm 5y ago

I tripped over this early in the article and had to rush back to write a comment. > Interestingly, Linux was the platform upon which we built everything a decade or more ago. Linux is still ubiquitous, and part of our stack, but few developers care much about it You definitely care about it when something breaks in a container and you have to figure out what's going on. Abstractions are fine until something doesn't work.

pjmlp 5y ago
That is when I open a cloud support ticket.

bazhova 5y ago

The problem is that these products are way too still-in-development. You want to use knative, istio, kubeflow and OIDC auth? Get ready to open some pull requests and start following Gitlab tickets you have no control over. I say this as someone who has rolled a full knative, istio, kubeflow stack in multiple clouds and on prem. Good luck making it bank-grade security compliant... The ecosystem just isn't there yet. There is so much pain involved still.

mt42or 5y ago

Lot of theories disconnected from the reality.

mt42or 5y ago

Istio is a big Kubernetes failure story. So much complicated although it could have been directly integrated to Kubernetes.

deleted 5y ago

[deleted]