6 ms·

A difference between the Linux movement and the Kubernetes ecosystem is that most of the open-source tooling developed for K8s is built by venture-funded startu

A difference between the Linux movement and the Kubernetes ecosystem is that most of the open-source tooling developed for K8s is built by venture-funded startups (e.g. Istio or Envoy). And the endgame of these companies is almost always an open-core product that is functional but in practice just broken enough to not be usable without buying premium support or the enterprise edition. So I'm a bit skeptical about the long-term viability of the platform. Personally I think we'll see new paradigms emerge that will mostly obsolete containers and complex orchestration platforms, you can e.g. already see some pretty interesting ideas around WASM. Before Kubernetes OpenStack was the big thing, but now it's mostly used as a substrate to run Kubernetes on.

rdli 5y ago
Istio was started by Google & IBM, while Envoy was created by Matt Klein at Lyft. One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. As a note, technically Google & Lyft are VC-funded, so that’s true, although I wouldn’t put Google in the startup category, and Lyft’s business model is unrelated to Envoy.
- ThePhysicist 5y ago
  I was under the impression that Istio and Envoy are significantly driven by Tetrate (https://www.tetrate.io/ https://www.tetrate.io/), which in my understanding was started by the Istio creators and sells enterprise editions of the two solutions [1]. So yeah it's open source, I can fork it and that's nice, but let's be honest Tetrate aims to monetize these products. Not saying this can't work and be great for the community at the same time, but I'm at least a bit skeptical. 1: https://www.businesswire.com/news/home/20210310005295/en/Tetrate-Started-by-Istio-Founders-Raises-40-Million-to-Help-Enterprises-With-Cloud-Native-Application-Networking-Platform https://www.businesswire.com/news/home/20210310005295/en/Tet...
  - bassarid 5y ago
    I've been running Istio for years and have never heard of Tetrate. Tetrate might be started by people who originally worked on Istio, but Istio itself is primarily driven by Google. Istio development is primarily driven by its steering committee [0][1], which is led by Google. If you look at the contributors [2], 6 of the top 10 are Google employees (perhaps more - not everyone lists their employer on GitHub). They have an open design process, posting design documents and similar to their Google Drive [3]. The team is likewise active at community events, where they solicit input that drives what they work on. [0]: https://istio.io/latest/blog/2020/steering-changes/ https://istio.io/latest/blog/2020/steering-changes/ [1]: https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqcmdWXDuCaPC0kPS21BuAfCL8/edit?usp=sharing https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqc... [2]: https://github.com/istio/istio/graphs/contributors https://github.com/istio/istio/graphs/contributors [3]: https://istio.io/latest/get-involved/ https://istio.io/latest/get-involved/
  - rdli 5y ago
    Envoy was originally written by Matt Klein @ Lyft, and the IP is owned by the Cloud Native Computing Foundation. Here's the initial video where Matt Klein introduces Envoy: https://www.microservices.com/talks/lyfts-envoy-monolith-service-mesh-matt-klein/ https://www.microservices.com/talks/lyfts-envoy-monolith-ser.... If you want to buy commercial support for Envoy & Istio, Tetrate certainly is an option. (Note: I am/was the CEO at Ambassador Labs, which organized the Microservices Practitioner Summit where this talk occurred.)
- otterley 5y ago
  > One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. I'm a little disappointed that you placed so much weight in that factor. I believe simplicity, reliability, and comprehensive documentation are far more important concerns than whether a commercial entity has control over a project. The flip side of community ownership is more politics, more complexity, bureaucratic decisionmaking, and slower evolution. Connection/service proxies are a dime a dozen. I suspect you could have picked nginx, haproxy, or the Linkerd proxy and been just as well off.
  - rdli 5y ago
    > The flip side of community ownership is more politics, more complexity, bureaucratic decisionmaking, and slower evolution. This of course may be true for some communities, but certainly not the case for L7 proxies. In fact, at the time, the opposite was true: NGINX & HAProxy were content in slowly supporting their (captive) customer bases, with little incentive to innovate. This is why Lyft went off to build Envoy Proxy. At GA, it supported features such as HTTP/2, native observability, hitless reloads — none of which were easy / possible to do in NGINX or HAProxy at the time. > Connection/service proxies are a dime a dozen. I suspect you could have picked nginx, haproxy, or the Linkerd proxy and been just as well off. Linkerd proxy didn’t exist then (there was an early version of it, written in Scala IIRC; the Rust version didn’t come along to quite some time later). I agree that there are many options for service proxies. I’ve found, though, that the Envoy community has been pushing innovation in the space quite aggressively.
jitl 5y ago
The dig about Envoy doesn’t read true to me. Envoy was open sourced by Lyft in 2017 and quickly replaced Haproxy in our stack because it was Good. Is there actually a venture backed startup steering it these days?
- ThePhysicist 5y ago
  See my other reply.
  - marcinzm 5y ago
    The responses to which indicate you're factually incorrect and that neither Envoy nor Istio are owned or controlled by venture backed startups. You can however get products built on top of them or enterprise support from venture backed startups. Akin to companies like Red Hat in regards to Linux.
    - ThePhysicist 5y ago
      I have never used Envoy or Istio, and maybe I shouldn't believe the PR hyperbole about Tetrate which says "Today, Tetrate is one of the largest contributors to the Istio open source project and second largest to Envoy Proxy.". In the other tread people have pointed out that Google is instead the main driver behind Istio these days, which I do not find more reassuring as they mostly use Kubernetes as a marketing tool to sell cloud services (as, in my understanding, they have an entirely different internal platform that they deploy their services on). I would say the point still holds, but of course it's fine to have different opinions on this. It's just that seeing containerization startups like Docker starting to monetize their core ecosystem to the detriment of the open-source community (e.g. by limiting container pulls on Dockerhub) does not give me confidence that the same thing won't happen to the Kubernetes ecosystem in the long term.
candiddevmike 5y ago
Any extensible platform will be driven into a complex slurry of options and configurations to support every niche vendor/use case. I think the successor of kubernetes will be a simpler version of Fleet: distributed systemd. The future is edge, and edge doesn't need to integrate with every series A startup, it needs a small footprint and stability. I've been tinkering with a prototype that does this (and if anyone is interested in collaborating, my email is in my profile).
- p_l 5y ago
  The strength of Kubernetes (vs Mesos, OpenStack, etc.) is that you can strip it down pretty well so that you only bring the niche bits that you need - whether it's edge, or something big that needs a lot of extras. The unified API mechanisms and moving drivers out of core helped a lot there. Also, in my experience, it turns out that you can quickly find yourself needing all those extra bits if you're not Google (which implements a lot of the same features in libraries linked to programs directly and depends on the fact that they run customized everything) or Netflix (similar).
paulgb 5y ago
> already see some pretty interesting ideas around WASM I'm excited about WASM, but I don't see how it's fundamentally different when it comes to being built by venture-funded startups. Pretty much all of the interesting stuff in server-side WASM is being driven by venture-backed startups.
- ridruejo 5y ago
  Agree, but also there's a lot of core work been done by the industry (Intel, Google, etc.) that we all benefit from. It definitely 'feels' different than K8s and closer to the early days of the web
Thaxll 5y ago
I don't know why people keep talking about WASM, WASM is 1% of what Kubernetes does, it's an alternative to runc nothing more. It's like comparing Docker to Kubernetes.
- boomskats 5y ago
  I don't think they were comparing WASM to Kubernetes. In the context of the comment you replied to, I read it as WASM potentially obsoleting _containers_ as opposed to _complex orchestration platforms_. > WASM is 1% of what Kubernetes does, it's an alternative to runc nothing more It'd be interesting if you could expand on this. I don't think WASM does _any_ of what Kubernetes does.
  - FooBarWidget 5y ago
    Why would WASM obsolete containers?
    - pjmlp 5y ago
      Because new generation startups are on the verge of selling WASM Application Servers....
    - boomskats 5y ago
      Hey, I didn't say I agreed with that statement - I was just pointing out what I thought they were trying to say.
    - ledgerdev 5y ago
      Of course containers will be around probably forever, just as we still have mainframes. Note that current serverless and container platforms start and manage long running instances and occasionally incur cold starts. WASM has near instant startup, making a cold start for each request feasible. I think this could be a key property/advantage that makes it preferable and more economically efficient in the long run. WASM I think has a stronger sandbox as well. Containers in a multi-tenant environment need to run inside a virtual machine. WASM can be run in multi-tenant without the need to isolate each tenant in a virtual machine, again making it more efficient in long run. edit: also recalled a relevant tweet from creator of docker https://twitter.com/solomonstre/status/1111004913222324225 https://twitter.com/solomonstre/status/1111004913222324225
yarky 5y ago
> broken enough to not be usable without buying premium support or the enterprise edition. How does kubernets do this ? I haven't hit that spot yet but I'm pretty sure I will. I guess I'll also find an open source solution to avoid switching licenses.
ghaff 5y ago
Istio came out of Google. And I'm not sure what Kubernetes has to do with OpenStack, other than early-on it was assumed you would need to run containers on top of cloud VMs because that's what the big public cloud providers did.
s-lambert 5y ago
>pretty interesting ideas around WASM I'm confused here, does WASM stand for something different besides WebAssembly because googling "kubernetes WASM' came up with nothing but WebAssembly. But I was under the impression that WebAssembly is just a virtual machine. I'm not much of a backend person but I don't see the overlap between container management (kubernetes) to a virtual machine (WASM).
- ThePhysicist 5y ago
  I meant more that technologies like WASM might obsolete the need for containers and thereby container orchestration platforms like Kubernetes. Of course Kubernetes might itself evolve to orchestrate WASM workloads.
  - pjmlp 5y ago
    I can already see it, a zip file containing application assets, a wasm binary and a couple of manifest files with configuration metadata. Where have I seen this already?
    - callahad 5y ago
      Sure, superficially it sounds like Java. But hey, Java was really damn successful for a very long time.
      - pjmlp 5y ago
        It would help if that wasn't sold as something new never attempted before.
  - kitd 5y ago
    Of course Kubernetes might itself evolve to orchestrate WASM workloads. Already been done: https://krustlet.dev/ https://krustlet.dev/
  - robert_dipaolo 5y ago
    The WASM concept has been around for decades, it's effectively jvm or .net in but in the browser. I don't get the relevance when discussing k8s or containers.
nunez 5y ago
tbf if a new Linux were starting in 2022, it would absolutely be VC-funded, since the dotcom era opened that Pandora's Box and it will never close again i think the other key difference b/w nix and k8s is that a very large swath of k8s core contributors are employed by very large companies (mostly Google, Red Hat, Microsoft, VMware, and IBM) who are also deeply invested in k8s-derivatives sold in the marketplace. from what i've gathered, and this might not be true anymore, many contributions to Linux still come from the academy (universities, etc)
- pjmlp 5y ago
  Same as Linux, it only took off because many UNIX shops like IBM saw a value in improving it, while cutting costs on their own. > 1998: Many major companies such as IBM, Compaq and Oracle announce their support for Linux > 2000: Dell announces that it is now the No. 2 provider of Linux-based systems worldwide and the first major manufacturer to offer Linux across its full product line > 2006: Oracle releases its own distribution of Red Hat Enterprise Linux. --- https://en.m.wikipedia.org/wiki/History_of_Linux https://en.m.wikipedia.org/wiki/History_of_Linux
pjmlp 5y ago
Given the failure of most pure FOSS business, and the love for non-copyleft licenses, it is quite clear that what most businesses care about is the model that ruled the industry before Linux, with open core being the new demo versions. No wonder that as IoT took off plenty of POSIX like clones with non-copyleft licenses sprung off. Even on Android, the Linux kernel is the only GPL piece of the puzzle left standing.

rdli 5y ago

Istio was started by Google & IBM, while Envoy was created by Matt Klein at Lyft. One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. As a note, technically Google & Lyft are VC-funded, so that’s true, although I wouldn’t put Google in the startup category, and Lyft’s business model is unrelated to Envoy.

ThePhysicist 5y ago
I was under the impression that Istio and Envoy are significantly driven by Tetrate (https://www.tetrate.io/ https://www.tetrate.io/), which in my understanding was started by the Istio creators and sells enterprise editions of the two solutions [1]. So yeah it's open source, I can fork it and that's nice, but let's be honest Tetrate aims to monetize these products. Not saying this can't work and be great for the community at the same time, but I'm at least a bit skeptical. 1: https://www.businesswire.com/news/home/20210310005295/en/Tetrate-Started-by-Istio-Founders-Raises-40-Million-to-Help-Enterprises-With-Cloud-Native-Application-Networking-Platform https://www.businesswire.com/news/home/20210310005295/en/Tet...
- bassarid 5y ago
  I've been running Istio for years and have never heard of Tetrate. Tetrate might be started by people who originally worked on Istio, but Istio itself is primarily driven by Google. Istio development is primarily driven by its steering committee [0][1], which is led by Google. If you look at the contributors [2], 6 of the top 10 are Google employees (perhaps more - not everyone lists their employer on GitHub). They have an open design process, posting design documents and similar to their Google Drive [3]. The team is likewise active at community events, where they solicit input that drives what they work on. [0]: https://istio.io/latest/blog/2020/steering-changes/ https://istio.io/latest/blog/2020/steering-changes/ [1]: https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqcmdWXDuCaPC0kPS21BuAfCL8/edit?usp=sharing https://docs.google.com/spreadsheets/d/1Dt-h9s8G7Wyt4r16ZVqc... [2]: https://github.com/istio/istio/graphs/contributors https://github.com/istio/istio/graphs/contributors [3]: https://istio.io/latest/get-involved/ https://istio.io/latest/get-involved/
- rdli 5y ago
  Envoy was originally written by Matt Klein @ Lyft, and the IP is owned by the Cloud Native Computing Foundation. Here's the initial video where Matt Klein introduces Envoy: https://www.microservices.com/talks/lyfts-envoy-monolith-service-mesh-matt-klein/ https://www.microservices.com/talks/lyfts-envoy-monolith-ser.... If you want to buy commercial support for Envoy & Istio, Tetrate certainly is an option. (Note: I am/was the CEO at Ambassador Labs, which organized the Microservices Practitioner Summit where this talk occurred.)
otterley 5y ago
> One of the reasons why we adopted Envoy at Ambassador for our API Gateway was precisely because there was no single _commercial_ entity who controlled Envoy. I'm a little disappointed that you placed so much weight in that factor. I believe simplicity, reliability, and comprehensive documentation are far more important concerns than whether a commercial entity has control over a project. The flip side of community ownership is more politics, more complexity, bureaucratic decisionmaking, and slower evolution. Connection/service proxies are a dime a dozen. I suspect you could have picked nginx, haproxy, or the Linkerd proxy and been just as well off.
- rdli 5y ago
  > The flip side of community ownership is more politics, more complexity, bureaucratic decisionmaking, and slower evolution. This of course may be true for some communities, but certainly not the case for L7 proxies. In fact, at the time, the opposite was true: NGINX & HAProxy were content in slowly supporting their (captive) customer bases, with little incentive to innovate. This is why Lyft went off to build Envoy Proxy. At GA, it supported features such as HTTP/2, native observability, hitless reloads — none of which were easy / possible to do in NGINX or HAProxy at the time. > Connection/service proxies are a dime a dozen. I suspect you could have picked nginx, haproxy, or the Linkerd proxy and been just as well off. Linkerd proxy didn’t exist then (there was an early version of it, written in Scala IIRC; the Rust version didn’t come along to quite some time later). I agree that there are many options for service proxies. I’ve found, though, that the Envoy community has been pushing innovation in the space quite aggressively.

jitl 5y ago

The dig about Envoy doesn’t read true to me. Envoy was open sourced by Lyft in 2017 and quickly replaced Haproxy in our stack because it was Good. Is there actually a venture backed startup steering it these days?

ThePhysicist 5y ago
See my other reply.
- marcinzm 5y ago
  The responses to which indicate you're factually incorrect and that neither Envoy nor Istio are owned or controlled by venture backed startups. You can however get products built on top of them or enterprise support from venture backed startups. Akin to companies like Red Hat in regards to Linux.
  - ThePhysicist 5y ago
    I have never used Envoy or Istio, and maybe I shouldn't believe the PR hyperbole about Tetrate which says "Today, Tetrate is one of the largest contributors to the Istio open source project and second largest to Envoy Proxy.". In the other tread people have pointed out that Google is instead the main driver behind Istio these days, which I do not find more reassuring as they mostly use Kubernetes as a marketing tool to sell cloud services (as, in my understanding, they have an entirely different internal platform that they deploy their services on). I would say the point still holds, but of course it's fine to have different opinions on this. It's just that seeing containerization startups like Docker starting to monetize their core ecosystem to the detriment of the open-source community (e.g. by limiting container pulls on Dockerhub) does not give me confidence that the same thing won't happen to the Kubernetes ecosystem in the long term.

candiddevmike 5y ago

Any extensible platform will be driven into a complex slurry of options and configurations to support every niche vendor/use case. I think the successor of kubernetes will be a simpler version of Fleet: distributed systemd. The future is edge, and edge doesn't need to integrate with every series A startup, it needs a small footprint and stability. I've been tinkering with a prototype that does this (and if anyone is interested in collaborating, my email is in my profile).

p_l 5y ago
The strength of Kubernetes (vs Mesos, OpenStack, etc.) is that you can strip it down pretty well so that you only bring the niche bits that you need - whether it's edge, or something big that needs a lot of extras. The unified API mechanisms and moving drivers out of core helped a lot there. Also, in my experience, it turns out that you can quickly find yourself needing all those extra bits if you're not Google (which implements a lot of the same features in libraries linked to programs directly and depends on the fact that they run customized everything) or Netflix (similar).

paulgb 5y ago

> already see some pretty interesting ideas around WASM I'm excited about WASM, but I don't see how it's fundamentally different when it comes to being built by venture-funded startups. Pretty much all of the interesting stuff in server-side WASM is being driven by venture-backed startups.

ridruejo 5y ago
Agree, but also there's a lot of core work been done by the industry (Intel, Google, etc.) that we all benefit from. It definitely 'feels' different than K8s and closer to the early days of the web

Thaxll 5y ago

I don't know why people keep talking about WASM, WASM is 1% of what Kubernetes does, it's an alternative to runc nothing more. It's like comparing Docker to Kubernetes.

boomskats 5y ago
I don't think they were comparing WASM to Kubernetes. In the context of the comment you replied to, I read it as WASM potentially obsoleting _containers_ as opposed to _complex orchestration platforms_. > WASM is 1% of what Kubernetes does, it's an alternative to runc nothing more It'd be interesting if you could expand on this. I don't think WASM does _any_ of what Kubernetes does.
- FooBarWidget 5y ago
  Why would WASM obsolete containers?
  - pjmlp 5y ago
    Because new generation startups are on the verge of selling WASM Application Servers....
  - boomskats 5y ago
    Hey, I didn't say I agreed with that statement - I was just pointing out what I thought they were trying to say.
  - ledgerdev 5y ago
    Of course containers will be around probably forever, just as we still have mainframes. Note that current serverless and container platforms start and manage long running instances and occasionally incur cold starts. WASM has near instant startup, making a cold start for each request feasible. I think this could be a key property/advantage that makes it preferable and more economically efficient in the long run. WASM I think has a stronger sandbox as well. Containers in a multi-tenant environment need to run inside a virtual machine. WASM can be run in multi-tenant without the need to isolate each tenant in a virtual machine, again making it more efficient in long run. edit: also recalled a relevant tweet from creator of docker https://twitter.com/solomonstre/status/1111004913222324225 https://twitter.com/solomonstre/status/1111004913222324225

yarky 5y ago

> broken enough to not be usable without buying premium support or the enterprise edition. How does kubernets do this ? I haven't hit that spot yet but I'm pretty sure I will. I guess I'll also find an open source solution to avoid switching licenses.

ghaff 5y ago

Istio came out of Google. And I'm not sure what Kubernetes has to do with OpenStack, other than early-on it was assumed you would need to run containers on top of cloud VMs because that's what the big public cloud providers did.

s-lambert 5y ago

>pretty interesting ideas around WASM I'm confused here, does WASM stand for something different besides WebAssembly because googling "kubernetes WASM' came up with nothing but WebAssembly. But I was under the impression that WebAssembly is just a virtual machine. I'm not much of a backend person but I don't see the overlap between container management (kubernetes) to a virtual machine (WASM).

ThePhysicist 5y ago
I meant more that technologies like WASM might obsolete the need for containers and thereby container orchestration platforms like Kubernetes. Of course Kubernetes might itself evolve to orchestrate WASM workloads.
- pjmlp 5y ago
  I can already see it, a zip file containing application assets, a wasm binary and a couple of manifest files with configuration metadata. Where have I seen this already?
  - callahad 5y ago
    Sure, superficially it sounds like Java. But hey, Java was really damn successful for a very long time.
    - pjmlp 5y ago
      It would help if that wasn't sold as something new never attempted before.
- kitd 5y ago
  Of course Kubernetes might itself evolve to orchestrate WASM workloads. Already been done: https://krustlet.dev/ https://krustlet.dev/
- robert_dipaolo 5y ago
  The WASM concept has been around for decades, it's effectively jvm or .net in but in the browser. I don't get the relevance when discussing k8s or containers.

nunez 5y ago

tbf if a new Linux were starting in 2022, it would absolutely be VC-funded, since the dotcom era opened that Pandora's Box and it will never close again i think the other key difference b/w nix and k8s is that a very large swath of k8s core contributors are employed by very large companies (mostly Google, Red Hat, Microsoft, VMware, and IBM) who are also deeply invested in k8s-derivatives sold in the marketplace. from what i've gathered, and this might not be true anymore, many contributions to Linux still come from the academy (universities, etc)

pjmlp 5y ago
Same as Linux, it only took off because many UNIX shops like IBM saw a value in improving it, while cutting costs on their own. > 1998: Many major companies such as IBM, Compaq and Oracle announce their support for Linux > 2000: Dell announces that it is now the No. 2 provider of Linux-based systems worldwide and the first major manufacturer to offer Linux across its full product line > 2006: Oracle releases its own distribution of Red Hat Enterprise Linux. --- https://en.m.wikipedia.org/wiki/History_of_Linux https://en.m.wikipedia.org/wiki/History_of_Linux

pjmlp 5y ago

Given the failure of most pure FOSS business, and the love for non-copyleft licenses, it is quite clear that what most businesses care about is the model that ruled the industry before Linux, with open core being the new demo versions. No wonder that as IoT took off plenty of POSIX like clones with non-copyleft licenses sprung off. Even on Android, the Linux kernel is the only GPL piece of the puzzle left standing.