3 ms·

I like the idea of "Building on Quicksand" as the analogy for Distributed Systems, but also maintaining your software dependencies. This article basically recom

I like the idea of "Building on Quicksand" as the analogy for Distributed Systems, but also maintaining your software dependencies. This article basically recommends trying to minimize your dependencies to keep reproducibility/portability high. I generally agree with this, but also carry an "all things within reason" mentality. But just as the article describes coworkers growing into their cluster, the complexity of what they run in their cluster will also grow over time and eventually they'll realize they've just built up their own "distribution". A few years ago, I've written a post asking people to think critically when they hear someone mention "Vanilla" Kubernetes[0]. The real problem they suffered is actually that Kubernetes isn't fundamentally designed for multi-tenancy. Instead, you're forced to make separate clusters to isolate different domains. Google themselves run multiple Borg clusters to isolate different domains, so it's natural that Kubernetes end up with a similar design. [0]: https://jzelinskie.com/posts/youre-not-running-vanilla-kubernetes/ https://jzelinskie.com/posts/youre-not-running-vanilla-kuber... Disclosure: I worked as an engineer and product manager on CoreOS Tectonic, the (now defunct) Kubernetes used in the post.

ffo 5y ago
Hey we used tectonic ;-) was a great tool at that time. Tectonic did influence some of the concepts around ORBOS. Just think of Tectonic combined with GitOps, minus the iPXE part. Disclaimer: I am working with ORBOS
GauntletWizard 5y ago
You're just wrong, unfortunately - Google runs dev and test and prod all on the same clusters. Kubernetes multi-tenancy works just fine, but the conventional definition of multi-tenancy includes things like "network isolation" that are misguided. Multi-tenancy should be set up (and is within Google) by understanding what is and isn't shared with the environment, and through cryptographic assertion of who you're speaking to. If you want to see the latter part nicely integrated, come to a SIGAUTH meeting and help me argue for it
- q3k 5y ago
  > If you want to see the latter part nicely integrated, come to a SIGAUTH meeting and help me argue for it. Invitation accepted! :) I've been dying to see an ALTS-like [1] thing that works with Kubernetes. I really should be able to talk encrypted and authenticated gRPC-to-gRPC without ever having to set up secrets or manually provision certificates, dammit. [1] - https://cloud.google.com/security/encryption-in-transit/application-layer-transport-security https://cloud.google.com/security/encryption-in-transit/appl...
- jzelinskie 5y ago
  All I said was that they use separate clusters to "isolate domains" which is a pretty vague description on purpose -- I did not intend to claim they do it for different deployment environments as you've described. It's fairly subjective what types of isolation define "multi-tenancy" which is why there hasn't been progress made despite SIG and WG efforts in the past. While you do not believe network isolation should be included, there are plenty of developers working on OpenShift Online which may disagree. OSO lets anyone on the internet sign up for free and instantly be given their own namespace on a shared cluster full of untrusted actors.

ffo 5y ago

Hey we used tectonic ;-) was a great tool at that time. Tectonic did influence some of the concepts around ORBOS. Just think of Tectonic combined with GitOps, minus the iPXE part. Disclaimer: I am working with ORBOS

GauntletWizard 5y ago

You're just wrong, unfortunately - Google runs dev and test and prod all on the same clusters. Kubernetes multi-tenancy works just fine, but the conventional definition of multi-tenancy includes things like "network isolation" that are misguided. Multi-tenancy should be set up (and is within Google) by understanding what is and isn't shared with the environment, and through cryptographic assertion of who you're speaking to. If you want to see the latter part nicely integrated, come to a SIGAUTH meeting and help me argue for it

q3k 5y ago
> If you want to see the latter part nicely integrated, come to a SIGAUTH meeting and help me argue for it. Invitation accepted! :) I've been dying to see an ALTS-like [1] thing that works with Kubernetes. I really should be able to talk encrypted and authenticated gRPC-to-gRPC without ever having to set up secrets or manually provision certificates, dammit. [1] - https://cloud.google.com/security/encryption-in-transit/application-layer-transport-security https://cloud.google.com/security/encryption-in-transit/appl...
jzelinskie 5y ago
All I said was that they use separate clusters to "isolate domains" which is a pretty vague description on purpose -- I did not intend to claim they do it for different deployment environments as you've described. It's fairly subjective what types of isolation define "multi-tenancy" which is why there hasn't been progress made despite SIG and WG efforts in the past. While you do not believe network isolation should be included, there are plenty of developers working on OpenShift Online which may disagree. OSO lets anyone on the internet sign up for free and instantly be given their own namespace on a shared cluster full of untrusted actors.