5 ms·

> The super perplexing thing is that cloud providers jumped on the kubernetes bandwagon so quickly and this is what made this commoditization possible. My view

> The super perplexing thing is that cloud providers jumped on the kubernetes bandwagon so quickly and this is what made this commoditization possible. My view remains that this was the purpose of Kubernetes: to scorch the earth ahead of ECS. Google had nothing to lose by doing so. > I've brought it up before, but the "Service Catalog" is basically identical to the user Operator (Controller + Custom Resource Definition) pattern, but for some reason it's being treated as different... The Service Catalog is based on the Open Service Broker API, which in turn was extracted from the Cloud Foundry Service Broker API. I hear folks say "it's not declarative", but typically this turns out to mean "it's not a YAML file". The fact is that OSBAPI is literally about telling the platform: hey, I need a service for this app. The brokerage process then works out how that's achieved. One such mechanism is Operators, there are also brokers that can apply Helm charts. The major difference is really about the CRDs as the point of interface. There's a fork or project somewhere which presents OSBAPI as CRDs (ie, YAML submitted to a remote REST endpoint) instead of as CLI commands (ie, JSON submitted to a remote REST endpoint). I feel like that's the best of both worlds. You keep the internal consistency of service discovery, service binding and service injection, plus the thing that really adds value: checking the YAML into revision control. Disclosure: I work for Pivotal. We compete with Red Hat / IBM. We also cooperate with them on OSBAPI and a number of other projects.

hardwaresofton 7y ago
> My view remains that this was the purpose of Kubernetes: to scorch the earth ahead of ECS. Google had nothing to lose by doing so. I agree, but avoided saying it to try to not seem like a crackpot. I'm fairly sure I have a comment to the same tune somewhere in the back of HN somewhere (or maybe somewhere else). Also in the sounds-like-what-a-crackpot-would-say category of thoughts -- I don't trust the CNCF. Similar to how I don't trust the OIN. > The Service Catalog is based on the Open Service Broker API, which in turn was extracted from the Cloud Foundry Service Broker API. I hear folks say "it's not declarative", but typically this turns out to mean "it's not a YAML file". The fact is that OSBAPI is literally about telling the platform: hey, I need a service for this app. The brokerage process then works out how that's achieved. One such mechanism is Operators, there are also brokers that can apply Helm charts. No comment about this, "declarative" is just tacked on onto things as marketing fluff these days. OSBAPI is plenty declarative to me; you're telling it what you want instead of how to provide what you want. > The major difference is really about the CRDs as the point of interface. There's a fork or project somewhere which presents OSBAPI as CRDs (ie, YAML submitted to a remote REST endpoint) instead of as CLI commands (ie, JSON submitted to a remote REST endpoint). I feel like that's the best of both worlds. You keep the internal consistency of service discovery, service binding and service injection, plus the thing that really adds value: checking the YAML into revision control. Though heavily simplified, do I have the difference (written below) correct? API version: 0) User wants cloud provided service X 1) User makes service catalog CLI request (a web request is made to the broker) CRD version: 0) User wants cloud provided service X 1) User creates a CRD 2) Controller sees created CRD and makes web request to the broker If this is right, it seems like a very small difference.
- jacques_chester 7y ago
  My understanding is that yes, you have a controller listening for CRDs which essentially stands in for the CLI/API step. To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum. Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to. After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!" And then OSBAPI will be reinvented (but ... declarative!) while I stand at a safe distance, teeth audibly grinding down to bone.
  - hardwaresofton 7y ago
    Sooooooo I'm about to basically go off > To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum. kubebuilder and the whole "ecosystem" for building controllers is unnecessarily hard to use and ideally shouldn't have been this way in the first place. Writing an operator is more painful than it should be, and I think many of the engineering decisions early on are what made it this way. IMO CRDs are actually the defining feature of k8s but no one realized it until later. They set out to build a cathedral when they should have been building a more orderly bazaar. > Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to. Right, but maybe this time people will solve that problem the right way -- by standardizing on the representation (CRD) in this case. I mean this in the general sense -- the case I think of is trying to capture all the different ways you can containerize something as they evolved over the years: - LXC exists (let's call this an `ubuntu.com/Container` entity) - Docker is introduced, people go wild (let's call this a `docker.com/Container` entity) - OCI container spec gets written (Let's call this an `opencontainers.org/Container` entity) - People want to just be able to make a "Container" that runs on whatever operator is available ==> They make a `foo.com/Container` that represents a the lowest common denominator This kind of progression is the only way you can keep your rapid iteration but have consistent behavior -- foo.com takes the bullet to provide and maintain consistent behavior while the other container types can iterate as fast as they desire. > After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!" This I think is a symptom is an orchestrator/operators that aren't robust enough -- operators should not be able to take out other operators, except through very well defined integration points. Hard to write the interface in practice but I think we can do better than what kubernetes currently is. I've been sitting on the concept for a kubernetes competitor that (I think) is simpler but haven't gotten around to writing it. Here are the broad strokes: - The only pattern is provider/resource (essentially what k8s knows as CRDs and operators) - Complexity increases only by composition (i.e. you want to write a `KafkaProvider`? looks like you'll need a `ContainerProvider`, `NetworkProvider`, etc) - No GRPC (good 'ol HTTP1.1, upgradable to 2/3 in the standard web-compatible ways, with "Content-Type" if you really feel like you need to squash your request/responses, and SSE if you need to stream stuff) - jsonschema + hyperschema + JSON-LD for automated everything (verification, request building, semantics parsing) - Rust - Single binary deployment (as in get your VM, maybe set some kernel flags, put a single binary on it, run that binary, and your cluster is up). I've got pages of notes on what this could be but at this point it's vaporware -- Who knows if I'll ever get to work on this idea but I sure wish someone would. I don't think there are any companies willing (essentially, dumb enough) to try to compete with Kubernetes so I don't think anyone is even considering trying to build a new but refined k8s.

hardwaresofton 7y ago

> My view remains that this was the purpose of Kubernetes: to scorch the earth ahead of ECS. Google had nothing to lose by doing so. I agree, but avoided saying it to try to not seem like a crackpot. I'm fairly sure I have a comment to the same tune somewhere in the back of HN somewhere (or maybe somewhere else). Also in the sounds-like-what-a-crackpot-would-say category of thoughts -- I don't trust the CNCF. Similar to how I don't trust the OIN. > The Service Catalog is based on the Open Service Broker API, which in turn was extracted from the Cloud Foundry Service Broker API. I hear folks say "it's not declarative", but typically this turns out to mean "it's not a YAML file". The fact is that OSBAPI is literally about telling the platform: hey, I need a service for this app. The brokerage process then works out how that's achieved. One such mechanism is Operators, there are also brokers that can apply Helm charts. No comment about this, "declarative" is just tacked on onto things as marketing fluff these days. OSBAPI is plenty declarative to me; you're telling it what you want instead of how to provide what you want. > The major difference is really about the CRDs as the point of interface. There's a fork or project somewhere which presents OSBAPI as CRDs (ie, YAML submitted to a remote REST endpoint) instead of as CLI commands (ie, JSON submitted to a remote REST endpoint). I feel like that's the best of both worlds. You keep the internal consistency of service discovery, service binding and service injection, plus the thing that really adds value: checking the YAML into revision control. Though heavily simplified, do I have the difference (written below) correct? API version: 0) User wants cloud provided service X 1) User makes service catalog CLI request (a web request is made to the broker) CRD version: 0) User wants cloud provided service X 1) User creates a CRD 2) Controller sees created CRD and makes web request to the broker If this is right, it seems like a very small difference.

jacques_chester 7y ago
My understanding is that yes, you have a controller listening for CRDs which essentially stands in for the CLI/API step. To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum. Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to. After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!" And then OSBAPI will be reinvented (but ... declarative!) while I stand at a safe distance, teeth audibly grinding down to bone.
- hardwaresofton 7y ago
  Sooooooo I'm about to basically go off > To be fair, OSBAPI is a bigger upfront effort for a service author than fiddling with kubebuilder and sorta-kinda doing some basic service-y stuff yourself. Plus CRDs are the cool thing right now and I expect they will continue to build in momentum. kubebuilder and the whole "ecosystem" for building controllers is unnecessarily hard to use and ideally shouldn't have been this way in the first place. Writing an operator is more painful than it should be, and I think many of the engineering decisions early on are what made it this way. IMO CRDs are actually the defining feature of k8s but no one realized it until later. They set out to build a cathedral when they should have been building a more orderly bazaar. > Where it will come back to bite everyone is that every service will have its own Operator and its own CRDs and these will often be different in shape, behaviour and conventions than other Operators and CRDs. It'll be like the bad old days of every piece of software shipping with its own pile of bash scripts for management and twenty different process managers. You know: the sort of thing Red Hat helped to bring an end to. Right, but maybe this time people will solve that problem the right way -- by standardizing on the representation (CRD) in this case. I mean this in the general sense -- the case I think of is trying to capture all the different ways you can containerize something as they evolved over the years: - LXC exists (let's call this an `ubuntu.com/Container` entity) - Docker is introduced, people go wild (let's call this a `docker.com/Container` entity) - OCI container spec gets written (Let's call this an `opencontainers.org/Container` entity) - People want to just be able to make a "Container" that runs on whatever operator is available ==> They make a `foo.com/Container` that represents a the lowest common denominator This kind of progression is the only way you can keep your rapid iteration but have consistent behavior -- foo.com takes the bullet to provide and maintain consistent behavior while the other container types can iterate as fast as they desire. > After we reach the point where you're spinning up whole clusters because the PostgreSQL operator doesn't play nice with the Kafka operator on Tuesdays when the breeze is too strong, someone wise and brave and famous on twitter will yell "this is insanity! Surely we can rationalise this into a single API, a single mechanism!" This I think is a symptom is an orchestrator/operators that aren't robust enough -- operators should not be able to take out other operators, except through very well defined integration points. Hard to write the interface in practice but I think we can do better than what kubernetes currently is. I've been sitting on the concept for a kubernetes competitor that (I think) is simpler but haven't gotten around to writing it. Here are the broad strokes: - The only pattern is provider/resource (essentially what k8s knows as CRDs and operators) - Complexity increases only by composition (i.e. you want to write a `KafkaProvider`? looks like you'll need a `ContainerProvider`, `NetworkProvider`, etc) - No GRPC (good 'ol HTTP1.1, upgradable to 2/3 in the standard web-compatible ways, with "Content-Type" if you really feel like you need to squash your request/responses, and SSE if you need to stream stuff) - jsonschema + hyperschema + JSON-LD for automated everything (verification, request building, semantics parsing) - Rust - Single binary deployment (as in get your VM, maybe set some kernel flags, put a single binary on it, run that binary, and your cluster is up). I've got pages of notes on what this could be but at this point it's vaporware -- Who knows if I'll ever get to work on this idea but I sure wish someone would. I don't think there are any companies willing (essentially, dumb enough) to try to compete with Kubernetes so I don't think anyone is even considering trying to build a new but refined k8s.