5 ms·

> do you have about a million dollars to spend on building and maintaining it all? Then you need Kubernetes. I think you're overstating the investment necessar

> do you have about a million dollars to spend on building and maintaining it all? Then you need Kubernetes. I think you're overstating the investment necessary to overcome the initial complication of Kubernetes and also understating the benefit of being on a platform with a massive and thriving community behind it. As an example, in a prior role, there were a set of data engineers that would receive data in the form of MS SQL server backups from which these engineers would need to query and transform data on an exploratory rather than production basis. Certainly one could use an "undifferentiated" service from a cloud provider, but it was also a roughly 5-minute process for me to use the rather high quality helm chart and docker image commonly available to stand up a new service for the benefit of each engineer that had the need. The process of creating that automation necessary to deploy the helm chart and restore the backups took approximately one hour and could be repeated ad nauseum in the aforementioned 5-minute time period. There are many, many other examples of this. Want a data-science platform complete out of the box with no vendor lock-in, how about data8.org? The list goes on.

peterwwillis 6y ago
What happens when your car's A/C stops working? Most people think, I'll just get a little can of R134a, fill up the system, and it'll be good as new. Somebody said they did that once and it worked just fine, so it should work for you too, right? I mean, it says so right on the can, and there's YouTube videos of it and everything. The trouble is, A/C is a complex system. There are moving pieces with specialized oils that oxidize and break down over time. There's a sealed system of pressurized gas. There's a pump, clutch, coils, fans, filter, thermostat, drain, belt, and electronics, Any of those parts could fail in a number of ways. Just to inspect it you need a custom gauge set, a tank of R134a, and a vacuum pump. Etcd is about as complex as an A/C system. That is one of a dozen components of a Kubernetes system, before we get into custom integrations, which you will need about another dozen of. The million dollars is to pay for everything needed to set up and maintain all of that, create the custom integrations that do not come turn-key from the community, create the custom integrations the community doesn't even have, integrate it with your development and deployment systems, business requirements, application-specific needs, and so on. A million is an average. You can get away with less, just like you can get away with pumping a pre-pressurized A/C system with extra coolant: if you're lucky it won't break. When it does, I hope you have either a lot of time, or a lot of money to pay a consultant.
- shaklee3 6y ago
  I would strongly disagree. Etcd, for what it does, is extremely simple. What is so complex about it? The configuration?
  - peterwwillis 6y ago
    It's a distributed decentralized database using self-signed certs. Just by itself it requires maintenance: upgrading the software, upgrading the host it runs on, rotating keys, networking, access control, key space maintenance, backup, etc. Here are the docs you need to know to run it: https://etcd.io/docs/v3.4.0/op-guide/ https://etcd.io/docs/v3.4.0/op-guide/ And there's another dozen docs not written there that the admin just sort of finds out over time. But it's part of other systems too, making the overall thing a system of systems. Interactions between systems of systems are complex and cause unexpected behavior. At some point you will run into an error in K8s that you can't resolve that will require you to debug Etcd. And "Bob" help you if the database gets corrupt or overwritten, or incompatible versions of software screw up what's in the database, etc. (My original analogy was inaccurate... Etcd is more like the engine than just A/C, because if it stops working, everything stops working) Do you know what happens if an Etcd certificate's SAN field does not include domain names but only IP addresses? The client requests HelloInfo with an empty ServerName so it doesn't trigger a TLS reload on handshake, making it more difficult to replace expired certs. That is a single random quirk in a single component of this software which underpins all of Kubernetes. I cannot sit here and explain every single reason why Etcd is complex; it must suffice to say that the software just is complex, and that this reality means that while it may sometimes be simple for some people to operate, it will definitely not always be simple to operate, and there will come a time that the true cost will emerge. Now, most people don't need to pay for that high cost of complexity. They can use a SaaS/PaaS product like AWS ECS/Fargate or others, where somebody at some other company is dealing with the cost of complexity for you. All you have to do at that point is run some API calls and everything just works. Not only is it easier, it's immensely cheaper, less time-consuming, and more reliable. ...But you might not even need ECS! There's a lot of work just to get a simple PoC up on Fargate with an NLB, ACM cert, RDS instance, security groups, VPCs, cluster, service, task, etc. Compare that to just spinning up a micro instance and running MariaDB and a Python on it, and the latter you can have done in 20 minutes. If you can avoid complexity and still meet your SLOs, do that.

peterwwillis 6y ago

What happens when your car's A/C stops working? Most people think, I'll just get a little can of R134a, fill up the system, and it'll be good as new. Somebody said they did that once and it worked just fine, so it should work for you too, right? I mean, it says so right on the can, and there's YouTube videos of it and everything. The trouble is, A/C is a complex system. There are moving pieces with specialized oils that oxidize and break down over time. There's a sealed system of pressurized gas. There's a pump, clutch, coils, fans, filter, thermostat, drain, belt, and electronics, Any of those parts could fail in a number of ways. Just to inspect it you need a custom gauge set, a tank of R134a, and a vacuum pump. Etcd is about as complex as an A/C system. That is one of a dozen components of a Kubernetes system, before we get into custom integrations, which you will need about another dozen of. The million dollars is to pay for everything needed to set up and maintain all of that, create the custom integrations that do not come turn-key from the community, create the custom integrations the community doesn't even have, integrate it with your development and deployment systems, business requirements, application-specific needs, and so on. A million is an average. You can get away with less, just like you can get away with pumping a pre-pressurized A/C system with extra coolant: if you're lucky it won't break. When it does, I hope you have either a lot of time, or a lot of money to pay a consultant.

shaklee3 6y ago
I would strongly disagree. Etcd, for what it does, is extremely simple. What is so complex about it? The configuration?
- peterwwillis 6y ago
  It's a distributed decentralized database using self-signed certs. Just by itself it requires maintenance: upgrading the software, upgrading the host it runs on, rotating keys, networking, access control, key space maintenance, backup, etc. Here are the docs you need to know to run it: https://etcd.io/docs/v3.4.0/op-guide/ https://etcd.io/docs/v3.4.0/op-guide/ And there's another dozen docs not written there that the admin just sort of finds out over time. But it's part of other systems too, making the overall thing a system of systems. Interactions between systems of systems are complex and cause unexpected behavior. At some point you will run into an error in K8s that you can't resolve that will require you to debug Etcd. And "Bob" help you if the database gets corrupt or overwritten, or incompatible versions of software screw up what's in the database, etc. (My original analogy was inaccurate... Etcd is more like the engine than just A/C, because if it stops working, everything stops working) Do you know what happens if an Etcd certificate's SAN field does not include domain names but only IP addresses? The client requests HelloInfo with an empty ServerName so it doesn't trigger a TLS reload on handshake, making it more difficult to replace expired certs. That is a single random quirk in a single component of this software which underpins all of Kubernetes. I cannot sit here and explain every single reason why Etcd is complex; it must suffice to say that the software just is complex, and that this reality means that while it may sometimes be simple for some people to operate, it will definitely not always be simple to operate, and there will come a time that the true cost will emerge. Now, most people don't need to pay for that high cost of complexity. They can use a SaaS/PaaS product like AWS ECS/Fargate or others, where somebody at some other company is dealing with the cost of complexity for you. All you have to do at that point is run some API calls and everything just works. Not only is it easier, it's immensely cheaper, less time-consuming, and more reliable. ...But you might not even need ECS! There's a lot of work just to get a simple PoC up on Fargate with an NLB, ACM cert, RDS instance, security groups, VPCs, cluster, service, task, etc. Compare that to just spinning up a micro instance and running MariaDB and a Python on it, and the latter you can have done in 20 minutes. If you can avoid complexity and still meet your SLOs, do that.