5 ms·

Most large orgs I'm looking at pretty much just have guys logging into tin, on a "secure" network, configuring things based on written instructions. Is that be

Most large orgs I'm looking at pretty much just have guys logging into tin, on a "secure" network, configuring things based on written instructions. Is that better than k8s? Is is VMware? Is it Ansible, Puppet or Chef? How are we dealing with node failures, are you paging and waking people up or just running everything in super HA? How are you making good useage of your compute, VMs? What is your better system?

cookiecaper 7y ago
>Is i[t] VMware? Is it Ansible, Puppet or Chef? The funny thing being that all of these things are applicable in a k8s-backed environment anyway. k8s is a container orchestrator and scheduler. You still need a platform (VMWare, AWS, GCloud) and configuration management separately. So, I guess the answer is "yes"? >How are we dealing with node failures First, in an ordinary system, "node failures" are rare. This thing that k8s encourages where the first response to a problem is "just kill the pod and hope it comes back OK on its own" is distasteful in many ways, and only further cements why MS-centric ops people are gung-ho. In my day, when there's a problem with a system, you divert traffic, take snapshot, analyze to figure out the reason it failed, and then prevent failures. Even in k8s-world, failures cost something, and it's bad practice to allow them to occur regularly in production (note that making "just reboot^H^H^H^Hschedule it" a routine is separate from being able to tolerate failures transparently). Second, you use the same systems that k8s ingresses use internally: load balancers with health checks like HAPROXY, nginx, Envoy, etc. > How are you making good useage of your compute, VMs? First, not dedicating a bunch of it to redundant and unnecessary distributed systems mechanisms. k8s is not cheap or easy to run. Second, there are robust toolkits and monitoring options for managing all kinds of workloads, including controlling scale. Depending on your platform and use case, there are a plethora of options. These problems are not new. The crux of the issue is that k8s is misunderstood as a generalized toolkit for every common-but-not-trivial operational task, because that's how Google promotes it to maximize adoption and thus wedge themselves into a position of more control v. AWS. k8s is a complex multi-node container orchestrator and scheduler. If you didn't need one of those before you knew what k8s was, you probably don't need one now.
- ownagefool 7y ago
  You didn't really answer the question and honestly I don't think you entirely understand the proposition, but it doesn't really matter... Honestly, I disagree with you. Putting k8s on tin is a fairly nice system that gives you much of the power of IaaS providers that you potentially don't need anymore. It makes it easy to create logical environments, allows packing of compute, and gives you standard APIs to develop against, allowing you to move providers, share platforms, etc. Now I do agree, you shouldn't generally have services that restart constantly, but some of the biggest headaches I've came across is sysadmins and that assumed their systems will just work, and now they don't and it's a clusterfuck. With regards to cheap, it doesn't cost any extra to get k8s as a managed service by google. It's fairly cheap to use on DO, and if you need to run it on tin, it's a damn sight easier than bootstrapping the underlying IaaS in my opinion, so frankly I don't think your rant represents reality. Theres a lot of hackers in the industry that are new to running systems. I'd rather they use a framework than build bespoke, but that's me. And thus I ask again, what's better?
  - cookiecaper 7y ago
    > And thus I ask again, what's better? I mean, if you're asking me to prescribe a one-size-fits-all infrastructure solution for any problem, that's not a thing. The answer is "it depends". What's better is to understand the mechanisms at work and use what you need to get a robust, reliable solution. If I were to prescribe a general infrastructure platform for your generic web app, the high-level would be a) use something for infrastructure-as-code to define the resources needed, so they can be rebuilt/spawned in new environments on demand; b) use something for config management and image construction, ideally something like NixOS that has reproduceability and dependency encapsulation as a fundamental part of the OS; c) use a production-grade load balancer like haproxy or Envoy and configure it properly for the infrastructure that it sits on top of; d) use a production-grade web server to serve requests, which is probably either Apache or nginx. Note that just saying "boom k8s" doesn't really resolve these concerns. k8s schedules, routes, and executes arbitrary containers, which have usually been built by developers who don't know what they're doing, and which are likely to contain tons of random outdated junk as a result of cascading FROMs in the Dockerfile and stray files in the developer and/or CI pipeline's build context (which are frequently sensitive btw). Container images should be just like any other image running in prod: constructed by competent admins and controlled with an appropriate configuration and patch management mechanism. The fact that applying those changes executes an entirely new image instead of just restarting a service is really an implementation detail, it's not a solution to anything. Your chosen k8s ingress is probably using nginx, haproxy, or Envoy under the covers, and you have to tune that either way, whether you call that a livenessProbe or a health check. There's nothing fundamentally better about the k8s YAML for this than the actual configuration file (and indeed, in the early days of k8s, I was hacking through the alpha-stage ingress plugin and editing haproxy configs by hand anyway), though I suppose if you have a use case where half of your applications need one load balancer and the other half need another, you may get some benefit here. That's pretty rare, though. If you ever do hit appreciable load, you may find that your container's `npm start` server has some inadequacies for which "omg pay $CLOUDPROVIDER more money and spin more pods" may not be an adequate solution. And lastly, k8s doesn't enter the picture until you have something to run it on so it doesn't do anything for your infrastructure-as-code problem, even if the answer to that is just "rent a Kube cluster from Google" -- something still has to actually go rent that, and it should be scripted. So I mean, yeah, if you like Kubernetes because Helm charts are convenient, by all means go ahead and run Kubernetes. Just don't pretend like that magically solves the problems involved in a robust architecture, especially if you ever expect to tune or profile your systems.

cookiecaper 7y ago

>Is i[t] VMware? Is it Ansible, Puppet or Chef? The funny thing being that all of these things are applicable in a k8s-backed environment anyway. k8s is a container orchestrator and scheduler. You still need a platform (VMWare, AWS, GCloud) and configuration management separately. So, I guess the answer is "yes"? >How are we dealing with node failures First, in an ordinary system, "node failures" are rare. This thing that k8s encourages where the first response to a problem is "just kill the pod and hope it comes back OK on its own" is distasteful in many ways, and only further cements why MS-centric ops people are gung-ho. In my day, when there's a problem with a system, you divert traffic, take snapshot, analyze to figure out the reason it failed, and then prevent failures. Even in k8s-world, failures cost something, and it's bad practice to allow them to occur regularly in production (note that making "just reboot^H^H^H^Hschedule it" a routine is separate from being able to tolerate failures transparently). Second, you use the same systems that k8s ingresses use internally: load balancers with health checks like HAPROXY, nginx, Envoy, etc. > How are you making good useage of your compute, VMs? First, not dedicating a bunch of it to redundant and unnecessary distributed systems mechanisms. k8s is not cheap or easy to run. Second, there are robust toolkits and monitoring options for managing all kinds of workloads, including controlling scale. Depending on your platform and use case, there are a plethora of options. These problems are not new. The crux of the issue is that k8s is misunderstood as a generalized toolkit for every common-but-not-trivial operational task, because that's how Google promotes it to maximize adoption and thus wedge themselves into a position of more control v. AWS. k8s is a complex multi-node container orchestrator and scheduler. If you didn't need one of those before you knew what k8s was, you probably don't need one now.

ownagefool 7y ago
You didn't really answer the question and honestly I don't think you entirely understand the proposition, but it doesn't really matter... Honestly, I disagree with you. Putting k8s on tin is a fairly nice system that gives you much of the power of IaaS providers that you potentially don't need anymore. It makes it easy to create logical environments, allows packing of compute, and gives you standard APIs to develop against, allowing you to move providers, share platforms, etc. Now I do agree, you shouldn't generally have services that restart constantly, but some of the biggest headaches I've came across is sysadmins and that assumed their systems will just work, and now they don't and it's a clusterfuck. With regards to cheap, it doesn't cost any extra to get k8s as a managed service by google. It's fairly cheap to use on DO, and if you need to run it on tin, it's a damn sight easier than bootstrapping the underlying IaaS in my opinion, so frankly I don't think your rant represents reality. Theres a lot of hackers in the industry that are new to running systems. I'd rather they use a framework than build bespoke, but that's me. And thus I ask again, what's better?
- cookiecaper 7y ago
  > And thus I ask again, what's better? I mean, if you're asking me to prescribe a one-size-fits-all infrastructure solution for any problem, that's not a thing. The answer is "it depends". What's better is to understand the mechanisms at work and use what you need to get a robust, reliable solution. If I were to prescribe a general infrastructure platform for your generic web app, the high-level would be a) use something for infrastructure-as-code to define the resources needed, so they can be rebuilt/spawned in new environments on demand; b) use something for config management and image construction, ideally something like NixOS that has reproduceability and dependency encapsulation as a fundamental part of the OS; c) use a production-grade load balancer like haproxy or Envoy and configure it properly for the infrastructure that it sits on top of; d) use a production-grade web server to serve requests, which is probably either Apache or nginx. Note that just saying "boom k8s" doesn't really resolve these concerns. k8s schedules, routes, and executes arbitrary containers, which have usually been built by developers who don't know what they're doing, and which are likely to contain tons of random outdated junk as a result of cascading FROMs in the Dockerfile and stray files in the developer and/or CI pipeline's build context (which are frequently sensitive btw). Container images should be just like any other image running in prod: constructed by competent admins and controlled with an appropriate configuration and patch management mechanism. The fact that applying those changes executes an entirely new image instead of just restarting a service is really an implementation detail, it's not a solution to anything. Your chosen k8s ingress is probably using nginx, haproxy, or Envoy under the covers, and you have to tune that either way, whether you call that a livenessProbe or a health check. There's nothing fundamentally better about the k8s YAML for this than the actual configuration file (and indeed, in the early days of k8s, I was hacking through the alpha-stage ingress plugin and editing haproxy configs by hand anyway), though I suppose if you have a use case where half of your applications need one load balancer and the other half need another, you may get some benefit here. That's pretty rare, though. If you ever do hit appreciable load, you may find that your container's `npm start` server has some inadequacies for which "omg pay $CLOUDPROVIDER more money and spin more pods" may not be an adequate solution. And lastly, k8s doesn't enter the picture until you have something to run it on so it doesn't do anything for your infrastructure-as-code problem, even if the answer to that is just "rent a Kube cluster from Google" -- something still has to actually go rent that, and it should be scripted. So I mean, yeah, if you like Kubernetes because Helm charts are convenient, by all means go ahead and run Kubernetes. Just don't pretend like that magically solves the problems involved in a robust architecture, especially if you ever expect to tune or profile your systems.