7 ms·

How do you implement healthcheck? Does the loadbalancer know how the healthceck is implemented? How do you determine it's time to scale? How do you implement

by 0xedd 4y ago

How do you implement healthcheck? Does the loadbalancer know how the healthceck is implemented? How do you determine it's time to scale? How do you implement always-on-process? service unit, initd, cron? How do you export the logs? How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? What about secrets? Service discovery? Is unbound/bind9? And on and on and on and on. These items are best done in a standard way. That's a major selling point in k8s. You define and implement these in a consistent way that also integrates seamlessly with other tools. Doubly so on cloud. Less reinventing the wheel means less money. The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview.

nurettin 4y ago
Set up large baremetal so that you don't have to scale for a long time and by the time you do, you have had your first million users and you can afford spending for inanely complex infra and don't care about aws billing anymore. Set up user services with systemctl, access your logs through journalctl, change your configs with rsync. And op was arguing against the cloud so giving cloud examples is beside the point. No need for all the sass.
- jb_gericke 4y ago
  And when your one big server fails and your first million customers start looking for more stable alternatives?
  - nurettin 4y ago
    Then you have a few hours of downtime. Happens to the best of us.
  - v3ss0n 4y ago
    One big server ( 6 Physical Core Xeon with 64GB of Ram , 1TB Nvme SSD) only cost 80$ a month . Can start with 2-3 big servers across 3 different DC. Equlivant specs would cost easily 300$ a month on AWS/GCP/Azure .
  - thejosh 4y ago
    Because all the cloud providers never have downtime, DNS issues, etc? :) At least when US east is down, half the internet is down.
  - RaitoBezarius 4y ago
    Then, you had disaster recovery plans, you devised a strategy of what is an acceptable data loss, execute the reconstruction plan on the new big server, under some hours, the new big server is up again. You apologize, offer some financial compensation, etc. And you learn from the situation, understand what are your new tolerance for data loss, failures, etc., then start new plans accordingly.
yashap 4y ago
100% this. Pre-K8s I’ve done EC2 auto-scaling with ansible and packer, our own in-house healthcheck framework (https://github.com/hootsuite/health-checks-api https://github.com/hootsuite/health-checks-api), our own rolling deploys, our own in-house service discovery (https://medium.com/hootsuite-engineering/how-our-service-mesh-eased-migrating-from-mesos-to-kubernetes-19a1a7f6d853 https://medium.com/hootsuite-engineering/how-our-service-mes...), our own in-house udp logging to an ELK stack (https://github.com/hootsuite/udp-logger https://github.com/hootsuite/udp-logger), our own init scripts, etc. It was a lot more complexity for us to manage than K8s, far more bugs/issues, far less uniform across teams/services. We switched to mesos, then K8s, and K8s was the simplest by far. My new company also uses K8s, and it’s also overall easier to manage with fewer issues than any alternative I’ve used. I think if you’re a small org with just a few devs and one or a handful of services, sure, K8s is overkill. But for a decent sized dev org, it’s probably way better and significantly simpler than what you’d rollout without it.
jb_gericke 4y ago
Couldn't agree more here. The amount of complexity involved in running moderately complex apps at scale on native compute and the cost to maintain over time vs. the ops savings from the standardization of most of that from Kubernetes is incomparable
- lstodd 4y ago
  I only can advise to try running more demanding apps. After all that's where the fun is. You will more-or-less instantly switch camps.
  - robertlagrant 4y ago
    I think you misread previous comment.
avl999 4y ago
> How do you implement healthcheck? > Does the loadbalancer know how the healthceck is implemented? Write a ping/sping endpoint like people did pre-Kubernetes? LBs provided by Cloud provider all have a trivial way of setting a healthcheck endpoint. If you are not in the cloud, nginx and HAProxy make it equally simple to configure a healthcheck. There is no k8s secret sauce behind healtchecks. > How do you determine it's time to scale? Monitor relevant metrics (CPU/Memory/DB), have alerts and add new vms or whatever when they hit a specified threshold. Or use your cloud provider's "elastic scaling". > How do you implement always-on-process? service unit, initd, cron? Yes. And the problem with that is? Processmanagers have been around forever, are simple to use and a well understood technology. > How do you export the logs? Cloud providers generally have a built in mechanism for pumping logs out, if you are not a cloud provider log export daemons exist that much simpler to work than running k8s deployment. > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? Do whatever your organizations feels comfortable with... config files, json, .properties files, env vars whatever, it doesn't really matter, since the code needs to be aware of the configuration mechanism anyways it doesn't make a difference. And again not worth the tradeoff of running full blown k8s deployment with its significant complexities > What about secrets? Cloud providers have convenient ways of solving this problem without involving k8s. If you are not on the cloud one can use cloud secret solutions (AWS Secret Manager, Google Secret Manager, Cloud Vault). All of which are much simpler to use than k8s secrets > Service discovery? Is unbound/bind9? DNS works well enough for most projects. Static configuration which lists domain names of whatever services you depend on (which are behind a load balancer) likely works for 90% of all applications. I am not trying to be argumentative just making a point that "standardizing" all these things you mentioned "the Kubernetes way" are not worth the significant complexity of running and managing k8s for most organizations. And certainly, non-k8s ways of doing these things are much simpler to reason about for developers. A team of 2-3 developers can handle of these things and be able to keep the entire state of their system in there head, whereas with k8s it is suicide to have a k8s cluster without having a dedicated devops team. And I disagree that there needs to be an industry wide standard of doing these things in the first place (and certainly not a standard like k8s that adds mountains of complexity). The standardization if needed, needs to happen at a team or organization level. Individual organizations should be free to pick and choose how they solve these problems rather than be pressured into the complexity of k8s.
sofixa 4y ago
Disclaimer: I work at Hashicorp but I have been saying what I'm about to say for years before joining (check my blog, linked in bio, for reference). There is a way to get all of those necessary things without having the massive complexity of k8s - use Nomad, a vastly simpler orchestrator that does many things better than k8s - e.g. integrated templating instead of go templates over yaml with Helm or other similar abominations; not using yaml for configuration; self-contained configuration and handling of updates (e.g. if your ConfigMap or Secret in k8s change, you have to manually restart/reload your pods; with Nomad you can configure it to handle that directly). It doesn't have nearly the same ecosystem, but there's enough of it through CSI, CNI, various load balancer and autoscaling integrations that you get very far with a production environment.
- v3ss0n 4y ago
  But Only problem of nomad is all examples are running on Vagrant which is designed for VMs . Everyone who is looking these days are containerization. There may be non-standard ways to run Nomad with containers but , putting Vagrant as main example is not a good idea.
  - zozbot234 4y ago
    Vagrant supports containers since v1.6 (released in 2014).
    - v3ss0n 4y ago
      yeah but examples uses vagrant vm, i am just checking nomad and use of vm discourages me / not sure how much compatible if i to placed them with nomad/lxd/dockers. Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.
  - sofixa 4y ago
    I'm not sure i follow. Nomad is an orchestrator which handles containers and other things (firecracker microVMs, LXC, etc.). The Vagrant examples are to get started and test it out, because running a container orchestrator in containers has some non-obvious pitfalls. You can just as well download the binary anywhere to play with it. That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).
mnahkies 4y ago
> How do you determine it's time to scale? This part is still really hard for a lot of workloads. We have some applications that have a really long startup time (5-10minutes) and also have on the whole predicable traffic patterns. It'd be great to be able to tell k8s to scale up/down based on time of day in addition to the usual metrics without having to resort to a hacky cronjob to adjust the min replicas. Sure it won't help with a unexpected traffic spike, and it would be better if the applications could come online faster (vendor app out of our control) but most days this feature would allow us to right size the workload way easier. There's been a few issues like https://github.com/kubernetes/kubernetes/issues/49931 https://github.com/kubernetes/kubernetes/issues/49931 but it doesn't seem to be going anywhere right now
- hardwaresofton 4y ago
  Maybe consider using KEDA[0]? [0]: https://keda.sh/ https://keda.sh/
  - mnahkies 4y ago
    That actually looks great - I could definitely make use of scaling based on message queue depths as well. Will have to look into it further, thanks!
zozbot234 4y ago
> These items are best done in a standard way. k8s is not a "standard" in any way. It is clunky, prototype-level code that has never been engineered for elegance or ease of use. It's pretty much the same problem systemd has, except to an even greater extent (because clustering is inherently a harder problem than tweaking a single system to enable the use cases systemd devs care about).
- delusional 4y ago
  Anything people agree on is a standard. It may be a horrible standard, but it's still a standard.
- bartvk 4y ago
  > It is clunky, prototype-level code that has never been engineered for elegance or ease of use How did you come to this conclusion? And what did you compare to? I'm not a devops so I don't have experience with Kubernetes.
KronisLV 4y ago
Nomad, Docker Swarm and other solutions support most of these out of the box, Kubernetes is just the most popular and flexible (with which comes a lot of complexity) solution, it seems. For example, even something as basic as Docker Swarm will see you a lot of the way through. > How do you implement healthcheck? Supported by Docker: https://docs.docker.com/engine/reference/builder/#healthcheck https://docs.docker.com/engine/reference/builder/#healthchec... > Does the loadbalancer know how the healthceck is implemented? When the health checks pass in accordance with the above config, the container state will change from "starting" to "healthy" and traffic will be able to be routed to it. Until then you can have a web server or whatever show a different page/implement circuit breaking or whatever. > How do you determine it's time to scale? Docker Swarm doesn't have an abstraction for autoscaling, though there are a few community projects. One can feasibly even write something like that themselves in an evening: https://github.com/jcwimer/docker-swarm-autoscaler https://github.com/jcwimer/docker-swarm-autoscaler That said, I mostly ignore this concern because I'm yet to see a workload that needs to dynamically scale in any number of private or government projects that I've worked on. Most of the time people want predictable infrastructure and being able to deal with backpressure (e.g. message queue), though that's different with startups. > How do you implement always-on-process? service unit, initd, cron? The service abstraction comes out of the box: https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/ https://docs.docker.com/engine/swarm/how-swarm-mode-works/se... You might also want to decide how to best schedule it: wherever available, on a particular node (hostname/tag/...) or on all nodes, which is actually what Portainer agent does! Example: https://docs.portainer.io/start/install/server/swarm/linux https://docs.portainer.io/start/install/server/swarm/linux > How do you export the logs? Docker supports multiple logging drivers: https://docs.docker.com/config/containers/logging/configure/ https://docs.docker.com/config/containers/logging/configure/ > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? Docker and Compose/Swarm support environment variables: https://docs.docker.com/compose/compose-file/#environment https://docs.docker.com/compose/compose-file/#environment If you need config files, you can also use bind mounts: https://docs.docker.com/storage/bind-mounts/ https://docs.docker.com/storage/bind-mounts/ > What about secrets? Docker supports secrets out of the box: https://docs.docker.com/engine/swarm/secrets/ https://docs.docker.com/engine/swarm/secrets/ > Service discovery? Is unbound/bind9? Docker Swarm supports built in DNS, even allows for multiple separate networks: https://docs.docker.com/engine/swarm/networking/ https://docs.docker.com/engine/swarm/networking/ > These items are best done in a standard way. Agreed! Though I'd say that the only two options being "running everything on *nix directly" and "running everything in Kubernetes" is a false narrative! The former can work but can also lead to non-standard and error-prone environments with a horrible waste of human resources, whereas the latter can work but can also lead to overcomplicated and hard to debug environments with a horrible waste of human resources. The best path for many folk probably lies somewhere in the middle, with Nomad/Swarm/Compose/Docker, regardless of what others might claim. The best path for folks interested in a DevOps career is probably running on cloud managed Kubernetes clusters and just using their APIs to lots of great results, not caring about how expensive that is or how easy it would be to self-host on-prem.
RaitoBezarius 4y ago
I'm tired to read this as if it was impossible. > How do you implement healthcheck? Most applications implement a simple endpoint which returns a known answer (whether it's an HTTP 200 code, a banner, etc.) You can just hit this endpoint using your favorite tool. > Does the loadbalancer know how the healthceck is implemented? Well, a lot of tooling supports native (periodic) healthchecks, e.g. NGINX or NGINX Plus (if you want to pay something else than managing K8S). > How do you determine it's time to scale? It is not that expensive to run performance monitoring such as netdata, sometimes, it is just a call to your distribution package manager, put it in your intranet, then you're done. > How do you implement always-on-process? service unit, initd, cron? Depending on what you're using in Linux distributions, why bother with initd and cron if you're running on a systemd-based distribution? Just use unit, timers, service, scope, slice and so on. > How do you export the logs? Again, if you're using a systemd-based distribution: `man systemd-journald-remote`. Otherwise, Loki, rsyslog, many solutions do exist for these scenarios and do not require that much fancy configuration. For mono-nodes, I would argue that the null logger exporter, called: "rsync your logs somewhere", is quite efficient. > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? I do not even know why people worry about that. If you're using a systemd-based distribution, your configuration should live more or less in the service unit using Environment, EnvironmentFile and LoadCredential(Encrypted), if your app want some files, sure, put it in /etc/xxx or /var/lib/xxx/config. As long as you do separate configuration from state (and use StateDirectory, etc.) and you do backup them, it is okay. > What about secrets? As said earlier, LoadCredential do the job, if you want to be fancy, you can run a UNIX socket server to distribute the secrets and some secret engine such as HashiCorp Vault. Uncertain that K8S has a better story than that. > Service discovery? Is unbound/bind9? Uncertain you actually need service discovery in early stages. Assuming you need, you could run envoy if you have advanced needs. Or you could run PowerDNS/nsd/unbound/BIND9 as you said with a thin wrapper to register your services. > And on and on and on and on. Well, I do not agree, those are items you choose. Not everyone has these requirements, and not everyone needs to solve them the k8s way. > These items are best done in a standard way. That's a major selling point in k8s. You define and implement these in a consistent way that also integrates seamlessly with other tools. Doubly so on cloud. Less reinventing the wheel means less money. The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview. That is a fallacy, k8s is not a standard way, they push for their standard way and it is in no case what everyone is doing. Defining and implementing thing in a "consistent way" is something we have been doing for a long time before k8s, through standardization process with IETF and W3C for example. k8s creates its own world, and it is indeed compatible with the external world sometimes, but I do not believe that a pile of complexity on the top of YAML files and YAML manipulation tooling is a "consistent way" to define and implement things, no offense here. > Doubly so on cloud. Yeah, because the "cloud ecosystem" is pushing for this, but it has not been a standard. > Less reinventing the wheel means less money. Using a "simple feature" is not reinventing the wheel, but if we go this way, I have a challenger: NixOS. Just reuse their expert modules database and do your classical YAML stuff, but this time with a simple functional programming language. > The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview. I think GP was discussing of an early stage and I do not think that all teams require to push at a very fast rate, there is definitely time for packaging and shipping it. BTW, in these organizations where I have worked, they adopted k8s and these "teams" could not use it because it was too complicated and wondered where their packaging team has gone. There is no silver bullet and choosing a deployment platform stack is not a figured out science, but there is a lot of value of choosing composable and simple tools before moving to more integrated ecosystems. > Take this into consideration before your next interview. That is an amazing level of arrogance, I only hope you may accept there are alternatives. :)
osigurdson 4y ago
I think behind most anti-k8s comments is the thought that these things are not needed for "your app". "Your app" is used sporadically by a very small number of people and they don't care if it is down for up to a day.
- lolinder 4y ago
  The "your" part of that is the problem. Other people are not in a position to know what my requirements are, so telling me what my app does or doesn't need comes off as arrogant. The anti-k8s crowd always talks as though the only reason anyone would choose k8s is that they're brainwashed by the hype. It's never possible that a team actually weighed all of their constraints and made a rational choice.
  - thunky 4y ago
    > The anti-k8s crowd always talks as though the only reason anyone would choose k8s is that they're brainwashed by the hype Maybe, but the pro-k8s camp always assumes everyone is building microservices or web services that need to scale.
    - p_l 4y ago
      If all you want to do is to scale, you can just use auto scaling with prebaked VM images on the cloud or whatever. K8s really shines when what you want to do is to optimize usage of your resources (I'd argue especially when running on bare metal)
ploppyploppy 4y ago
Finally a sane response on a HN thread about k8s.

by 0xedd 4y ago

nurettin 4y ago

Set up large baremetal so that you don't have to scale for a long time and by the time you do, you have had your first million users and you can afford spending for inanely complex infra and don't care about aws billing anymore. Set up user services with systemctl, access your logs through journalctl, change your configs with rsync. And op was arguing against the cloud so giving cloud examples is beside the point. No need for all the sass.

jb_gericke 4y ago
And when your one big server fails and your first million customers start looking for more stable alternatives?
- nurettin 4y ago
  Then you have a few hours of downtime. Happens to the best of us.
- v3ss0n 4y ago
  One big server ( 6 Physical Core Xeon with 64GB of Ram , 1TB Nvme SSD) only cost 80$ a month . Can start with 2-3 big servers across 3 different DC. Equlivant specs would cost easily 300$ a month on AWS/GCP/Azure .
- thejosh 4y ago
  Because all the cloud providers never have downtime, DNS issues, etc? :) At least when US east is down, half the internet is down.
- RaitoBezarius 4y ago
  Then, you had disaster recovery plans, you devised a strategy of what is an acceptable data loss, execute the reconstruction plan on the new big server, under some hours, the new big server is up again. You apologize, offer some financial compensation, etc. And you learn from the situation, understand what are your new tolerance for data loss, failures, etc., then start new plans accordingly.

yashap 4y ago

100% this. Pre-K8s I’ve done EC2 auto-scaling with ansible and packer, our own in-house healthcheck framework (https://github.com/hootsuite/health-checks-api https://github.com/hootsuite/health-checks-api), our own rolling deploys, our own in-house service discovery (https://medium.com/hootsuite-engineering/how-our-service-mesh-eased-migrating-from-mesos-to-kubernetes-19a1a7f6d853 https://medium.com/hootsuite-engineering/how-our-service-mes...), our own in-house udp logging to an ELK stack (https://github.com/hootsuite/udp-logger https://github.com/hootsuite/udp-logger), our own init scripts, etc. It was a lot more complexity for us to manage than K8s, far more bugs/issues, far less uniform across teams/services. We switched to mesos, then K8s, and K8s was the simplest by far. My new company also uses K8s, and it’s also overall easier to manage with fewer issues than any alternative I’ve used. I think if you’re a small org with just a few devs and one or a handful of services, sure, K8s is overkill. But for a decent sized dev org, it’s probably way better and significantly simpler than what you’d rollout without it.

jb_gericke 4y ago

Couldn't agree more here. The amount of complexity involved in running moderately complex apps at scale on native compute and the cost to maintain over time vs. the ops savings from the standardization of most of that from Kubernetes is incomparable

lstodd 4y ago
I only can advise to try running more demanding apps. After all that's where the fun is. You will more-or-less instantly switch camps.
- robertlagrant 4y ago
  I think you misread previous comment.

avl999 4y ago

> How do you implement healthcheck? > Does the loadbalancer know how the healthceck is implemented? Write a ping/sping endpoint like people did pre-Kubernetes? LBs provided by Cloud provider all have a trivial way of setting a healthcheck endpoint. If you are not in the cloud, nginx and HAProxy make it equally simple to configure a healthcheck. There is no k8s secret sauce behind healtchecks. > How do you determine it's time to scale? Monitor relevant metrics (CPU/Memory/DB), have alerts and add new vms or whatever when they hit a specified threshold. Or use your cloud provider's "elastic scaling". > How do you implement always-on-process? service unit, initd, cron? Yes. And the problem with that is? Processmanagers have been around forever, are simple to use and a well understood technology. > How do you export the logs? Cloud providers generally have a built in mechanism for pumping logs out, if you are not a cloud provider log export daemons exist that much simpler to work than running k8s deployment. > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? Do whatever your organizations feels comfortable with... config files, json, .properties files, env vars whatever, it doesn't really matter, since the code needs to be aware of the configuration mechanism anyways it doesn't make a difference. And again not worth the tradeoff of running full blown k8s deployment with its significant complexities > What about secrets? Cloud providers have convenient ways of solving this problem without involving k8s. If you are not on the cloud one can use cloud secret solutions (AWS Secret Manager, Google Secret Manager, Cloud Vault). All of which are much simpler to use than k8s secrets > Service discovery? Is unbound/bind9? DNS works well enough for most projects. Static configuration which lists domain names of whatever services you depend on (which are behind a load balancer) likely works for 90% of all applications. I am not trying to be argumentative just making a point that "standardizing" all these things you mentioned "the Kubernetes way" are not worth the significant complexity of running and managing k8s for most organizations. And certainly, non-k8s ways of doing these things are much simpler to reason about for developers. A team of 2-3 developers can handle of these things and be able to keep the entire state of their system in there head, whereas with k8s it is suicide to have a k8s cluster without having a dedicated devops team. And I disagree that there needs to be an industry wide standard of doing these things in the first place (and certainly not a standard like k8s that adds mountains of complexity). The standardization if needed, needs to happen at a team or organization level. Individual organizations should be free to pick and choose how they solve these problems rather than be pressured into the complexity of k8s.

sofixa 4y ago

Disclaimer: I work at Hashicorp but I have been saying what I'm about to say for years before joining (check my blog, linked in bio, for reference). There is a way to get all of those necessary things without having the massive complexity of k8s - use Nomad, a vastly simpler orchestrator that does many things better than k8s - e.g. integrated templating instead of go templates over yaml with Helm or other similar abominations; not using yaml for configuration; self-contained configuration and handling of updates (e.g. if your ConfigMap or Secret in k8s change, you have to manually restart/reload your pods; with Nomad you can configure it to handle that directly). It doesn't have nearly the same ecosystem, but there's enough of it through CSI, CNI, various load balancer and autoscaling integrations that you get very far with a production environment.

v3ss0n 4y ago
But Only problem of nomad is all examples are running on Vagrant which is designed for VMs . Everyone who is looking these days are containerization. There may be non-standard ways to run Nomad with containers but , putting Vagrant as main example is not a good idea.
- zozbot234 4y ago
  Vagrant supports containers since v1.6 (released in 2014).
  - v3ss0n 4y ago
    yeah but examples uses vagrant vm, i am just checking nomad and use of vm discourages me / not sure how much compatible if i to placed them with nomad/lxd/dockers. Any good tutorials with Noman + Containers would be nice for CI/CD/Executor setup.
- sofixa 4y ago
  I'm not sure i follow. Nomad is an orchestrator which handles containers and other things (firecracker microVMs, LXC, etc.). The Vagrant examples are to get started and test it out, because running a container orchestrator in containers has some non-obvious pitfalls. You can just as well download the binary anywhere to play with it. That doesn't mean Nomad can't orchestrate containers (in fact that's the main use case for it).

mnahkies 4y ago

> How do you determine it's time to scale? This part is still really hard for a lot of workloads. We have some applications that have a really long startup time (5-10minutes) and also have on the whole predicable traffic patterns. It'd be great to be able to tell k8s to scale up/down based on time of day in addition to the usual metrics without having to resort to a hacky cronjob to adjust the min replicas. Sure it won't help with a unexpected traffic spike, and it would be better if the applications could come online faster (vendor app out of our control) but most days this feature would allow us to right size the workload way easier. There's been a few issues like https://github.com/kubernetes/kubernetes/issues/49931 https://github.com/kubernetes/kubernetes/issues/49931 but it doesn't seem to be going anywhere right now

hardwaresofton 4y ago
Maybe consider using KEDA[0]? [0]: https://keda.sh/ https://keda.sh/
- mnahkies 4y ago
  That actually looks great - I could definitely make use of scaling based on message queue depths as well. Will have to look into it further, thanks!

zozbot234 4y ago

> These items are best done in a standard way. k8s is not a "standard" in any way. It is clunky, prototype-level code that has never been engineered for elegance or ease of use. It's pretty much the same problem systemd has, except to an even greater extent (because clustering is inherently a harder problem than tweaking a single system to enable the use cases systemd devs care about).

delusional 4y ago
Anything people agree on is a standard. It may be a horrible standard, but it's still a standard.
bartvk 4y ago
> It is clunky, prototype-level code that has never been engineered for elegance or ease of use How did you come to this conclusion? And what did you compare to? I'm not a devops so I don't have experience with Kubernetes.

KronisLV 4y ago

Nomad, Docker Swarm and other solutions support most of these out of the box, Kubernetes is just the most popular and flexible (with which comes a lot of complexity) solution, it seems. For example, even something as basic as Docker Swarm will see you a lot of the way through. > How do you implement healthcheck? Supported by Docker: https://docs.docker.com/engine/reference/builder/#healthcheck https://docs.docker.com/engine/reference/builder/#healthchec... > Does the loadbalancer know how the healthceck is implemented? When the health checks pass in accordance with the above config, the container state will change from "starting" to "healthy" and traffic will be able to be routed to it. Until then you can have a web server or whatever show a different page/implement circuit breaking or whatever. > How do you determine it's time to scale? Docker Swarm doesn't have an abstraction for autoscaling, though there are a few community projects. One can feasibly even write something like that themselves in an evening: https://github.com/jcwimer/docker-swarm-autoscaler https://github.com/jcwimer/docker-swarm-autoscaler That said, I mostly ignore this concern because I'm yet to see a workload that needs to dynamically scale in any number of private or government projects that I've worked on. Most of the time people want predictable infrastructure and being able to deal with backpressure (e.g. message queue), though that's different with startups. > How do you implement always-on-process? service unit, initd, cron? The service abstraction comes out of the box: https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/ https://docs.docker.com/engine/swarm/how-swarm-mode-works/se... You might also want to decide how to best schedule it: wherever available, on a particular node (hostname/tag/...) or on all nodes, which is actually what Portainer agent does! Example: https://docs.portainer.io/start/install/server/swarm/linux https://docs.portainer.io/start/install/server/swarm/linux > How do you export the logs? Docker supports multiple logging drivers: https://docs.docker.com/config/containers/logging/configure/ https://docs.docker.com/config/containers/logging/configure/ > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? Docker and Compose/Swarm support environment variables: https://docs.docker.com/compose/compose-file/#environment https://docs.docker.com/compose/compose-file/#environment If you need config files, you can also use bind mounts: https://docs.docker.com/storage/bind-mounts/ https://docs.docker.com/storage/bind-mounts/ > What about secrets? Docker supports secrets out of the box: https://docs.docker.com/engine/swarm/secrets/ https://docs.docker.com/engine/swarm/secrets/ > Service discovery? Is unbound/bind9? Docker Swarm supports built in DNS, even allows for multiple separate networks: https://docs.docker.com/engine/swarm/networking/ https://docs.docker.com/engine/swarm/networking/ > These items are best done in a standard way. Agreed! Though I'd say that the only two options being "running everything on *nix directly" and "running everything in Kubernetes" is a false narrative! The former can work but can also lead to non-standard and error-prone environments with a horrible waste of human resources, whereas the latter can work but can also lead to overcomplicated and hard to debug environments with a horrible waste of human resources. The best path for many folk probably lies somewhere in the middle, with Nomad/Swarm/Compose/Docker, regardless of what others might claim. The best path for folks interested in a DevOps career is probably running on cloud managed Kubernetes clusters and just using their APIs to lots of great results, not caring about how expensive that is or how easy it would be to self-host on-prem.

RaitoBezarius 4y ago

I'm tired to read this as if it was impossible. > How do you implement healthcheck? Most applications implement a simple endpoint which returns a known answer (whether it's an HTTP 200 code, a banner, etc.) You can just hit this endpoint using your favorite tool. > Does the loadbalancer know how the healthceck is implemented? Well, a lot of tooling supports native (periodic) healthchecks, e.g. NGINX or NGINX Plus (if you want to pay something else than managing K8S). > How do you determine it's time to scale? It is not that expensive to run performance monitoring such as netdata, sometimes, it is just a call to your distribution package manager, put it in your intranet, then you're done. > How do you implement always-on-process? service unit, initd, cron? Depending on what you're using in Linux distributions, why bother with initd and cron if you're running on a systemd-based distribution? Just use unit, timers, service, scope, slice and so on. > How do you export the logs? Again, if you're using a systemd-based distribution: `man systemd-journald-remote`. Otherwise, Loki, rsyslog, many solutions do exist for these scenarios and do not require that much fancy configuration. For mono-nodes, I would argue that the null logger exporter, called: "rsync your logs somewhere", is quite efficient. > How do you inject configs? /etc/environment, profile.d, systemd config, /etc/bestestapp/config? I do not even know why people worry about that. If you're using a systemd-based distribution, your configuration should live more or less in the service unit using Environment, EnvironmentFile and LoadCredential(Encrypted), if your app want some files, sure, put it in /etc/xxx or /var/lib/xxx/config. As long as you do separate configuration from state (and use StateDirectory, etc.) and you do backup them, it is okay. > What about secrets? As said earlier, LoadCredential do the job, if you want to be fancy, you can run a UNIX socket server to distribute the secrets and some secret engine such as HashiCorp Vault. Uncertain that K8S has a better story than that. > Service discovery? Is unbound/bind9? Uncertain you actually need service discovery in early stages. Assuming you need, you could run envoy if you have advanced needs. Or you could run PowerDNS/nsd/unbound/BIND9 as you said with a thin wrapper to register your services. > And on and on and on and on. Well, I do not agree, those are items you choose. Not everyone has these requirements, and not everyone needs to solve them the k8s way. > These items are best done in a standard way. That's a major selling point in k8s. You define and implement these in a consistent way that also integrates seamlessly with other tools. Doubly so on cloud. Less reinventing the wheel means less money. The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview. That is a fallacy, k8s is not a standard way, they push for their standard way and it is in no case what everyone is doing. Defining and implementing thing in a "consistent way" is something we have been doing for a long time before k8s, through standardization process with IETF and W3C for example. k8s creates its own world, and it is indeed compatible with the external world sometimes, but I do not believe that a pile of complexity on the top of YAML files and YAML manipulation tooling is a "consistent way" to define and implement things, no offense here. > Doubly so on cloud. Yeah, because the "cloud ecosystem" is pushing for this, but it has not been a standard. > Less reinventing the wheel means less money. Using a "simple feature" is not reinventing the wheel, but if we go this way, I have a challenger: NixOS. Just reuse their expert modules database and do your classical YAML stuff, but this time with a simple functional programming language. > The more teams, the faster this becomes exponentially expensive. Take this into consideration before your next interview. I think GP was discussing of an early stage and I do not think that all teams require to push at a very fast rate, there is definitely time for packaging and shipping it. BTW, in these organizations where I have worked, they adopted k8s and these "teams" could not use it because it was too complicated and wondered where their packaging team has gone. There is no silver bullet and choosing a deployment platform stack is not a figured out science, but there is a lot of value of choosing composable and simple tools before moving to more integrated ecosystems. > Take this into consideration before your next interview. That is an amazing level of arrogance, I only hope you may accept there are alternatives. :)

osigurdson 4y ago

I think behind most anti-k8s comments is the thought that these things are not needed for "your app". "Your app" is used sporadically by a very small number of people and they don't care if it is down for up to a day.

lolinder 4y ago
The "your" part of that is the problem. Other people are not in a position to know what my requirements are, so telling me what my app does or doesn't need comes off as arrogant. The anti-k8s crowd always talks as though the only reason anyone would choose k8s is that they're brainwashed by the hype. It's never possible that a team actually weighed all of their constraints and made a rational choice.
- thunky 4y ago
  > The anti-k8s crowd always talks as though the only reason anyone would choose k8s is that they're brainwashed by the hype Maybe, but the pro-k8s camp always assumes everyone is building microservices or web services that need to scale.
  - p_l 4y ago
    If all you want to do is to scale, you can just use auto scaling with prebaked VM images on the cloud or whatever. K8s really shines when what you want to do is to optimize usage of your resources (I'd argue especially when running on bare metal)

ploppyploppy 4y ago

Finally a sane response on a HN thread about k8s.