21 ms·

You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done fa

You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done faster, then all the better. However, the question is comparing Kubernetes to traditional VM-based infrastructure (especially with pet nodes) whereas you're comparing Kubernetes to a higher-level abstraction. For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so: kubectl patch deployment foo -p '{"spec":{"containers":[{"name":"main","image":"new-image"}]}}' Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.

sgarland 4y ago
Please, do not manage deployments with imperative kubectl commands, I beg of you.
- I_dev_outdoors 4y ago
  I probably wouldn't do this, but what problem does this cause?
  - ryapric 4y ago
    The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.
    - throwaway894345 4y ago
      It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.
      - xbar 4y ago
        So upon deploy the CD pipeline calls kubectl with the proper deployment image and that's ok?
        throwaway894345 4y ago
        Yes.
  - sgarland 4y ago
    Drift, as the other child comment mentions, but also loss of version control. I do not want to have to trawl through someone's shell history to figure out what they changed, nor do I want to have to redirect `kubectl get foo -o yaml` output into diff. If everything is in code, and you have a reasonable branching strategy, it's much easier to control change, to rollback bad merges, to run pre-hooks like security checks and configuration validation tools, etc.
    - thinkmassive 4y ago
      You know kubectl has a built in diff subcommand?
      - sgarland 4y ago
        I do now! Thank you.
    - throwaway894345 4y ago
      It's not drift, because the infra-as-code doesn't manage the image field (the application's CD pipeline does). You don't trawl through someone's shell history', you look at the CD pipeline history. Rollbacks are easy--you just deploy the prior version via your CD tool. I think you're assuming that invoking kubectl means invoking it directly from a user's command line, but kubectl can also be called in a CD script.
      - sgarland 4y ago
        If you're using ArgoCD or something then sure, but bear in mind the original statement you made was directed at someone who is new to K8s, and given a command that can be executed from their shell, they would likely assume that's what you meant.
        throwaway894345 4y ago
        It doesn’t have to be Argo, it can be Jenkins. Whether or not you use a CD pipeline is orthogonal to whether or not you use k8s. The best practice is to use a pipeline whether you’re targeting k8s or bare VMs or a higher level PaaS abstraction.
  - alias_neo 4y ago
    The declarative approach is a more sustainable way to run Kubernetes. If you define some desired state in manifests and apply them to a cluster, they can be applied again to new clusters or the same one and Kubernetes will attempt to maintain the desired state. This state can be version controlled, written in stone, whatever you prefer and it can always be attained. When administrators start issuing imperative commands to a cluster, state starts being changed and there is no record[0] of the state Kubernetes is being asked to maintain. [0] Not entirely true, the state can always be retrieved from the cluster so long as it hasn't failed.
    - jen20 4y ago
      I find this to be a common misconception, stemming from a misunderstanding of what "declarative" means (especially common when people are discussing tools like Terraform). Firstly as you point out, there is a record of the state Kubernetes is being asked to maintain: it's in the API server as the spec of each resource. Secondly, using `kubectl` "patch" in the manner described is not making changes to the cluster state directly, it's making changes to the specification of what should be maintained, and the various controllers effect the state changes. Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models. I'm not advocating actually doing this on a day-to-day basis, but the arguments against it are not ones of imperative vs declarative.
      - sgarland 4y ago
        Given that Kubernetes' docs[0] discuss using imperative commands, I think it's a fairly reasonable way to describe it. [0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/imperative-command/ https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
        throwaway894345 4y ago
        Why is applying a full manifest “declarative” but applying a patch is “imperative”? That’s clearly an error.
- mihaigalos 4y ago
  Spot on!
- throwaway894345 4y ago
  This is a declarative kubectl command.
  - sgarland 4y ago
    Kubernetes' docs[0] disagree. [0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/imperative-command/#how-to-update-objects https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
    - throwaway894345 4y ago
      They’re not being precise with their language. Applying a full manifest is no more “declarative” than applying a patch.
  - deleted 4y ago
    [deleted]
- jrx 4y ago
  The question is: what tool to use? I'm a solo developer running a very small kube cluster for my hobby project. I very much wanted to have a declarative version controlled state of my cluster. Every time I try googling solutions I get flooded with some enterprise Saas offerings that do nothing I want. I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky. Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.
  - edude03 4y ago
    Fluxcd, or argo CD if you want a nice UI
  - lukeschlather 4y ago
    I'm a fan of emrichen, but simple text templating (Jinja) works almost as well. (j2cli means you can just provide a yaml file with per-stage configs.)
  - kube-system 4y ago
    kubectl apply on a directory full of yaml works fairly well for small stuff. Check it into git and it's version controlled. If you need something more flexible than that, try making your own helm chart. Helm will give you some text templating, pre and post hooks, some basic functions, and some versioning and rollback functionality. You can start simple by just pasting in your existing k8s yaml, and then just pull out the pieces you need into variables in your values file. If you need to change an image version, then you just update the variable and `helm upgrade mychart ./mychart`
    - throwaway894345 4y ago
      kubectl apply on a directory doesn’t work because deleting a resource manifest won’t delete the corresponding resources.
epivosism 4y ago
As an outsider, that command looks really easy to mess up. * shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name? Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right. * var x = new List<Dictionary<CustomType,int?>>() This will light up warnings if you get anything wrong; You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong So structurally, 1) in the code realm, for doing minor work we have really strong guarantees, 2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker. I think this is behind the feeling that the k8s system is not ready yet.
- kuschku 4y ago
  1. You’ll use `kubectl set image deployment/gitlab gitlab=gitlab/gitlab-ce:14.9.2-ce.0` in production instead 2. kubernetes will abort the change if it doesn’t syntax check or the types don’t match.
  - throwaway894345 4y ago
    Hah, I didn't even know about `set image`.
- throwaway894345 4y ago
  You put it in a bash script that gets called from CI and move on with life. No one is typing this into their terminal every time they want to do a deploy. > I think this is behind the feeling that the k8s system is not ready yet. Whether you patch the deployment image from bash or from C# doesn't indict the k8s ecosystem.
- throwaway2037 4y ago
  About quoting rules: We desperately needs static analysis or IDEs (Emacs plugins, whatever) that "explain" exactly what quotes do in a particular context. When I returned to C++ recently, I was blown away by Clang-Tidy (JetBrains CLion integration). Literally: Clang-Tidy seems to "know what you really want" and give intelligent suggestions. For someone whom is a very average C++ programmer, it instantly leveled me up! I could see the same for someone writing /bin/sh, /bin/ksh, /bin/bash shell commands. If the IDE could give some hints, it would be incredibly helpful to many.
throwaway2037 4y ago
You wrote: <<if you can find a higher-level abstraction>> This a stretch, but to me Kubernetes is like the C programming language for infra. If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I assume any commercial (or non-!) "higher-level abstraction" for infra is already (or will be soon) built on top of Kubernetes. I am OK with it. I write this post as someone who is uber-techincal, but I know nothing about actually using Kubernetes. I can do vanilla "hand-coded/snowflake" infra just fine in my constrained, private cloud environments, but nothing that scales like Kubernetes.
- throwaway894345 4y ago
  Kubernetes may eventually underpin lots of higher level infrastructure, but most stuff currently isn't running atop Kubernetes. None of the higher level container-as-a-service offerings by the major cloud providers run on Kubernetes, for example. Nor does Heroku. And moreover a lot of people are still working with lower-level abstractions (VMs and auto-scaling groups) or no abstractions at all (pet servers).
- rustyminnow 4y ago
  > If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I might be nitpicking, but I'm not sure that's necessarily true. You could in theory write a compiler for a new language in C (or even assembly!), and once you have a working language, re-write the compiler in that new language. Now that there is no C code involved in the stack anymore, would that still count as a C-turtle? Haskell for example, has some "bits" written in C, but a lot is written in Haskell or assembly[1]. So if you look at the WHOLE stack you'll find C _somewhere_ (probably most places), but I don't think _everything_ boils down to C. Granted, a LOT of stuff is written on top of C. [1] https://stackoverflow.com/questions/23912375/what-language-is-haskell-written-in https://stackoverflow.com/questions/23912375/what-language-i...
  - throwaway2037 4y ago
    Great point and not a nitpick at all! I hope the next generation of languages is written on top Python, Ruby, C++ (ultra modern), Java, or DotNet/C#. I wish more languages would figure out if they can host in JVM/CLR, which would provide some crazy interactions! After seeing so much great work done with JavaScript in the form of "transpilers", I think a lot can be done in that area. I feel Zig is a crazy good idea: A brand new language that produces binaries that are 100% compatible with C linkers. If all goes well, in a few years, why would anyone use C over Zig? It seems like the future. Lots of people think C++ is bat sh-t crazy complex (me too, but I still like the mental gymnastics!). What if there were different C++ dialects supported by transpilers that intentionally restricted features? I think kernel and embedded programmers would the primary audience.
kuschku 4y ago
Don’t do that, use kubectl set image deployment/foo main=main:new-image instead
- throwaway894345 4y ago
  TIL. Thanks for the advice.
- mdaniel 4y ago
  It also accepts "*" if you don't know or care about the container name(s) kubectl set image deployment/foo "*=the/new/image" and then, my other favorite (albeit horribly named, IMHO) "rollout status" to watch it progress: kubectl rollout status deployment/foo
tauwauwau 4y ago
It took me 4-5 hours to read and understand first few chapters of Kubernetes in action, I annotated a lot text as notes on the Oreily site, but didn't have to go back to it. I didn't have to do any security and account management stuff, it was being done by someone else. After 3 years of working with plain Kubernetes and OpenShift, I still didn't have to go back to the book. Basic concepts in Kubernetes are easy to understand, if you're working as a developer to deploy your apps in it and not in DevOps managing it. https://www.oreilly.com/library/view/kubernetes-in-action/9781617293726/ https://www.oreilly.com/library/view/kubernetes-in-action/97...

sgarland 4y ago

Please, do not manage deployments with imperative kubectl commands, I beg of you.

I_dev_outdoors 4y ago
I probably wouldn't do this, but what problem does this cause?
- ryapric 4y ago
  The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.
  - throwaway894345 4y ago
    It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.
    - xbar 4y ago
      So upon deploy the CD pipeline calls kubectl with the proper deployment image and that's ok?
      - throwaway894345 4y ago
        Yes.
- sgarland 4y ago
  Drift, as the other child comment mentions, but also loss of version control. I do not want to have to trawl through someone's shell history to figure out what they changed, nor do I want to have to redirect `kubectl get foo -o yaml` output into diff. If everything is in code, and you have a reasonable branching strategy, it's much easier to control change, to rollback bad merges, to run pre-hooks like security checks and configuration validation tools, etc.
  - thinkmassive 4y ago
    You know kubectl has a built in diff subcommand?
    - sgarland 4y ago
      I do now! Thank you.
  - throwaway894345 4y ago
    It's not drift, because the infra-as-code doesn't manage the image field (the application's CD pipeline does). You don't trawl through someone's shell history', you look at the CD pipeline history. Rollbacks are easy--you just deploy the prior version via your CD tool. I think you're assuming that invoking kubectl means invoking it directly from a user's command line, but kubectl can also be called in a CD script.
    - sgarland 4y ago
      If you're using ArgoCD or something then sure, but bear in mind the original statement you made was directed at someone who is new to K8s, and given a command that can be executed from their shell, they would likely assume that's what you meant.
      - throwaway894345 4y ago
        It doesn’t have to be Argo, it can be Jenkins. Whether or not you use a CD pipeline is orthogonal to whether or not you use k8s. The best practice is to use a pipeline whether you’re targeting k8s or bare VMs or a higher level PaaS abstraction.
- alias_neo 4y ago
  The declarative approach is a more sustainable way to run Kubernetes. If you define some desired state in manifests and apply them to a cluster, they can be applied again to new clusters or the same one and Kubernetes will attempt to maintain the desired state. This state can be version controlled, written in stone, whatever you prefer and it can always be attained. When administrators start issuing imperative commands to a cluster, state starts being changed and there is no record[0] of the state Kubernetes is being asked to maintain. [0] Not entirely true, the state can always be retrieved from the cluster so long as it hasn't failed.
  - jen20 4y ago
    I find this to be a common misconception, stemming from a misunderstanding of what "declarative" means (especially common when people are discussing tools like Terraform). Firstly as you point out, there is a record of the state Kubernetes is being asked to maintain: it's in the API server as the spec of each resource. Secondly, using `kubectl` "patch" in the manner described is not making changes to the cluster state directly, it's making changes to the specification of what should be maintained, and the various controllers effect the state changes. Fundamentally, the argument seems to come down to "you don't have a record of what you once asked the API server to do", and that's fair enough - you don't. But that has nothing to do with imperative or declarative models. I'm not advocating actually doing this on a day-to-day basis, but the arguments against it are not ones of imperative vs declarative.
    - sgarland 4y ago
      Given that Kubernetes' docs[0] discuss using imperative commands, I think it's a fairly reasonable way to describe it. [0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/imperative-command/ https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
      - throwaway894345 4y ago
        Why is applying a full manifest “declarative” but applying a patch is “imperative”? That’s clearly an error.
mihaigalos 4y ago
Spot on!
throwaway894345 4y ago
This is a declarative kubectl command.
- sgarland 4y ago
  Kubernetes' docs[0] disagree. [0] https://kubernetes.io/docs/tasks/manage-kubernetes-objects/imperative-command/#how-to-update-objects https://kubernetes.io/docs/tasks/manage-kubernetes-objects/i...
  - throwaway894345 4y ago
    They’re not being precise with their language. Applying a full manifest is no more “declarative” than applying a patch.
- deleted 4y ago
  [deleted]
jrx 4y ago
The question is: what tool to use? I'm a solo developer running a very small kube cluster for my hobby project. I very much wanted to have a declarative version controlled state of my cluster. Every time I try googling solutions I get flooded with some enterprise Saas offerings that do nothing I want. I managed my stateful sets/services for a while with terraform, but my experience was absolutely terrible and I have stopped that eventually. I now use "kubectl patch" and "kubectl apply" with handwritten yaml, but the workflow feels very clunky. Intuitively it seems obvious to me that there must be a tool helping with that, but for some reason I was absolutely not able to find anything that would be even a little bit helpful. I am considering writing a couple python scripts that will automate it.
- edude03 4y ago
  Fluxcd, or argo CD if you want a nice UI
- lukeschlather 4y ago
  I'm a fan of emrichen, but simple text templating (Jinja) works almost as well. (j2cli means you can just provide a yaml file with per-stage configs.)
- kube-system 4y ago
  kubectl apply on a directory full of yaml works fairly well for small stuff. Check it into git and it's version controlled. If you need something more flexible than that, try making your own helm chart. Helm will give you some text templating, pre and post hooks, some basic functions, and some versioning and rollback functionality. You can start simple by just pasting in your existing k8s yaml, and then just pull out the pieces you need into variables in your values file. If you need to change an image version, then you just update the variable and `helm upgrade mychart ./mychart`
  - throwaway894345 4y ago
    kubectl apply on a directory doesn’t work because deleting a resource manifest won’t delete the corresponding resources.

epivosism 4y ago

As an outsider, that command looks really easy to mess up. * shell interactions with quotes * double quotes * interpolating into image namespace with no autocomplete * easy to forget an argument * do you get autocomplete against the deployment name? Comparison: C# declaration of a complex type - it's less complex than the `kubectl` command above, but IDEs offer way more support to get it right. * var x = new List<Dictionary<CustomType,int?>>() This will light up warnings if you get anything wrong; You get: * go to definition of `CustomType` * autocomplete on classnames * highlighting on type mismatches * warnings about unused vars. * if you initialize the var, the IDE will try hard to not let you do it wrong So structurally, 1) in the code realm, for doing minor work we have really strong guarantees, 2) in deployment realm, even though it's likely going to hit customers harder, the guarantees are weaker. I think this is behind the feeling that the k8s system is not ready yet.

kuschku 4y ago
1. You’ll use `kubectl set image deployment/gitlab gitlab=gitlab/gitlab-ce:14.9.2-ce.0` in production instead 2. kubernetes will abort the change if it doesn’t syntax check or the types don’t match.
- throwaway894345 4y ago
  Hah, I didn't even know about `set image`.
throwaway894345 4y ago
You put it in a bash script that gets called from CI and move on with life. No one is typing this into their terminal every time they want to do a deploy. > I think this is behind the feeling that the k8s system is not ready yet. Whether you patch the deployment image from bash or from C# doesn't indict the k8s ecosystem.
throwaway2037 4y ago
About quoting rules: We desperately needs static analysis or IDEs (Emacs plugins, whatever) that "explain" exactly what quotes do in a particular context. When I returned to C++ recently, I was blown away by Clang-Tidy (JetBrains CLion integration). Literally: Clang-Tidy seems to "know what you really want" and give intelligent suggestions. For someone whom is a very average C++ programmer, it instantly leveled me up! I could see the same for someone writing /bin/sh, /bin/ksh, /bin/bash shell commands. If the IDE could give some hints, it would be incredibly helpful to many.

throwaway2037 4y ago

You wrote: <<if you can find a higher-level abstraction>> This a stretch, but to me Kubernetes is like the C programming language for infra. If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I assume any commercial (or non-!) "higher-level abstraction" for infra is already (or will be soon) built on top of Kubernetes. I am OK with it. I write this post as someone who is uber-techincal, but I know nothing about actually using Kubernetes. I can do vanilla "hand-coded/snowflake" infra just fine in my constrained, private cloud environments, but nothing that scales like Kubernetes.

throwaway894345 4y ago
Kubernetes may eventually underpin lots of higher level infrastructure, but most stuff currently isn't running atop Kubernetes. None of the higher level container-as-a-service offerings by the major cloud providers run on Kubernetes, for example. Nor does Heroku. And moreover a lot of people are still working with lower-level abstractions (VMs and auto-scaling groups) or no abstractions at all (pet servers).
rustyminnow 4y ago
> If you look at the entire software stack today, drill down (all the turtles), and you will eventually find C (everything goes back to libc or a C/C++ kernel). I might be nitpicking, but I'm not sure that's necessarily true. You could in theory write a compiler for a new language in C (or even assembly!), and once you have a working language, re-write the compiler in that new language. Now that there is no C code involved in the stack anymore, would that still count as a C-turtle? Haskell for example, has some "bits" written in C, but a lot is written in Haskell or assembly[1]. So if you look at the WHOLE stack you'll find C _somewhere_ (probably most places), but I don't think _everything_ boils down to C. Granted, a LOT of stuff is written on top of C. [1] https://stackoverflow.com/questions/23912375/what-language-is-haskell-written-in https://stackoverflow.com/questions/23912375/what-language-i...
- throwaway2037 4y ago
  Great point and not a nitpick at all! I hope the next generation of languages is written on top Python, Ruby, C++ (ultra modern), Java, or DotNet/C#. I wish more languages would figure out if they can host in JVM/CLR, which would provide some crazy interactions! After seeing so much great work done with JavaScript in the form of "transpilers", I think a lot can be done in that area. I feel Zig is a crazy good idea: A brand new language that produces binaries that are 100% compatible with C linkers. If all goes well, in a few years, why would anyone use C over Zig? It seems like the future. Lots of people think C++ is bat sh-t crazy complex (me too, but I still like the mental gymnastics!). What if there were different C++ dialects supported by transpilers that intentionally restricted features? I think kernel and embedded programmers would the primary audience.

kuschku 4y ago

Don’t do that, use kubectl set image deployment/foo main=main:new-image instead

throwaway894345 4y ago
TIL. Thanks for the advice.
mdaniel 4y ago
It also accepts "*" if you don't know or care about the container name(s) kubectl set image deployment/foo "*=the/new/image" and then, my other favorite (albeit horribly named, IMHO) "rollout status" to watch it progress: kubectl rollout status deployment/foo

tauwauwau 4y ago

It took me 4-5 hours to read and understand first few chapters of Kubernetes in action, I annotated a lot text as notes on the Oreily site, but didn't have to go back to it. I didn't have to do any security and account management stuff, it was being done by someone else. After 3 years of working with plain Kubernetes and OpenShift, I still didn't have to go back to the book. Basic concepts in Kubernetes are easy to understand, if you're working as a developer to deploy your apps in it and not in DevOps managing it. https://www.oreilly.com/library/view/kubernetes-in-action/9781617293726/ https://www.oreilly.com/library/view/kubernetes-in-action/97...