10 ms·

I don't know my dude, all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platf

I don't know my dude, all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice. I agree with some other comments in this thread about a general fervor in the Enterprise space to "modernize" needlessly. This conversation usually lands on the company copying what everyone else is doing or what Gartner tells them to do. Cue "DevOps". 100 percent agree with your comments on something simpler. I can't tell you how many times I've debated with our Analytics teams to just use Docker Compose/Swarm.

Avalaxy 4y ago
> all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff. Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.
- sk8serboi 4y ago
  Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away. Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago. Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms. Guess it is someone else’s management problem now. :shrug:
- throwaway894345 4y ago
  You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done faster, then all the better. However, the question is comparing Kubernetes to traditional VM-based infrastructure (especially with pet nodes) whereas you're comparing Kubernetes to a higher-level abstraction. For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so: kubectl patch deployment foo -p '{"spec":{"containers":[{"name":"main","image":"new-image"}]}}' Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.
  - sgarland 4y ago
    Please, do not manage deployments with imperative kubectl commands, I beg of you.
    - I_dev_outdoors 4y ago
      I probably wouldn't do this, but what problem does this cause?
      - ryapric 4y ago
        The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.
        throwaway894345 4y ago
        It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.
- jokethrowaway 4y ago
  Azure is generally the worst (their new container offering looks better though) If you try digital ocean k8s offering it's fairly straightforward. Google cloud was the first to offer a decent k8s as a service, if I recall correctly. We didn't have DevOps back in 2015 and we were on GCloud. Personally I don't pick k8s just because it's heavy to run and I don't want to waste machines (plain docker is good enough for a large part of what people actually need). Sometimes in a project when I can't figure something out with just docker, I just bite the bullet and install k0s.
- crucialfelix 4y ago
  And you probably didn't even get to TLS and or authenticated communication between containers, CI/CD, canary deployments, observability, monitoring etc. What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes. The core system is great, but we need to replace these DevOps with a framework or platform.
  - neurostimulant 4y ago
    > What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes. Doesn't Rancher fit this description? It's pretty resource-heavy though.
    - crucialfelix 4y ago
      Rancher seems to now be a "multi-cloud container management platform". - Deploys kubernetes clusters - includes Fleet for CI/CD - install apps with helm - ISTIO (awesome) It feels like it is positioned for orgs with large needs. I'm looking for k8s for small nimble orgs. I will look at Rancher more, thanks for reminding me of it! Google's Anthos is also hard to describe now as they cover similar "everything" product features. GCP now has Autopilot which lets you pay just for the cpu you use (no cluster management at all). Anthos includes ISTIO which may at some point work on Autopilot. This would mean not having to fiddle with GKE Ingress (which I found unpleasant) and instead use the new standard Gateway. I believe that eventually GKE Autopilot will offer running individual pods on GPU/TPU, pay as you go. But when is this all as easy as using Next.js ?
- neurostimulant 4y ago
  I feel you. Took me 2 years of dabbling with k8s every once in a while on the side to finally "get" it. and if I stop looking for a few months, suddenly it already gained another set of features and deprecating some other.
- astrospective 4y ago
  Container instances are pretty good really, the app service for containers is pretty good too. I’ve been playing with k8s because that’s what everyone thinks they want and need to be able to speak to it, and use it when the time comes, but I’ve yet to run into a case where I really thought it was necessary, for the platform I work on (millions of users, big number of transactions per second).
  - Avalaxy 4y ago
    Do you use ACI at that scale? It's interesting how Microsoft also promotes AKS rather than ACI. For Azure ML they even explicitly state that ACI should not be used for production purposes, which I find quite curious.
- heretogetout 4y ago
  I hear you about feeling dumb. I think some early decisions in the k8s ecosystem led to a lot of wasted time and effort, and this frustration. YAML: significant whitespace is always unwelcome but YAML also introduces unexpected problems, like how it deals with booleans. For example, say you have this array: foo: - x - y - z You might think this is an array of strings, but you'd be wrong. It's also difficult to read through a YAML config and understand the parent of each key, especially in code reviews or just on GH. I believe k8s life would have been easier with JSON configs, where it's impossible to confuse e.g. booleans for strings and where it's easier to understand the object's hierarchy. Helm's use of gotpl: this choice exacerbates the problems with YAML. Now you're treating a structured language with a text template library. You have to spend energy thinking about indentation levels and how the values will be interpreted by both the templater and k8s. I think helm would be less frustrating if they chose some templating library that made objects first class citizens. Where you can inject values at specific locations with one-liners or simple blocks of code (e.g. `ingress.spec.rules[0].append(yadda yadda)`) I'm sure there was debate about these choices early on and I don't have any unique ideas here, so I don't want to be too critical. These are just a couple of pain points I've personally experienced.
  - aliswe 4y ago
    for what its worth, and if im not mostaken, there is some support for JSON, ie. kubectl get deployment xyz -o json
    - neurostimulant 4y ago
      JSON has a different set of problems, e.g. not having multi-line string support.
  - jtchang 4y ago
    Is that yaml not an array of strings? ["x","y","z"]
    - Filligree 4y ago
      I don't happen to have a YAML parser at hand, but I believe it's... ["x", True, "z"]
- jljljl 4y ago
  Definitely a pain point I've seen in my past work as well. Even though spinning up a basic cluster has gotten easier with canned services like EKS, deploying and developing on the cluster is a major challenging for most developers without a higher level framework. My cofounders and I are working on a solution to this at https://www.jetpack.io/ https://www.jetpack.io/. If you're interested in early access, we'd love your feedback!
- Art9681 4y ago
  There are tools that let you convert a running docker/podman container to a kubernetes deployment manifest which you can then deploy with one line command to your cluster. Kubernetes is the new data center and I don’t expect anyone with <10 years of enterprise experience to master it. Certainly one can use it and deploy things to it if you work at a place where there are mature pipelines, but deploying your own cluster and all associated services plus networking plus security is the domain of experienced/senior engineers. How quickly one achieves that “status” is up to the individual. It took me 15 years before I could build an enterprise network from scratch. It took me two years to understand how to do the same with K8s.
throwaway7865 4y ago
Literally a one-liner for AWS EKS: eksctl create cluster --name mycluster --region us-west1 --with-oidc --fargate --external-dns-access --asg-access --full-ecr-access --alb-ingress-access
- belter 4y ago
  That is a quick and short line. Now the fun starts:"Kubernetes Failure Stories" https://k8s.af/ https://k8s.af/
  - throwaway7865 4y ago
    I’ve made a comment below, but long story short we’ve moved to Kubernetes running on Fargate and we don’t have downtime anymore. Sure, one can break anything, but our anecdotal experience is we’re now focused on actually delivering code rather than fretting about node failures. https://news.ycombinator.com/item?id=31581372 https://news.ycombinator.com/item?id=31581372
    - belter 4y ago
      You can't run all types of workloads on Fargate. At least not yet.
      - throwaway7865 4y ago
        That’s fair, for example I couldn’t manage to run clustered Redis. Something with EFS file system that Fargate nodes use.
dagss 4y ago
We have a team to manage our Azure k8s. Blue/green clusters, switching traffic between clusters to be able to upgrade k8s, etc Definitely a lot of work.
jasiek 4y ago
The reason why they provide it, is because everyone expects it. The reason everyone expects it, is because it was "cool" and an ecosystem grew around it. I use AWS ECS and it's really really good and easy to understand.
eeZah7Ux 4y ago
> 3 major clouds offer "canned" k8s services ...until you need to debug something somewhere in the enormous stack.
OJFord 4y ago
You mean like AWS' EKS? That spares you kubeadm stuff in setup. Arguably complicated upgrade, because you're less familiar with what you have/need/rely on (their own upgrade docs point you to upstream changelogs/release notes etc.). You're still left with a kubernetes cluster to deploy stuff to, decide how stuff scales, etc., which is a lot of what is generally meant by 'DevOps' anyway? The lower level infrastructure/platform/kubeadm type stuff isn't really 'Dev' related at all.

Avalaxy 4y ago

> all 3 major clouds offer "canned" k8s services that you can set up in a ridiculously short amount of time with Terraform and your CI platform of choice I don't agree. I spun up a Kubernetes cluster in Azure, which was indeed easy. But then I had to figure out how to write the correct deployment scripts to deploy my docker containers to it, and how to configure all the security stuff. After more than a week of trying to figure it out, I decided to ditch the whole solution and go for Azure Container Instances instead. It was too much for me to learn about all the concepts of Kubernetes, how you configure them, how to make it work for solutions that are not as simple as the example on the website, and how to navigate through the various different methods of deploying stuff. Maybe I'm just too dumb. But I wasn't going to invest a month of my time into doing something that should be simple enough for an average developer to accomplish.

sk8serboi 4y ago
Similarly, we switched from self-managed k8s on EC2 to Fargate. Took about 2 months part-time with some consultants to get thing squared away. Once we deployed, we ran into all sorts of SRE-issues. Turns out AWS sets all those “sane limits” that our own folks never did. Still hunting ghosts from the rollout 6 months ago. Makes for good resume fodder, and makes me laugh at the prestigious titles and positions the folks who built this system went on to receive at big name firms. Guess it is someone else’s management problem now. :shrug:
throwaway894345 4y ago
You're not dumb, you're just new to it, and it's fundamentally hard stuff anyways, and if you can find a higher-level abstraction that lets you get work done faster, then all the better. However, the question is comparing Kubernetes to traditional VM-based infrastructure (especially with pet nodes) whereas you're comparing Kubernetes to a higher-level abstraction. For what it's worth, deploying in Kubernetes is pretty easy once you figure it out (and often finding the information is the hardest part). All you need to do is update the Deployment resource's "image" parameter. You can do that with `kubectl patch` like so: kubectl patch deployment foo -p '{"spec":{"containers":[{"name":"main","image":"new-image"}]}}' Kubernetes will handle bringing up a new replicaset, validating health checks, draining the old replicaset, etc.
- sgarland 4y ago
  Please, do not manage deployments with imperative kubectl commands, I beg of you.
  - I_dev_outdoors 4y ago
    I probably wouldn't do this, but what problem does this cause?
    - ryapric 4y ago
      The same problems any imperative management within declarative config causes -- drift. If the tool you're using supports declarative configuration, all changes should be made exclusively via the declarative interface to prevent that drift. In this example, the new image should be added to the original manifest itself, not via a CLI update.
      - throwaway894345 4y ago
        It depends on how you manage your changes. A lot of people don't have their infra-as-code manage the deployment's image field--rather, that's updated by the application's CD pipeline. There's no drift to worry about.
jokethrowaway 4y ago
Azure is generally the worst (their new container offering looks better though) If you try digital ocean k8s offering it's fairly straightforward. Google cloud was the first to offer a decent k8s as a service, if I recall correctly. We didn't have DevOps back in 2015 and we were on GCloud. Personally I don't pick k8s just because it's heavy to run and I don't want to waste machines (plain docker is good enough for a large part of what people actually need). Sometimes in a project when I can't figure something out with just docker, I just bite the bullet and install k0s.
crucialfelix 4y ago
And you probably didn't even get to TLS and or authenticated communication between containers, CI/CD, canary deployments, observability, monitoring etc. What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes. The core system is great, but we need to replace these DevOps with a framework or platform.
- neurostimulant 4y ago
  > What we need is a Next.js for Kubernetes. Something that delivers a full stack solution on top of base Kubernetes. Doesn't Rancher fit this description? It's pretty resource-heavy though.
  - crucialfelix 4y ago
    Rancher seems to now be a "multi-cloud container management platform". - Deploys kubernetes clusters - includes Fleet for CI/CD - install apps with helm - ISTIO (awesome) It feels like it is positioned for orgs with large needs. I'm looking for k8s for small nimble orgs. I will look at Rancher more, thanks for reminding me of it! Google's Anthos is also hard to describe now as they cover similar "everything" product features. GCP now has Autopilot which lets you pay just for the cpu you use (no cluster management at all). Anthos includes ISTIO which may at some point work on Autopilot. This would mean not having to fiddle with GKE Ingress (which I found unpleasant) and instead use the new standard Gateway. I believe that eventually GKE Autopilot will offer running individual pods on GPU/TPU, pay as you go. But when is this all as easy as using Next.js ?
neurostimulant 4y ago
I feel you. Took me 2 years of dabbling with k8s every once in a while on the side to finally "get" it. and if I stop looking for a few months, suddenly it already gained another set of features and deprecating some other.
astrospective 4y ago
Container instances are pretty good really, the app service for containers is pretty good too. I’ve been playing with k8s because that’s what everyone thinks they want and need to be able to speak to it, and use it when the time comes, but I’ve yet to run into a case where I really thought it was necessary, for the platform I work on (millions of users, big number of transactions per second).
- Avalaxy 4y ago
  Do you use ACI at that scale? It's interesting how Microsoft also promotes AKS rather than ACI. For Azure ML they even explicitly state that ACI should not be used for production purposes, which I find quite curious.
heretogetout 4y ago
I hear you about feeling dumb. I think some early decisions in the k8s ecosystem led to a lot of wasted time and effort, and this frustration. YAML: significant whitespace is always unwelcome but YAML also introduces unexpected problems, like how it deals with booleans. For example, say you have this array: foo: - x - y - z You might think this is an array of strings, but you'd be wrong. It's also difficult to read through a YAML config and understand the parent of each key, especially in code reviews or just on GH. I believe k8s life would have been easier with JSON configs, where it's impossible to confuse e.g. booleans for strings and where it's easier to understand the object's hierarchy. Helm's use of gotpl: this choice exacerbates the problems with YAML. Now you're treating a structured language with a text template library. You have to spend energy thinking about indentation levels and how the values will be interpreted by both the templater and k8s. I think helm would be less frustrating if they chose some templating library that made objects first class citizens. Where you can inject values at specific locations with one-liners or simple blocks of code (e.g. `ingress.spec.rules[0].append(yadda yadda)`) I'm sure there was debate about these choices early on and I don't have any unique ideas here, so I don't want to be too critical. These are just a couple of pain points I've personally experienced.
- aliswe 4y ago
  for what its worth, and if im not mostaken, there is some support for JSON, ie. kubectl get deployment xyz -o json
  - neurostimulant 4y ago
    JSON has a different set of problems, e.g. not having multi-line string support.
- jtchang 4y ago
  Is that yaml not an array of strings? ["x","y","z"]
  - Filligree 4y ago
    I don't happen to have a YAML parser at hand, but I believe it's... ["x", True, "z"]
jljljl 4y ago
Definitely a pain point I've seen in my past work as well. Even though spinning up a basic cluster has gotten easier with canned services like EKS, deploying and developing on the cluster is a major challenging for most developers without a higher level framework. My cofounders and I are working on a solution to this at https://www.jetpack.io/ https://www.jetpack.io/. If you're interested in early access, we'd love your feedback!
Art9681 4y ago
There are tools that let you convert a running docker/podman container to a kubernetes deployment manifest which you can then deploy with one line command to your cluster. Kubernetes is the new data center and I don’t expect anyone with <10 years of enterprise experience to master it. Certainly one can use it and deploy things to it if you work at a place where there are mature pipelines, but deploying your own cluster and all associated services plus networking plus security is the domain of experienced/senior engineers. How quickly one achieves that “status” is up to the individual. It took me 15 years before I could build an enterprise network from scratch. It took me two years to understand how to do the same with K8s.

throwaway7865 4y ago

Literally a one-liner for AWS EKS: eksctl create cluster --name mycluster --region us-west1 --with-oidc --fargate --external-dns-access --asg-access --full-ecr-access --alb-ingress-access

belter 4y ago
That is a quick and short line. Now the fun starts:"Kubernetes Failure Stories" https://k8s.af/ https://k8s.af/
- throwaway7865 4y ago
  I’ve made a comment below, but long story short we’ve moved to Kubernetes running on Fargate and we don’t have downtime anymore. Sure, one can break anything, but our anecdotal experience is we’re now focused on actually delivering code rather than fretting about node failures. https://news.ycombinator.com/item?id=31581372 https://news.ycombinator.com/item?id=31581372
  - belter 4y ago
    You can't run all types of workloads on Fargate. At least not yet.
    - throwaway7865 4y ago
      That’s fair, for example I couldn’t manage to run clustered Redis. Something with EFS file system that Fargate nodes use.

dagss 4y ago

We have a team to manage our Azure k8s. Blue/green clusters, switching traffic between clusters to be able to upgrade k8s, etc Definitely a lot of work.

jasiek 4y ago

The reason why they provide it, is because everyone expects it. The reason everyone expects it, is because it was "cool" and an ecosystem grew around it. I use AWS ECS and it's really really good and easy to understand.

eeZah7Ux 4y ago

> 3 major clouds offer "canned" k8s services ...until you need to debug something somewhere in the enormous stack.

OJFord 4y ago

You mean like AWS' EKS? That spares you kubeadm stuff in setup. Arguably complicated upgrade, because you're less familiar with what you have/need/rely on (their own upgrade docs point you to upstream changelogs/release notes etc.). You're still left with a kubernetes cluster to deploy stuff to, decide how stuff scales, etc., which is a lot of what is generally meant by 'DevOps' anyway? The lower level infrastructure/platform/kubeadm type stuff isn't really 'Dev' related at all.