4 ms·

From what I've seen with Kubernetes, the problem is that in order to define the resources and links with a schema, it needs several abstractions and extra tooli

From what I've seen with Kubernetes, the problem is that in order to define the resources and links with a schema, it needs several abstractions and extra tooling. So you not only need to define the schema (which isn't that trivial) but also understand what Kubernetes does, how to work with it, and how to solve problems. It's a tool that adds complexity and difficulty, if you're not using the advantages it provides (mainly multi-node management and scale capabilities) you're just handicapping yourself.

jpgvm 4y ago
This is a common misconception. k8s isn't about scale, multi-node or even reliability/resiliency. We had solutions for all of that before it came along. It's about having a standard API for deployment artifacts. The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP. There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns. Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes. Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.
- ramraj07 4y ago
  It would be great if otherwise smart people Also learn to know when what’s trivial to them is not trivial to the masses. K8s is not trivial, not to me at the least. I’m not some super duper engineer but I do alright? That’s all I can say at the least.
  - jpgvm 4y ago
    Have you made a good faith attempt to use k8s? Or are you just regurgitating how hard it is to use based on what you hear on the Internet? My experience is even mediocre engineers are capable of understanding k8s concepts with minimal assistance assuming things are either sufficiently standard (i.e google-able) or well documented with any company specific practices. K8s == hard to run, k8s != hard to use.
    - ramraj07 4y ago
      I have made a good faith attempt at using it, yes. Is spending an entire week on it good faith? I did get everything running but also saw so many settings and features I had to take on faith as being handled without understanding it. I just chose not to bother further because this seems like a minefield of problems in the future as we on boarded other engineers. The number of times I’ve seen production deployments go down due to k8s misconfiguration on other teams has only validated my concerns.
      - jpgvm 4y ago
        So what you are saying is everything worked? That sounds like k8s did its job. You aren't meant to fully understand it in a week, anymore than you are expected to fully understand all of sysadmin in a week. Just because you don't know what every single directive in nginx config does doesn't mean you can't use it effectively and learn what they mean when the time comes. k8s isn't much different. You don't need to know what a liveliness probe is the first time you use it, you can learn about it as you go (most likely when you run into a badly behaved program that needs forced restarts when it hangs). Ofcourse if you are running it yourself that is entirely different, you really do need to know how it works to do that but you should be using hosted k8s unless you have hardcore systems folk and actually need that.
        ramraj07 4y ago
        You’re right that nginx is also complicated , but seems like we are talking about different alternatives. The solution I went with, given my limited knowledge of some of these things, was to use elastic beanstalk. You write your flask application, upload the zip and that’s it pretty much. You get a UI to do all the configurations and for the most part nothing there is hard to decipher or google. The only hiccup might be when you’re trying to connect it to RDS and to the external internet but even that is straightforward as long as you follow the right instructions. We run apps that power entire SaaS organizations and this system seems to be more than sufficient. Why complicate further? We have other fish to fry anyway.
- gjulianm 4y ago
  > Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes. I do not agree with this. I have tried to start Kubernetes and it's far more confusing to set up than just setting up something on a standalone box, mainly because it looks like the set of knowledge I need to set up anything on Kubernetes is a superset of what I need to set up the same thing in bare Linux.
  - jpgvm 4y ago
    It's really not unless you are doing a half-assed job of setting up bare boxes. Lets see to get something even half-reasonable on a bare box you need the minimum: process monitor: For this you can use systemd, supervisord, etc. logging: rsyslog or similar. http reverse proxy: nginx or haproxy deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at this level plus a bunch of bash to properly restart things. backups: depends on persistence, litestream for sqllite these days, pgdump/whatever for RDBMS, wal shipping if you are running the DB on your own boxes. access control: some mechanism to manage either multiple users + homes or shared user account with shared authorized_keys. security: lock down sshd, reverse proxy, etc. stay on top of patching. So are bare minimum there is a ton of concepts you need to know, you are essentially regressing back into the days of hardcore sysadmins... just without all the hardcore sysadmins to do the job correctly (because they are now all busy running k8s setups). Meanwhile to deploy a similarly simply app to k8s (assuming you are buying managed k8s) you need to know the following things: container + registry: i.e Dockerfile + docker build + docker push gcr.io/mything. deployment: Describes how to run your docker image, what args, how much resources, does it need volumes, etc. You can go ghetto and not use service accounts/workload-identity and still be in better shape than bare boxes. service: how your service exposes itself, ports etc. ingress: how to route traffic from internet into your service, usually HTTP host, paths etc. Generally this is equivalent but with way less moving parts. There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you. You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc. K8s reputation as being hard to run is deserved. K8s reputation as being hard to use is grossly undeserved.

jpgvm 4y ago

This is a common misconception. k8s isn't about scale, multi-node or even reliability/resiliency. We had solutions for all of that before it came along. It's about having a standard API for deployment artifacts. The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP. There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns. Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes. Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.

ramraj07 4y ago
It would be great if otherwise smart people Also learn to know when what’s trivial to them is not trivial to the masses. K8s is not trivial, not to me at the least. I’m not some super duper engineer but I do alright? That’s all I can say at the least.
- jpgvm 4y ago
  Have you made a good faith attempt to use k8s? Or are you just regurgitating how hard it is to use based on what you hear on the Internet? My experience is even mediocre engineers are capable of understanding k8s concepts with minimal assistance assuming things are either sufficiently standard (i.e google-able) or well documented with any company specific practices. K8s == hard to run, k8s != hard to use.
  - ramraj07 4y ago
    I have made a good faith attempt at using it, yes. Is spending an entire week on it good faith? I did get everything running but also saw so many settings and features I had to take on faith as being handled without understanding it. I just chose not to bother further because this seems like a minefield of problems in the future as we on boarded other engineers. The number of times I’ve seen production deployments go down due to k8s misconfiguration on other teams has only validated my concerns.
    - jpgvm 4y ago
      So what you are saying is everything worked? That sounds like k8s did its job. You aren't meant to fully understand it in a week, anymore than you are expected to fully understand all of sysadmin in a week. Just because you don't know what every single directive in nginx config does doesn't mean you can't use it effectively and learn what they mean when the time comes. k8s isn't much different. You don't need to know what a liveliness probe is the first time you use it, you can learn about it as you go (most likely when you run into a badly behaved program that needs forced restarts when it hangs). Ofcourse if you are running it yourself that is entirely different, you really do need to know how it works to do that but you should be using hosted k8s unless you have hardcore systems folk and actually need that.
      - ramraj07 4y ago
        You’re right that nginx is also complicated , but seems like we are talking about different alternatives. The solution I went with, given my limited knowledge of some of these things, was to use elastic beanstalk. You write your flask application, upload the zip and that’s it pretty much. You get a UI to do all the configurations and for the most part nothing there is hard to decipher or google. The only hiccup might be when you’re trying to connect it to RDS and to the external internet but even that is straightforward as long as you follow the right instructions. We run apps that power entire SaaS organizations and this system seems to be more than sufficient. Why complicate further? We have other fish to fry anyway.
gjulianm 4y ago
> Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes. I do not agree with this. I have tried to start Kubernetes and it's far more confusing to set up than just setting up something on a standalone box, mainly because it looks like the set of knowledge I need to set up anything on Kubernetes is a superset of what I need to set up the same thing in bare Linux.
- jpgvm 4y ago
  It's really not unless you are doing a half-assed job of setting up bare boxes. Lets see to get something even half-reasonable on a bare box you need the minimum: process monitor: For this you can use systemd, supervisord, etc. logging: rsyslog or similar. http reverse proxy: nginx or haproxy deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at this level plus a bunch of bash to properly restart things. backups: depends on persistence, litestream for sqllite these days, pgdump/whatever for RDBMS, wal shipping if you are running the DB on your own boxes. access control: some mechanism to manage either multiple users + homes or shared user account with shared authorized_keys. security: lock down sshd, reverse proxy, etc. stay on top of patching. So are bare minimum there is a ton of concepts you need to know, you are essentially regressing back into the days of hardcore sysadmins... just without all the hardcore sysadmins to do the job correctly (because they are now all busy running k8s setups). Meanwhile to deploy a similarly simply app to k8s (assuming you are buying managed k8s) you need to know the following things: container + registry: i.e Dockerfile + docker build + docker push gcr.io/mything. deployment: Describes how to run your docker image, what args, how much resources, does it need volumes, etc. You can go ghetto and not use service accounts/workload-identity and still be in better shape than bare boxes. service: how your service exposes itself, ports etc. ingress: how to route traffic from internet into your service, usually HTTP host, paths etc. Generally this is equivalent but with way less moving parts. There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you. You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc. K8s reputation as being hard to run is deserved. K8s reputation as being hard to use is grossly undeserved.