15 ms·

The list of things mentioned in the article to do and learn for a simple, personal project with k8s is absolutely staggering, in my opinion. Having used it, th

by sklivvz1971 8y ago

The list of things mentioned in the article to do and learn for a simple, personal project with k8s is absolutely staggering, in my opinion. Having used it, there's a sizeable amount of further work needed which the article doesn't mention (e.g. learning how to use the pretty confusing google interface, finding the right logs and using their tools). So the overhead is really huge. Furthermore, the whole system is slow. Want to run a SQL query against your postgres? You need to use a google cloud command that changes the firewall and ssh's you in on the machine... and this takes a couple of minutes, just enough to make me desist unless I _really_ need to run that query. Abysmal. Finally, and this is a pet peeve against many advocacy blog posts, they just show you the happy path! Sure, _in the best of cases_ you just edit a file. In a more realistic case, you'll be stuck with a remote management system which is incredibly rich but also a really steep learning curve. Your setup is not performant? Good luck. Need to tweak or fine tune? Again, best of luck. We've tried to adopt k8s 3-4 times at work and every single time productivity dropped significantly without having significant benefits over normal provisioning of machines. {Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it!} ...which in turn is usually significantly slower than building your own home server (but that's another story!)

arountheworld 8y ago
If you design your app properly you won't ever need to run SQL queries by hand. This is a bad practice and it is good that k8s discourages such behaviour. I agree with your other points though.
- barrkel 8y ago
  You have no idea what you're talking about to make a blanket statement like that.
- reacharavindh 8y ago
  SQL is too low level?! I don't think you understand what is going on with the tools you use.
  - rmdashrfroot 8y ago
    He's advocating using ORMs rather than writing raw SQL queries
    - reacharavindh 8y ago
      Okay, now what does Kubernetes have to do with ORMs? Why does it even have a say in what applications we're running on top of it? Am I missing something? In another tangent, I still use raw SQL queries in most cases over ORM. That's just me. May be I'm a control freak or I just dont know how to magically use the abstraction of ORM and still get the most optimised results.
    - Insanity 8y ago
      In a complex environment, that will not always work. We use an ORM at work for all simple CRUD stuff, but when you want something more complex and performant, we end up writing sql anyway. (On a db the with millions of daily transactions)
    - arountheworld 8y ago
      I am not. I am sorry if that sounded ambiguous. By "By hand" I meant SSH into a container and then pasting raw query into SQL client. ORM will not always help you achieve what you want so writing queries is alright, but you need to have tests too. You can create a command for your app (like a Django command for example) that performs the query, but you also create tests that prove it is doing what you think it is doing. Then you run this command in the cluster. This way you can replicate it to different environments etc.
  - arountheworld 8y ago
    I am happy to write SQL, but you need to write it in such a way that you have tests confirming your query does what you intend to. Then run a query as a short lived application on the cluster. You don't need to SSH even for low level stuff. I am sorry if I didn't make it clear.
    - notyourday 8y ago
- candiodari 8y ago
  "I have a problem doing X" "Then don't do it. It only shows you're an idiot anyway." Not a good answer.
- sklivvz1971 8y ago
  I think you are confusing the experience as a user of the app (who shouldn't really ever touch SQL) and the experience of a developer (who should be knowing SQL inside out and using it in their day-to-day work).
- PeterisP 8y ago
  "If you design your app properly" applies if you're talking about using k8s to deploy an app. If you're doing individual development, however, it's quite likely that you're also doing development of the database, and need to do ad-hoc data analysis before you include those queries in that app.
- collyw 8y ago
  Never needed to use SQL for debugging, or just generally checking that what you have in the database is what you expect?
reacharavindh 8y ago
This. Time and again. The number of people who adopt complicated stuff like Kubernetes for what is essentially a couple of web servers and a database is too high. They're Google wannabies that thinks in Google's scale but forget that it is utterly unnecessary in their case. I know a bio scientist who spent two months working on containers and Docker and what not, for what are essentially independent shell scripts that are best run as batch jobs. I spoke with him in length, and he realized at the end of the day that what he really needed was better understanding of standard *nix processes and not containers...
- codegladiator 8y ago
  I had an excellent time working with kubernetes and I am practically a one person company. Kubernetes frees my mind from so many things that now I hate to work without it. Couple of those things include: - automated ssl - centralized logging - super easy scaling up and down (going from 1 instance to 2 instance is a headache manually) - centralized auth (have a service which doesnt have built in auth?) - super easy recovery (containers recreated/volumes attached automatically/apps just work unless you have db type of application which shouldn't be just restarted, which is rare) - Smooth CI/CD (gogs/jenkins/private docker image repository) As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in. > I know a bio scientist who spent Super obscure example. Kubernetes is definitely not for non-tech people. And I didn't pick k8s overnight. Spent a year doing playful POCs before deploying in a production environment. If the only thing thats stopping you from using k8s is the learning curve, I suggest you to go ahead and learn it. Its a huge boon.
  - bthornbury 8y ago
    The learning curve is sharp. The amount of things happening you don't have awareness of is also worrisome (to me). Source: I am also using Kubernetes in production, migrating off it soon.
    - austinjp 8y ago
      It would be interesting if you could briefly say your reasons for moving away from K8s and what you're moving to.
redmorphium 8y ago
You don't need to run the gcloud command to connect to a Cloud SQL instance. You can run the cloud_sql_proxy as a background process on your machine and then you can use mysql or psql to connect to it instantly.
- eecc 8y ago
  I guess your answer supports the parent's case: the happy path is a charm, but you need to learn a whole lot more to use it effectively. Paraphrasing the tag-line of Mesosphere, it's a new OS, one that is targeted at data centres, not small pet-servers setups.
movedx 8y ago
tl;dr if you're a team of developers sitting around a table shouting at your monitor because K8s isn't working, hire someone to do the job right for you. I think you'll find it's pretty easy and utterly worth every penny invested in it. > Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it! Or it could mean you're not the right people for it? Do you have a dedicated individual or team that represents infrastructure engineering? Are you a team of developers trying to play the Ops game? More often than not people/companies that struggle to get K8s going or use it are a team of developers who's expertise is code and not infrastructure/automation. K8s is hard when done from scratch (there's a book and a video course about that by THE K8s guy.) It's not hard when you use Kops, Tectonic (discontinued, I believe), EKS, or one of the other managed services. Implementing a YAML file and launching it isn't hard. Also, you don't have to understand it inside out; you only have to understand how-to use it. Can you tell me how every component in your microwave works or do you just get on with cooking your food? And to say you're not the use case for it - that's highly unlikely. I mean, it's possible, but it's unlikely. Your work loads and software solutions would have to be highly unique and or specialised because hey, it's not like Google's setup is simple, right?
- tannhaeuser 8y ago
  But for personal projects, which is the context of TFA, you're not going to hire a k8s guru. As others have already said, as long as your k8s setup is working as expected, everything is fine (the happy path). But if there ever is a problem with it, then you're confronted with insane complexity, and said tools/setups on top of k8s won't fix it for you; they're even working against you as you won't have learnt the k8s foo for diagnosing wtf is the problem. And in my experience, problems occur rather sooner than later. For example, just the other week a customer of mine had their IPs exhausted on their Azure cloud. Azure and k8s will just silently fail with timeouts and all kinds of weird errors but won't tell you the root cause, and it took two experts, a fresh setup of the k8s cluster from scratch, and almost a week of downtime to fix it. It doesn't help that you only have limited diagnostics on other people's clouds either.
  - movedx 8y ago
grillorafael 8y ago
Where I work we adopted DCOS and it has been nothing but a pleasure to use and we have been updating the cluster quite often. Always very smooth and no pain at all. We thought about k8s a couple of times but it always looked too over complicated and no gain compared to our use of DCOS
tnolet 8y ago
Agreed 100x. I find it very hard to understand why some people talk themselves into thinking they need a distributed container orchestration and management platform, when they don’t have a container orchestration problem: they just want to run an app or two with a DB under it. We need to go back and understand the problem we’re trying to solve, and for 99% of smallish companies and projects this is not container orchestration or hardware resource abstraction.
- snaky 8y ago
  > an app or two with a DB under it And most often all those app or two are doing is converting JSON to SQL and SQL to JSON actually. And PostgREST would be pretty much enough for that.
mockingbirdy 8y ago
I can recommend CaptainDuckDuck for simpler use cases: One-Click-Deployments (e.g. MySQL, PostgreSQL, MongoDB, ...), easy-to-use CLI tool, automatic HTTPS, Docker Swarm support (cluster and scaling), web GUI and open-source. https://captainduckduck.com https://captainduckduck.com https://github.com/githubsaturn/captainduckduck https://github.com/githubsaturn/captainduckduck It made it so much easier to deploy new changes (just `captainduckduck deploy` and done). We also use minio.io (open-source S3) and are extremely productive with those tools. It's perfect for web agencies. It's not sophisticated enough for k8s' use case, but it's extremely easy to use (you can just add nodes and it will restart containers automatically [1]). [1]: https://captainduckduck.com/docs/app-scaling-and-cluster.html https://captainduckduck.com/docs/app-scaling-and-cluster.htm...
- sidmitra 8y ago
  This looks interesting. I hadn't heard of CaptainDuckDuck. Have you tried Dokku or Flynn(flynn.io)? I had looked at them and settled for Flynn because it had better support for multiple-host clusters.
  - pythonaut_16 8y ago
    I looked at Dokku and Flynn for personal (single host) projects and find both to be too complicated for my needs. I generally prefer deploying Docker containers over Heroku buildpacks. I ended up just running a single Digital Ocean droplet with Traefik as a loadbalancer/entry point, and then running each of my projects with docker-compose. (And building each project with Docker) With Traefik I can set up reverse proxying for each project just by adding a few labels to its docker containers, and Traefik manages LetsEncrypt for me.
  - mockingbirdy 8y ago
    Tried Dokku in the past, but it was too barebones for me. Flynn looked also good, but it seemed to complicated (more features than we need) and we settled for CDD.
    - josegonzalez 8y ago
ljm 8y ago
When I got involved with it (and to be fair, most devops things) the value was more in what I was learning for myself, as opposed to what was suitable for the business. Because it's pretty exciting tech and everyone's talking about it, and it doesn't feel like a rabbit hole until you pull yourself back out and see the light again. So, what happened almost every time is that the business unknowingly pivoted into innovating on PaaS, because Docker/Kubernetes was only the first step. After that, you had to figure out the best approach to CI, how to set up your deploy pipelines across multiple environments, how to replicate what you were used to seeing with Heroku, etc. And of course the cost balloons at that stage because your cheap 3 node setup ends up exposing half a dozen load balancers that each cost money, so you start to think about hosting Traefik or nginx-ingress-controller so you can manage load balancing yourself and save some money, and because it's so easy to add more services once you know how, you start wanting to self-host all the things. Meanwhile your startup hasn't even launched to the public yet and the sunk cost fallacy has fully embedded itself in the team mentality: they've just put months of time and effort into setting up the perfect K8S/Docker architecture and workflow, that now requires actual devops experience to maintain, and you can't push back on it because it's all about scaling when things go live, and self-hosting, and how convenient it all is. Except, you know, that's 3-6 months of runway down the drain because leadership didn't shift the focus back to an MVP and let the fantasy continue. And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for.
- mooreds 8y ago
  > And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for. Exactly. I was following along with the article until he mentioned that startups are doing this. Using k8s at a startup (especially a startup building an MVP) is just like using microservices. It is a disservice to the business by technologists. Building the monolith on heroku will get you to market faster. Yagni. (Until you do, and thrn by all means pay off that technical debt.)
  - nugator 8y ago
    Totally disagree. Our startup use Kubernetes and Google Cloud and shipped an MVP in a few months. No operations people only devs and devops. Kubernetes is not perfectly set up yet and the logging has failed sometimes but we ship code to production many times per day and are really happy about not having to care about infrastructure that much. Three nodes and Cloud SQL (PostgreSQL) goes a long way.
cdoxsey 8y ago
Can't really speak to the cloudsql issue... that's not really kubernetes. Running SQL as a container you can just kubectl port-forward.... but it sounds like a specific issue you ran into? I'll grant that there can be headaches that I don't really see anymore because I was deeply immersed in it for a while. But running a linux server by itself was hard till I learned that too. Performance is a subtler point, not sure I follow what you're trying to say there. Container overhead? At least in my experience gke seemed responsive, and deployments settled pretty quick. Tweaking is usually a strong point of k8s. Just reapply... so I'm probably misunderstanding what you were getting at here. Kubernetes needs to improve their accessibility to newcomers but the happy path isn't unheard, for simple stateless apps it mostly just works. And when it doesn't I can usually kubectl log and kubectl delete pod my way to a solution and there are docs and stack overflow to help. Not perfect but I was surprised it worked as well as it did.
- markdog12 8y ago
  > Can't really speak to the cloudsql issue... that's not really kubernetes. Running SQL as a container you can just kubectl port-forward Best solution is Cloud SQL Proxy: https://cloud.google.com/sql/docs/postgres/sql-proxy https://cloud.google.com/sql/docs/postgres/sql-proxy
bitL 8y ago
Don't forget that K8s changes rapidly so whatever you learned a year ago might be completely useless now. So better keep those people dealing with K8s learning by all means or have fun with broken production.
- loriverkutya 8y ago
  That is not true, based on my experience. I started to learn and use kubernetes about a year ago (now we are running our production systems on kubernetes) and we are usually on the latest version. I cannot mention anything which I learned about kubernetes and because useless, because something changed.
  - mds 8y ago
    It's less true now, but a lot of older blog posts and books (at the time, even the most recently published books) talked about managing via replica sets instead of deployments. That was a major source of confusion for me when I was getting started.
erickj 8y ago
welcome to what it's like to work at Google
- dekhn 8y ago
  borg is easy compared to k8s and even a hosted service like gke.
outside1234 8y ago
You are right in the large. That said, once you learn all of these k8s concepts and mechanisms, and probably script nearly all of them, they all become like breathing and don't really take much time. I frankly now am at the point where it is faster to spin up a small project on k8s than it would be to do so with the cloud providers PaaS platform.
Birch-san 8y ago
I've had good experiences using Rancher (https://rancher.com/ https://rancher.com/) to operate k8s clusters. It provides a nice management interface, which handles common use-cases really well. Grabbing per-container logs or getting a shell into a container becomes trivially easy. Admittedly, operation via Rancher comes with its own complexities: setting up security groups and IAM roles. These are documented of course, but as you say: minimum effort remains non-trivial.
theo31 8y ago
Hi, It can be rough to manage Kubernetes when you just want to focus on your app. I was having that same problem when building side projects and focused on building a platform to deploy dockerfiles on a managed Kubernetes instance. https://usekube.co https://usekube.co
jaytaylor 8y ago
For context: I was extremely lucky to get to work with Caleb back in 2012-13. He introduced me to Go and my life has never been the same :D Properly hosting custom apps on the web in a resilient way isn't easy. This being the case, it seems an error to interpret the thoroughness of the article as a signal that the proposed solution or situation is unreasonably complex. I'll take detailed and exhaustive over sparse and hand-wavy any day! Kubernetes certainly has a learning curve, just like most worthwhile things in life.
- fulafel 8y ago
  Hosting personal projects being the task at hand, super high availability isn't usually a headline requirement. That being said, there are certainly existing solutions for doing that, for cheap or even free tiers with less effort - Heroku, Google App Engine, etc.
auslander 8y ago
> and every single time productivity dropped significantly I'm skipping job postings which require k8s. It just tells me they're not competent, falling for the hype. For AWS, ECS is way simpler and is a fully featured orchestration tool. I had an interview today, where I tried to educate why simple is better, and why they don't need k8s. No luck :)

arountheworld 8y ago

If you design your app properly you won't ever need to run SQL queries by hand. This is a bad practice and it is good that k8s discourages such behaviour. I agree with your other points though.

barrkel 8y ago
You have no idea what you're talking about to make a blanket statement like that.
reacharavindh 8y ago
SQL is too low level?! I don't think you understand what is going on with the tools you use.
- rmdashrfroot 8y ago
  He's advocating using ORMs rather than writing raw SQL queries
  - reacharavindh 8y ago
    Okay, now what does Kubernetes have to do with ORMs? Why does it even have a say in what applications we're running on top of it? Am I missing something? In another tangent, I still use raw SQL queries in most cases over ORM. That's just me. May be I'm a control freak or I just dont know how to magically use the abstraction of ORM and still get the most optimised results.
  - Insanity 8y ago
    In a complex environment, that will not always work. We use an ORM at work for all simple CRUD stuff, but when you want something more complex and performant, we end up writing sql anyway. (On a db the with millions of daily transactions)
  - arountheworld 8y ago
    I am not. I am sorry if that sounded ambiguous. By "By hand" I meant SSH into a container and then pasting raw query into SQL client. ORM will not always help you achieve what you want so writing queries is alright, but you need to have tests too. You can create a command for your app (like a Django command for example) that performs the query, but you also create tests that prove it is doing what you think it is doing. Then you run this command in the cluster. This way you can replicate it to different environments etc.
- arountheworld 8y ago
  I am happy to write SQL, but you need to write it in such a way that you have tests confirming your query does what you intend to. Then run a query as a short lived application on the cluster. You don't need to SSH even for low level stuff. I am sorry if I didn't make it clear.
  - notyourday 8y ago
candiodari 8y ago
"I have a problem doing X" "Then don't do it. It only shows you're an idiot anyway." Not a good answer.
sklivvz1971 8y ago
I think you are confusing the experience as a user of the app (who shouldn't really ever touch SQL) and the experience of a developer (who should be knowing SQL inside out and using it in their day-to-day work).
PeterisP 8y ago
"If you design your app properly" applies if you're talking about using k8s to deploy an app. If you're doing individual development, however, it's quite likely that you're also doing development of the database, and need to do ad-hoc data analysis before you include those queries in that app.
collyw 8y ago
Never needed to use SQL for debugging, or just generally checking that what you have in the database is what you expect?

reacharavindh 8y ago

This. Time and again. The number of people who adopt complicated stuff like Kubernetes for what is essentially a couple of web servers and a database is too high. They're Google wannabies that thinks in Google's scale but forget that it is utterly unnecessary in their case. I know a bio scientist who spent two months working on containers and Docker and what not, for what are essentially independent shell scripts that are best run as batch jobs. I spoke with him in length, and he realized at the end of the day that what he really needed was better understanding of standard *nix processes and not containers...

codegladiator 8y ago
I had an excellent time working with kubernetes and I am practically a one person company. Kubernetes frees my mind from so many things that now I hate to work without it. Couple of those things include: - automated ssl - centralized logging - super easy scaling up and down (going from 1 instance to 2 instance is a headache manually) - centralized auth (have a service which doesnt have built in auth?) - super easy recovery (containers recreated/volumes attached automatically/apps just work unless you have db type of application which shouldn't be just restarted, which is rare) - Smooth CI/CD (gogs/jenkins/private docker image repository) As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in. > I know a bio scientist who spent Super obscure example. Kubernetes is definitely not for non-tech people. And I didn't pick k8s overnight. Spent a year doing playful POCs before deploying in a production environment. If the only thing thats stopping you from using k8s is the learning curve, I suggest you to go ahead and learn it. Its a huge boon.
- bthornbury 8y ago
  The learning curve is sharp. The amount of things happening you don't have awareness of is also worrisome (to me). Source: I am also using Kubernetes in production, migrating off it soon.
  - austinjp 8y ago
    It would be interesting if you could briefly say your reasons for moving away from K8s and what you're moving to.

redmorphium 8y ago

You don't need to run the gcloud command to connect to a Cloud SQL instance. You can run the cloud_sql_proxy as a background process on your machine and then you can use mysql or psql to connect to it instantly.

eecc 8y ago
I guess your answer supports the parent's case: the happy path is a charm, but you need to learn a whole lot more to use it effectively. Paraphrasing the tag-line of Mesosphere, it's a new OS, one that is targeted at data centres, not small pet-servers setups.

movedx 8y ago

tl;dr if you're a team of developers sitting around a table shouting at your monitor because K8s isn't working, hire someone to do the job right for you. I think you'll find it's pretty easy and utterly worth every penny invested in it. > Edit: this does not mean k8s is bad, but rather that we are probably not the right use case for it! Or it could mean you're not the right people for it? Do you have a dedicated individual or team that represents infrastructure engineering? Are you a team of developers trying to play the Ops game? More often than not people/companies that struggle to get K8s going or use it are a team of developers who's expertise is code and not infrastructure/automation. K8s is hard when done from scratch (there's a book and a video course about that by THE K8s guy.) It's not hard when you use Kops, Tectonic (discontinued, I believe), EKS, or one of the other managed services. Implementing a YAML file and launching it isn't hard. Also, you don't have to understand it inside out; you only have to understand how-to use it. Can you tell me how every component in your microwave works or do you just get on with cooking your food? And to say you're not the use case for it - that's highly unlikely. I mean, it's possible, but it's unlikely. Your work loads and software solutions would have to be highly unique and or specialised because hey, it's not like Google's setup is simple, right?

tannhaeuser 8y ago
But for personal projects, which is the context of TFA, you're not going to hire a k8s guru. As others have already said, as long as your k8s setup is working as expected, everything is fine (the happy path). But if there ever is a problem with it, then you're confronted with insane complexity, and said tools/setups on top of k8s won't fix it for you; they're even working against you as you won't have learnt the k8s foo for diagnosing wtf is the problem. And in my experience, problems occur rather sooner than later. For example, just the other week a customer of mine had their IPs exhausted on their Azure cloud. Azure and k8s will just silently fail with timeouts and all kinds of weird errors but won't tell you the root cause, and it took two experts, a fresh setup of the k8s cluster from scratch, and almost a week of downtime to fix it. It doesn't help that you only have limited diagnostics on other people's clouds either.
- movedx 8y ago

grillorafael 8y ago

Where I work we adopted DCOS and it has been nothing but a pleasure to use and we have been updating the cluster quite often. Always very smooth and no pain at all. We thought about k8s a couple of times but it always looked too over complicated and no gain compared to our use of DCOS

tnolet 8y ago

Agreed 100x. I find it very hard to understand why some people talk themselves into thinking they need a distributed container orchestration and management platform, when they don’t have a container orchestration problem: they just want to run an app or two with a DB under it. We need to go back and understand the problem we’re trying to solve, and for 99% of smallish companies and projects this is not container orchestration or hardware resource abstraction.

snaky 8y ago
> an app or two with a DB under it And most often all those app or two are doing is converting JSON to SQL and SQL to JSON actually. And PostgREST would be pretty much enough for that.

mockingbirdy 8y ago

I can recommend CaptainDuckDuck for simpler use cases: One-Click-Deployments (e.g. MySQL, PostgreSQL, MongoDB, ...), easy-to-use CLI tool, automatic HTTPS, Docker Swarm support (cluster and scaling), web GUI and open-source. https://captainduckduck.com https://captainduckduck.com https://github.com/githubsaturn/captainduckduck https://github.com/githubsaturn/captainduckduck It made it so much easier to deploy new changes (just `captainduckduck deploy` and done). We also use minio.io (open-source S3) and are extremely productive with those tools. It's perfect for web agencies. It's not sophisticated enough for k8s' use case, but it's extremely easy to use (you can just add nodes and it will restart containers automatically [1]). [1]: https://captainduckduck.com/docs/app-scaling-and-cluster.html https://captainduckduck.com/docs/app-scaling-and-cluster.htm...

sidmitra 8y ago
This looks interesting. I hadn't heard of CaptainDuckDuck. Have you tried Dokku or Flynn(flynn.io)? I had looked at them and settled for Flynn because it had better support for multiple-host clusters.
- pythonaut_16 8y ago
  I looked at Dokku and Flynn for personal (single host) projects and find both to be too complicated for my needs. I generally prefer deploying Docker containers over Heroku buildpacks. I ended up just running a single Digital Ocean droplet with Traefik as a loadbalancer/entry point, and then running each of my projects with docker-compose. (And building each project with Docker) With Traefik I can set up reverse proxying for each project just by adding a few labels to its docker containers, and Traefik manages LetsEncrypt for me.
- mockingbirdy 8y ago
  Tried Dokku in the past, but it was too barebones for me. Flynn looked also good, but it seemed to complicated (more features than we need) and we settled for CDD.
  - josegonzalez 8y ago

ljm 8y ago

When I got involved with it (and to be fair, most devops things) the value was more in what I was learning for myself, as opposed to what was suitable for the business. Because it's pretty exciting tech and everyone's talking about it, and it doesn't feel like a rabbit hole until you pull yourself back out and see the light again. So, what happened almost every time is that the business unknowingly pivoted into innovating on PaaS, because Docker/Kubernetes was only the first step. After that, you had to figure out the best approach to CI, how to set up your deploy pipelines across multiple environments, how to replicate what you were used to seeing with Heroku, etc. And of course the cost balloons at that stage because your cheap 3 node setup ends up exposing half a dozen load balancers that each cost money, so you start to think about hosting Traefik or nginx-ingress-controller so you can manage load balancing yourself and save some money, and because it's so easy to add more services once you know how, you start wanting to self-host all the things. Meanwhile your startup hasn't even launched to the public yet and the sunk cost fallacy has fully embedded itself in the team mentality: they've just put months of time and effort into setting up the perfect K8S/Docker architecture and workflow, that now requires actual devops experience to maintain, and you can't push back on it because it's all about scaling when things go live, and self-hosting, and how convenient it all is. Except, you know, that's 3-6 months of runway down the drain because leadership didn't shift the focus back to an MVP and let the fantasy continue. And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for.

mooreds 8y ago
> And it would be hard to justify anything like Kubernetes for pushing an MVP out of the door; that's what Heroku and cheap Digital Ocean boxes are for. Exactly. I was following along with the article until he mentioned that startups are doing this. Using k8s at a startup (especially a startup building an MVP) is just like using microservices. It is a disservice to the business by technologists. Building the monolith on heroku will get you to market faster. Yagni. (Until you do, and thrn by all means pay off that technical debt.)
- nugator 8y ago
  Totally disagree. Our startup use Kubernetes and Google Cloud and shipped an MVP in a few months. No operations people only devs and devops. Kubernetes is not perfectly set up yet and the logging has failed sometimes but we ship code to production many times per day and are really happy about not having to care about infrastructure that much. Three nodes and Cloud SQL (PostgreSQL) goes a long way.

cdoxsey 8y ago

Can't really speak to the cloudsql issue... that's not really kubernetes. Running SQL as a container you can just kubectl port-forward.... but it sounds like a specific issue you ran into? I'll grant that there can be headaches that I don't really see anymore because I was deeply immersed in it for a while. But running a linux server by itself was hard till I learned that too. Performance is a subtler point, not sure I follow what you're trying to say there. Container overhead? At least in my experience gke seemed responsive, and deployments settled pretty quick. Tweaking is usually a strong point of k8s. Just reapply... so I'm probably misunderstanding what you were getting at here. Kubernetes needs to improve their accessibility to newcomers but the happy path isn't unheard, for simple stateless apps it mostly just works. And when it doesn't I can usually kubectl log and kubectl delete pod my way to a solution and there are docs and stack overflow to help. Not perfect but I was surprised it worked as well as it did.

markdog12 8y ago
> Can't really speak to the cloudsql issue... that's not really kubernetes. Running SQL as a container you can just kubectl port-forward Best solution is Cloud SQL Proxy: https://cloud.google.com/sql/docs/postgres/sql-proxy https://cloud.google.com/sql/docs/postgres/sql-proxy

bitL 8y ago

Don't forget that K8s changes rapidly so whatever you learned a year ago might be completely useless now. So better keep those people dealing with K8s learning by all means or have fun with broken production.

loriverkutya 8y ago
That is not true, based on my experience. I started to learn and use kubernetes about a year ago (now we are running our production systems on kubernetes) and we are usually on the latest version. I cannot mention anything which I learned about kubernetes and because useless, because something changed.
- mds 8y ago
  It's less true now, but a lot of older blog posts and books (at the time, even the most recently published books) talked about managing via replica sets instead of deployments. That was a major source of confusion for me when I was getting started.

erickj 8y ago

welcome to what it's like to work at Google

dekhn 8y ago
borg is easy compared to k8s and even a hosted service like gke.

outside1234 8y ago

You are right in the large. That said, once you learn all of these k8s concepts and mechanisms, and probably script nearly all of them, they all become like breathing and don't really take much time. I frankly now am at the point where it is faster to spin up a small project on k8s than it would be to do so with the cloud providers PaaS platform.

Birch-san 8y ago

I've had good experiences using Rancher (https://rancher.com/ https://rancher.com/) to operate k8s clusters. It provides a nice management interface, which handles common use-cases really well. Grabbing per-container logs or getting a shell into a container becomes trivially easy. Admittedly, operation via Rancher comes with its own complexities: setting up security groups and IAM roles. These are documented of course, but as you say: minimum effort remains non-trivial.

theo31 8y ago

Hi, It can be rough to manage Kubernetes when you just want to focus on your app. I was having that same problem when building side projects and focused on building a platform to deploy dockerfiles on a managed Kubernetes instance. https://usekube.co https://usekube.co

jaytaylor 8y ago

For context: I was extremely lucky to get to work with Caleb back in 2012-13. He introduced me to Go and my life has never been the same :D Properly hosting custom apps on the web in a resilient way isn't easy. This being the case, it seems an error to interpret the thoroughness of the article as a signal that the proposed solution or situation is unreasonably complex. I'll take detailed and exhaustive over sparse and hand-wavy any day! Kubernetes certainly has a learning curve, just like most worthwhile things in life.

fulafel 8y ago
Hosting personal projects being the task at hand, super high availability isn't usually a headline requirement. That being said, there are certainly existing solutions for doing that, for cheap or even free tiers with less effort - Heroku, Google App Engine, etc.

auslander 8y ago

> and every single time productivity dropped significantly I'm skipping job postings which require k8s. It just tells me they're not competent, falling for the hype. For AWS, ECS is way simpler and is a fully featured orchestration tool. I had an interview today, where I tried to educate why simple is better, and why they don't need k8s. No luck :)