6 ms·

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deploym

Kubernetes has to be most complex software I've ever tried to learn. I eventually gave up and decided to stick with simple single machine docker-compose deployments. I figure by the time any of my personal projects actually need to scale beyond 1 machine, I'd probably have enough revenue that I can afford to hire someone else to worry about it.

arcturus17 6y ago
Maybe your starting instances are workhorses, but if you go from something like a t3.medium to two it’s a ~$60 / mth increase... Not something I’d personally optimize for. Also why Docker in the first place? I’m genuinely wondering - in the stacks I run (Express / Python) it doesn’t seem necessary at low scale. Elastic Beanstalk, Heroku, Digital Ocean etc all offer facilities for single-command deploys that work out of the box.
- TobTobXX 6y ago
  I like Docker for the 'keeping it clean' aspect. Install php, composer and stuff juat because one of the projects you host requires it? Nope. Have to make excessive configuration on a system component just to run another application? Nope. Forgot how you set it up and now you are struggeling at the new machine? Use the Dockerfile.
  - yyx 6y ago
    vagrant and ansible can help.
- AnHonestComment 6y ago
  I use Docker without the network virtualization as a package manager. Docker make it easy to run the same version of code in different places and let’s things run next to each other without version conflicts. Also, I think you’re in a very small minority not to care about $720/yr increases in your hobbies.
  - arcturus17 6y ago
    GP was talking about projects that had revenue, and about "hiring someone" past a single instance. I replied that beyond a single instance, you can probably get away with not hiring a K8s devops person and just spinning another instance. I'm not sure you've read this whole thing right. And yes, I certainly wouldn't mind paying an additional $720 / yr for a project that had revenue; I almost certainly wouldn't want to spend money hiring a specialist, or spend time hyperoptimizing that myself - I make that in about a dozen hours of work, so counting how far one can go down the rabbit hole of optimizing server costs, and the associated cost of opportunity, the economics are crystal clear. I don't have any successful personal projects but I have significant experience working with clients, and they are sold on the reasoning pretty much every single time ("I can charge you $3,000 for developing this feature, or we can use a paid service for $720 a year"). I also don't see how Docker is going to save you that much money; if you need a certain amount of compute, you need a certain amount of compute. AWS ElasticBeanstalk for instance charges nothing for spinning up an additional instance compared to EC2; there is no overhead for the PaaS aspect of it, like there would be in Heroku. Digital Ocean app platform is the same as EB, AFAIK.
    - deleted 6y ago
      [deleted]
    - AnHonestComment 6y ago
      GP talked about using Docker to do complex single host deployments until they needed horizontal scaling, which given max VM power, is after they can afford someone to manage it for them. That makes me think they have multiple services of variable workload packed onto a single host, eg web server, async, and DB all on a single host via Docker. That’s the antithesis of EB, which can only do horizontal scaling. Docker provides a way to replicate those multiple services in a deployment configuration when you want to set up a host image. Not using Docker as an easy way to pack it all onto a single box as long as they can is just wasted expense.
NathanFlurry 6y ago
I’d recommend giving Docker Swarm + Traefik a shot. It’s dead simple to set up manually and has very little “magic” in how things work under the hood. Plus much of your existing Docker Compose config will work out of the box. It vastly simplified the deployment process too. I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed k8s would be a safer bet than managing my own Swarm cluster, but if you’ve used anybody’s managed k8s (or read https://k8s.af https://k8s.af), you’ll realize that every cloud provider has their own closed source fork of k8s with plenty of nasty bugs that you can’t do anything about.
- fy20 6y ago
  I use Swarm at home (only on a single node, because it turns out 3 are overkill for my needs) and it's been running great for 6 months so far. Before that I tried various incarnations of k8s and eventually they'd just destroy themselves up and require a rebuild (the main issue was persistent storage). My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host networking). I have a few containers (wireguard and minidlna) which need this, so those are running through docker-compose. I've tried macvlan but wasn't able to get that working in Swarm mode.
  - lub 6y ago
    Did you try this? networks: - host https://docs.docker.com/network/host/ https://docs.docker.com/network/host/
    - fy20 6y ago
      Maybe it has changed since I built it, but I wasn't able to get this working with Swarm services. I had to convert them to docker-compose to make it work. The docs suggest it should work with Swarm mode though, so maybe I need to try again. Ideally I'd like to give each service its own IP on the network, which was possible with how I had k8s setup.
  - diarrhea 6y ago
dtech 6y ago
Yes, k8s is not built for solo outfits and small projects. In places with 10+ developers and 10+ services on a single cluster it works surprisingly well from the user (developer) side.
- je42 6y ago
  I think it is more like: 10+ servers OR 10+ developers. you can have 10+ services as a single developer ;)
- bavell 6y ago
  Hmm, I guess I should shut down my business then since I'm a solo founder using k8s exclusively for my hosting for 3+ years... Definitely a learning curve but honestly not bad if you like ops. Absolutely possible and beneficial for any size team. As always, it depends on what your goals are and how you want to use k8s.
- wdenniss 6y ago
  In my opinion (and I'm biased as I work on GKE Autopilot), GKE is viable for a 1 developer project, especially with Autopilot mode (I have my own hobby projects deployed on GKE). If you were self-hosting k8s, then I'd agree with you.
dilyevsky 6y ago
Long time ago I worked on HA setups for telecom (Wimax/LTE) equipment. Kubernetes is complicated but has nothing on those systems. Just to give you some idea - https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-ims-core-reference-architecture-.jpg?width=800&height=637&name=3gpp-ts-23-228-ims-core-reference-architecture-.jpg https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-... (doesn’t even cover everything)
- 2sk21 6y ago
  The very term "HA" still gives me nightmares. It can be very hard to get HA to work correctly. Many years ago, I worked in a startup and one of our main offerings was an HA network device. It was unbelievably finicky to get it to work in the first place and even harder to update the software on an HA cluster.
  - isbvhodnvemrwvn 6y ago
    It's HA by intimidation. The cluster is complex enough that nobody even wants to touch it, and since human errors are the most common type of error out there, it breaks much less often.
  - dilyevsky 6y ago
    Yes I believe this is why you see things like k3s in some iot/edge deployment scenarios. Because other alternatives for HA like OpenSAF have been severely lacking for years
jillesvangurp 6y ago
It's the right attitude. The management fees alone for this autopilot thing are 0.10$ per hour. Or about 70$/month. It's a bargain considering all the hidden costs kubernetes imposes in terms of requiring people that know how to tame the complexity associated with it (i.e. very expensive devops people costing magnitudes more than that). Automating those people away is worth money. I like Cloud Run for the same reason because I can use it without needing a lot of devops skills in my team or without sacrificing my own time (because I have those skills but have more valuable things to do). It allows me to focus on keeping my CI/CD pipeline (cloud run sets that up with a button click) busy with new functionality. And our hosting cost are close to 0$ because we stay below the freemium layer until we actually need to scale. Edit. corrected the typo 700->70
- mping 6y ago
  I think you mean 70$, not 700$
  - jillesvangurp 6y ago
    Eh yeah, thanks for correcting me.
- je42 6y ago
  yep, cloud run is pretty good. unfortunately, it doesnt cover all cases. (i.e. stateful stuff like: websockets and chunking, and recurring jobs) for these case, i still have gke cluster around.
  - flakiness 6y ago
    Nitpick FYI: WebSocket is coming. https://cloud.google.com/blog/products/serverless/cloud-run-gets-websockets-http-2-and-grpc-bidirectional-streams https://cloud.google.com/blog/products/serverless/cloud-run-... (Still agree that Cloud Run isn't for everyone.)
- wdenniss 6y ago
  Hey, it's William from Google here. You're right about the costs, I just wanted to point out that Autopilot does include one cluster in GKE's free tier. So you'll only pay the ~$73/month if you have more than 1 cluster. There's (almost) no limit to what you can run in one cluster too, and Kubernetes namespaces can help to separate different environments to allow for sharing. Cloud Run sounds like the perfect solution for your workloads though!
emiliobool 6y ago
docker swarm is a good alternative for you then, comes out of the box with docker and you only need to add a few lines to your docker compose files to make them swarm compatible
justsomeuser 6y ago
I’m leaning towards using Ubuntu/systemd to start all my services, and using a single container per project with SQLite. This way: - I can easily move from dev to prod by using the private container registry - I have apt-get on the server and just use the default Ubuntu - No distributed/network state I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own config standard. At least with Ubuntu they have LTS and a bunch of Google pages for fixing stuff.
- notyourday 6y ago
  systemd-nspawn is the most underrated piece of software on any modern link system. It just works and does exactly what it needs to do. Not to mention using a directory structure as a container image?! What is this sorcery?!
- deleted 6y ago
  [deleted]
Seb-C 6y ago
This is a honest question, but may I ask what you found to be hard about it? For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws. Maybe being full stack and having a strong knowledge of Linux and Docker helped. Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.
einpoklum 6y ago
> simple ... docker-compose deployments So, not simple.

arcturus17 6y ago

Maybe your starting instances are workhorses, but if you go from something like a t3.medium to two it’s a ~$60 / mth increase... Not something I’d personally optimize for. Also why Docker in the first place? I’m genuinely wondering - in the stacks I run (Express / Python) it doesn’t seem necessary at low scale. Elastic Beanstalk, Heroku, Digital Ocean etc all offer facilities for single-command deploys that work out of the box.

TobTobXX 6y ago
I like Docker for the 'keeping it clean' aspect. Install php, composer and stuff juat because one of the projects you host requires it? Nope. Have to make excessive configuration on a system component just to run another application? Nope. Forgot how you set it up and now you are struggeling at the new machine? Use the Dockerfile.
- yyx 6y ago
  vagrant and ansible can help.
AnHonestComment 6y ago
I use Docker without the network virtualization as a package manager. Docker make it easy to run the same version of code in different places and let’s things run next to each other without version conflicts. Also, I think you’re in a very small minority not to care about $720/yr increases in your hobbies.
- arcturus17 6y ago
  GP was talking about projects that had revenue, and about "hiring someone" past a single instance. I replied that beyond a single instance, you can probably get away with not hiring a K8s devops person and just spinning another instance. I'm not sure you've read this whole thing right. And yes, I certainly wouldn't mind paying an additional $720 / yr for a project that had revenue; I almost certainly wouldn't want to spend money hiring a specialist, or spend time hyperoptimizing that myself - I make that in about a dozen hours of work, so counting how far one can go down the rabbit hole of optimizing server costs, and the associated cost of opportunity, the economics are crystal clear. I don't have any successful personal projects but I have significant experience working with clients, and they are sold on the reasoning pretty much every single time ("I can charge you $3,000 for developing this feature, or we can use a paid service for $720 a year"). I also don't see how Docker is going to save you that much money; if you need a certain amount of compute, you need a certain amount of compute. AWS ElasticBeanstalk for instance charges nothing for spinning up an additional instance compared to EC2; there is no overhead for the PaaS aspect of it, like there would be in Heroku. Digital Ocean app platform is the same as EB, AFAIK.
  - deleted 6y ago
    [deleted]
  - AnHonestComment 6y ago
    GP talked about using Docker to do complex single host deployments until they needed horizontal scaling, which given max VM power, is after they can afford someone to manage it for them. That makes me think they have multiple services of variable workload packed onto a single host, eg web server, async, and DB all on a single host via Docker. That’s the antithesis of EB, which can only do horizontal scaling. Docker provides a way to replicate those multiple services in a deployment configuration when you want to set up a host image. Not using Docker as an easy way to pack it all onto a single box as long as they can is just wasted expense.

NathanFlurry 6y ago

I’d recommend giving Docker Swarm + Traefik a shot. It’s dead simple to set up manually and has very little “magic” in how things work under the hood. Plus much of your existing Docker Compose config will work out of the box. It vastly simplified the deployment process too. I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed k8s would be a safer bet than managing my own Swarm cluster, but if you’ve used anybody’s managed k8s (or read https://k8s.af https://k8s.af), you’ll realize that every cloud provider has their own closed source fork of k8s with plenty of nasty bugs that you can’t do anything about.

fy20 6y ago
I use Swarm at home (only on a single node, because it turns out 3 are overkill for my needs) and it's been running great for 6 months so far. Before that I tried various incarnations of k8s and eventually they'd just destroy themselves up and require a rebuild (the main issue was persistent storage). My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host networking). I have a few containers (wireguard and minidlna) which need this, so those are running through docker-compose. I've tried macvlan but wasn't able to get that working in Swarm mode.
- lub 6y ago
  Did you try this? networks: - host https://docs.docker.com/network/host/ https://docs.docker.com/network/host/
  - fy20 6y ago
    Maybe it has changed since I built it, but I wasn't able to get this working with Swarm services. I had to convert them to docker-compose to make it work. The docs suggest it should work with Swarm mode though, so maybe I need to try again. Ideally I'd like to give each service its own IP on the network, which was possible with how I had k8s setup.
- diarrhea 6y ago

dtech 6y ago

Yes, k8s is not built for solo outfits and small projects. In places with 10+ developers and 10+ services on a single cluster it works surprisingly well from the user (developer) side.

je42 6y ago
I think it is more like: 10+ servers OR 10+ developers. you can have 10+ services as a single developer ;)
bavell 6y ago
Hmm, I guess I should shut down my business then since I'm a solo founder using k8s exclusively for my hosting for 3+ years... Definitely a learning curve but honestly not bad if you like ops. Absolutely possible and beneficial for any size team. As always, it depends on what your goals are and how you want to use k8s.
wdenniss 6y ago
In my opinion (and I'm biased as I work on GKE Autopilot), GKE is viable for a 1 developer project, especially with Autopilot mode (I have my own hobby projects deployed on GKE). If you were self-hosting k8s, then I'd agree with you.

dilyevsky 6y ago

Long time ago I worked on HA setups for telecom (Wimax/LTE) equipment. Kubernetes is complicated but has nothing on those systems. Just to give you some idea - https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-ims-core-reference-architecture-.jpg?width=800&height=637&name=3gpp-ts-23-228-ims-core-reference-architecture-.jpg https://www.metaswitch.com/hs-fs/hubfs/Blogs/3gpp-ts-23-228-... (doesn’t even cover everything)

2sk21 6y ago
The very term "HA" still gives me nightmares. It can be very hard to get HA to work correctly. Many years ago, I worked in a startup and one of our main offerings was an HA network device. It was unbelievably finicky to get it to work in the first place and even harder to update the software on an HA cluster.
- isbvhodnvemrwvn 6y ago
  It's HA by intimidation. The cluster is complex enough that nobody even wants to touch it, and since human errors are the most common type of error out there, it breaks much less often.
- dilyevsky 6y ago
  Yes I believe this is why you see things like k3s in some iot/edge deployment scenarios. Because other alternatives for HA like OpenSAF have been severely lacking for years

jillesvangurp 6y ago

It's the right attitude. The management fees alone for this autopilot thing are 0.10$ per hour. Or about 70$/month. It's a bargain considering all the hidden costs kubernetes imposes in terms of requiring people that know how to tame the complexity associated with it (i.e. very expensive devops people costing magnitudes more than that). Automating those people away is worth money. I like Cloud Run for the same reason because I can use it without needing a lot of devops skills in my team or without sacrificing my own time (because I have those skills but have more valuable things to do). It allows me to focus on keeping my CI/CD pipeline (cloud run sets that up with a button click) busy with new functionality. And our hosting cost are close to 0$ because we stay below the freemium layer until we actually need to scale. Edit. corrected the typo 700->70

mping 6y ago
I think you mean 70$, not 700$
- jillesvangurp 6y ago
  Eh yeah, thanks for correcting me.
je42 6y ago
yep, cloud run is pretty good. unfortunately, it doesnt cover all cases. (i.e. stateful stuff like: websockets and chunking, and recurring jobs) for these case, i still have gke cluster around.
- flakiness 6y ago
  Nitpick FYI: WebSocket is coming. https://cloud.google.com/blog/products/serverless/cloud-run-gets-websockets-http-2-and-grpc-bidirectional-streams https://cloud.google.com/blog/products/serverless/cloud-run-... (Still agree that Cloud Run isn't for everyone.)
wdenniss 6y ago
Hey, it's William from Google here. You're right about the costs, I just wanted to point out that Autopilot does include one cluster in GKE's free tier. So you'll only pay the ~$73/month if you have more than 1 cluster. There's (almost) no limit to what you can run in one cluster too, and Kubernetes namespaces can help to separate different environments to allow for sharing. Cloud Run sounds like the perfect solution for your workloads though!

emiliobool 6y ago

docker swarm is a good alternative for you then, comes out of the box with docker and you only need to add a few lines to your docker compose files to make them swarm compatible

justsomeuser 6y ago

I’m leaning towards using Ubuntu/systemd to start all my services, and using a single container per project with SQLite. This way: - I can easily move from dev to prod by using the private container registry - I have apt-get on the server and just use the default Ubuntu - No distributed/network state I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own config standard. At least with Ubuntu they have LTS and a bunch of Google pages for fixing stuff.

notyourday 6y ago
systemd-nspawn is the most underrated piece of software on any modern link system. It just works and does exactly what it needs to do. Not to mention using a directory structure as a container image?! What is this sorcery?!
deleted 6y ago
[deleted]

Seb-C 6y ago

This is a honest question, but may I ask what you found to be hard about it? For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws. Maybe being full stack and having a strong knowledge of Linux and Docker helped. Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.

einpoklum 6y ago

> simple ... docker-compose deployments So, not simple.