4 ms·

For fun, I run a bare metal k8s cluster with my blog and other projects running on it. My last three nights have been fighting bugs. Bugs with volumes not attac

For fun, I run a bare metal k8s cluster with my blog and other projects running on it. My last three nights have been fighting bugs. Bugs with volumes not attaching, nginx magically configuring itself incorrectly, and a whole bunch of other crap. This just magically started happening, but crap like this seems to happen at least once a month. It’s to the point where I spend at least one night a week babysitting the cluster. I don’t have to pay someone else to handle this, but if did, I would get rid of k8s in a heartbeat. I’ve seen a devops team of only a few people manage tens of thousands of traditional servers, but I doubt such a small team could handle a k8s cluster of the same size. I’m considering moving back to traditional architecture for my blog and other projects. K8s has been fun, but there’s too much magic everywhere.

red-iron-pine 4y ago
> I don’t have to pay someone else to handle this, but if did, I would get rid of k8s in a heartbeat. I’ve seen a devops team of only a few people manage tens of thousands of traditional servers, but I doubt such a small team could handle a k8s cluster of the same size. This has been my experience with a lot of the "we need to be cloud native! containers!" mantra in the enterprise. Some exec gets it in their head it's a good idea (and probably gets non-trivial "referral agent fees") this is a must do, and all of the young, hip developer types are happy to cheerlead it. Two years later OpEx is exploding, most of the processes haven't yet been converted to be in the cloud, and the environment isn't noticeably better or different. It sucks, just sucks in a new and more expensive way that gives you less control of your data. Seen this at 3 x F500 orgs and with multiple cloud providers, including the big 3 + one of the well known second tiers.
thunky 4y ago
> My last three nights have been fighting bugs > I spend at least one night a week babysitting the cluster ... > K8s has been fun This is why everything sucks now.
metacatdud 4y ago
Hey, you can try nomad. It works nice for small-med projects. Works well with terraform and extending with nomad clients it's a breeze. I set my personal bare metals with nomad infra and never looked back.
- withinboredom 4y ago
  How do you solve logging and storage? Those were two issues that caused me to leave it behind. With k8s, there is longhorn for storage so I can move databases around and have volumes replicated to deal with disk failures. Is there anything like that for Nomad?
  - metacatdud 4y ago
    Do you mean this? https://developer.hashicorp.com/nomad/tutorials/stateful-workloads/stateful-workloads-host-volumes https://developer.hashicorp.com/nomad/tutorials/stateful-wor... Honestly I didn't got into the position where this to become critical and always managed to stay ahead. So far(2yrs) so good. I am mostly one man show and I found k8s a bit too much, there is always something. Maybe I was doing smth wrong or didn't knew how to plan better, idk :) Edit//typos
asdfman123 4y ago
No one has ever explained the point of it to me either. I’ve heard it’s supposed to solve the problem of programs running differently on different machines. That’s a problem I’ve never encountered in my 12 years of experience. But the types of issues you describe are very real and very time consuming.
- mrkeen 4y ago
  > No one has ever explained the point of it to me either. It makes sense if you use docker. Docker containers need somewhere to live. If you want two copies of your service alive at all times, K8s is the thing which will listen for crashes, and restart them, etc.
  - asdfman123 4y ago
    Well, sure, but what’s the point of the whole ecosystem?
    - DSingularity 4y ago
      Consider the alternative in conditions where you need various forms of scalability in a cloud agnostic way. Especially when you have a complicated system of many services.
    - mrkeen 4y ago
      The ecosystem isn't really anything more than the sum of its features. I already mentioned K8s as an automatic container runner/restarter. But if you run two copies of a service, you need a load balancer to route traffic to them. You can program your own (more work), or download & run someone else's (less work). Or you can see what K8s provides [0] and do even less work than that. If your services talk to one another, they could talk by hard-coded IP (maintenance nightmare), or by hostname. If they talk by hostname, then they need DNS to resolve those host names. Again, you can roll DNS yourself, or you can see what K8s gives you [1]. And on and on. Firewalls, https, permissions, password/secrets management. There's one more thing to say about K8s which is that it has become a bit of a defacto standard. So you don't need to relearn a completely new way of doing this stuff if you decide to switch jobs / cloud providers. [0] https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr... [1] https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ https://kubernetes.io/docs/concepts/services-networking/dns-...
badpun 4y ago
K8s was never meant to be used for running a blog :) It was built to support Google-scale deployment, with probably dozens of engineers just supporting the live clusters as they stumble into various bizzare states.
- withinboredom 4y ago
  Well, yeah. I just stuck my blog on there to reduce infrastructure costs for myself. The cluster runs much bigger things beside my blog :)
amir734jj 4y ago
Use docker compose or portainer
deleted 4y ago
[deleted]
dikei 4y ago
Well, for a counter anecdote, my bare metal K8S cluster haven't bugged out on me for months, living through multiple versions upgrade. To each his own, I guess.
- withinboredom 4y ago
  I’m quite jealous. How big is your cluster? I’ve got several hundred cpus, nodes just for storage, and multi-region services. It’s quite a beast.
  - dikei 4y ago
    I've 80 nodes, totaling around ~700 CPU cores, all in the same DC though.
puppymaster 4y ago
Nowadays when i try to scaffold quick ideas, I just start a cloudflare worker. You get a url, cron, key-value store and express-like JS server going with a click of a button. I don't even have to npm -i. I can definitely see the appeal of tinkering with 'advance tech' for personal hobby tho. Because now I am pretty sure you know more about K8 than me :)

red-iron-pine 4y ago

> I don’t have to pay someone else to handle this, but if did, I would get rid of k8s in a heartbeat. I’ve seen a devops team of only a few people manage tens of thousands of traditional servers, but I doubt such a small team could handle a k8s cluster of the same size. This has been my experience with a lot of the "we need to be cloud native! containers!" mantra in the enterprise. Some exec gets it in their head it's a good idea (and probably gets non-trivial "referral agent fees") this is a must do, and all of the young, hip developer types are happy to cheerlead it. Two years later OpEx is exploding, most of the processes haven't yet been converted to be in the cloud, and the environment isn't noticeably better or different. It sucks, just sucks in a new and more expensive way that gives you less control of your data. Seen this at 3 x F500 orgs and with multiple cloud providers, including the big 3 + one of the well known second tiers.

thunky 4y ago

> My last three nights have been fighting bugs > I spend at least one night a week babysitting the cluster ... > K8s has been fun This is why everything sucks now.

metacatdud 4y ago

Hey, you can try nomad. It works nice for small-med projects. Works well with terraform and extending with nomad clients it's a breeze. I set my personal bare metals with nomad infra and never looked back.

withinboredom 4y ago
How do you solve logging and storage? Those were two issues that caused me to leave it behind. With k8s, there is longhorn for storage so I can move databases around and have volumes replicated to deal with disk failures. Is there anything like that for Nomad?
- metacatdud 4y ago
  Do you mean this? https://developer.hashicorp.com/nomad/tutorials/stateful-workloads/stateful-workloads-host-volumes https://developer.hashicorp.com/nomad/tutorials/stateful-wor... Honestly I didn't got into the position where this to become critical and always managed to stay ahead. So far(2yrs) so good. I am mostly one man show and I found k8s a bit too much, there is always something. Maybe I was doing smth wrong or didn't knew how to plan better, idk :) Edit//typos

asdfman123 4y ago

No one has ever explained the point of it to me either. I’ve heard it’s supposed to solve the problem of programs running differently on different machines. That’s a problem I’ve never encountered in my 12 years of experience. But the types of issues you describe are very real and very time consuming.

mrkeen 4y ago
> No one has ever explained the point of it to me either. It makes sense if you use docker. Docker containers need somewhere to live. If you want two copies of your service alive at all times, K8s is the thing which will listen for crashes, and restart them, etc.
- asdfman123 4y ago
  Well, sure, but what’s the point of the whole ecosystem?
  - DSingularity 4y ago
    Consider the alternative in conditions where you need various forms of scalability in a cloud agnostic way. Especially when you have a complicated system of many services.
  - mrkeen 4y ago
    The ecosystem isn't really anything more than the sum of its features. I already mentioned K8s as an automatic container runner/restarter. But if you run two copies of a service, you need a load balancer to route traffic to them. You can program your own (more work), or download & run someone else's (less work). Or you can see what K8s provides [0] and do even less work than that. If your services talk to one another, they could talk by hard-coded IP (maintenance nightmare), or by hostname. If they talk by hostname, then they need DNS to resolve those host names. Again, you can roll DNS yourself, or you can see what K8s gives you [1]. And on and on. Firewalls, https, permissions, password/secrets management. There's one more thing to say about K8s which is that it has become a bit of a defacto standard. So you don't need to relearn a completely new way of doing this stuff if you decide to switch jobs / cloud providers. [0] https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr... [1] https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ https://kubernetes.io/docs/concepts/services-networking/dns-...

badpun 4y ago

K8s was never meant to be used for running a blog :) It was built to support Google-scale deployment, with probably dozens of engineers just supporting the live clusters as they stumble into various bizzare states.

withinboredom 4y ago
Well, yeah. I just stuck my blog on there to reduce infrastructure costs for myself. The cluster runs much bigger things beside my blog :)

amir734jj 4y ago

Use docker compose or portainer

deleted 4y ago

[deleted]

dikei 4y ago

Well, for a counter anecdote, my bare metal K8S cluster haven't bugged out on me for months, living through multiple versions upgrade. To each his own, I guess.

withinboredom 4y ago
I’m quite jealous. How big is your cluster? I’ve got several hundred cpus, nodes just for storage, and multi-region services. It’s quite a beast.
- dikei 4y ago
  I've 80 nodes, totaling around ~700 CPU cores, all in the same DC though.

puppymaster 4y ago

Nowadays when i try to scaffold quick ideas, I just start a cloudflare worker. You get a url, cron, key-value store and express-like JS server going with a click of a button. I don't even have to npm -i. I can definitely see the appeal of tinkering with 'advance tech' for personal hobby tho. Because now I am pretty sure you know more about K8 than me :)