4 ms·

I have desperately avoided K8s since it became popular, but this seems to be pretty opposite of everything else I've read? Can you share what specifically makes

I have desperately avoided K8s since it became popular, but this seems to be pretty opposite of everything else I've read? Can you share what specifically makes plain docker so terrible?

p_l 7y ago
Plain docker (no swarm etc.) for practical purposes resulted in containers that were in weird position when it comes to decoupling from the host OS. Integration with init system was abysmal. Docker daemon had its own conventions it wanted to follow. Unless you ensure that the state of the docker daemon is deleted on reboot, you could have weird conditions when it tried to handle starting the containers by itself. A very easy thing to use for developer, a pretty shitty tool (assuming no external wrappers) on server. One of the greatest joys of k8s for me was always "it abstracts docker away so I don't have to deal with it, and it drops pretty much all broken features of docker"
- mdaniel 7y ago
  > One of the greatest joys of k8s for me was always "it abstracts docker away so I don't have to deal with it, and it drops pretty much all broken features of docker" It also offers portability away from docker via the Container Runtime Interface; we use containerd and it has been absolutely rock solid, without the weird "what happens to my containers if I have to restart a wedged dockerd?" situation
  - p_l 7y ago
    Indeed - Back when the two alternatives were very alpha quality rktnetes and hypernetes, I was already full of joy that I was seeing the end of docker on server. Since then we got CRI-O and life looks even better.
- simplecto 7y ago
  I always start and keep my containers stateless. the docker run --rm command line switch tell docker to remove the container when it dies. Never a problem on restarts.
- pnako 7y ago
  You've decoupled from the host OS (good thinking), and you've abstracted away from Docker (even better thinking), but how did you decouple from k8s?
Spivak 7y ago
If you are a user of k8s then the experience relatively smooth and painless since all the complexity is hidden behind the API you're calling to provision resources for you. If you are an operator or k8s then you've entered the nightmare zone where you have to make all of those endpoints actually work and do the right thing with code that written 17 days ago. Unlimited terrible middleware to try and form static services into dynamic boxes. k8s was not designed for someone to deploy on-prem that doesn't have a dedicated team of developers and ops people to work on just k8s.
- alexbanks 7y ago
  Since you've made this distinction, can you explain more? How would the person I replied to not be both? In what scenario are they different people?
- p_l 7y ago
  I mostly use it, but I also tend to keep to stable features so that running on-prem is also relatively simple. My biggest cryparty story when it comes to on-prem kubernetes is not actually due to kubernetes, but due to Red Hat. There are words I could say about their official OpenShift deployment scripts and their authors, but I would be rightly banned for writing them. Biggest issues I've encountered involve things on the interface between "classic distro" and running k8s on top of it, and that goes down when you move towards base OS that is more optimized for k8s (for example flatcar). When it comes to the size of team involved, I'd say keeping "classic" stack with loadbalancers, pacemaker, custom deployment methods etc. was comparable effort - at least if we're matching feature for feature what I'd see on "base" k8s on-prem setup (and based on "what I wish I had, or that I could replace with k8s, back in 2016 for an on-prem project). There's one thing, however, where it gets much harder to deploy k8s and I won't hide it - when you're dealing with random broken classic infrastructure with no authority to change it. K8s gets noticeably harder when you need to deal with annoying pre-existing IP layouts because you can't get IP address space allocated, when you have to sit on badly stretched L2 domains, where the local idea of routing is OSPF - or worse, static routes to gateway and a stretched L2. To the point that sometimes it's easier to setup a base "private cloud" (I like VMware for this) on the physical machines, then setup the hosts on top of that - even if you're only going to have one VM per hypervisor. The benefits of abstracting possibly broken infrastructure are too big.

I have desperately avoided K8s since it became popular, but this seems to be pretty opposite of everything else I've read? Can you share what specifically makes plain docker so terrible?

p_l 7y ago

Plain docker (no swarm etc.) for practical purposes resulted in containers that were in weird position when it comes to decoupling from the host OS. Integration with init system was abysmal. Docker daemon had its own conventions it wanted to follow. Unless you ensure that the state of the docker daemon is deleted on reboot, you could have weird conditions when it tried to handle starting the containers by itself. A very easy thing to use for developer, a pretty shitty tool (assuming no external wrappers) on server. One of the greatest joys of k8s for me was always "it abstracts docker away so I don't have to deal with it, and it drops pretty much all broken features of docker"

mdaniel 7y ago
> One of the greatest joys of k8s for me was always "it abstracts docker away so I don't have to deal with it, and it drops pretty much all broken features of docker" It also offers portability away from docker via the Container Runtime Interface; we use containerd and it has been absolutely rock solid, without the weird "what happens to my containers if I have to restart a wedged dockerd?" situation
- p_l 7y ago
  Indeed - Back when the two alternatives were very alpha quality rktnetes and hypernetes, I was already full of joy that I was seeing the end of docker on server. Since then we got CRI-O and life looks even better.
simplecto 7y ago
I always start and keep my containers stateless. the docker run --rm command line switch tell docker to remove the container when it dies. Never a problem on restarts.
pnako 7y ago
You've decoupled from the host OS (good thinking), and you've abstracted away from Docker (even better thinking), but how did you decouple from k8s?

Spivak 7y ago

If you are a user of k8s then the experience relatively smooth and painless since all the complexity is hidden behind the API you're calling to provision resources for you. If you are an operator or k8s then you've entered the nightmare zone where you have to make all of those endpoints actually work and do the right thing with code that written 17 days ago. Unlimited terrible middleware to try and form static services into dynamic boxes. k8s was not designed for someone to deploy on-prem that doesn't have a dedicated team of developers and ops people to work on just k8s.

alexbanks 7y ago
Since you've made this distinction, can you explain more? How would the person I replied to not be both? In what scenario are they different people?
p_l 7y ago
I mostly use it, but I also tend to keep to stable features so that running on-prem is also relatively simple. My biggest cryparty story when it comes to on-prem kubernetes is not actually due to kubernetes, but due to Red Hat. There are words I could say about their official OpenShift deployment scripts and their authors, but I would be rightly banned for writing them. Biggest issues I've encountered involve things on the interface between "classic distro" and running k8s on top of it, and that goes down when you move towards base OS that is more optimized for k8s (for example flatcar). When it comes to the size of team involved, I'd say keeping "classic" stack with loadbalancers, pacemaker, custom deployment methods etc. was comparable effort - at least if we're matching feature for feature what I'd see on "base" k8s on-prem setup (and based on "what I wish I had, or that I could replace with k8s, back in 2016 for an on-prem project). There's one thing, however, where it gets much harder to deploy k8s and I won't hide it - when you're dealing with random broken classic infrastructure with no authority to change it. K8s gets noticeably harder when you need to deal with annoying pre-existing IP layouts because you can't get IP address space allocated, when you have to sit on badly stretched L2 domains, where the local idea of routing is OSPF - or worse, static routes to gateway and a stretched L2. To the point that sometimes it's easier to setup a base "private cloud" (I like VMware for this) on the physical machines, then setup the hosts on top of that - even if you're only going to have one VM per hypervisor. The benefits of abstracting possibly broken infrastructure are too big.