6 ms·

Can someone explain how logging is supposed to work after this change? I’m complete bloody lost. Actually I’m nearly always lost with kubernetes. It’s either b

by qz2 6y ago

Can someone explain how logging is supposed to work after this change? I’m complete bloody lost. Actually I’m nearly always lost with kubernetes. It’s either broken or changing.

throwaway201103 6y ago
> I’m nearly always lost with kubernetes. It’s either broken or changing. Glad I'm not the only one. I'm sure I'm not the smartest engineer/sysadmin in the world, but I'm also not the dumbest and I have never gotten an on-premises Kubernetes installation to work. The way I manage containers is lxc and shell scripts. I understand it, and it works.
- freedomben 6y ago
  I work for Red Hat as an OpenShift Consultant, so I'm on the bleeding edge of change and am constantly pushing boundaries. Don't tell the boss or the customers, but most of the time when release notes for a new version come out, I look at them and go "WTF, why do we need that, I better do some research." It's fast changing, complex as hell, and absolutely brutal. That said most things are there for a reason and once I dig in I usually see the need. That said I do love it despite its warts. There's no doubt some Stockholm Syndrome at play here, but I love the API (which is pretty curlable btw, a mark of a great API IMHO) and the principles (declarative, everything's a YAML/JSON object in etcd, etc). I see it the same way I did C++ (which I also loved). It gives you great power which you can use to build an elegant, robust system, or you can create an unmaintainable, complex, monster of a nightmare. It's up to you.
  - jacques_chester 6y ago
    The full-time job of keeping up with Kubernetes was published in February 2018[0] and things have only gotten faster since then. [0] https://goteleport.com/blog/kubernetes-release-cycle/ https://goteleport.com/blog/kubernetes-release-cycle/ , HN discussion: https://news.ycombinator.com/item?id=16285192 https://news.ycombinator.com/item?id=16285192
- chucky_z 6y ago
  Give nomad, docker swarm, or lxd a shot.
  - yjftsjthsd-h 6y ago
    I'm currently involved in an effort to rip out docker swarm at work because its overlay networks are shockingly unreliable. LXD looks interesting but https://linderud.dev/blog/packaging-lxd-for-arch-linux/ https://linderud.dev/blog/packaging-lxd-for-arch-linux/ convinced me that it's another Canonical NIH special and probably best avoided (in particular, "only distributed for Ubuntu via snaps" means "forces auto-updates" which means "not going in my environment"). Need to try Nomad; I'm cautiously optimistic since the rest of HashiCorp's stuff is good.
    - chucky_z 6y ago
      Nomad 1.0 is the bomb. I've ran Nomad since 0.7 and it was usable but pretty rough in those days. 1.0 is such a good, smooth thing. Super awesome even if you don't want to use Docker at all, you can just execute random stuff through exec/raw_exec, or straight Java. Heck, run it in a BSD homelab and use one of the jailshell executors. It's not Kubernetes but it doesn't try to be at all.
    - albatruss 6y ago
      Is this on 19.x swarm mode? I thought it got a lot more reliable with that release.
- mschuster91 6y ago
  Getting Kubernetes itself running on bare metal ("running" as in "you have containers and can access them") is half a day of work. What is deadly difficult is getting networking to work. Even a comparably "easy" thing with a couple of two-NIC machines (one external, internet-routable, one DMZ) cost me a fucking week. What's even worse is when one has to obey corporate restrictions - for example, only having external interfaces on "loadbalancer" nodes: - First of all, MetalLB only has one active Speaker node which means your bandwidth is limited to that node's uplink and you're wasting the resources of the other loadbalancers. - Second, you can taint your nodes to only schedule the MetalLB speaker on your "loadbalancer" nodes via tolerations... but how the f..k do you convince MetalLB to change the speaker node once you apply that change?! - Third, what do you do when you want to expose N services but only have one or two external IPs? DC/OS was way more flexible, you had one set of loadbalancers (haproxy) that did all the routing, and could run an entire cluster on four machines - two LBs, one master, one worker. There is no way to replicate this with Kubernetes. None.
  - throwaway201103 6y ago
    Yes, it's the networking that never works for me either. I'm one guy, wear many hats, and don't have time to chase rabbits down holes. If I follow published instructions and it doesn't work, I pretty much stop there.
disgruntledphd2 6y ago
Welcome to modern software development, apparently.
- qz2 6y ago
  Modern software development makes me wish I was born in the late 1940s.
  - deleted 6y ago
    [deleted]
  - von_neumann 6y ago
    You like punch-cards? Because that is how you get them!
    - lokedhs 6y ago
      If you're born in the late 40's, you'd probably have been exposed to some punch cards but when starting work, it'd be the 70's already. I'm sure there was still punch cards around at the time, but at least you had printing terminals if not monitor-based terminals. That said, depending on where you were working, things could also change fast. You could find yourself finding that a kernel system call had changed because someone patched it the evening before.
    - qz2 6y ago
      Yes. Plenty of time to think before doing.
jacques_chester 6y ago
Loosely, my understanding Kubernetes works like this: You have a Pod definition, which is basically a gang of containers. In that Pod definition you have included one or more container image references. You send the Pod definition to the API Server. The API Server informs listeners for Pod updates that there is a new Pod definition. One of these listeners is the scheduler, which decides which Node should get the Pod. It creates an update for the Pod's "status", essentially annotating it with the name of the Node that should run the Pod. Each Node has a "kubelet". These too subscribe to Pod definitions. When a change shows up saying "Pod 'foo' should run on Node 27", the kubelet in Node 27 perks up its ears. The kubelet converts the Pod definition into descriptions of containers -- image reference, RAM limits, which disks to attach etc. It then turns to its container runtime through the "Container Runtime Interface" (CRI). In the early days this was a Docker daemon. The container runtime now acts on the descriptions it got. Most notably, it will check to see if it has an image in its local cache; if it doesn't then it will try to pull that image from a registry. Now: The CRI is distinct from the Docker daemon API. The CRI is abstracted because since the Docker daemon days, other alternatives have emerged (and some have withered), such as rkt, podman and containerd. This update says "we are not going to maintain the Docker daemon option for CRI". You can use containerd. From a Kubernetes end-user perspective, nothing should change. From an operator perspective, all that happens is that you have a smaller footprint with less attack surface.
- rsanheim 6y ago
  I'm sure all this complexity makes sense for all sorts of reasons buried in the history of Kubernetes development at Google. "Things are the way they are because they got that way." The fact that so many other orgs, many of which are startups or just small to medium sized tech companies, use a system this complex is ludicrous to me.
  - jacques_chester 6y ago
    Well, I might have designed it differently, but I wasn't there. For what it does, this architecture works well. More to the point: none of it is visible to end-users of Kubernetes. You send a Pod definition, some magic happens, pow! running software.

by qz2 6y ago

Can someone explain how logging is supposed to work after this change? I’m complete bloody lost. Actually I’m nearly always lost with kubernetes. It’s either broken or changing.

throwaway201103 6y ago

> I’m nearly always lost with kubernetes. It’s either broken or changing. Glad I'm not the only one. I'm sure I'm not the smartest engineer/sysadmin in the world, but I'm also not the dumbest and I have never gotten an on-premises Kubernetes installation to work. The way I manage containers is lxc and shell scripts. I understand it, and it works.

freedomben 6y ago
I work for Red Hat as an OpenShift Consultant, so I'm on the bleeding edge of change and am constantly pushing boundaries. Don't tell the boss or the customers, but most of the time when release notes for a new version come out, I look at them and go "WTF, why do we need that, I better do some research." It's fast changing, complex as hell, and absolutely brutal. That said most things are there for a reason and once I dig in I usually see the need. That said I do love it despite its warts. There's no doubt some Stockholm Syndrome at play here, but I love the API (which is pretty curlable btw, a mark of a great API IMHO) and the principles (declarative, everything's a YAML/JSON object in etcd, etc). I see it the same way I did C++ (which I also loved). It gives you great power which you can use to build an elegant, robust system, or you can create an unmaintainable, complex, monster of a nightmare. It's up to you.
- jacques_chester 6y ago
  The full-time job of keeping up with Kubernetes was published in February 2018[0] and things have only gotten faster since then. [0] https://goteleport.com/blog/kubernetes-release-cycle/ https://goteleport.com/blog/kubernetes-release-cycle/ , HN discussion: https://news.ycombinator.com/item?id=16285192 https://news.ycombinator.com/item?id=16285192
chucky_z 6y ago
Give nomad, docker swarm, or lxd a shot.
- yjftsjthsd-h 6y ago
  I'm currently involved in an effort to rip out docker swarm at work because its overlay networks are shockingly unreliable. LXD looks interesting but https://linderud.dev/blog/packaging-lxd-for-arch-linux/ https://linderud.dev/blog/packaging-lxd-for-arch-linux/ convinced me that it's another Canonical NIH special and probably best avoided (in particular, "only distributed for Ubuntu via snaps" means "forces auto-updates" which means "not going in my environment"). Need to try Nomad; I'm cautiously optimistic since the rest of HashiCorp's stuff is good.
  - chucky_z 6y ago
    Nomad 1.0 is the bomb. I've ran Nomad since 0.7 and it was usable but pretty rough in those days. 1.0 is such a good, smooth thing. Super awesome even if you don't want to use Docker at all, you can just execute random stuff through exec/raw_exec, or straight Java. Heck, run it in a BSD homelab and use one of the jailshell executors. It's not Kubernetes but it doesn't try to be at all.
  - albatruss 6y ago
    Is this on 19.x swarm mode? I thought it got a lot more reliable with that release.
mschuster91 6y ago
Getting Kubernetes itself running on bare metal ("running" as in "you have containers and can access them") is half a day of work. What is deadly difficult is getting networking to work. Even a comparably "easy" thing with a couple of two-NIC machines (one external, internet-routable, one DMZ) cost me a fucking week. What's even worse is when one has to obey corporate restrictions - for example, only having external interfaces on "loadbalancer" nodes: - First of all, MetalLB only has one active Speaker node which means your bandwidth is limited to that node's uplink and you're wasting the resources of the other loadbalancers. - Second, you can taint your nodes to only schedule the MetalLB speaker on your "loadbalancer" nodes via tolerations... but how the f..k do you convince MetalLB to change the speaker node once you apply that change?! - Third, what do you do when you want to expose N services but only have one or two external IPs? DC/OS was way more flexible, you had one set of loadbalancers (haproxy) that did all the routing, and could run an entire cluster on four machines - two LBs, one master, one worker. There is no way to replicate this with Kubernetes. None.
- throwaway201103 6y ago
  Yes, it's the networking that never works for me either. I'm one guy, wear many hats, and don't have time to chase rabbits down holes. If I follow published instructions and it doesn't work, I pretty much stop there.

disgruntledphd2 6y ago

Welcome to modern software development, apparently.

qz2 6y ago
Modern software development makes me wish I was born in the late 1940s.
- deleted 6y ago
  [deleted]
- von_neumann 6y ago
  You like punch-cards? Because that is how you get them!
  - lokedhs 6y ago
    If you're born in the late 40's, you'd probably have been exposed to some punch cards but when starting work, it'd be the 70's already. I'm sure there was still punch cards around at the time, but at least you had printing terminals if not monitor-based terminals. That said, depending on where you were working, things could also change fast. You could find yourself finding that a kernel system call had changed because someone patched it the evening before.
  - qz2 6y ago
    Yes. Plenty of time to think before doing.

jacques_chester 6y ago

Loosely, my understanding Kubernetes works like this: You have a Pod definition, which is basically a gang of containers. In that Pod definition you have included one or more container image references. You send the Pod definition to the API Server. The API Server informs listeners for Pod updates that there is a new Pod definition. One of these listeners is the scheduler, which decides which Node should get the Pod. It creates an update for the Pod's "status", essentially annotating it with the name of the Node that should run the Pod. Each Node has a "kubelet". These too subscribe to Pod definitions. When a change shows up saying "Pod 'foo' should run on Node 27", the kubelet in Node 27 perks up its ears. The kubelet converts the Pod definition into descriptions of containers -- image reference, RAM limits, which disks to attach etc. It then turns to its container runtime through the "Container Runtime Interface" (CRI). In the early days this was a Docker daemon. The container runtime now acts on the descriptions it got. Most notably, it will check to see if it has an image in its local cache; if it doesn't then it will try to pull that image from a registry. Now: The CRI is distinct from the Docker daemon API. The CRI is abstracted because since the Docker daemon days, other alternatives have emerged (and some have withered), such as rkt, podman and containerd. This update says "we are not going to maintain the Docker daemon option for CRI". You can use containerd. From a Kubernetes end-user perspective, nothing should change. From an operator perspective, all that happens is that you have a smaller footprint with less attack surface.

rsanheim 6y ago
I'm sure all this complexity makes sense for all sorts of reasons buried in the history of Kubernetes development at Google. "Things are the way they are because they got that way." The fact that so many other orgs, many of which are startups or just small to medium sized tech companies, use a system this complex is ludicrous to me.
- jacques_chester 6y ago
  Well, I might have designed it differently, but I wasn't there. For what it does, this architecture works well. More to the point: none of it is visible to end-users of Kubernetes. You send a Pod definition, some magic happens, pow! running software.