8 ms·

Kubernetes SidecarContainers feature is merged

217 points by xdasf 3y ago

xdasf 3y ago
KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers https://github.com/kubernetes/enhancements/tree/master/keps/... TLDR: Introduce a restartPolicy field to init containers and use it to indicate that an init container is a sidecar container. Kubelet will start init containers with restartPolicy=Always in the order with other init containers, but instead of waiting for its completion, it will wait for the container startup completion.
- hunta2097 3y ago
  Hopefully these changes should make Envoy sidecars (and sidecar co-existence in general) more reliable.
  - tommiegannert 3y ago
    What's the use-case for Envoy as a sidecar? (As someone using Envoy Gateway.)
    - pluies 3y ago
      Service mesh. Istio for example injects an envoy sidecar to each pod, and manages everything "meshy" (routing, retries, mTLS, etc) via this sidecar. This is also how linkerd works, though they're using their own purpose-built sidecar rather than envoy.
      - rad_gruchalski 3y ago
        Isn’t the new ambient mesh available in istio 1.18 sidecar-less?
        phrotoma 3y ago
        Yep, it’s also alpha, under intense development, and by every account (including those vendors who are chomping at the bit to start selling it to customers) absolutely not production ready.
        rad_gruchalski 3y ago
        Oh, good to know. I was about to pack a spike into an upcoming sprint.
- plagiarist 3y ago
  It seems like there could be a better marker for this. Maybe my skill with Kubernetes is too low for it to make sense.
raesene9 3y ago
Worth noting that this is hitting Alpha in Kubernetes 1.28, so won't be available by default at this stage. If you've got self-managed clusters, it'd be possible to enable with a feature gate on the API server, but it's unlikely to be available on managed Kubernetes until it gets to GA.
- numbsafari 3y ago
  [flagged]
CSDude 3y ago
It's a shame it took so long. If the main container shutdown (i.e connection drain, processing inflight queue items) takes a while, and your service mesh dies (nice go binary) and main container cannot communicate with internet anymore. But I'm not sure about initContainers being used. init keyword implies it'd run and die in order for others to continue. Using restartPolicy with init instead of a dedicated sideCars field feels weird.
- zeeZ 3y ago
  Without restart policy, a failing init container is retried forever. With a policy of never, the entire pod is marked as having failed. The init containers still have to run and succeed before the main pod continues.
- smarterclayton 3y ago
  We did that to leave open more complex ordering of both init containers and sidecars (regular containers do not have a restart order). For instance, you might have a service mesh that needs a vault secret - those both might be sidecars, and you may need to ensure the vault sidecar starts first if both go down. Eventually we may want to add parallelism to that start order, and a separate field would prevent simple ordering from working now. Also, these are mostly init containers that run longer, and you want a sidecar not starting to be able to block regular pods, and adding a new container type (like ephemeral containers) is extremely disruptive to other parts of the system (security, observability, and UI), so we looked to minimize that disruption.
MrStonedOne 3y ago
[dead]
sidcool 3y ago
Any documentation on this? What does this mean?
- tecleandor 3y ago
  So, until now, a sidecar container was just the idea of running containers in you Kubernetes pod, along with your main service, that were 'helpers' for something: connection to databases or vpns, mesh networking, pulling secrets or config, debugging... But they didn't have special status, they were just regular containers in your pod. This sometimes posed some problems because they weren't available for the full life cycle of the pod, notably on the init process. So if your init containers needed secrets, connections, networking... that was being provided via a sidecar container, you were going to have a hard time. With this change, among other things, sidecars containers are going to be available for the whole life cycle of the pod. There are other implications, probably, but I still haven't finished reading the KEP [0]. Check it out, and there you'll find its motivation and several interesting examples. 0: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers Edit: corrected syntax
  - deleted 3y ago
    [deleted]
- chdefrene 3y ago
  The KEP (Kubernetes Enhancement Proposal) is linked to in the PR [1]. From the summary: > Sidecar containers are a new type of containers that start among the Init containers, run through the lifecycle of the Pod and don’t block pod termination. Kubelet makes a best effort to keep them alive and running while other containers are running. [1] https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers https://github.com/kubernetes/enhancements/tree/master/keps/...
yla92 3y ago
A very welcome change. It's gonna be helpful for the case where the database proxy (CloudSQL) and the main container got terminated out of order. https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine https://cloud.google.com/sql/docs/postgres/connect-kubernete...
- giovannibonetti 3y ago
  That is very annoying. I remember having spent some time with this same issue in Google App Engine as well, which also runs Cloud SQL Proxy as a sidecar container. https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/396 https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues...
- numbsafari 3y ago
  The fact that this is needed for so many different things Google pushes, and it has been so slow to make it in, has been very frustrating and telling.
- aranelsurion 3y ago
  Just FYI for people who don't know about it yet: with cloudsql-proxy v2 there's a new parameter called "--quitquitquit" that starts up an HTTP endpoint to be used for graceful shutdowns. Basically your main container makes a POST to this endpoint, and sidecar exits.
fnord77 3y ago
> Pod is terminated even if sidecar is still running this is great for things like Jobs and Istio eliminates the scheme where the main container had to signal to the sidecar it was exiting otherwise the pod would hang
- jeremy_k 3y ago
  Yep, I was looking into running Jobs with Sidecars awhile back and came across this issue. I was actually surprised this morning to see a link on HN be in the "already read" state. Nice to see this feature merged, however our Cluster is on 1.25 I think? Probably a ways away from being able to use this.
Jameskibi07 3y ago
[dead]
jauntywundrkind 3y ago
On the one hand, great. The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about defining pieces of state, but managing blocks of state & multiple things at once is left almost entirely to external tools. The ability to compose &sequence multiple containers feels like a very specific example of a much broader general capability. There's bedevilling infinite complexity to trying to figure out a fully expressive state of state management system - I get why refining a couple specialized existing capabilities is the way - but it does make me a little sad to see a lack of appetite for the broader crosscutting system problem at the root here.
- penciltwirler 3y ago
  There's lots of tools built on top of K8s to accomplish this tho. For example, Argo, Tekton, Flyte etc.
  - jauntywundrkind 3y ago
    Absolutely, no shortage of things atop. Helm is probably the most well used composition tool. It seems unideal to me to forever bunt on this topic, leaving it out of core forever. Especially when we are slowly adding im very specialized composition orchestration tools in core.
    - numbsafari 3y ago
      Helm really solves a different use case than this. This is about describing the desired coordination among running containers. Helm is about how you template or generate your declarative state. You could certainly add this description to your templates with Helm, but you couldn't actually implement this feature with Helm itself.
      - jauntywundrkind 3y ago
        I bundled both composition & orchestration under the same header. It so happens that pods have multiple containers, which is another example of Kubernetes having a specialized specific composition or orchestration implementation. One that started as composition, and here iterates towards orchestration.
nodesocket 3y ago
How does the syntax look for defining a sidecar in a deployment? Is it similar to initContainers?
AtNightWeCode 3y ago
When I first learned about the sidecar pattern I thought it was great. I am not sure about it anymore. Most of it could be propagated to custom images or layers at the boundary. To me this feels a bit sketchy. Too have containers that kinda is part of the mesh but then does not share the same lifecycle as the mesh.
- verst 3y ago
  If you create a custom image you would need to create a complex health endpoint that is essentially only considered healthy if all the components baked into your image are considered healthy. This gets harder when you are not the author of the sidecar process on which you rely. With a single image it would be easier to run into a situation where the sidecar process (baked into your image) is in an unhealthy state but your container is not restarted because the app itself is not reporting unhealthy status.
  - AtNightWeCode 3y ago
    Monolith apps can have many dependency checks and it is not really an issue but I get your point. It can become messy. What I have seen gone into sidecars is TLS-termination, caching, authentication, service clients, metrics and logging. Things I would prefer to have in a dedicated proxy layer or in the images.
nrmitchi 3y ago
While this is a very welcome improvement in terms of functionality, I can't help by feel that the re-use of "restartPolicy" to mean something similar, but different, when used in a different context, is a very poor decision. Kubernetes already has an issue with having a (perceived) high barrier to entry, and I'm not sure that "restartPolicy on a container means this, unless isn't used in this list of containers, in which case it means this". I would have preferred to see a separate attribute (such as `sidecar: true`), rather than overloading (and in my opinion, abusing) the existing `restartPolicy`.
- deleted 3y ago
  [deleted]
- 0xbadcafebee 3y ago
  It's par for the course. Most of K8s's design has been shoving whatever crap they feel like in, regardless of confusion, difficulty, complexity, etc for the end user.
  - perryizgr8 3y ago
    At some level it seems deliberate so that administration of the complexity can be sold to you for a price once you realise that you can't hack it on your own, but are now too invested to back out.
- smarterclayton 3y ago
  The challenge with a separate attribute is that it is not forward compatible with new features we might add to pods around ordering and lifecycle. If we used a simple boolean, eventually we’d have to have it interact with other fields and deal with conflicting behaviors between what “sidecar” means and more flexibility. The only difference today between init containers and regular containers is: a) init containers have an implicit default restart policy of OnFailure, and regular containers inherit the pods restartPolicy b) init containers are serial, regular containers are parallel We are leaving room for the possibility that init containers can fail the pod, and be parallelized, as well as regular containers having unique restartPolicies. Both of those would allow more control for workflow / job engines to break apart monolith containers and get better isolation. The key design point was that “sidecars aren’t special containers” - because we want to leave room for future growth.
cacois 3y ago
In case anyone else was looking for a clear, concise summary of the new feature: "The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination."
- rafaelturk 3y ago
  thank you!
annexrichmond 3y ago
The lack of native sidecar support was my biggest surprise when moving from ECS to EKS, and it was not fun hacking with shared process IDs to accomplish sidecars. I'm glad this is finally in but also curious how it takes roughly 3ish years(?) from KEP proposal to merge?
tmzt 3y ago
Is there a clean way to share an emptyDir between sidecar(s) and main container(s)? Looking at the logging usecase and want to be able to add a log shipper sidecar to a pod with ephemeral storage.
- FridgeSeal 3y ago
  An easier solution for you might be something like vector-which will automatically harvest the logs from pods, and has excellent routing capabilities. You wouldn’t need a sidecar-per-pod this way either.
sargun 3y ago
This is great. My team at Netflix (I'm not longer there) sponsored some of the work behind this, via Kinvolk (now acquired by MSFT). Great to see that it finally shipped. At the time, this was a blocker to us using Kubelet, and we thought it might take a few...months to sort out. Turns out it was closer to a few years, but its a tricky API, and important to get right.

217 points by xdasf 3y ago

KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers https://github.com/kubernetes/enhancements/tree/master/keps/... TLDR: Introduce a restartPolicy field to init containers and use it to indicate that an init container is a sidecar container. Kubelet will start init containers with restartPolicy=Always in the order with other init containers, but instead of waiting for its completion, it will wait for the container startup completion.

hunta2097 3y ago
Hopefully these changes should make Envoy sidecars (and sidecar co-existence in general) more reliable.
- tommiegannert 3y ago
  What's the use-case for Envoy as a sidecar? (As someone using Envoy Gateway.)
  - pluies 3y ago
    Service mesh. Istio for example injects an envoy sidecar to each pod, and manages everything "meshy" (routing, retries, mTLS, etc) via this sidecar. This is also how linkerd works, though they're using their own purpose-built sidecar rather than envoy.
    - rad_gruchalski 3y ago
      Isn’t the new ambient mesh available in istio 1.18 sidecar-less?
      - phrotoma 3y ago
        Yep, it’s also alpha, under intense development, and by every account (including those vendors who are chomping at the bit to start selling it to customers) absolutely not production ready.
        rad_gruchalski 3y ago
        Oh, good to know. I was about to pack a spike into an upcoming sprint.
plagiarist 3y ago
It seems like there could be a better marker for this. Maybe my skill with Kubernetes is too low for it to make sense.

raesene9 3y ago

Worth noting that this is hitting Alpha in Kubernetes 1.28, so won't be available by default at this stage. If you've got self-managed clusters, it'd be possible to enable with a feature gate on the API server, but it's unlikely to be available on managed Kubernetes until it gets to GA.

numbsafari 3y ago
[flagged]

CSDude 3y ago

It's a shame it took so long. If the main container shutdown (i.e connection drain, processing inflight queue items) takes a while, and your service mesh dies (nice go binary) and main container cannot communicate with internet anymore. But I'm not sure about initContainers being used. init keyword implies it'd run and die in order for others to continue. Using restartPolicy with init instead of a dedicated sideCars field feels weird.

zeeZ 3y ago
Without restart policy, a failing init container is retried forever. With a policy of never, the entire pod is marked as having failed. The init containers still have to run and succeed before the main pod continues.
smarterclayton 3y ago
We did that to leave open more complex ordering of both init containers and sidecars (regular containers do not have a restart order). For instance, you might have a service mesh that needs a vault secret - those both might be sidecars, and you may need to ensure the vault sidecar starts first if both go down. Eventually we may want to add parallelism to that start order, and a separate field would prevent simple ordering from working now. Also, these are mostly init containers that run longer, and you want a sidecar not starting to be able to block regular pods, and adding a new container type (like ephemeral containers) is extremely disruptive to other parts of the system (security, observability, and UI), so we looked to minimize that disruption.

MrStonedOne 3y ago

[dead]

sidcool 3y ago

Any documentation on this? What does this mean?

tecleandor 3y ago
So, until now, a sidecar container was just the idea of running containers in you Kubernetes pod, along with your main service, that were 'helpers' for something: connection to databases or vpns, mesh networking, pulling secrets or config, debugging... But they didn't have special status, they were just regular containers in your pod. This sometimes posed some problems because they weren't available for the full life cycle of the pod, notably on the init process. So if your init containers needed secrets, connections, networking... that was being provided via a sidecar container, you were going to have a hard time. With this change, among other things, sidecars containers are going to be available for the whole life cycle of the pod. There are other implications, probably, but I still haven't finished reading the KEP [0]. Check it out, and there you'll find its motivation and several interesting examples. 0: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers Edit: corrected syntax
- deleted 3y ago
  [deleted]
chdefrene 3y ago
The KEP (Kubernetes Enhancement Proposal) is linked to in the PR [1]. From the summary: > Sidecar containers are a new type of containers that start among the Init containers, run through the lifecycle of the Pod and don’t block pod termination. Kubelet makes a best effort to keep them alive and running while other containers are running. [1] https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers https://github.com/kubernetes/enhancements/tree/master/keps/...

yla92 3y ago

A very welcome change. It's gonna be helpful for the case where the database proxy (CloudSQL) and the main container got terminated out of order. https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine https://cloud.google.com/sql/docs/postgres/connect-kubernete...

giovannibonetti 3y ago
That is very annoying. I remember having spent some time with this same issue in Google App Engine as well, which also runs Cloud SQL Proxy as a sidecar container. https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/396 https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues...
numbsafari 3y ago
The fact that this is needed for so many different things Google pushes, and it has been so slow to make it in, has been very frustrating and telling.
aranelsurion 3y ago
Just FYI for people who don't know about it yet: with cloudsql-proxy v2 there's a new parameter called "--quitquitquit" that starts up an HTTP endpoint to be used for graceful shutdowns. Basically your main container makes a POST to this endpoint, and sidecar exits.

fnord77 3y ago

> Pod is terminated even if sidecar is still running this is great for things like Jobs and Istio eliminates the scheme where the main container had to signal to the sidecar it was exiting otherwise the pod would hang

jeremy_k 3y ago
Yep, I was looking into running Jobs with Sidecars awhile back and came across this issue. I was actually surprised this morning to see a link on HN be in the "already read" state. Nice to see this feature merged, however our Cluster is on 1.25 I think? Probably a ways away from being able to use this.

Jameskibi07 3y ago

[dead]

jauntywundrkind 3y ago

On the one hand, great. The other hand, one of the main criticisms of Kubernetes is that it has no composition or orchestration capabilities. It's great about defining pieces of state, but managing blocks of state & multiple things at once is left almost entirely to external tools. The ability to compose &sequence multiple containers feels like a very specific example of a much broader general capability. There's bedevilling infinite complexity to trying to figure out a fully expressive state of state management system - I get why refining a couple specialized existing capabilities is the way - but it does make me a little sad to see a lack of appetite for the broader crosscutting system problem at the root here.

penciltwirler 3y ago
There's lots of tools built on top of K8s to accomplish this tho. For example, Argo, Tekton, Flyte etc.
- jauntywundrkind 3y ago
  Absolutely, no shortage of things atop. Helm is probably the most well used composition tool. It seems unideal to me to forever bunt on this topic, leaving it out of core forever. Especially when we are slowly adding im very specialized composition orchestration tools in core.
  - numbsafari 3y ago
    Helm really solves a different use case than this. This is about describing the desired coordination among running containers. Helm is about how you template or generate your declarative state. You could certainly add this description to your templates with Helm, but you couldn't actually implement this feature with Helm itself.
    - jauntywundrkind 3y ago
      I bundled both composition & orchestration under the same header. It so happens that pods have multiple containers, which is another example of Kubernetes having a specialized specific composition or orchestration implementation. One that started as composition, and here iterates towards orchestration.

nodesocket 3y ago

How does the syntax look for defining a sidecar in a deployment? Is it similar to initContainers?

AtNightWeCode 3y ago

When I first learned about the sidecar pattern I thought it was great. I am not sure about it anymore. Most of it could be propagated to custom images or layers at the boundary. To me this feels a bit sketchy. Too have containers that kinda is part of the mesh but then does not share the same lifecycle as the mesh.

verst 3y ago
If you create a custom image you would need to create a complex health endpoint that is essentially only considered healthy if all the components baked into your image are considered healthy. This gets harder when you are not the author of the sidecar process on which you rely. With a single image it would be easier to run into a situation where the sidecar process (baked into your image) is in an unhealthy state but your container is not restarted because the app itself is not reporting unhealthy status.
- AtNightWeCode 3y ago
  Monolith apps can have many dependency checks and it is not really an issue but I get your point. It can become messy. What I have seen gone into sidecars is TLS-termination, caching, authentication, service clients, metrics and logging. Things I would prefer to have in a dedicated proxy layer or in the images.

nrmitchi 3y ago

While this is a very welcome improvement in terms of functionality, I can't help by feel that the re-use of "restartPolicy" to mean something similar, but different, when used in a different context, is a very poor decision. Kubernetes already has an issue with having a (perceived) high barrier to entry, and I'm not sure that "restartPolicy on a container means this, unless isn't used in this list of containers, in which case it means this". I would have preferred to see a separate attribute (such as `sidecar: true`), rather than overloading (and in my opinion, abusing) the existing `restartPolicy`.

deleted 3y ago
[deleted]
0xbadcafebee 3y ago
It's par for the course. Most of K8s's design has been shoving whatever crap they feel like in, regardless of confusion, difficulty, complexity, etc for the end user.
- perryizgr8 3y ago
  At some level it seems deliberate so that administration of the complexity can be sold to you for a price once you realise that you can't hack it on your own, but are now too invested to back out.
smarterclayton 3y ago
The challenge with a separate attribute is that it is not forward compatible with new features we might add to pods around ordering and lifecycle. If we used a simple boolean, eventually we’d have to have it interact with other fields and deal with conflicting behaviors between what “sidecar” means and more flexibility. The only difference today between init containers and regular containers is: a) init containers have an implicit default restart policy of OnFailure, and regular containers inherit the pods restartPolicy b) init containers are serial, regular containers are parallel We are leaving room for the possibility that init containers can fail the pod, and be parallelized, as well as regular containers having unique restartPolicies. Both of those would allow more control for workflow / job engines to break apart monolith containers and get better isolation. The key design point was that “sidecars aren’t special containers” - because we want to leave room for future growth.

cacois 3y ago

In case anyone else was looking for a clear, concise summary of the new feature: "The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination."

rafaelturk 3y ago
thank you!

annexrichmond 3y ago

The lack of native sidecar support was my biggest surprise when moving from ECS to EKS, and it was not fun hacking with shared process IDs to accomplish sidecars. I'm glad this is finally in but also curious how it takes roughly 3ish years(?) from KEP proposal to merge?

tmzt 3y ago

Is there a clean way to share an emptyDir between sidecar(s) and main container(s)? Looking at the logging usecase and want to be able to add a log shipper sidecar to a pod with ephemeral storage.

FridgeSeal 3y ago
An easier solution for you might be something like vector-which will automatically harvest the logs from pods, and has excellent routing capabilities. You wouldn’t need a sidecar-per-pod this way either.

sargun 3y ago

This is great. My team at Netflix (I'm not longer there) sponsored some of the work behind this, via Kinvolk (now acquired by MSFT). Great to see that it finally shipped. At the time, this was a blocker to us using Kubelet, and we thought it might take a few...months to sort out. Turns out it was closer to a few years, but its a tricky API, and important to get right.