13 ms·

eBPF will help solve service mesh by getting rid of sidecars

237 points by tgraf 5y ago

unmole 5y ago
Offtopic: I really like the style of the diagrams. I remember seeing something similar elsewhere. Are this manually drawn or is this the result of some tool?
- tgraf 5y ago
  OP here: It's whimsical.com. I really love it.
  - unmole 5y ago
    Thank you, Thomas! I really admire all that you have done with Cilium.
dijit 5y ago
Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. Since some number of persistent connections will get force terminated on scale down or node replacement events... Cilium and eBPF looks like a pretty good solution to this though since you can then advertise your pods directly on the network and load balance those instead of every node.
- p_l 5y ago
  Whether load balancer can or can-not tell the nodes apart depends on load balancer and method you use to expose your service to it, as well as what kind of networking setup you use (i.e. is pod networking sensibly exposed to load balancer or ... weirdly) Each "Service" object provides (by default, can be disabled) load-balanced IP address that by default uses kube-proxy as you described, a DNS A record pointing to said address, DNS SRV records pointing to actual direct connections (whether NodePorts or PodIP/port combinations) plus API access to get the same data out. There are even replacement kube-proxy implementations that route everything through F5 load balancer boxes, but they are less known.
- q3k 5y ago
  > Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. There can be a difference, if your LoadBalancer-type service integration is well implemented. The externalTrafficPolicy knob determines whether all nodes should attract traffic from outside or only nodes that contain pods backing this service. For example, metallb (which attracts traffic by /32 BGP announcements to given external peers) will do this correctly. Within the cluster itself, only nodes which have pods backing a given service will be part of the iptables/ipvs/... Pod->Service->Pod mesh, so you won't end up with scenic routes anyway. Same for Pod->Pod networking, as these addresses are already clustered by host node.
  - kklimonda 5y ago
    How do you keep ecmp hashing stable between rollouts?
    - dharmab 5y ago
      If you're asking about connection stability in general: - Ideally, you avoid it in your application design. - If you need it, you set up SIGTERM handling in the application to wait for all connections to close before the process exits. You also set up "connection draining" at the load balancer to keep existing sessions to terminating Pods open but send new sessions to the new Pods. The tradeoff is that rollouts take much longer- if the session time is unbounded, you may need to enforce a deadline to break connections eventua.
      - dilyevsky 5y ago
        You dont just wait until all connections exit, you first need to withdraw bgp announcement to the edge router, then start the wait. It’s not that simple with metal LBs. On the other hand it’s not that simple with cloud LBs either bc they also break long tcp streams when they please
- dharmab 5y ago
  That's if you're using a NodePort service, which the documentation explains is for niche use cases such as if you don't have a compatible dedicated load balancer. In most professional setups you do have such a load balancer and can use other types of routing that avoid this. https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types https://kubernetes.io/docs/concepts/services-networking/serv...
  - topspin 5y ago
    > In most professional setups you do have such a load balancer May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? Does IPv6 address any of these issues? It seems to me that IPv6 is capable of providing every component in the system its own globally routable address, identity (mTLS perhaps) and transparent encryption with no extra sidecars, eBPF pieces, etc.
    - shosti 5y ago
      Ingresses on EKS will set up an ALB that sends traffic directly to pods instead of nodes (basically skips the whole K8s Service/NodePort networking setup). You have to use ` alb.ingress.kubernetes.io/target-type: ip` as an annotation I think (see https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress...).
    - deleted 5y ago
      [deleted]
    - dharmab 5y ago
      > May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? The AWS cloud controller will automatically set up an ALB for you if you configure a LoadBalancer service in Kubernetes. I've also done custom setups with AWS NLBs. > Does IPv6 address any of these issues? It could address some issues- you could conceivably create a CNI plugin which allocates an externally addressable IP to your Pods. Although you would probably still want a load balancer for custom routing rules and the improved reliability over DNS round robin.
- pm90 5y ago
  This is a concern only if you have ungraceful node termination Ie you suddenly yoink the node. In most cases when you terminate the node, k8s will (attempt to) cordon and drain the nodes, letting the pods gracefully terminate the connections before getting evicted. If you didn’t have k8s and just used an autoscaling group of VMs you would have the same issue…
zdw 5y ago
So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in arbitrarily fancy service discovery and load-balancing. " Which seems like a better solution, if requiring some reengineering of apps.
- jrockway 5y ago
  The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authorization decisions. (I suppose in some sense I like to make sure that "kubectl port-forward" doesn't have magical enhanced privileges, which it does if your app is oblivious to the mTLS going on in the background. You could disable that specifically in your k8s setup, but generally security through remembering to disable default features seems like a losing battle to me. Easier to have the app say "yeah you need a key". Just make sure you build the feature to let oncall get a key, or they will be very sad.) For that reason, I really do think that this is a temporary hack while client libraries are brought up to speed in popular languages. It is really easy to sell stuff with "just add another component to your house of cards to get feature X", but eventually it's all too much and you'll have to just edit your code. I personally don't use service meshes. I have played with Istio but the code is legitimately awful, so the anecdotes of "I've never seen it work" make perfect sense to me. I have, in fact, never seen it work. (Read the xDS spec, then read Istio's implementation. Errors? Just throw them away! That's the core goal of the project, it seems. I wrote my own xDS implementation that ... handles errors and NACKs correctly. Wow, such an engineering marvel and so difficult...) I do stick Envoy in front of things when it seems appropriate. For example, I'll put Envoy in front of a split frontend/backend application to provide one endpoint that serves both the frontend or backend. That way production is identical to your local development environment, avoiding surprises at the worst possible time. I also put it in front of applications that I don't feel like editing and rebuilding to get metrics and traces. The one feature that I've been missing from service meshes, Kubernetes networking plugins, etc. is the ability to make all traffic leave the cluster through a single set of services, who can see the cleartext of TLS transactions. (I looked at Istio specifically, because it does have EgressGateways, but it's implemented at the TCP level and not the HTTP level. So you don't see outgoing URLs, just outgoing IP addresses. And if someone is exfiltrating data, you can't log that.) My biggest concern with running things in production is not so much internal security, though that is a big concern, but rather "is my cluster abusing someone else". That's the sort of thing that gets your cloud account shut down without appeal, and I feel like I don't have good tooling to stop that right now.
codetrotter 5y ago
> Identity-based Security: Relying on network identifiers to achieve security is no longer sufficient, both the sending and receiving services must be able to authenticate each other based on identities instead of a network identifier. Kinda semi-offtopic but I am curious to know if anyone has used identity part of a WireGuard setup for this purpose. So say you have a bunch of machines all connected in a WireGuard VPN. And then instead of your application knowing host names or IP addresses as the primary identifier of other nodes, your application refers to other nodes by their WireGuard public key? I use WireGuard but haven’t tried anything like that. Don’t know if it would be possible or sensible. Just thinking and wondering.
- q3k 5y ago
  You'd be adding a whole new layer of what would effectively be dynamic routing. It's doable, but it's not a trivial amount of effort. Especially if you want everything to be transparent and automagic. There's earlier projects like CJDNS which provide pubkey-addressed networking, but they're limited in usability as they route based on a DHT.
- madjam002 5y ago
  I too am interested in this. I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags". Tailscale seems to be pretty much this, but while it seems great for the dev/user facing side of things (developer machine connectivity), it doesn't seem like it's suited for the service to service communication side? It would be nice to have one unified connectivity solution with identity based security rather than e.g Consul Connect for services, Tailscale / Wireguard for dev machine connectivity, etc.
  - starfallg 5y ago
    >I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags". That's exactly what Scalable Group Tags (SGTs) are - https://tools.ietf.org/id/draft-smith-kandula-sxp-07.html https://tools.ietf.org/id/draft-smith-kandula-sxp-07.html Cisco implements this as a part of TrustSec
outside1234 5y ago
There is a good talk about this (and more) from KubeCon: https://www.youtube.com/watch?v=KY5qujcujfI https://www.youtube.com/watch?v=KY5qujcujfI
xmodem 5y ago
Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.
- mixedCase 5y ago
  There are good reasons more often than not. Being able to pick up something generic rather than something language-specific. Not having to do process supervision (which includes handling monitoring and logs) within your application. Not making the application lifecycle subservient to needs such as log shipping and request rerouting. People get sig traps wrong suprisingly often.
  - taeric 5y ago
    My gut is that using sidecars doesn't really solve these problems straight up. Just moved them to the orchestrator. Which is not bad. But that area is also often misconfigured for supervision. And trapping signals remains mostly broken in all sidecars.
- cfors 5y ago
  You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of the potential SDK's for teams. Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that. Also, there is a decent amount of work being done on gRPC to speak XDS which also removes the need for the sidecar [0]. [0] https://istio.io/latest/blog/2021/proxyless-grpc/ https://istio.io/latest/blog/2021/proxyless-grpc/
  - 5y ago
Matthias247 5y ago
I understand how BPF works for transparently steering TCP connections. But the article mentions gRPC - which means HTTP2. How can the BPF module be a replacement for a proxy here. My understanding is it would need to understand http2 framing and having buffers - which all sound like capabilities that require more than BPF? Are they implementing a http2 capable proxy in native kernel C code and making APIs to that accessible via bpf?
- tgraf 5y ago
  The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of security benefits because it runs in a sandboxed environment. We are doing both. Aspect 2) is currently done for HTTP visibility and we will be working on connection splicing and HTTP header mutation going forward.
  - tptacek 5y ago
    What does an HTTP parser written in BPF look like? Bounded loops only --- meaning no string libraries --- seems like a hell of a constraint there.
    - tgraf 5y ago
      It looks not too different from the majority of HTTP parsers out there written in C. Here is an example of NodeJS [0]. [0] https://github.com/nodejs/http-parser/blob/main/http_parser.c https://github.com/nodejs/http-parser/blob/main/http_parser....
      - tptacek 5y ago
        Node's HTTP parser doesn't have to placate the BPF verifier, is why I'm asking.
davewritescode 5y ago
From a resource perspective this makes sense but from a security perspective this drives me a little bit crazy. Sidecars aren't just for managing traffic, they're also a good way to automate managing the security context of the pod itself. The current security model in Istio delivers a pod specific SPIFFE cert to only that pod and pod identity is conveyed via that certificate. That feels like a whole bunch of eggs in 1 basket.
- tgraf 5y ago
  What the proposed architecture allows is to continue using SPIFFE or another certificate management solution to generate and distribute the certificates but use either a per-node proxy or an eBPF implementation to enforce it. Even if the authentication handshake remains in a proxy but data encryption moves to the kernel then that is a massive benefit from an overhead perspective. This already exists and is called kTLS.
manvendrasingh 5y ago
I am wondering how would this solve the problem of mTLS while still supporting service level identities? Is it possible to move the mTLS to listeners instead of sidecar or some other mechanism?
ko27 5y ago
Not convinced that this a better solution then just implementing these features as part of the protocol. For example, most languages have libraries that support grpc load balancing. https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md https://github.com/grpc/proposal/blob/master/A27-xds-global-...
zinclozenge 5y ago
It's not clear how eBPF will deal with mTLS. I actually asked that when interviewing at a company using eBPF for observability into Kubernetes the answer was they didn't know. Yea, if you're getting TLS termination at the load balancer prior to k8s ingress then it's pretty nice.
- GauntletWizard 5y ago
  The answer to this is simple - TLS will start being terminated at the pods themselves. The frontend load balancer will also terminate TLS - to the public sphere, and then will authenticate it's connection to your backends as well. Kubernetes will provide x509 certificates suitable for service-to-service communications to pods automatically. The work is still in the early phases, so the exact form this will take has yet to be hammered out, but there's broad agreement that this functionality will be first-class in k8s in the future. If you want to keep running proxies for the other feature they provide, great - They'll be able to use the certificates provided by k8s for identity. If you'd like to know more, come to on of the SIGAUTH Meetings :)
- tgraf 5y ago
  Then you should interview again but with us. This is not too different from wpa_supplicant used by several operating for key management for wireless networks. The complicated key negotiation and authentication can remain in user space, the encryption of the negotiated key can be done in the kernel (kTLS) or, when eBPF can control both sides, it can even be done without using TLS but encrypting using a network level encapsulation format to it works for non-TCP as well. Hint: We are hiring.

237 points by tgraf 5y ago

Offtopic: I really like the style of the diagrams. I remember seeing something similar elsewhere. Are this manually drawn or is this the result of some tool?

tgraf 5y ago
OP here: It's whimsical.com. I really love it.
- unmole 5y ago
  Thank you, Thomas! I really admire all that you have done with Cilium.

dijit 5y ago

Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. Since some number of persistent connections will get force terminated on scale down or node replacement events... Cilium and eBPF looks like a pretty good solution to this though since you can then advertise your pods directly on the network and load balance those instead of every node.

p_l 5y ago
Whether load balancer can or can-not tell the nodes apart depends on load balancer and method you use to expose your service to it, as well as what kind of networking setup you use (i.e. is pod networking sensibly exposed to load balancer or ... weirdly) Each "Service" object provides (by default, can be disabled) load-balanced IP address that by default uses kube-proxy as you described, a DNS A record pointing to said address, DNS SRV records pointing to actual direct connections (whether NodePorts or PodIP/port combinations) plus API access to get the same data out. There are even replacement kube-proxy implementations that route everything through F5 load balancer boxes, but they are less known.
q3k 5y ago
> Honestly after I learned that the majority of Kubernetes nodes just proxy traffic between each other using iptables and that a load balancer can't tell the nodes apart (ones where your app lives vs ones that will proxy connection to your app) I got really worried about any kind of persistent connection in k8s land. There can be a difference, if your LoadBalancer-type service integration is well implemented. The externalTrafficPolicy knob determines whether all nodes should attract traffic from outside or only nodes that contain pods backing this service. For example, metallb (which attracts traffic by /32 BGP announcements to given external peers) will do this correctly. Within the cluster itself, only nodes which have pods backing a given service will be part of the iptables/ipvs/... Pod->Service->Pod mesh, so you won't end up with scenic routes anyway. Same for Pod->Pod networking, as these addresses are already clustered by host node.
- kklimonda 5y ago
  How do you keep ecmp hashing stable between rollouts?
  - dharmab 5y ago
    If you're asking about connection stability in general: - Ideally, you avoid it in your application design. - If you need it, you set up SIGTERM handling in the application to wait for all connections to close before the process exits. You also set up "connection draining" at the load balancer to keep existing sessions to terminating Pods open but send new sessions to the new Pods. The tradeoff is that rollouts take much longer- if the session time is unbounded, you may need to enforce a deadline to break connections eventua.
    - dilyevsky 5y ago
      You dont just wait until all connections exit, you first need to withdraw bgp announcement to the edge router, then start the wait. It’s not that simple with metal LBs. On the other hand it’s not that simple with cloud LBs either bc they also break long tcp streams when they please
dharmab 5y ago
That's if you're using a NodePort service, which the documentation explains is for niche use cases such as if you don't have a compatible dedicated load balancer. In most professional setups you do have such a load balancer and can use other types of routing that avoid this. https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types https://kubernetes.io/docs/concepts/services-networking/serv...
- topspin 5y ago
  > In most professional setups you do have such a load balancer May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? Does IPv6 address any of these issues? It seems to me that IPv6 is capable of providing every component in the system its own globally routable address, identity (mTLS perhaps) and transparent encryption with no extra sidecars, eBPF pieces, etc.
  - shosti 5y ago
    Ingresses on EKS will set up an ALB that sends traffic directly to pods instead of nodes (basically skips the whole K8s Service/NodePort networking setup). You have to use ` alb.ingress.kubernetes.io/target-type: ip` as an annotation I think (see https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress...).
  - deleted 5y ago
    [deleted]
  - dharmab 5y ago
    > May I ask what one might use in an AWS cloud environment to provide that load balancer within a Region? The AWS cloud controller will automatically set up an ALB for you if you configure a LoadBalancer service in Kubernetes. I've also done custom setups with AWS NLBs. > Does IPv6 address any of these issues? It could address some issues- you could conceivably create a CNI plugin which allocates an externally addressable IP to your Pods. Although you would probably still want a load balancer for custom routing rules and the improved reliability over DNS round robin.
pm90 5y ago
This is a concern only if you have ungraceful node termination Ie you suddenly yoink the node. In most cases when you terminate the node, k8s will (attempt to) cordon and drain the nodes, letting the pods gracefully terminate the connections before getting evicted. If you didn’t have k8s and just used an autoscaling group of VMs you would have the same issue…

zdw 5y ago

So instead of making the applications use a good RPC library, we're going to shove more crap into the kernel? No thanks, from a security context and complexity perspective. Per https://blog.dave.tf/post/new-kubernetes/ https://blog.dave.tf/post/new-kubernetes/ , the way that this was solved in Borg was: > "Borg solves that complexity by fiat, decreeing that Thou Shalt Use Our Client Libraries For Everything, so there’s an obvious point at which to plug in arbitrarily fancy service discovery and load-balancing. " Which seems like a better solution, if requiring some reengineering of apps.

jrockway 5y ago
The big secret is that sidecars can only help so much. If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars). mTLS is similar; it's great to encrypt your internal traffic on the wire, but that needs to get propagated up to the application to make internal authorization decisions. (I suppose in some sense I like to make sure that "kubectl port-forward" doesn't have magical enhanced privileges, which it does if your app is oblivious to the mTLS going on in the background. You could disable that specifically in your k8s setup, but generally security through remembering to disable default features seems like a losing battle to me. Easier to have the app say "yeah you need a key". Just make sure you build the feature to let oncall get a key, or they will be very sad.) For that reason, I really do think that this is a temporary hack while client libraries are brought up to speed in popular languages. It is really easy to sell stuff with "just add another component to your house of cards to get feature X", but eventually it's all too much and you'll have to just edit your code. I personally don't use service meshes. I have played with Istio but the code is legitimately awful, so the anecdotes of "I've never seen it work" make perfect sense to me. I have, in fact, never seen it work. (Read the xDS spec, then read Istio's implementation. Errors? Just throw them away! That's the core goal of the project, it seems. I wrote my own xDS implementation that ... handles errors and NACKs correctly. Wow, such an engineering marvel and so difficult...) I do stick Envoy in front of things when it seems appropriate. For example, I'll put Envoy in front of a split frontend/backend application to provide one endpoint that serves both the frontend or backend. That way production is identical to your local development environment, avoiding surprises at the worst possible time. I also put it in front of applications that I don't feel like editing and rebuilding to get metrics and traces. The one feature that I've been missing from service meshes, Kubernetes networking plugins, etc. is the ability to make all traffic leave the cluster through a single set of services, who can see the cleartext of TLS transactions. (I looked at Istio specifically, because it does have EgressGateways, but it's implemented at the TCP level and not the HTTP level. So you don't see outgoing URLs, just outgoing IP addresses. And if someone is exfiltrating data, you can't log that.) My biggest concern with running things in production is not so much internal security, though that is a big concern, but rather "is my cluster abusing someone else". That's the sort of thing that gets your cloud account shut down without appeal, and I feel like I don't have good tooling to stop that right now.

codetrotter 5y ago

> Identity-based Security: Relying on network identifiers to achieve security is no longer sufficient, both the sending and receiving services must be able to authenticate each other based on identities instead of a network identifier. Kinda semi-offtopic but I am curious to know if anyone has used identity part of a WireGuard setup for this purpose. So say you have a bunch of machines all connected in a WireGuard VPN. And then instead of your application knowing host names or IP addresses as the primary identifier of other nodes, your application refers to other nodes by their WireGuard public key? I use WireGuard but haven’t tried anything like that. Don’t know if it would be possible or sensible. Just thinking and wondering.

q3k 5y ago
You'd be adding a whole new layer of what would effectively be dynamic routing. It's doable, but it's not a trivial amount of effort. Especially if you want everything to be transparent and automagic. There's earlier projects like CJDNS which provide pubkey-addressed networking, but they're limited in usability as they route based on a DHT.
madjam002 5y ago
I too am interested in this. I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags". Tailscale seems to be pretty much this, but while it seems great for the dev/user facing side of things (developer machine connectivity), it doesn't seem like it's suited for the service to service communication side? It would be nice to have one unified connectivity solution with identity based security rather than e.g Consul Connect for services, Tailscale / Wireguard for dev machine connectivity, etc.
- starfallg 5y ago
  >I long for the day where Kubernetes services, virtual machines, dedicated servers and developer machines can all securely talk to eachother in some kind of service mesh, where security and firewalls can be implemented with "tags". That's exactly what Scalable Group Tags (SGTs) are - https://tools.ietf.org/id/draft-smith-kandula-sxp-07.html https://tools.ietf.org/id/draft-smith-kandula-sxp-07.html Cisco implements this as a part of TrustSec

outside1234 5y ago

There is a good talk about this (and more) from KubeCon: https://www.youtube.com/watch?v=KY5qujcujfI https://www.youtube.com/watch?v=KY5qujcujfI

xmodem 5y ago

Doing this with eBPF is definitely an improvement, but when I look at some of the sidecars we run in production, I often wonder why we can't just... integrate them into the application.

mixedCase 5y ago
There are good reasons more often than not. Being able to pick up something generic rather than something language-specific. Not having to do process supervision (which includes handling monitoring and logs) within your application. Not making the application lifecycle subservient to needs such as log shipping and request rerouting. People get sig traps wrong suprisingly often.
- taeric 5y ago
  My gut is that using sidecars doesn't really solve these problems straight up. Just moved them to the orchestrator. Which is not bad. But that area is also often misconfigured for supervision. And trapping signals remains mostly broken in all sidecars.
cfors 5y ago
You can! There are downsides though for any sufficiently polyglot organization, which is maintaining all the different client SDK's that need to use that. Sidecars are often useful for platform-centric teams that would like to have access to help manage something like secrets, mTLS, or traffic shaping in the case of Envoy. The team that's responsible for that just needs to maintain a single sidecar rather than all of the potential SDK's for teams. Especially if you have specific sidecars that only work on a specific infrastructure, for example if you have a Vault sidecar that deals with secrets for your service over EKS IAM permissions, you suddenly can't start your service without a decent amount of mocking and feature flags. Its nice to not have to burden your client code with all of that. Also, there is a decent amount of work being done on gRPC to speak XDS which also removes the need for the sidecar [0]. [0] https://istio.io/latest/blog/2021/proxyless-grpc/ https://istio.io/latest/blog/2021/proxyless-grpc/
- 5y ago

Matthias247 5y ago

I understand how BPF works for transparently steering TCP connections. But the article mentions gRPC - which means HTTP2. How can the BPF module be a replacement for a proxy here. My understanding is it would need to understand http2 framing and having buffers - which all sound like capabilities that require more than BPF? Are they implementing a http2 capable proxy in native kernel C code and making APIs to that accessible via bpf?

tgraf 5y ago
The model I'm describing contains two pieces: 1) Moving away from sidecars to per-node proxies that can be better integrated into the Linux kernel concept of namespacing instead of artificially injecting them with complicated iptables redirection logic at the network level. 2) Providing the HTTP awareness directly with eBPF using eBPF-based protocol parsers. The parser itself is written in eBPF which has a ton of security benefits because it runs in a sandboxed environment. We are doing both. Aspect 2) is currently done for HTTP visibility and we will be working on connection splicing and HTTP header mutation going forward.
- tptacek 5y ago
  What does an HTTP parser written in BPF look like? Bounded loops only --- meaning no string libraries --- seems like a hell of a constraint there.
  - tgraf 5y ago
    It looks not too different from the majority of HTTP parsers out there written in C. Here is an example of NodeJS [0]. [0] https://github.com/nodejs/http-parser/blob/main/http_parser.c https://github.com/nodejs/http-parser/blob/main/http_parser....
    - tptacek 5y ago
      Node's HTTP parser doesn't have to placate the BPF verifier, is why I'm asking.

davewritescode 5y ago

From a resource perspective this makes sense but from a security perspective this drives me a little bit crazy. Sidecars aren't just for managing traffic, they're also a good way to automate managing the security context of the pod itself. The current security model in Istio delivers a pod specific SPIFFE cert to only that pod and pod identity is conveyed via that certificate. That feels like a whole bunch of eggs in 1 basket.

tgraf 5y ago
What the proposed architecture allows is to continue using SPIFFE or another certificate management solution to generate and distribute the certificates but use either a per-node proxy or an eBPF implementation to enforce it. Even if the authentication handshake remains in a proxy but data encryption moves to the kernel then that is a massive benefit from an overhead perspective. This already exists and is called kTLS.

manvendrasingh 5y ago

I am wondering how would this solve the problem of mTLS while still supporting service level identities? Is it possible to move the mTLS to listeners instead of sidecar or some other mechanism?

ko27 5y ago

Not convinced that this a better solution then just implementing these features as part of the protocol. For example, most languages have libraries that support grpc load balancing. https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md https://github.com/grpc/proposal/blob/master/A27-xds-global-...

zinclozenge 5y ago

It's not clear how eBPF will deal with mTLS. I actually asked that when interviewing at a company using eBPF for observability into Kubernetes the answer was they didn't know. Yea, if you're getting TLS termination at the load balancer prior to k8s ingress then it's pretty nice.

GauntletWizard 5y ago
The answer to this is simple - TLS will start being terminated at the pods themselves. The frontend load balancer will also terminate TLS - to the public sphere, and then will authenticate it's connection to your backends as well. Kubernetes will provide x509 certificates suitable for service-to-service communications to pods automatically. The work is still in the early phases, so the exact form this will take has yet to be hammered out, but there's broad agreement that this functionality will be first-class in k8s in the future. If you want to keep running proxies for the other feature they provide, great - They'll be able to use the certificates provided by k8s for identity. If you'd like to know more, come to on of the SIGAUTH Meetings :)
tgraf 5y ago
Then you should interview again but with us. This is not too different from wpa_supplicant used by several operating for key management for wireless networks. The complicated key negotiation and authentication can remain in user space, the encryption of the negotiated key can be done in the kernel (kTLS) or, when eBPF can control both sides, it can even be done without using TLS but encrypting using a network level encapsulation format to it works for non-TCP as well. Hint: We are hiring.