6 ms·

Retina – eBPF distributed networking observability tool for Kubernetes

166 points by boratanrikulu 3y ago

maxboone 3y ago
Probably this means it's not for me, but what is this useful for? Anyone using this in their prod set-up and has a scenario where they found this useful?
- p_l 3y ago
  Just today at $DAYJOB we had a complaint that one teams Azure Kubernetes clusters was "slow". About only metric out of norm was network traffic - but lack of detailed instrumentation meant we couldn't really isolate the cause to specific container or process
  - hacker_newz 3y ago
    If the only explanation someone can provide is that "a cluster is slow", the issue isn't with network observability. They need to do at least the minimum level of analysis before escalating.
    - p_l 3y ago
      Unfortunately we were the right team for escalation before escalating to network team.
    - chronid 3y ago
      Yes, that would be great, but unfortunately there are application teams (particularly in the enterprise) lacking such tact when blaming infrastructure for issues. Good old silos are alive and well, and ownership is not always part of the culture.
      - p_l 3y ago
        In our case the expected golden path is that once our team figures the proper procedure, we will establish it for the downstream teams that are direct supports of the application teams. So at least in theory things are somewhat well set up, but there's too much siloing at our level (wildly separate network teams, teams for specific clouds, etc.)
- nijave 3y ago
  Haven't used this but I tried out Pixie trying to debug where outgoing traffic was coming from and where it was going and was fairly successful although Pixie wasn't very stable/had a lot of issues causing crashes. In this case, we had a couple services talking to 3rd party services running on AWS so it wasn't obvious from generic flow logs. I also used Lacework a couple years ago which is eBPF based and it was pretty trivial to see things phoning home or one off maintenance where a new connection was being initiated.
- FridgeSeal 3y ago
  It’s like Cilium + Hubble but useful for you don’t/can’t run cilium. Uses eBPF to collect metrics and stats on what flows where, can record an impressive amount of stuff, without any required instrumentation of your applications. Amazingly handy for when you run both first party and 3rd party apps in your K8s cluster. The network maps these tools produce are handy too. Although, Cilium is pretty great, so not sure why you wouldn’t run it, given the option…
  - hosh 3y ago
    Cillium has been bought out by Cisco, so its monetization is only a matter of time. Also, not everyone needs to implement a service mesh.
    - FridgeSeal 3y ago
      Neither Cilium nor Hubble are service meshes. Cilium is a CNI - the functionality that provides the K8s cluster inter-pod networking. The fact that it uses eBPF to deliver its functionality is what gives it the impressive observability you usually only get from a service mesh. I agree that not everyone needs a service mesh.
      - nullify88 3y ago
        You are right that cilium is a CNI but one of its many features is also providing a sidecar-less service mesh using eBPF. https://cilium.io/use-cases/service-mesh/ https://cilium.io/use-cases/service-mesh/
    - candiddevmike 3y ago
      Cillium is a CNCF project and Apache 2 licensed. Isovalent and their enterprise product were bought by Cisco.
      - deleted 3y ago
        [deleted]
- emmelaich 3y ago
  The linked docs provide more info: https://retina.sh/docs/intro https://retina.sh/docs/intro
orisho 3y ago
See also: Network Mapper - low privileges, no-eBPF network observability tool for K8s https://news.ycombinator.com/item?id=39761114 https://news.ycombinator.com/item?id=39761114
scottlamb 3y ago
Sigh, anyone have a name suggestion for a (Rust) RTSP library? https://crates.io/crates/retina https://crates.io/crates/retina
- emmanueloga_ 3y ago
  https://deadcells.fandom.com/wiki/Conjunctivius https://deadcells.fandom.com/wiki/Conjunctivius :-p
- karolist 3y ago
  The name collisions for opensource projects stopped mattering a long time ago
- nextaccountic 3y ago
  Don't change the name of this crate, it is in an unrelated domain. (Also unrelated to retina displays and uhh.. this Brazilian intrusion detection system https://sunsoftware.com.br/retina/ https://sunsoftware.com.br/retina/ and whatever this thing is https://retina.ai/ https://retina.ai/ among other things)
emmanueloga_ 3y ago
There is a flood of observability tools based on eBPF coming out these days [1]. eBPF is used to collect metrics without the need to instrument the code. -- 1: https://ebpf.io/applications/ https://ebpf.io/applications/
- sharangxy 3y ago
  DeepFlow [1][2] is one of them, where we implemented distributed tracing for microservices using eBPF. 1. https://deepflow.io https://deepflow.io 2. https://github.com/deepflowio/deepflow https://github.com/deepflowio/deepflow
xyst 3y ago
Speaking of observability tools. Anybody here know how to gather more in-depth metrics on mTLS requests? Have an internal (self signed) CA and just want to know which issued certs are presented to nodes. Would be nice to get cert serial number and other metadata as well
- vamsi1105 3y ago
  https://github.com/microsoft/retina/issues/85 https://github.com/microsoft/retina/issues/85 That is a very interesting ask, let me raise an issue against the repo and see how we can solve this with eBPF in this repo. I am pretty sure this is a very common problem for a lot of kube admins.
  - xyst 3y ago
    oh, thank you! Will follow the issue
tempaccount420 3y ago
What's the performance impact of using VM-based eBPF for these tasks instead of native code?
- tptacek 3y ago
  Linux eBPF is JIT'd. It has native performance.
  - tempaccount420 3y ago
    Even WASM doesn't have native performance, so I question this claim.
jookat 3y ago
Red Hat has a similar eBPF based tool https://github.com/netobserv/network-observability-operator https://github.com/netobserv/network-observability-operator (Disclaimer: i work on it) - the cool thing imho with retina or redhat netobserv or pixie is they aren't tied to a specific CNI. Now one of the problems that arises is potential conflicts and lack of collaboration between eBPF based tools, as there are more and more. Something called bpfman aims to address this aspect
- worthless-trash 3y ago
  I've used this in OpenShift and it is -very- neat. Saved incredible amounts of time trying to solve problems.
- sharangxy 3y ago
  You can also check out DeepFlow [1], where we implemented distributed tracing for microservices using eBPF, which of course also includes observability of K8s networks. 1. https://deepflow.io https://deepflow.io

166 points by boratanrikulu 3y ago

Probably this means it's not for me, but what is this useful for? Anyone using this in their prod set-up and has a scenario where they found this useful?

p_l 3y ago
Just today at $DAYJOB we had a complaint that one teams Azure Kubernetes clusters was "slow". About only metric out of norm was network traffic - but lack of detailed instrumentation meant we couldn't really isolate the cause to specific container or process
- hacker_newz 3y ago
  If the only explanation someone can provide is that "a cluster is slow", the issue isn't with network observability. They need to do at least the minimum level of analysis before escalating.
  - p_l 3y ago
    Unfortunately we were the right team for escalation before escalating to network team.
  - chronid 3y ago
    Yes, that would be great, but unfortunately there are application teams (particularly in the enterprise) lacking such tact when blaming infrastructure for issues. Good old silos are alive and well, and ownership is not always part of the culture.
    - p_l 3y ago
      In our case the expected golden path is that once our team figures the proper procedure, we will establish it for the downstream teams that are direct supports of the application teams. So at least in theory things are somewhat well set up, but there's too much siloing at our level (wildly separate network teams, teams for specific clouds, etc.)
nijave 3y ago
Haven't used this but I tried out Pixie trying to debug where outgoing traffic was coming from and where it was going and was fairly successful although Pixie wasn't very stable/had a lot of issues causing crashes. In this case, we had a couple services talking to 3rd party services running on AWS so it wasn't obvious from generic flow logs. I also used Lacework a couple years ago which is eBPF based and it was pretty trivial to see things phoning home or one off maintenance where a new connection was being initiated.
FridgeSeal 3y ago
It’s like Cilium + Hubble but useful for you don’t/can’t run cilium. Uses eBPF to collect metrics and stats on what flows where, can record an impressive amount of stuff, without any required instrumentation of your applications. Amazingly handy for when you run both first party and 3rd party apps in your K8s cluster. The network maps these tools produce are handy too. Although, Cilium is pretty great, so not sure why you wouldn’t run it, given the option…
- hosh 3y ago
  Cillium has been bought out by Cisco, so its monetization is only a matter of time. Also, not everyone needs to implement a service mesh.
  - FridgeSeal 3y ago
    Neither Cilium nor Hubble are service meshes. Cilium is a CNI - the functionality that provides the K8s cluster inter-pod networking. The fact that it uses eBPF to deliver its functionality is what gives it the impressive observability you usually only get from a service mesh. I agree that not everyone needs a service mesh.
    - nullify88 3y ago
      You are right that cilium is a CNI but one of its many features is also providing a sidecar-less service mesh using eBPF. https://cilium.io/use-cases/service-mesh/ https://cilium.io/use-cases/service-mesh/
  - candiddevmike 3y ago
    Cillium is a CNCF project and Apache 2 licensed. Isovalent and their enterprise product were bought by Cisco.
    - deleted 3y ago
      [deleted]
emmelaich 3y ago
The linked docs provide more info: https://retina.sh/docs/intro https://retina.sh/docs/intro

orisho 3y ago

See also: Network Mapper - low privileges, no-eBPF network observability tool for K8s https://news.ycombinator.com/item?id=39761114 https://news.ycombinator.com/item?id=39761114

scottlamb 3y ago

Sigh, anyone have a name suggestion for a (Rust) RTSP library? https://crates.io/crates/retina https://crates.io/crates/retina

emmanueloga_ 3y ago
https://deadcells.fandom.com/wiki/Conjunctivius https://deadcells.fandom.com/wiki/Conjunctivius :-p
karolist 3y ago
The name collisions for opensource projects stopped mattering a long time ago
nextaccountic 3y ago
Don't change the name of this crate, it is in an unrelated domain. (Also unrelated to retina displays and uhh.. this Brazilian intrusion detection system https://sunsoftware.com.br/retina/ https://sunsoftware.com.br/retina/ and whatever this thing is https://retina.ai/ https://retina.ai/ among other things)

emmanueloga_ 3y ago

There is a flood of observability tools based on eBPF coming out these days [1]. eBPF is used to collect metrics without the need to instrument the code. -- 1: https://ebpf.io/applications/ https://ebpf.io/applications/

sharangxy 3y ago
DeepFlow [1][2] is one of them, where we implemented distributed tracing for microservices using eBPF. 1. https://deepflow.io https://deepflow.io 2. https://github.com/deepflowio/deepflow https://github.com/deepflowio/deepflow

xyst 3y ago

Speaking of observability tools. Anybody here know how to gather more in-depth metrics on mTLS requests? Have an internal (self signed) CA and just want to know which issued certs are presented to nodes. Would be nice to get cert serial number and other metadata as well

vamsi1105 3y ago
https://github.com/microsoft/retina/issues/85 https://github.com/microsoft/retina/issues/85 That is a very interesting ask, let me raise an issue against the repo and see how we can solve this with eBPF in this repo. I am pretty sure this is a very common problem for a lot of kube admins.
- xyst 3y ago
  oh, thank you! Will follow the issue

tempaccount420 3y ago

What's the performance impact of using VM-based eBPF for these tasks instead of native code?

tptacek 3y ago
Linux eBPF is JIT'd. It has native performance.
- tempaccount420 3y ago
  Even WASM doesn't have native performance, so I question this claim.

jookat 3y ago

Red Hat has a similar eBPF based tool https://github.com/netobserv/network-observability-operator https://github.com/netobserv/network-observability-operator (Disclaimer: i work on it) - the cool thing imho with retina or redhat netobserv or pixie is they aren't tied to a specific CNI. Now one of the problems that arises is potential conflicts and lack of collaboration between eBPF based tools, as there are more and more. Something called bpfman aims to address this aspect

worthless-trash 3y ago
I've used this in OpenShift and it is -very- neat. Saved incredible amounts of time trying to solve problems.
sharangxy 3y ago
You can also check out DeepFlow [1], where we implemented distributed tracing for microservices using eBPF, which of course also includes observability of K8s networks. 1. https://deepflow.io https://deepflow.io