5 ms·

Show HN: Pixie, Instant Kubernetes-Native Application Observability Using EBPF

108 points by zasgar 6y ago

devangs 6y ago
Having used an earlier version of this, I highly recommend it if you're carrying a pager in micro-service land, and don't have Brendan Gregg on your team.
airocker 6y ago
Would love to see intrumentation without any code changes. Is it possible to write EBPF based tracers for JVM and Node/Python interpreters?
- blasrodri 6y ago
  Check bcc-tools: https://github.com/iovisor/bcc/tree/master/tools https://github.com/iovisor/bcc/tree/master/tools
- zasgar 6y ago
  Co-founder/CEO of Pixie here. EBPF allows you to access static tracepoints that are defined in many runtime, and can be used to capture information about the state of the runtime. Since most VM’s/runtimes allow monkey patching you can usually get to the same level of information without using EBPF. We plan to add support for this in Pixie in the future and provide a seamless experience regardless of what underlying tracing technology is used. Lots of good stuff from Brendan Gregg here: http://www.brendangregg.com/Slides/Velocity2017_BPF_superpowers.pdf http://www.brendangregg.com/Slides/Velocity2017_BPF_superpow...
  - airocker 6y ago
    Great to see that zasgar. Monkey patching is pretty gnarly to replace functions at load time or in JVM/.NET CLR. Python is easier. Every version change, the monkey patch has to be updated. But you can get a lot of good value from it in the form of performance charts and call graphs of an application. It would be great value if you could reduce the gnarliness of the monkey patching work and possibly replace it with a simple configuration file. Have to look into eBPF in more detail.
- oazizi 6y ago
  Founding engineer at Pixie here. To add to what zasgar mentioned, I just wanted to point out that our instrumentation-free approach does apply to JVM and Node/Python applications for many of the traces we gather. For example, we use EBPF to trace protocols like HTTP as the data passes through the kernel. By gathering the data in the kernel, these EBPF tracers are completely language agnostic.
  - justsomeuser 6y ago
    What do you do for HTTPS which would be encrypted as it passes through the kernel and is only decrypted in the application process?
    - oazizi 6y ago
      Great question. You're right that tracing in the kernel doesn't work for encrypted traffic (that means anything over TLS, including HTTPS). For encrypted connections, we still want to give the no-manual-instrumentation-required experience to our users, so what we do is trace the SSL/TLS library to capture the traffic. Right now, for example, we trace traffic going through OpenSSL. This has the benefit of covering a wide array of programs in different languages, including any dynamic languages that use OpenSSL. And we plan on adding more TLS libraries soon (e.g. GoTLS) to fill in the gaps. We'll be publishing a blog post on this soon, so please stay tuned. In the meanwhile, this other post (https://docs.pixielabs.ai/tutorials/simple-go-tracing/ https://docs.pixielabs.ai/tutorials/simple-go-tracing/) gives an idea of how one can use EBPF user-space probes to trace applications and system libraries.
djstein 6y ago
this is really amazing. the marketing page is also top notch from a visual perspective
kahrensdd 6y ago
I love the idea of instrumenting my clusters with less work. Signed up for the BETA and starting to get data out of one of my GKE clusters right away. Also I like how the cli just interacted with the cluster itself instead of tons of piping the output to other commands. Good luck Pixie team!
hapless 6y ago
I closed the tab as soon as I saw "bash -c "$(curl ..." Fuck that, forever. That is a waving red flag that the authors are targeting users too foolish to have any business using a computer.
- meddlepal 6y ago
  Lol. I love how irrationally upset people get about this. How is this any worse than downloading a random tarball and running a makefile blindly or installing a deb with god knows what pre or post scripts exist. If youre that concerned download it and read the script.
  - waste_monk 6y ago
    >If youre that concerned download it and read the script. It's possible to detect the use of curl|bash and serve different content accordingly [1]. This adds a burden to to the person checking it trying to make sure that any trickery has been accounted for and that you're actually getting a non-malicious installer. Aside from that, I don't want scripts shitting files all over my file system - I am firmly of the opinion that software installation should be handled in the package manager so it can be cleanly removed later on, upgraded, and so on using the standard tools. I also see curl|bash as a red flag because it indicates that either they don't have the skill required to build a deb/rpm/etc. package, or they simply don't care to do so - which I feel indicates they have a lazy or uncaring attitude towards software quality and craftsmanship. This is less of a red flag if they say something "debian/ubuntu users download .deb here [link], everyone else please use curl|bash" to limit the variety of package management systems they have to support, but still concerning in my opinion. [1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b... and related discussion https://news.ycombinator.com/item?id=17636032 https://news.ycombinator.com/item?id=17636032
    - meddlepal 6y ago
      I doubt the skill problem. Building a deb or rpm is trivial with fpm. Anecdotal but from past experience at a tiny startup delivering dev-tools we found our pipeline to get people to use our tool was significantly better when we just had them curl|bash a script. Getting people to use your junk is more important than doing it "The Right Way".
villgax 6y ago
Whats with .ai for an ops tooling?
- zasgar 6y ago
  Co-founder/CEO of Pixie here. At Pixie, we are building a new type of data system that can deal with the data volume that we collect and process. We have AI/ML models to help classify traffic and make analysis easier. Over the next few weeks, we will release some blog posts that will discuss how this works.

108 points by zasgar 6y ago

Having used an earlier version of this, I highly recommend it if you're carrying a pager in micro-service land, and don't have Brendan Gregg on your team.

airocker 6y ago

Would love to see intrumentation without any code changes. Is it possible to write EBPF based tracers for JVM and Node/Python interpreters?

blasrodri 6y ago
Check bcc-tools: https://github.com/iovisor/bcc/tree/master/tools https://github.com/iovisor/bcc/tree/master/tools
zasgar 6y ago
Co-founder/CEO of Pixie here. EBPF allows you to access static tracepoints that are defined in many runtime, and can be used to capture information about the state of the runtime. Since most VM’s/runtimes allow monkey patching you can usually get to the same level of information without using EBPF. We plan to add support for this in Pixie in the future and provide a seamless experience regardless of what underlying tracing technology is used. Lots of good stuff from Brendan Gregg here: http://www.brendangregg.com/Slides/Velocity2017_BPF_superpowers.pdf http://www.brendangregg.com/Slides/Velocity2017_BPF_superpow...
- airocker 6y ago
  Great to see that zasgar. Monkey patching is pretty gnarly to replace functions at load time or in JVM/.NET CLR. Python is easier. Every version change, the monkey patch has to be updated. But you can get a lot of good value from it in the form of performance charts and call graphs of an application. It would be great value if you could reduce the gnarliness of the monkey patching work and possibly replace it with a simple configuration file. Have to look into eBPF in more detail.
oazizi 6y ago
Founding engineer at Pixie here. To add to what zasgar mentioned, I just wanted to point out that our instrumentation-free approach does apply to JVM and Node/Python applications for many of the traces we gather. For example, we use EBPF to trace protocols like HTTP as the data passes through the kernel. By gathering the data in the kernel, these EBPF tracers are completely language agnostic.
- justsomeuser 6y ago
  What do you do for HTTPS which would be encrypted as it passes through the kernel and is only decrypted in the application process?
  - oazizi 6y ago
    Great question. You're right that tracing in the kernel doesn't work for encrypted traffic (that means anything over TLS, including HTTPS). For encrypted connections, we still want to give the no-manual-instrumentation-required experience to our users, so what we do is trace the SSL/TLS library to capture the traffic. Right now, for example, we trace traffic going through OpenSSL. This has the benefit of covering a wide array of programs in different languages, including any dynamic languages that use OpenSSL. And we plan on adding more TLS libraries soon (e.g. GoTLS) to fill in the gaps. We'll be publishing a blog post on this soon, so please stay tuned. In the meanwhile, this other post (https://docs.pixielabs.ai/tutorials/simple-go-tracing/ https://docs.pixielabs.ai/tutorials/simple-go-tracing/) gives an idea of how one can use EBPF user-space probes to trace applications and system libraries.

djstein 6y ago

this is really amazing. the marketing page is also top notch from a visual perspective

kahrensdd 6y ago

I love the idea of instrumenting my clusters with less work. Signed up for the BETA and starting to get data out of one of my GKE clusters right away. Also I like how the cli just interacted with the cluster itself instead of tons of piping the output to other commands. Good luck Pixie team!

hapless 6y ago

I closed the tab as soon as I saw "bash -c "$(curl ..." Fuck that, forever. That is a waving red flag that the authors are targeting users too foolish to have any business using a computer.

meddlepal 6y ago
Lol. I love how irrationally upset people get about this. How is this any worse than downloading a random tarball and running a makefile blindly or installing a deb with god knows what pre or post scripts exist. If youre that concerned download it and read the script.
- waste_monk 6y ago
  >If youre that concerned download it and read the script. It's possible to detect the use of curl|bash and serve different content accordingly [1]. This adds a burden to to the person checking it trying to make sure that any trickery has been accounted for and that you're actually getting a non-malicious installer. Aside from that, I don't want scripts shitting files all over my file system - I am firmly of the opinion that software installation should be handled in the package manager so it can be cleanly removed later on, upgraded, and so on using the standard tools. I also see curl|bash as a red flag because it indicates that either they don't have the skill required to build a deb/rpm/etc. package, or they simply don't care to do so - which I feel indicates they have a lazy or uncaring attitude towards software quality and craftsmanship. This is less of a red flag if they say something "debian/ubuntu users download .deb here [link], everyone else please use curl|bash" to limit the variety of package management systems they have to support, but still concerning in my opinion. [1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b... and related discussion https://news.ycombinator.com/item?id=17636032 https://news.ycombinator.com/item?id=17636032
  - meddlepal 6y ago
    I doubt the skill problem. Building a deb or rpm is trivial with fpm. Anecdotal but from past experience at a tiny startup delivering dev-tools we found our pipeline to get people to use our tool was significantly better when we just had them curl|bash a script. Getting people to use your junk is more important than doing it "The Right Way".

villgax 6y ago

Whats with .ai for an ops tooling?

zasgar 6y ago
Co-founder/CEO of Pixie here. At Pixie, we are building a new type of data system that can deal with the data volume that we collect and process. We have AI/ML models to help classify traffic and make analysis easier. Over the next few weeks, we will release some blog posts that will discuss how this works.