4 ms·

Show HN: Trayce – Network tab for Docker containers

138 points by ev_dev3 2y ago

Trayce (https://github.com/evanrolfe/trayce_gui https://github.com/evanrolfe/trayce_gui) is an open source desktop application which monitors HTTP(S) traffic to Docker containers on your machine. It uses EBPF to achieve zero-configuration sniffing of TLS-encrypted traffic. As a backend developer I wanted something which was similar to Wireshark or the Chrome network tab, but which intercepted requests & responses to my containers for debugging in a local dev environment. Wireshark is a great tool but it seems more geared towards lower level networking tasks. When I'm developing APIs or microservices I dont care about packets, I'm only concerned with HTTP requests and their responses. I also didn't want to have to configure a pre-shared master key to intercept TLS, I wanted it to work out-of-the-box. Trayce is in beta phase so feedback is very welcome, bug reports too. The frontend GUI is written in Python with the QT framework. The TrayceAgent which is what does the intercepting of traffic is written in Go and EBPF.

upon_drumhead 2y ago
This is super cool! I'm gonna give this a try shortly. One thing that we've been craving for our infrastructure is something like Little Snitch for containers. We make extensive use of third party containers and have been very concerned about any sort of data exfiltration attempts via supply chain attacks. We have a pile of iptable rules right now, but they're error prone and difficult to rationalize about. If we had something like Little Snitch where we would get the feed of connections and allow us to approve/reject them, it would make us all sleep a lot better. Best of luck with your endeavors!
deleted 2y ago
[deleted]
thomasfromcdnjs 2y ago
I wanted to have a play but crashed when running the container command libbpf: sec '.reluprobe/gotls_exit_read_register': relo #5: insn #148 against 'active_go_read_args_map' libbpf: prog 'probe_exit_go_tls_read': found map 17 (active_go_read_args_map, sec 30, off 528) for insn #148 libbpf: Error in bpf_object__probe_loading():Function not implemented(38). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value. libbpf: failed to load object 'main.bpf.o' failed to load BPF object: function not implemented
- deleted 2y ago
  [deleted]
- ev_dev3 2y ago
  Are you running on Mac or Linux? What linux kernel version do you have in docker, run: `uname -r`. If you are using Mac with Docker Desktop then the first thing I would do is make sure thats on the latest version. This is a common problem however when trying to run ebpf programs in docker, see "CONFIG_BPF_SYSCALL and RLIMIT_MEMLOCK issues" here: http://andreybleme.com/2022-05-22/running-ebpf-programs-on-docker-containers/ http://andreybleme.com/2022-05-22/running-ebpf-programs-on-d... Also see "Configuring limits in the container" here too: https://blog.bitsofnetworks.org/debug-bpf-docker-l2tpns/ https://blog.bitsofnetworks.org/debug-bpf-docker-l2tpns/
- quibono 2y ago
  +1 - I get the same error. Running on Ubuntu 20.04
iamcreasy 2y ago
Cool! > It uses EBPF to achieve zero-configuration sniffing of TLS-encrypted traffic. Can someone ELI5 this?
- ThaDood 2y ago
  IIRC EBPF is an enhanced version of the Berkeley Packet filter. In this scenario I believe it is being used for sandboxing a low level process to allow for TLS "decryption" on network connections related to Docker.
- deleted 2y ago
  [deleted]
- ev_dev3 2y ago
  I won't explain what ebpf is because google can do a better job of that than me but essentially this program hooks into the low-level system calls being made by your programs, so its able to grab un-encrypted network traffic before its encrypted and sent over the network. "zero-configuration" here mean you dont have to do much manual configuration to get this working.
  - Varriount 2y ago
    Huh, is TLS being handled in the kernel these days? I thought most systems still used usermode libraries.
    - ev_dev3 2y ago
      I'm using the term "low level system calls" loosely here since the poster asked for ELI5. Trayce actually uses a combination of kprobes for system calls and uprobes to monitor calls to OpenSSL or the Go crypto/tls package. More details are here: https://github.com/evanrolfe/trayce_agent/blob/main/docs/README.md https://github.com/evanrolfe/trayce_agent/blob/main/docs/REA...
      - deleted 2y ago
        [deleted]
inyorgroove 2y ago
This tool has really cool potential! Just one problem I noticed imminently that prevents me from using this, the docker agent container[1] isn't multi-architecture, this will be an issue on Apple Silicon devices. This is something I have some experience setting up if you are looking for help, though will take some research to figure out how to get going in github actions etc. 1: https://github.com/evanrolfe/trayce_agent/ https://github.com/evanrolfe/trayce_agent/ EDIT: quick search found this post, tested on a side project repo it works great: https://depot.dev/blog/multi-platform-docker-images-in-github-actions https://depot.dev/blog/multi-platform-docker-images-in-githu...
- deleted 2y ago
  [deleted]
- ev_dev3 2y ago
  Good point, thanks. Its only ever been tested on a Mac with an Intel chip. I will try and sort this out ASAP!
  - inyorgroove 2y ago
    I did submit a PR with a partial implementation, well the Docker and Github side. I am useless at the C and low level code.
    - westurner 2y ago
      Who has CI build runners for the given architectures?
      - inyorgroove 2y ago
        It looks[1] like github actions only run on amd64 hosts, so if you use a platform matrix like in this example[2] I am fairly certain it is running under qemu just based on the fact that is takes roughly 10 times longer to run. I am aware of Blaze[3] has arm64 runners. If you are willing to pay for it, you can also setup a runner that uses AWS Graviton EC2 instances[4], we do that at my workplace for our multi architecture builds. 1: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#cloud-hosts-used-by-github-hosted-runners https://docs.github.com/en/actions/using-github-hosted-runne... 2: https://docs.docker.com/build/ci/github-actions/multi-platform/ https://docs.docker.com/build/ci/github-actions/multi-platfo... 3: https://www.runblaze.dev/ https://www.runblaze.dev/ 4: https://aws.amazon.com/ec2/graviton/ https://aws.amazon.com/ec2/graviton/
globular-toast 2y ago
Licence? Assuming GPL if it's using Qt?
- ev_dev3 2y ago
  Yes, GPL3. I've added the license to the repo.
  - globular-toast 2y ago
    Thanks. Looks awesome. You mention the Chrome network tab but you should also check out the Firefox network tab which is probably better and more powerful, for some more inspiration.
piyushtechsavy 2y ago
This seems super cool debugging tool. Is this purely for dockerized applications or can also be used with simple microservices.
gyre007 2y ago
Looks very cool. I think you should write a docker extension https://www.docker.com/products/extensions/ https://www.docker.com/products/extensions/
- notpushkin 2y ago
  As long as it's in addition to the standalone app. Not everybody uses Docker Desktop.
  - westurner 2y ago
    Podman Desktop supports Docker Desktop extensions and also its own extension API. https://podman-desktop.io/extend https://podman-desktop.io/extend "Developing a Podman Desktop extension" https://podman-desktop.io/docs/extensions/developing https://podman-desktop.io/docs/extensions/developing
deleted 2y ago
[deleted]
sesm 2y ago
Slightly off topic: is there a developer proxy that has a similar UI? I don't want to wrap the app in docker just see network logs in a nice UI.
- zokier 2y ago
  At least conceptually I think the same approach, using ebpf to capture traffic, should work pretty much the same with or without containers. So it could be fairly minor adjustment to get it working outside containers too?
viralpraxis 2y ago
Can it be used to trace containers with «host» network driver?
skeptrune 2y ago
This is an amazing and insanely helpful product
esafak 2y ago
Consider contributing it to macports when it's out of beta.
thebuilderjr 2y ago
[flagged]
alexliu518 2y ago
Very useful project, keep it up
ancras 2y ago
Sounds cool. Would it maintain a history of requests which I could filter by time?

138 points by ev_dev3 2y ago

upon_drumhead 2y ago

This is super cool! I'm gonna give this a try shortly. One thing that we've been craving for our infrastructure is something like Little Snitch for containers. We make extensive use of third party containers and have been very concerned about any sort of data exfiltration attempts via supply chain attacks. We have a pile of iptable rules right now, but they're error prone and difficult to rationalize about. If we had something like Little Snitch where we would get the feed of connections and allow us to approve/reject them, it would make us all sleep a lot better. Best of luck with your endeavors!

deleted 2y ago

[deleted]

thomasfromcdnjs 2y ago

I wanted to have a play but crashed when running the container command libbpf: sec '.reluprobe/gotls_exit_read_register': relo #5: insn #148 against 'active_go_read_args_map' libbpf: prog 'probe_exit_go_tls_read': found map 17 (active_go_read_args_map, sec 30, off 528) for insn #148 libbpf: Error in bpf_object__probe_loading():Function not implemented(38). Couldn't load trivial BPF program. Make sure your kernel supports BPF (CONFIG_BPF_SYSCALL=y) and/or that RLIMIT_MEMLOCK is set to big enough value. libbpf: failed to load object 'main.bpf.o' failed to load BPF object: function not implemented

deleted 2y ago
[deleted]
ev_dev3 2y ago
Are you running on Mac or Linux? What linux kernel version do you have in docker, run: `uname -r`. If you are using Mac with Docker Desktop then the first thing I would do is make sure thats on the latest version. This is a common problem however when trying to run ebpf programs in docker, see "CONFIG_BPF_SYSCALL and RLIMIT_MEMLOCK issues" here: http://andreybleme.com/2022-05-22/running-ebpf-programs-on-docker-containers/ http://andreybleme.com/2022-05-22/running-ebpf-programs-on-d... Also see "Configuring limits in the container" here too: https://blog.bitsofnetworks.org/debug-bpf-docker-l2tpns/ https://blog.bitsofnetworks.org/debug-bpf-docker-l2tpns/
quibono 2y ago
+1 - I get the same error. Running on Ubuntu 20.04

iamcreasy 2y ago

Cool! > It uses EBPF to achieve zero-configuration sniffing of TLS-encrypted traffic. Can someone ELI5 this?

ThaDood 2y ago
IIRC EBPF is an enhanced version of the Berkeley Packet filter. In this scenario I believe it is being used for sandboxing a low level process to allow for TLS "decryption" on network connections related to Docker.
deleted 2y ago
[deleted]
ev_dev3 2y ago
I won't explain what ebpf is because google can do a better job of that than me but essentially this program hooks into the low-level system calls being made by your programs, so its able to grab un-encrypted network traffic before its encrypted and sent over the network. "zero-configuration" here mean you dont have to do much manual configuration to get this working.
- Varriount 2y ago
  Huh, is TLS being handled in the kernel these days? I thought most systems still used usermode libraries.
  - ev_dev3 2y ago
    I'm using the term "low level system calls" loosely here since the poster asked for ELI5. Trayce actually uses a combination of kprobes for system calls and uprobes to monitor calls to OpenSSL or the Go crypto/tls package. More details are here: https://github.com/evanrolfe/trayce_agent/blob/main/docs/README.md https://github.com/evanrolfe/trayce_agent/blob/main/docs/REA...
    - deleted 2y ago
      [deleted]

inyorgroove 2y ago

This tool has really cool potential! Just one problem I noticed imminently that prevents me from using this, the docker agent container[1] isn't multi-architecture, this will be an issue on Apple Silicon devices. This is something I have some experience setting up if you are looking for help, though will take some research to figure out how to get going in github actions etc. 1: https://github.com/evanrolfe/trayce_agent/ https://github.com/evanrolfe/trayce_agent/ EDIT: quick search found this post, tested on a side project repo it works great: https://depot.dev/blog/multi-platform-docker-images-in-github-actions https://depot.dev/blog/multi-platform-docker-images-in-githu...

deleted 2y ago
[deleted]
ev_dev3 2y ago
Good point, thanks. Its only ever been tested on a Mac with an Intel chip. I will try and sort this out ASAP!
- inyorgroove 2y ago
  I did submit a PR with a partial implementation, well the Docker and Github side. I am useless at the C and low level code.
  - westurner 2y ago
    Who has CI build runners for the given architectures?
    - inyorgroove 2y ago
      It looks[1] like github actions only run on amd64 hosts, so if you use a platform matrix like in this example[2] I am fairly certain it is running under qemu just based on the fact that is takes roughly 10 times longer to run. I am aware of Blaze[3] has arm64 runners. If you are willing to pay for it, you can also setup a runner that uses AWS Graviton EC2 instances[4], we do that at my workplace for our multi architecture builds. 1: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#cloud-hosts-used-by-github-hosted-runners https://docs.github.com/en/actions/using-github-hosted-runne... 2: https://docs.docker.com/build/ci/github-actions/multi-platform/ https://docs.docker.com/build/ci/github-actions/multi-platfo... 3: https://www.runblaze.dev/ https://www.runblaze.dev/ 4: https://aws.amazon.com/ec2/graviton/ https://aws.amazon.com/ec2/graviton/

globular-toast 2y ago

Licence? Assuming GPL if it's using Qt?

ev_dev3 2y ago
Yes, GPL3. I've added the license to the repo.
- globular-toast 2y ago
  Thanks. Looks awesome. You mention the Chrome network tab but you should also check out the Firefox network tab which is probably better and more powerful, for some more inspiration.

piyushtechsavy 2y ago

This seems super cool debugging tool. Is this purely for dockerized applications or can also be used with simple microservices.

gyre007 2y ago

Looks very cool. I think you should write a docker extension https://www.docker.com/products/extensions/ https://www.docker.com/products/extensions/

notpushkin 2y ago
As long as it's in addition to the standalone app. Not everybody uses Docker Desktop.
- westurner 2y ago
  Podman Desktop supports Docker Desktop extensions and also its own extension API. https://podman-desktop.io/extend https://podman-desktop.io/extend "Developing a Podman Desktop extension" https://podman-desktop.io/docs/extensions/developing https://podman-desktop.io/docs/extensions/developing

deleted 2y ago

[deleted]

sesm 2y ago

Slightly off topic: is there a developer proxy that has a similar UI? I don't want to wrap the app in docker just see network logs in a nice UI.

zokier 2y ago
At least conceptually I think the same approach, using ebpf to capture traffic, should work pretty much the same with or without containers. So it could be fairly minor adjustment to get it working outside containers too?

viralpraxis 2y ago

Can it be used to trace containers with «host» network driver?

skeptrune 2y ago

This is an amazing and insanely helpful product

esafak 2y ago

Consider contributing it to macports when it's out of beta.

thebuilderjr 2y ago

[flagged]

alexliu518 2y ago

Very useful project, keep it up

ancras 2y ago

Sounds cool. Would it maintain a history of requests which I could filter by time?