4 ms·

How can it tie requests arriving at a service and generating additional downstream requests? Distributed tracing needs some common token all requests share to

How can it tie requests arriving at a service and generating additional downstream requests? Distributed tracing needs some common token all requests share to identify all RPCs that should be associated with a specific incoming request.

archivator 3y ago
Take a look at Core Feature #2 in this post - https://deepflow.io/ebpf-the-key-technology-to-observability/#0x2-Three-Core-Features-of-DeepFlow-Based-on-eBPF https://deepflow.io/ebpf-the-key-technology-to-observability... It looks like it's using tcp flow tuple + tcp_seq to join things.
Eridrus 3y ago
It looks like it depends on applications either using threads or go routines for concurrency: > When collecting invocation logs through eBPF and cBPF, DeepFlow calculates information such as syscall_trace_id, thread_id, goroutine_id, cap_seq, tcp_seq based on the system call context. This allows for distributed tracing without modifying application code or injecting TraceID and SpanID. Currently, DeepFlow can achieve Zero Code distributed tracing for all cases except for cross-thread communication (through memory queues or channels) and asynchronous invocations.
sharangxy 3y ago
VP of DeepFlow here. Thank you for your interest in DeepFlow! Yes, we have implemented distributed tracing using eBPF. In simple terms, we use thread-id, coroutine-id, and tcp-seq to automatically correlate all spans. Most importantly, we use eBPF to calculate a syscall-trace-id (without the need to propagate it between upstream and downstream), enabling automatic correlation of a service's ingress and egress requests. For more details, you can refer to our paper presented at SIGCOMM'23: https://dl.acm.org/doi/10.1145/3603269.3604823 https://dl.acm.org/doi/10.1145/3603269.3604823. Of course, this kind of Zero Code distributed tracing currently has some limitations. For specific details, please see: https://deepflow.io/docs/features/distributed-tracing/auto-tracing/#current-limitations https://deepflow.io/docs/features/distributed-tracing/auto-t... These limitations are not entirely insurmountable. We are actively working on resolving them and continually making breakthroughs.
- inhumantsar 3y ago
  would it be reasonable to assume that because this entirely network-based, it would work best with systems which really emphasize the "micro" in microservices? how well does this work if, say, my system has a legacy monolith in addition to microservices?
  - sharangxy 3y ago
    I believe the current situation is like this. The advantage of eBPF lies in *request granularity* (i.e. PRC, API, SQL, etc ...) distributed tracing. To trace the internal functions of an application, instrumentation is still required for coverage. Therefore, the finer the service decomposition, the more effective eBPF's distributed tracing becomes.

archivator 3y ago

Take a look at Core Feature #2 in this post - https://deepflow.io/ebpf-the-key-technology-to-observability/#0x2-Three-Core-Features-of-DeepFlow-Based-on-eBPF https://deepflow.io/ebpf-the-key-technology-to-observability... It looks like it's using tcp flow tuple + tcp_seq to join things.

Eridrus 3y ago

It looks like it depends on applications either using threads or go routines for concurrency: > When collecting invocation logs through eBPF and cBPF, DeepFlow calculates information such as syscall_trace_id, thread_id, goroutine_id, cap_seq, tcp_seq based on the system call context. This allows for distributed tracing without modifying application code or injecting TraceID and SpanID. Currently, DeepFlow can achieve Zero Code distributed tracing for all cases except for cross-thread communication (through memory queues or channels) and asynchronous invocations.

sharangxy 3y ago

VP of DeepFlow here. Thank you for your interest in DeepFlow! Yes, we have implemented distributed tracing using eBPF. In simple terms, we use thread-id, coroutine-id, and tcp-seq to automatically correlate all spans. Most importantly, we use eBPF to calculate a syscall-trace-id (without the need to propagate it between upstream and downstream), enabling automatic correlation of a service's ingress and egress requests. For more details, you can refer to our paper presented at SIGCOMM'23: https://dl.acm.org/doi/10.1145/3603269.3604823 https://dl.acm.org/doi/10.1145/3603269.3604823. Of course, this kind of Zero Code distributed tracing currently has some limitations. For specific details, please see: https://deepflow.io/docs/features/distributed-tracing/auto-tracing/#current-limitations https://deepflow.io/docs/features/distributed-tracing/auto-t... These limitations are not entirely insurmountable. We are actively working on resolving them and continually making breakthroughs.

inhumantsar 3y ago
would it be reasonable to assume that because this entirely network-based, it would work best with systems which really emphasize the "micro" in microservices? how well does this work if, say, my system has a legacy monolith in addition to microservices?
- sharangxy 3y ago
  I believe the current situation is like this. The advantage of eBPF lies in *request granularity* (i.e. PRC, API, SQL, etc ...) distributed tracing. To trace the internal functions of an application, instrumentation is still required for coverage. Therefore, the finer the service decomposition, the more effective eBPF's distributed tracing becomes.