3 ms·

I would love to see a tracer that works across processes and machines. For example, a request hits the web server, the web server tickles an application server

I would love to see a tracer that works across processes and machines. For example, a request hits the web server, the web server tickles an application server, the application server makes a number of database queries, and the results propagate back the way they came. I'd love to see something that could follow this request through all of the layers and across all of the machines, without the luxury of having the source code for most of the components. I don't see why this wouldn't be possible, but I certainly see why it wouldn't be EASY!

ariwilson 11y ago
e.g. Google Dapper? http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf http://static.googleusercontent.com/media/research.google.co...
kyrra 11y ago
Google published a paper similar to this 6 years ago: http://research.google.com/pubs/pub36356.html http://research.google.com/pubs/pub36356.html Dapper will follow the call path of an RPC and report the details of it.
bg451 11y ago
Seeing that people have already mentioned Dapper, I'd add that there are already a few open source implementations of Dapper available. Appdash[1] is a very lightweight tracer that isn't too fancy, but gets the job done. There's also the much bigger, more popular tracer Zipkin[2], which was built by Twitter. [1] https://github.com/sourcegraph/appdash https://github.com/sourcegraph/appdash [2] https://github.com/openzipkin/zipkin https://github.com/openzipkin/zipkin
- Mister_Snuggles 11y ago
  It looks like both of these require changes to the application. Dapper appears to have the same issue. My use case is for understanding a closed source application which has a number of separate pieces. Since I don't have the source code, these don't look like viable options.
jonathan_mace 11y ago
Other people have mentioned end-to-end tracing frameworks, which are heavyweight application-level frameworks that are similar to logging. However, lightweight cross-component dynamic instrumentation that correctly captures causality is still an open research problem. This is actually something I have done some work on recently in my system Pivot Tracing[1] which won a best paper award at the conference it was presented at (SOSP). Pivot Tracing combines dynamic instrumentation with causal tracing in order to address precisely the problem you describe. The code for the system is open source and available on GitHub[2]. [1] http://cs.brown.edu/~jcmace/papers/mace15pivot.pdf http://cs.brown.edu/~jcmace/papers/mace15pivot.pdf [2] https://github.com/brownsys/tracing-framework https://github.com/brownsys/tracing-framework
- Mister_Snuggles 11y ago
  This looks good, but requires changes to the system being traced. In my case, the system I'm most interested in tracing is a closed-source black box, so changing it isn't an option.

ariwilson 11y ago

e.g. Google Dapper? http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf http://static.googleusercontent.com/media/research.google.co...

kyrra 11y ago

Google published a paper similar to this 6 years ago: http://research.google.com/pubs/pub36356.html http://research.google.com/pubs/pub36356.html Dapper will follow the call path of an RPC and report the details of it.

bg451 11y ago

Seeing that people have already mentioned Dapper, I'd add that there are already a few open source implementations of Dapper available. Appdash[1] is a very lightweight tracer that isn't too fancy, but gets the job done. There's also the much bigger, more popular tracer Zipkin[2], which was built by Twitter. [1] https://github.com/sourcegraph/appdash https://github.com/sourcegraph/appdash [2] https://github.com/openzipkin/zipkin https://github.com/openzipkin/zipkin

Mister_Snuggles 11y ago
It looks like both of these require changes to the application. Dapper appears to have the same issue. My use case is for understanding a closed source application which has a number of separate pieces. Since I don't have the source code, these don't look like viable options.

jonathan_mace 11y ago

Other people have mentioned end-to-end tracing frameworks, which are heavyweight application-level frameworks that are similar to logging. However, lightweight cross-component dynamic instrumentation that correctly captures causality is still an open research problem. This is actually something I have done some work on recently in my system Pivot Tracing[1] which won a best paper award at the conference it was presented at (SOSP). Pivot Tracing combines dynamic instrumentation with causal tracing in order to address precisely the problem you describe. The code for the system is open source and available on GitHub[2]. [1] http://cs.brown.edu/~jcmace/papers/mace15pivot.pdf http://cs.brown.edu/~jcmace/papers/mace15pivot.pdf [2] https://github.com/brownsys/tracing-framework https://github.com/brownsys/tracing-framework

Mister_Snuggles 11y ago
This looks good, but requires changes to the system being traced. In my case, the system I'm most interested in tracing is a closed-source black box, so changing it isn't an option.