5 ms·
If you're interested in distributed system tracing there's a lot going on. As a starting point, I would recommend reading Google's paper on their project "Dapp
by devbug 10y ago
If you're interested in distributed system tracing there's a lot going on.
As a starting point, I would recommend reading Google's paper on their project "Dapper." [1] It's essentially the core of most distributed tracing systems. At least those I've encountered.
There's a lot of tooling out there that take their cues from Dapper. I've recently been looking into integrating OpenZipkin[2] with our systems. I see at as a more viable alternative (no tie in!) to Yet Another Propriety Thing in AWS (YAPTA). There's other as well, like AppDash.
Recently there has been a push towards an open-standard for the collection side called OpenTracing[3]. I came across it when investigating LightStep[4]. Ideally that means no vendor lock-in, which of course has lots of knock-on effects.
If you know of any, I'd love to be pointed in the direction of _different_ and not just divergent techniques.
[1]: http://static.googleusercontent.com/media/research.google.com/en//archive/papers/dapper-2010-1.pdf http://static.googleusercontent.com/media/research.google.co...
[2]: http://zipkin.io http://zipkin.io
[3]: http://opentracing.io http://opentracing.io
[4]: http://lightstep.com http://lightstep.com - Impressive team behind this.
- Roritharr 10y agoThose are the comments I come here for. I was reading this and dreaded the lock-in. Thanks for pointing out sane alternatives!
- tonyarkles 10y agoMy masters thesis is X-trace derived, as well as a paper we published at ICPE a few years ago. I'm on mobile so don't have a link handy. You can probably guess my name from my username.
- benzewdu 10y agohttps://ecommons.usask.ca/xmlui/bitstream/handle/10388/ETD-2015-06-2185/ARKLES-THESIS.pdf https://ecommons.usask.ca/xmlui/bitstream/handle/10388/ETD-2...
- SEJeff 10y agoAppdash[5] :) [5] https://github.com/sourcegraph/appdash https://github.com/sourcegraph/appdash
- seanp2k2 10y agoSomewhat related: https://github.com/Netflix/vizceral https://github.com/Netflix/vizceral
- dgacmu 10y agoOn the research front, going back prior to Dapper is x-trace: http://www.x-trace.net/wiki/doku.php http://www.x-trace.net/wiki/doku.php and their follow on work, Pivot Tracing: http://pivottracing.io/ http://pivottracing.io/ (which got a best paper award at SOSP'15.)
- cce_ 10y agoTracelytics was a startup I co-founded based on X-Trace, which ended up as AppNeta's Traceview and is now part of the Solarwinds monitoring cloud suite of products (joining Librato, Pingdom, Papertrail). So look out for more tracing from us as well.
- j_s 10y agoI'm sure this is not new, but still useful to get the word out about the power of universal X-Request-Id: https://twitter.com/dhh/status/795732523640766464 https://twitter.com/dhh/status/795732523640766464
- AYBABTME 10y agoRequest IDs and their variants are only giving you correlated events, they don't encode enough information to connect causality and ordering of events. Basically: with request IDs you get a unordered bag of events, with proper tracing (trace id/span id/parent id) you get a DAG of events. In practice, the analysis a DAG allows you to perform are way more powerful. So if you're in the process of setting that request ID stuff up, consider doing the tracing way.
- dangets 10y agoWe've been instrumenting Zipkin in our environment over the last couple of months and it has been excellent to work with. While there are a couple of snags here and there, it is _amazing_ at how quickly it can bring bottlenecks and bad network calls to light.
- allengeorge 10y agoAre you guys using finagle?
- mmclean 10y agoStackdriver Trace [1] is Google's external implementation of Dapper with additional analysis features. It's available for free, even for workloads not running on Google Cloud Platform. (Disclosure: I work on it) [1]: https://cloud.google.com/trace/ https://cloud.google.com/trace/
- tonyhb 10y agoThis is awesome! Can Google update the docs so that the links for external zipkin integrations work? They currently link in circles and there's no info on how to integrate into OpenTracing/Zipkin
- mmclean 10y agoUh oh - are you referring to the docs on GitHub or the actual docs site (https://cloud.google.com/trace/docs/zipkin https://cloud.google.com/trace/docs/zipkin)?
- mmclean 10y agoNevermind, I think we found it. Fixing now.
- tonyhb 10y agoThis part: https://cloud.google.com/trace/docs/zipkin#configure_zipkin_tracers https://cloud.google.com/trace/docs/zipkin#configure_zipkin_... It doesn't show zipkin configuration :)
- mmclean 10y agoYep, an issue with the anchor links that'll be updated soon. It's supposed to just link to a lower part of the page. If you scroll down you'll see a top-level header also titled "Configure Zipkin tracers" that describes how to configure the Brave tracer. You don't really need to do anything special here - just point your Zipkin tracers at the Stackdriver Zipkin Collector rather than your existing one.
- roncohen 10y agoAnd if you're curious about how this sort of thing could look on the client/browser side of things, check out https://opbeat.com/react https://opbeat.com/react (I work on it)
- AYBABTME 10y agoBeen using OpenTracing with LightStep and they have a really solid product, and a super well thought out API design in OpenTracing. Highly recommend checking them out, especially since OpenTracing saves you from vendor lock-in and works using AppDash/Zipkin/etc.
- mkj 10y agoYAPTA is a nice acronym, thanks!