3 ms·

We built something similar last year for our internal services, but ran into problems when dealing with asynchronous code and threading context. How do you pro

by dpratt 13y ago

We built something similar last year for our internal services, but ran into problems when dealing with asynchronous code and threading context. How do you propagate the Zipkin info from thread to thread inside your code? An example - a request comes in, we generate a new request ID and pass this down into the processing code. Part of this code executes an async call to an external service with essentially a callback (in reality, it's a scala.concurrent.Future executing on an arbitrary context or an akka actor) - how do you properly rehydrate the Zipkin info when the response comes back? The only way we could think of is some sort of fiendishly complex custom ExecutionContext that inspected the thread local state at creation time and recreates that in the thread running the callback, or just have pretty much every method take an implicit context parameter. Neither of those solutions worked well, so we've largely bailed on the concept for the bits of our code that don't execute in a linear/blocking fashion.

bmdhacks 13y ago
Twitter util to the rescue! https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Local.scala https://github.com/twitter/util/blob/master/util-core/src/ma...
- dpratt 13y ago
  We actually built something quite similar to that, but how do you avoid having to artificially insert a save() and restore(context) call all over the place? It looks to me like you'd have to do something like val context = Local.save() val eventuallyFoo = someServiceClient.makeCall("data").map { result => Local.restore(context) //do work } for every single async interaction.
  - bmdhacks 13y ago
    That's what we do, but it's built into the RPC library we use, finagle: https://github.com/twitter/finagle/blob/master/finagle-core/src/main/scala/com/twitter/finagle/tracing/Trace.scala#L181-L184 https://github.com/twitter/finagle/blob/master/finagle-core/... BTW, to the readers at home, note how almost all our core infrastructure is open-sourced.
    - ludwigvan 13y ago
      That's intriguing. Could you please elaborate on that? Does this mean that Twitter sees no danger in exposing these? Which parts of the infrastructure would be considered secret sauce and not open source? Or does it not matter when the company is as big as Twitter, since the core strength lies in user base, not technical infrastructure? What does Twitter primarily seek to achieve when it open sources its stuff? Talent acquisition or brand image or other benefits of open source such as collaboration?
  - mariusae 13y ago
    More or less 100% of asynchronous composition is done via Futures[1] whose default implementation[2] does this for you. [1] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Future.scala https://github.com/twitter/util/blob/master/util-core/src/ma... [2] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Promise.scala#L128 https://github.com/twitter/util/blob/master/util-core/src/ma...
    - mariusae 13y ago
      (For fun, I just searched through our entire codebase. The only manipulation of locals anywhere is in the util library.)
    - dpratt 13y ago
      I mean this as politely and constructively as possible, but this looks like it has quite a few easy to fall into failure modes. If I'm not using a twitter Future, or if I've been given a Future from somebody else, it would look like I need to ensure to surround every compositional operation with this save and restore, otherwise my context is permanently lost. Additionally, I have to trust that any code I hand a closure to will do the right thing, otherwise my context is potentially lost since I have no guarantee on which thread that closure will actually be executed on. It seems like it would be virtually impossible to avoid this happening at least once in even a simple codebase, and when context is lost, it happens silently.
      - theatrus2 13y ago
        Yes, if you "go off the reservation" and outside of the Twitter Future, you will lose your automatic trace identifiers. This isn't a unique problem, but using consistent libraries goes a long way (which works well internally at Twitter).
      - mariusae 13y ago
        You are right that this fails when you move outside of the model. However, we don't. You may be surprised (amazed?) to learn that, internally, 100% of composition happens in this manner. We have a massive code base, and we've not seen this be an issue. Further, we've worked with the Scala community to standardize the idea of an "execution context" which helps make these ideas portable, the particular of the implementation transparent to arbitrary producers and consumers of futures, so long as they comply to the standard Scala future API. (Twitter futures will when we migrate to Scala 2.10.)

by dpratt 13y ago

bmdhacks 13y ago

Twitter util to the rescue! https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Local.scala https://github.com/twitter/util/blob/master/util-core/src/ma...

dpratt 13y ago
We actually built something quite similar to that, but how do you avoid having to artificially insert a save() and restore(context) call all over the place? It looks to me like you'd have to do something like val context = Local.save() val eventuallyFoo = someServiceClient.makeCall("data").map { result => Local.restore(context) //do work } for every single async interaction.
- bmdhacks 13y ago
  That's what we do, but it's built into the RPC library we use, finagle: https://github.com/twitter/finagle/blob/master/finagle-core/src/main/scala/com/twitter/finagle/tracing/Trace.scala#L181-L184 https://github.com/twitter/finagle/blob/master/finagle-core/... BTW, to the readers at home, note how almost all our core infrastructure is open-sourced.
  - ludwigvan 13y ago
    That's intriguing. Could you please elaborate on that? Does this mean that Twitter sees no danger in exposing these? Which parts of the infrastructure would be considered secret sauce and not open source? Or does it not matter when the company is as big as Twitter, since the core strength lies in user base, not technical infrastructure? What does Twitter primarily seek to achieve when it open sources its stuff? Talent acquisition or brand image or other benefits of open source such as collaboration?
- mariusae 13y ago
  More or less 100% of asynchronous composition is done via Futures[1] whose default implementation[2] does this for you. [1] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Future.scala https://github.com/twitter/util/blob/master/util-core/src/ma... [2] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Promise.scala#L128 https://github.com/twitter/util/blob/master/util-core/src/ma...
  - mariusae 13y ago
    (For fun, I just searched through our entire codebase. The only manipulation of locals anywhere is in the util library.)
  - dpratt 13y ago
    I mean this as politely and constructively as possible, but this looks like it has quite a few easy to fall into failure modes. If I'm not using a twitter Future, or if I've been given a Future from somebody else, it would look like I need to ensure to surround every compositional operation with this save and restore, otherwise my context is permanently lost. Additionally, I have to trust that any code I hand a closure to will do the right thing, otherwise my context is potentially lost since I have no guarantee on which thread that closure will actually be executed on. It seems like it would be virtually impossible to avoid this happening at least once in even a simple codebase, and when context is lost, it happens silently.
    - theatrus2 13y ago
      Yes, if you "go off the reservation" and outside of the Twitter Future, you will lose your automatic trace identifiers. This isn't a unique problem, but using consistent libraries goes a long way (which works well internally at Twitter).
    - mariusae 13y ago
      You are right that this fails when you move outside of the model. However, we don't. You may be surprised (amazed?) to learn that, internally, 100% of composition happens in this manner. We have a massive code base, and we've not seen this be an issue. Further, we've worked with the Scala community to standardize the idea of an "execution context" which helps make these ideas portable, the particular of the implementation transparent to arbitrary producers and consumers of futures, so long as they comply to the standard Scala future API. (Twitter futures will when we migrate to Scala 2.10.)