I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud en

I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud environments) is a real pain here. The google white paper on Dapper [0] and all its implementations [1][2][3] in one form or another are appearing but usually target the Web environments. That means that for my legacy stack with multiple nodes connected via networks of different kinds I have to profile on multiple systems, aggregate different log types manually, think of time syncs (not everything logs utc..) and so on. To be honest we just reached the point where we try to get a European subsidiary project rolling where we address this problem. The time lost to trace down unexplainable time gaps between first displayed result in app and request are weeks, and in multiple occasions months. Just because we have no easy way to profile our distributed ecosystem. After identifying who is to blame: network, db nodes, processing nodes, application logic, cache (invalidation); we can talk about fixing. For fixing of course it would be great if you could extract a meta model of the application and another of the hardware. Changes to those models and some nice prediction of impact would be the ideal end goal. Or, if we are wishing anyway, some nice DSE on those models to optimize would be even better. [0] https://research.google.com/pubs/pub36356.html https://research.google.com/pubs/pub36356.html [1] http://opentracing.io/ http://opentracing.io/ [2] http://lightstep.com/ http://lightstep.com/ [3] https://aws.amazon.com/xray/ https://aws.amazon.com/xray/