> 9.2 Performance Tracing - We also have an internal tool called EEG (not included in the initial open source release in November, 2015) that we use to collect

> 9.2 Performance Tracing - We also have an internal tool called EEG (not included in the initial open source release in November, 2015) that we use to collect and visualize very fine-grained information about the exact ordering and performance characteristics of the execution of TensorFlow graphs. This tool works in both our single machine and distributed implementations, and is very useful for understanding the bottlenecks in the computation and communication patterns of a TensorFlow program. The flow graphs look very similar to a talk given by Dick Sites [2, 3], where he talks about performance monitoring across the Google fleet. Highly recommend watching this if you are at all interested in performance monitoring (he has a clear passion and deep understanding in this area). What's really interesting about this talk, is that it appears Google engineers do not need to bake this into their distributed applications, it works at the OS level, along with supporting tracing systems (tagging, log collections, aggregation into a central tool, analytics, etc [4 - Dapper]). So, you get it for free basically if you are running in the Google environment. Compare this with something you want to debug outside of Google, you need to add all types of debug bits into your code, and then do the analysis in your own little silo. Very thought provoking idea. I think this is why they are not releasing it. This is a guess based off a few things I've read though. So, I could be wrong. ps - if you want to get really crazy about this, you might read into this that they "would like to" or "have built" their own CPUs, to address some of the limitations they see from the talk. [1] https://en.wikipedia.org/wiki/Electroencephalography https://en.wikipedia.org/wiki/Electroencephalography [2] http://www.pdl.cmu.edu/SDI/2015/slides/DatacenterComputers.pdf http://www.pdl.cmu.edu/SDI/2015/slides/DatacenterComputers.p... [3] https://vimeo.com/121396406 https://vimeo.com/121396406 [4] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36356.pdf http://static.googleusercontent.com/media/research.google.co...