I guess this is as good a time as any to make my post-PyParallel pet project public (holy alliteration Batman): https://github.com/tpn/tracer https://github.com

I guess this is as good a time as any to make my post-PyParallel pet project public (holy alliteration Batman): https://github.com/tpn/tracer https://github.com/tpn/tracer. I started it about 2.5 years ago because I wasn't really happy with existing Python tracing tools. I didn't like: a) the performance hit when tracing non-trivial programs, b) the inability to say: "hey, trace all code living in modules mymodule1 and mymodule2; ignore everything else", and c) the inability to say: "screw it, trace everything", and have an efficient means to persist tens of gigabytes worth of data (i.e. don't separate in-memory representation from on-disk representation). So I came up with this tracing project, creatively named: tracer. It has relatively low overhead, efficiently and optimally persists tens of gigabytes worth of data to disk with minimal overhead, and allows for module-based tracing directives so that you can capture info about your project and ignore all the other stuff (like stdlib and package code). Like all fun pet projects, once the easy stuff was done, this one devolved into writing assembly to do remote thread injection (https://github.com/tpn/tracer/blob/master/Asm/InjectionThunk.asm#L357 https://github.com/tpn/tracer/blob/master/Asm/InjectionThunk...), coming up with new "string table" data structure as an excuse to use some AVX2 intrinsics (https://github.com/tpn/tracer/blob/master/StringTable/PrefixSearchStringTable.c https://github.com/tpn/tracer/blob/master/StringTable/Prefix...), and writing a sqlite3 virtual table interface to the flat-file memory maps, complete with CUDA kernels (https://github.com/tpn/tracer/blob/master/TraceStore/TraceStoreSqlite3ExtInit.c#L228 https://github.com/tpn/tracer/blob/master/TraceStore/TraceSt...). Edit: something pretty fascinating I observed in the wild when running this code: a tracing-enabled Python run of a large project ran faster than a normal, non-traced version of Python in a low-memory environment (a VM with 8GB of RAM). My guess is that splaying the PyCodeObject structures each frame invocation has the very beneficial side effect of reducing the likelihood the OS swaps the underlying pages out at a later date (https://github.com/tpn/tracer/blob/master/Python/PythonFunction.c https://github.com/tpn/tracer/blob/master/Python/PythonFunct...).