Great points about data provenance and sanity checking at the right level. This is all valid for any kind of data and not just python stuff. > Doing the task i

Great points about data provenance and sanity checking at the right level. This is all valid for any kind of data and not just python stuff. > Doing the task in vanilla Python does have the advantage of not needing to load the whole file in memory - however, pandas does things behind the scenes to optimize I/O and performance. There's a neat way around this, just set iterator=True or chunksize and pandas'll return you an iterable TextFileReader object: http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunk... Another tip is that even before you get into cythonizing stuff, see how much of the computation you can push down into high performance libraries like numpy - use a map() with a vectorized numpy function, and store your stuff in a numpy data structure instead of a manual for loop on a regular array-of-arrays, etc. Sidenote, is there a decorator version of cythonmagic? Basically I want to annotate my functions with it since sometimes the non-typed basic cython is still much faster than the pure python version, and I don't have to manage the compilation step.