as a rule of thumb, pure python code is often 1000x slower than naively written unoptimized native code. code in pandas can be very slow (standard pure-python

as a rule of thumb, pure python code is often 1000x slower than naively written unoptimized native code. code in pandas can be very slow (standard pure-python speed) or "fast-for-python" depending on if you are going with or against the grain. a pandas dataframe is basically a bunch of numpy arrays, one array per column. if you do columnar calculations that can be reduced to numpy operations, like summing over a column, then numpy will execute the operation in native code, and it will be fast-for-python, but there will still be some overhead due to wrapping things in python, as well as perhaps temporary array allocation etc. if you do something against the grain, such as expressing all your pandas calculations as on row-wise operations instead of column-wise operations, or using "apply(lambda x: pure_python_expression_of(x))", then pandas cannot execute it efficiently, as the operations are going against the grain of how things are stored in memory, and the operations cannot be reduced to native code primitives on the column arrays that are implemented in numpy. another alternative to switching to rust is using cython to define a native python module. by starting with python code and using many of the same optimization rules of thumb in this post (static typing! avoid frequent tiny allocation, preallocate stuff! avoid hashing complex things, prefer arrays with indexes!), you can translate idiomatic (and very slow) pure python code into simple code that looks closer to array-oriented fortran-in-C, that runs very fast and compiles to a native python module that is easy to integrate.