Nice post! I have written several versions of a scientific simulation program in python, first using MPI (single machine) and then native multiprocessing. Usin

Nice post! I have written several versions of a scientific simulation program in python, first using MPI (single machine) and then native multiprocessing. Using numpy and Pandas for all the heavy data operations. I really valued the ergonomics of python and native multiprocessing and found the MPI code to be not very human friendly. However, I ran into some very strange bugs relating to MKL and multiprocessing that just hung the program on large input sizes with MKL_NUM_THREADS above 1. I also had to bounce the input data to the worker processes to files due to mp not being able to pickle data when it got too large. As soon as Apache arrow support in rust gets more feature complete I'll probably try to redo it there.