The original article said that one reason it doesn't matter that pure Python's performance is poor is that you can use numpy (and pandas) to vectorise things, w

The original article said that one reason it doesn't matter that pure Python's performance is poor is that you can use numpy (and pandas) to vectorise things, which then has native code performance. It goes on to say that his current problem is that the things he's doing today can't be vectorised with numpy – so that poor performance does matter after all. If he can't even express his code in terms of numpy operations (and other C-based libraries like scipy), I doubt they're going to be expressible in terms of Spark's primitives, which are a considerably smaller subset.