2 ms·
Inability to scale easily to clusters, unless the problem is embarrassingly parallel (ours isn't always), and no equivalent to the conceptual beauty and efficie
by vegabook 11y ago
Inability to scale easily to clusters, unless the problem is embarrassingly parallel (ours isn't always), and no equivalent to the conceptual beauty and efficiency of distributed directed acyclic graphs that Storm amongst others uses. We could hack Python up to do this, but if the problem became TRULY huge, hundreds or thousands of nodes, we'd probably end up spending more time building a processing infrastructure, and repaying technical debt, than doing the actual algorithms, while Spark / Storm etc already have massive scaling ability built in. Still love Python and especially the Numpy ecosystem though and we stream thousands of ticks per second through it no problem, with a bit of help from multiprocessing, MKL libs and Numbapro. But as we get towards tens of thousands we're hitting the buffers, and most importanly, problem complexity is rising so the DAG approach is looking very attractive.
- infinite8s 11y agoI'm looking to build something like Naiad on top of python, disco and numba. Mind if I pick your brain on the kind of tooling you'd like to see around big data python? My email is in my profile.