3 ms·
If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because i
by kthejoker2 6y ago
If you are already doing Big Data / ML / deep learning work in Spark or Dask, Arrow is (with caveats) more or less a drop-in performance optimization (because it's optimized for in-memory vectors) and complexity reducer (because it does all the interop between JVM, Python, TF, etc)
So by itself it doesn't necessarily introduce new use cases that weren't there before, except where performance was a bottleneck.
But we're talking like 500x performance increase on workloads vs ODBC, so night and day for a lot of data scientist pipelines.