26 ms·
Could this same problem be solved by using Apache Arrow, to convert to/from pandas, and cut down the complexity in the process?
by iblaine 7y ago
Could this same problem be solved by using Apache Arrow, to convert to/from pandas, and cut down the complexity in the process?
- lacksconfidence 7y agoThis does use Apache arrow, that's how the data is transferred from spark to pandas and back.
- iblaine 7y agoGreat, glad this was confirmed as I am about to solve a similar problem and my expectation is that Apache Arrow will be needed. So pyarrow = Apache Arrow.