4 ms·
This does use Apache arrow, that's how the data is transferred from spark to pandas and back.
by lacksconfidence 7y ago
This does use Apache arrow, that's how the data is transferred from spark to pandas and back.
- iblaine 7y agoGreat, glad this was confirmed as I am about to solve a similar problem and my expectation is that Apache Arrow will be needed. So pyarrow = Apache Arrow.