4 ms·
As one example, I use Arrow in production to share DataFrames between Python and Julia processes. My previous method was saving to disk and loading, this reduce
by SatvikBeri 5y ago
As one example, I use Arrow in production to share DataFrames between Python and Julia processes. My previous method was saving to disk and loading, this reduces the time from ~2 minutes to ~5 seconds.
- nerdponx 5y agoYou're actually sharing the memory between the processes? I'd love to know how that works.
- SatvikBeri 5y agoI actually don't know very much about how it works (which is a testament to the usability of the libraries!) I use PyCall to use Python from Julia, and PyJulia to do the reverse. PyCall & PyJulia have functionality to easily share arrays. PyCall is pretty seamless, PyJulia is a bit more work but still solid. To share a DataFrame, I convert it to Arrow, get a bytearray representation of that, use PyCall/PyJulia to access the array from a different process, and reinterpret it as Arrow data within that process. PyCall: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl PyJulia: https://pyjulia.readthedocs.io/en/latest/ https://pyjulia.readthedocs.io/en/latest/