4 ms·
That is not correct, I just refer to the buffers/memory, 0 copying going on. Vaex is not really opinionated about the memory model actually. The only exception
by maartenbreddels 8y ago
That is not correct, I just refer to the buffers/memory, 0 copying going on. Vaex is not really opinionated about the memory model actually. The only exception is the bitmasks that are being copied for now because of an incompatibility with numpy.
But if I get a 50GB Arrow dataset, vaex leaves the structure intact.
Thanks for your work on Arrow, I hope to support and contribute more to it in the future.
- wesm 8y agoI'm looking at the code I linked, and you are serializing in the general case, it is not zero copy. Unpacking a bitmap is not free.
- maartenbreddels 8y agothe 'convert' name is misleading perhaps, maybe we can agree the proof is in the execution time https://youtu.be/TlTcQJPUL3M?t=478 https://youtu.be/TlTcQJPUL3M?t=478 Anyway, let us celebrate a wider adoption of Arrow! :)