5 ms·
Check out apache arrow in rust for transferring large amounts data between rust and python. With it you can get zero copy transfers between the languages. Afaik
by rdedev 5y ago
Check out apache arrow in rust for transferring large amounts data between rust and python. With it you can get zero copy transfers between the languages. Afaik there is some performance penalty involved when converting python types to native rust types.
It used to be hard to do such transfers between rust and python using arrow but from v3 or v4, arrow implementation in rust started supporting the c data interface. Creating the array in rust and sending it to python might involve leaking a boxed object though
Check this out if you want to see how
https://github.com/jhoekx/python-rust-arrow-interop-example/blob/master/src/lib.rs https://github.com/jhoekx/python-rust-arrow-interop-example/...
- wrcwill 5y agoi was using rust numpy for this, i wonder how much faster this is and if it supports passing arrays of strings
- rdedev 5y agoYup. It supports arrays of strings. https://docs.rs/arrow/6.2.0/arrow/array/type.LargeStringArray.html https://docs.rs/arrow/6.2.0/arrow/array/type.LargeStringArra... As for performance, YMMV but I think it could be faster since storing a list of strings as a numpy array would involve padding. Arrow arrays can get away without padding