8 ms·
If we are doing wishlists: I would love to see better support for fast creation of Arrow/Dataframes/numpy objects from query results (possibly through the COPY
by mangecoeur 7y ago
If we are doing wishlists: I would love to see better support for fast creation of Arrow/Dataframes/numpy objects from query results (possibly through the COPY support, maybe through some kind of plugin or compile time optional support). See for example Turbodbc (https://turbodbc.readthedocs.io/en/latest/pages/advanced_usage.html#apache-arrow-support https://turbodbc.readthedocs.io/en/latest/pages/advanced_usa...).
At the moment we are forced to have query data converted from the binary stream into (boxed) python objects, then unbox them back into arrays - this can add a lot of overhead. I did some very rough experiments in Cython and got 3x speedup for bulk loads of queries into arrays.