3 ms·

I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I i

I agree 100%! DataFusion is beautiful and easy to extend in any direction. For the second version of my "out-of-core" graph algorithms project, for example, I implemented my own "co-partitioning" to speed up joins and achieved a performance improvement of two times! It was also easy to modify the physical plan and declare partitioning.

stuhood 2mo ago
Co-partitioning of range-partitioned data is now supported on `main` too: see https://github.com/apache/datafusion/issues/23183 https://github.com/apache/datafusion/issues/23183 and the epic https://github.com/apache/datafusion/issues/22395 https://github.com/apache/datafusion/issues/22395