3 ms·
Should Citus reevaluate cstore_fdw in light of this extensions API? I wonder if a foreign data wrapper is similarly ill-fitted for this application.
by dharbin 11y ago
Should Citus reevaluate cstore_fdw in light of this extensions API? I wonder if a foreign data wrapper is similarly ill-fitted for this application.
- mslot 11y agoFor basic reads and writes the FDW APIs are quite suitable and they work well for cstore_fdw. However, to get the most out of columnar storage you also need vectorized execution that can calculate aggregates on blocks of values. The FDW APIs are not well-suited for this purpose, since they pull one row of data at a time. We've done some experiments in which we used the executor hooks to implement a vectorized executor for cstore_fdw [1] and it can give dramatic performance gains. However, the SQL support of this solution is limited as the APIs force you to build a new executor from scratch. With sharding, the distributed executor can work on the SQL level and translate one SQL query to many SQL queries. In that case your distributed executor can focus on issues such as partitioning and merging data, while the low-level work is still being done by the regular PostgreSQL executor, so you don't have to implement it from scratch. [1] https://github.com/citusdata/postgres_vectorization_test https://github.com/citusdata/postgres_vectorization_test