3 ms·
ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may c
by brancz 4y ago
ArcticDB has a general purpose dataframe-like query builder that Parca uses to build its queries. The query engine scans parquet row groups and those that may contain interesting data are converted to arrow and go through the query plans the query planner creates. The query plans other than the table scan expect arrow frames.
Hope that explains it, but happy to elaborate more!
- bigcat12345678 4y agoIIUC: The query written by users of arcticdb is in a dataframe-like python-ish language. The query engine first go through the inmemory parquet rows to get a subset that contains the relevant data. These relevant data is returned as arrow frames. Then the query planer produces a query plan and the engine execute the query plan on the previously returned arrow frames. Does the query engine produce the query plan before selecting the arrow frames, and uses the plan for the selecting process as well?
- brancz 4y agoAs it stands, right now the query planner doesn’t look at any of the data (yet) when putting together the plan, but this will inevitably happen.