4 ms·
> which potentially means huge volumes of data moving into this physical processing layer You get to cheat at that, because the naive implementation loads too
by gopalv 10y ago
> which potentially means huge volumes of data moving into this physical processing layer
You get to cheat at that, because the naive implementation loads too much data, while most of it is going to drop off at the filter level.
The ideal TPC-DS schema stores data partitioned by date and maintains a lookup index by item, store and clustered by store's state.
In a database organized like that, neither part of the Query-3
item.i_manufact_id = 436
or
dt.d_moy=12
should actually look at every row in the data-set in brute-force fashion - you get to skip massive parts of the data by just inspecting the index or going to the specific date partition and do it much faster than a naive C++ full-scan.
However, if you were limited to a full-load + scan, this would really be killer to have a columnar format which can be handled by an FPGA. The really interesting quote in the blog (for me) is
> the data for the queries is pushed to the accelerator card in columnar format (which is blazingly fast for queries)
I sure could use FPGA filters for columnar data ... even after I've done all my indexing, since I can't index down AND clauses or OR clauses - unlike a traditional CPU, the FPGA will be able to evaluate my entire condition in one sequential flow instead of check + branching.