4 ms·
I guess you do end up a "full adapter" but you can do it piecemeal. For most of my use cases my custom tables don't need to be highly optimal, I can just give a
by kevincox 4y ago
I guess you do end up a "full adapter" but you can do it piecemeal. For most of my use cases my custom tables don't need to be highly optimal, I can just give a rough expectation of cost, and if I don't provide a batch get then obviously you can't do push-down of predicates.
Some metadata can be provided by the definition, and maybe some can be learned on-the-fly (like latency and data distribution?)
- obi1kenobi 4y agoI designed Trustfall[0] out of frustration with existing systems that either: - didn't allow optimizations like batch get and predicate pushdown even when I knew they'd help, or - made it really difficult to add them piecemeal after-the-fact on an existing schema, by either demanding everything be batched and optimized, or that nothing be batched and optimized. In my experience, it's usually a small component of the schema that is causing most of the runtime cost, and it's not always predictable ahead of time where it's going to be. It's dependent on the queries and the data size and shape, so it changes over time. So I wanted to be able to write a working schema and adapter quickly, and then optimize the parts that were slow in practice for my workload and not just ones that "hypothetically might be slow," and this is what Trustfall lets me do. More info in my reply on an ancestor comment in the thread. [0]: https://github.com/obi1kenobi/trustfall https://github.com/obi1kenobi/trustfall
- michaelmior 4y agoFWIW, you can do this with Calcite. All you really need to write an adapter is something which gives all the data in a table. If that isn't efficient enough, you can write some rules for the planner which allow operators to pushed down.