4 ms·
Gotcha. Just some confusion around what you meant by in-memory OLAP engine. It seems that many storage format libraries get integrated into other platforms that
by zten 9y ago
Gotcha. Just some confusion around what you meant by in-memory OLAP engine. It seems that many storage format libraries get integrated into other platforms that bring the other supporting features (and the other undesirable heavyweight things you didn't want) with them.
With a little bit more poking: Apache Calcite?
- sixdimensional 9y agoCalcite is more a query optimization framework. Actually, I think parent has a good question, which is, how to use these tools, formats, etc. from something simpler than an entire full blown distributed technology - and I think the answer to that is, there aren't too many technologies out of the box that do that right now. I would think maybe extending SQLite to use columnar storage would be one possible path that might make sense. There are columnar DBMS engines you can embed like MonetDB, but I personally haven't seen a lightweight query engine sitting right on top of Parquet/ORC, or implementing Arrow for in-memory columnar. You can get Apache Spark up and running pretty quickly on a single node, but I haven't seen anything even easier than that.