3 ms·
I gave this a try. Looks neat however one thing I noticed is that it does not seem very performant in querying parquet files. Take this simple query for example
by Cwizard 4y ago
I gave this a try. Looks neat however one thing I noticed is that it does not seem very performant in querying parquet files. Take this simple query for example:
'select * from ./order.parquet limit 5;'
It takes roughly 38seconds to execute on my machine (I ran it a few times). The order file is about 300mb with 7.5m records in it.
To give some comparison DuckDB can run this query in less than a second (while also directly accessing the parquet file).
- cube2222 4y agoThanks for writing about this! This is a performance regression in the v0.8.0 release it seems, where the outer LIMIT clause doesn't correctly short-circuit evaluation. Overall, for a `LIMIT 10` query, it should basically be instantaneous (and is, with v0.7.3). Will fix this asap. That said, DuckDB will *definitely* be much faster for processing Parquet files, that's for sure.
- cube2222 4y agoThis is now fixed[0]. I've released a new version (v0.8.1) and now a query like this should once again take just ~tens of milliseconds to complete. [0]: https://github.com/cube2222/octosql/commit/ba4a8896be35ffdcc47c208faa86dbfff154c663 https://github.com/cube2222/octosql/commit/ba4a8896be35ffdcc...