2 ms·
Hi, this looks like a great addition to the dataflow toolbox! I wondered if OctoSQL supports memory-mapping parquet files like Vaex, or the whole file has to b
by alexisread 4y ago
Hi, this looks like a great addition to the dataflow toolbox!
I wondered if OctoSQL supports memory-mapping parquet files like Vaex, or the whole file has to be loaded into memory first?
- cube2222 4y agoI'm not sure how Vaex works, but for what it's worth, OctoSQL doesn't load whole Parquet files into memory. It will only read the columns that are used in your query and will read those columns page by page (Parquet columns are organized into pages on disk) and process pages record by record. It won't ever buffer the whole thing if it's not necessary for the query as a whole.