I'm not sure how Vaex works, but for what it's worth, OctoSQL doesn't load whole Parquet files into memory. It will only read the columns that are used in your

I'm not sure how Vaex works, but for what it's worth, OctoSQL doesn't load whole Parquet files into memory. It will only read the columns that are used in your query and will read those columns page by page (Parquet columns are organized into pages on disk) and process pages record by record. It won't ever buffer the whole thing if it's not necessary for the query as a whole.