3 ms·

Many years back I found that the actual performance of orc and parquet and avro was dominated by the optimisations of the client libraries you used to access th

Many years back I found that the actual performance of orc and parquet and avro was dominated by the optimisations of the client libraries you used to access them rather than the theoretical pros and cons of the formats themselves. (IIRC on spark chose parquet, on presto choose orc etc?) Hopefully the gaping performance gaps have closed.

sanderjd 3y ago
I have spent hours trying to find what I believe must be a memory leak in snowflake's python client that seems to make a query that gets the serialized data (<10GB) locally in under 10 seconds, but requires over a minute to return to the caller. So, yep, this has been my conclusion too!
- chrisjc 3y ago
  Which Python Client? The one using ADBC/nanoarrow? If not, the issue might be related to converting columnar data to row-based. https://www.youtube.com/watch?v=9FOp160CZYE https://www.youtube.com/watch?v=9FOp160CZYE
ayhanfuat 3y ago
This came up in a Twitter discussion, too. Contains a link to an interesting paper which I copied below. > As someone who work with C++ parquet readers and writers I say that different configurations can easily result in 10x differences in size -- the default behavior is usually very poor (Tony Wang / @marsupialtail_2) https://twitter.com/marsupialtail_2/status/1702185015503888391 https://twitter.com/marsupialtail_2/status/17021850155038883... > TUM have written a great summary about performance niches around parquet file IO performance, @DatabendLabs adopted a lot of optimizations mentioned in the summary and result is great in practice. https://dl.gi.de/server/api/core/bitstreams/9c8435ee-d478-4b0e-9e3f-94f39a9e7090/content https://dl.gi.de/server/api/core/bitstreams/9c8435ee-d478-4b... (zhihanz / @zhihanz1205) https://twitter.com/zhihanz1205/status/1702196118472536166 https://twitter.com/zhihanz1205/status/1702196118472536166

sanderjd 3y ago

I have spent hours trying to find what I believe must be a memory leak in snowflake's python client that seems to make a query that gets the serialized data (<10GB) locally in under 10 seconds, but requires over a minute to return to the caller. So, yep, this has been my conclusion too!

chrisjc 3y ago
Which Python Client? The one using ADBC/nanoarrow? If not, the issue might be related to converting columnar data to row-based. https://www.youtube.com/watch?v=9FOp160CZYE https://www.youtube.com/watch?v=9FOp160CZYE

ayhanfuat 3y ago

This came up in a Twitter discussion, too. Contains a link to an interesting paper which I copied below. > As someone who work with C++ parquet readers and writers I say that different configurations can easily result in 10x differences in size -- the default behavior is usually very poor (Tony Wang / @marsupialtail_2) https://twitter.com/marsupialtail_2/status/1702185015503888391 https://twitter.com/marsupialtail_2/status/17021850155038883... > TUM have written a great summary about performance niches around parquet file IO performance, @DatabendLabs adopted a lot of optimizations mentioned in the summary and result is great in practice. https://dl.gi.de/server/api/core/bitstreams/9c8435ee-d478-4b0e-9e3f-94f39a9e7090/content https://dl.gi.de/server/api/core/bitstreams/9c8435ee-d478-4b... (zhihanz / @zhihanz1205) https://twitter.com/zhihanz1205/status/1702196118472536166 https://twitter.com/zhihanz1205/status/1702196118472536166