4 ms·

This summer I was preparing the ClickBench: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ When I tried to use DuckDB on the same dataset

This summer I was preparing the ClickBench: https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ When I tried to use DuckDB on the same dataset as ClickHouse, it simply did not work due to OOM: https://github.com/duckdb/duckdb/issues/3969 https://github.com/duckdb/duckdb/issues/3969 I also told them about our experience of using various memory allocators, and why you should never use the GLibC's malloc. This issue was fixed.

vgt 4y ago
Thanks, sounds like relative immaturity. I did not catch you address correctness? FWIW I have some experience with Clickhouse as I ran product at Firebolt and played a critical role in being more transparent about their foundations and giving credit where it is due. However, I do have some first hand experience with Clickhouse, which was jarring, considering my previous experience was with BigQuery.
glogla 4y ago
The memory aspect is pretty interesting. I tried to play with "serverless ELT" (the other kind of serverless) where I would define AWS Lambda that turns incoming CSV files into Parquet for archiving and querying. It seems that in DuckDB, the amount of memory it needs to do that is always at least a few gigs, and the memory goes up proportionally with the file size (which I think is due to data type inference?), which is both expensive and annoying because you are either overpaying or you need to go and increase a size of your Lambda when things start crashing. I wonder if ClickHouse-local or some other tool can do that with constant memory, no matter the file size. I know Spark can, but Spark is kind of pain to work with. (Yes, I do realize this is a bizarre use case.)
POPOSYS 4y ago
https://developers.redhat.com/articles/2021/08/25/securing-malloc-glibc-why-malloc-hooks-had-go https://developers.redhat.com/articles/2021/08/25/securing-m...

vgt 4y ago

Thanks, sounds like relative immaturity. I did not catch you address correctness? FWIW I have some experience with Clickhouse as I ran product at Firebolt and played a critical role in being more transparent about their foundations and giving credit where it is due. However, I do have some first hand experience with Clickhouse, which was jarring, considering my previous experience was with BigQuery.

glogla 4y ago

The memory aspect is pretty interesting. I tried to play with "serverless ELT" (the other kind of serverless) where I would define AWS Lambda that turns incoming CSV files into Parquet for archiving and querying. It seems that in DuckDB, the amount of memory it needs to do that is always at least a few gigs, and the memory goes up proportionally with the file size (which I think is due to data type inference?), which is both expensive and annoying because you are either overpaying or you need to go and increase a size of your Lambda when things start crashing. I wonder if ClickHouse-local or some other tool can do that with constant memory, no matter the file size. I know Spark can, but Spark is kind of pain to work with. (Yes, I do realize this is a bizarre use case.)

POPOSYS 4y ago

https://developers.redhat.com/articles/2021/08/25/securing-malloc-glibc-why-malloc-hooks-had-go https://developers.redhat.com/articles/2021/08/25/securing-m...