qoega

7 ms·

31.
by qoega 4y ago
Look on ClickHouse. I think you will be pleasantly surprised how fast and effective it would be. By the way, you have to pay for this access or there are some other options available? It is interesting to ingest all this data by ourselves
32.
by qoega 4y ago
Is it really the case? Assume you have column with schema name and one with json object. And your materialised view/JsonExtract can be dispatched by schema name for a row. I see the only suboptimal part if different schemas have differ
33.
by qoega 4y ago
Interesting! As ClickHouse is not the fastest in this scenario, we can improve it! Probably initialisation of clickhouse-local process takes more time than jq.
34.
by qoega 4y ago
ClickHouse has no such optimizations for reuse of columns.
35.
by qoega 5y ago
Have you tried ClickHouse for dataset above? Interesting if it is faster or not
36.
by qoega 5y ago
Looks, it is not
37.
by qoega 5y ago
Hi! Why do you share it today? You have updated something recently in it?
38.
by qoega 5y ago
You can add even without mentioning. I did almost nothing - it worked almost without diff.
39.
by qoega 5y ago
The rest of the queries https://gist.github.com/qoega/32c4f54da6b0324aef8cd8d910dd7c...
40.
by qoega 5y ago
It will probably be even faster: > clickhouse-client --time --query "INSERT INTO rides FROM INFILE 'nyc_data_rides.csv' FORMAT CSV;"
41.
by qoega 5y ago
You can use clickhouse/clickhouse-server image. It has 21.10 version: https://hub.docker.com/layers/clickhouse/clickhouse-server/2...
42.
by qoega 5y ago
You can enable fsync in ClickHouse. And it will not decrease bandwidth.
43.
by qoega 5y ago
Why use FDW if you can work with ClickHouse directly? What is the use case? By the way I can see there are some already but I do not use them https://github.com/Infinidat/infi.clickhouse_fdw/ https://gi
44.
by qoega 5y ago
Nice to hear! We are looking for cases where ClickHouse is not as fast as it could be. And we you can share your experience with ClickHouse on GitHub issues or by mail. If something was not intuitive or worked not as expected. Or lack in fu
45.
by qoega 5y ago
ClickHouse has no limit on sting data length. MaterializedPostgreSQL engine is just very recent feature with not large adoption rate. I believe if community will use it frequently it will become more bulletproof and more edge cases will be
46.
by qoega 5y ago
From docs it seems they use forked ClickHouse code.
47.
by qoega 5y ago
Have you tried generateRandom? It supports almost all types. https://clickhouse.tech/docs/en/sql-reference/table-function...