4 ms·
Using ClickHouse you can also process local files in one line using clickhouse-local command tool. And it will look a lot easier: clickhouse local -q "SELECT p
by avogar 4y ago
Using ClickHouse you can also process local files in one line using clickhouse-local command tool. And it will look a lot easier:
clickhouse local -q "SELECT passenger_count, COUNT(*), AVG(total_amount) FROM file(taxi.csv, 'CSVWithNames') GROUP BY passenger_count"
And ClickHouse supports a lot of different file formats both for import and export (you can see all of them here https://clickhouse.com/docs/en/interfaces/formats/ https://clickhouse.com/docs/en/interfaces/formats/).
There is an example of using clickhouse-local with taxi dataset mentioned in the post: https://colab.research.google.com/drive/1tiOUCjTnwUIFRxovpRXQ2xVOHLgN_WL8?usp=sharing https://colab.research.google.com/drive/1tiOUCjTnwUIFRxovpRX...
- eatonphil 4y agoClickhouse-local is incredible. It does the best of any similar tool I've benchmarked. But the reason I took it out of the linked benchmarks in OP's post is because it's 2+GB. That's a massive binary. It's the whole server. I'm not sure you want to be distributing this all over the place in general. It's just not in the same category IMO. Disclaimer: I build another tool that does similar things.
- qoega 4y agoIt is that big only with debug symbols. If you take packaged version of ClickHouse it will be much smaller. Or just strip large binary manually if you already have it.
- zX41ZdbW 4y agoIt is shipped with debug info and symbol tables just in case. Without debug info, it will be 350 MB, and compressed can fit in 50 MB: https://github.com/ClickHouse/ClickHouse/issues/29378 https://github.com/ClickHouse/ClickHouse/issues/29378 It is definitely a worth improvement.
- eatonphil 4y agoAha, thanks for the clarification and link. I'll be following that issue.