5 ms·
This and similar tasks can be solved efficiently with clickhouse-local [1]. Example: ch --input-format LineAsString --query "SELECT line, count() AS c GROU
by zX41ZdbW 1y ago
This and similar tasks can be solved efficiently with clickhouse-local [1]. Example:
ch --input-format LineAsString --query "SELECT line, count() AS c GROUP BY line ORDER BY c DESC" < data.txt
I've tested it and it is faster than both sort and this Rust code:
time LC_ALL=C sort data.txt | uniq -c | sort -rn > /dev/null
32 sec.
time hist data.txt > /dev/null
14 sec.
time ch --input-format LineAsString --query "SELECT line, count() AS c GROUP BY line ORDER BY c DESC" < data.txt > /dev/null
2.7 sec.
It is like a Swiss Army knife for data processing: it can solve various tasks, such as joining data from multiple files and data sources, processing various binary and text formats, converting between them, and accessing external databases.
[1] https://clickhouse.com/docs/operations/utilities/clickhouse-local https://clickhouse.com/docs/operations/utilities/clickhouse-...
- nasretdinov 1y agoTo be more fair you could also add SETTINGS max_threads=1 though?
- supermatt 1y agoHow is that “more fair”?
- deleted 1y ago[deleted]
- nasretdinov 1y agoWell, fair in a sense that we'd compare which implementation is more efficient. Surely, ClickHouse is faster, but is it because it's using actually superior algorithms or is it just that it executes stuff in parallel by default? I'd like to believe it's both, but without "user%" it's hard to tell
- mickeyp 1y agoLast time I checked, writing efficient, contention-free and correct parallel code is hard and often harder than pulling an algorithm out of a book.
- reppap 1y agoWould you take half the wheels off a car to compare it to a motorcycle?
- Almondsetat 1y agoMotorcycles are faster than cars though
- Etheryte 1y agoNot necessarily, that really depends on what you mean by fast. Cars definitely go higher in top speed than bikes do for example. If I'm not mistaken, racing electric cars also accelerate comparable or faster than bikes. A bike can generally go around a track faster than a car, but that only holds true in dry conditions. Etc, many ways to define fast and what you actually mean.
- AtlasBarfed 1y agoI thought all the land speed records were basically motorcycles. Jet motorcycles but motorcycles
- wang_li 1y agoMusk's roadster is currently going in excess of 10,000 mph. Which bike is faster than that? :)
- gigatexal 1y agoExactly. I love this and DuckDb and other such amazing tools.
- da_chicken 1y agoI'd not heard of clickhouse before. It does seem interesting, but I just can't get behind a project that says: > The easiest way to download the latest version is with the following command: > curl https://clickhouse.com/ https://clickhouse.com/ | sh Like, sure, there is some risk downloading a binary or running an arbitrary installer. But this is just nuts.
- gigatexal 1y agoChdb is just a binary. You can just grab that. Also pipe to sh is used by a ton of projects
- bflesch 1y agoit's used by many projects but still regarded as an anti-pattern and security issue
- monerozcash 1y agoit's really exactly the same as wget file;./file and not a real anti-pattern in any way
- Etheryte 1y agoA ton of people drink and drive too, doesn't make it any more fine.
- gigatexal 1y agoY’all are so pure. Just don’t install it that way. Sheesh.
- Aefiam 1y agohow is this any less secure than running a binary/installer? the binary could run this inside?
- trollbridge 1y agoIt's Apache licenced and you could also install it via your favourite package installer. Given all the crazy supply chain attacks going on, I don't really feel this is any worse than downloading a binary from a distro archive, and specifically this pipe | sh doesn't expect you to run it as root (which a lot of other cut-and-paste installers do).
- nsteel 1y agoJust noting that in your benchmark (which we know nothing about), your "naive" data point is just 2.29x slower than hist. In their testing it was 27x slower! And it's not quite the same naive shell command, which isn't helpful.
- danlark1 1y agoDisclaimer: the author of the comment is the founder and CTO of ClickHouse
- LtdJorge 1y agoWhen using clickhouse-local like this, does it build a logical plan and run the optimizer on it? Does it have any kind of code generation, since it knows the query (and physical data layout) ahead of time?