zX41ZdbW

49 ms·

691.
by zX41ZdbW 4y ago
If you are interested in the research on technologies used on the Internet, I recommend playing with the "Minicrawl" dataset. It contains data about ~7 million top websites, and for every website, it also contains: - the full cont
692.
by zX41ZdbW 4y ago
For a dynamic set of strings, I recommend checking the StringHashTable in ClickHouse. The basic idea is: to use a bunch of hash tables with fixed-size keys but split the strings into power-of-two size-classes. This basic idea originated in
693.
by zX41ZdbW 4y ago
Another idea - why don't plug in faker.js as an UDF?
694.
by zX41ZdbW 4y ago
This post does not mention `clickhouse-obfuscator`. It is a small tool that generates datasets based on another dataset - with the same structure and the same distribution of values, including the mutual distributions. It allows data anonym
695.
by zX41ZdbW 4y ago
There are only a few ways to determine it before running the query - by the amount of rows/bytes to scan. But most of the logic happens when query is already started - by the possibility to stop it and throw exception if some metrics b
696.
by zX41ZdbW 4y ago
We do it exactly this way in ClickHouse, and it works perfectly. You can even expose the database server to the internet and get away with it if you configure the user access control, quotas, and limitations on query complexity. For example
697.
by zX41ZdbW 4y ago
mcedit
698.
by zX41ZdbW 4y ago
I've also thought about this problem but ended up with curl | sh: curl https://clickhouse.com/ | sh The arguments are: - it's 'sh', not 'sudo sh', so it's not that bad; - you can re
699.
by zX41ZdbW 4y ago
clickhouse-local[1] saved me countless hours. I'm using it for format conversion, data processing, querying external data, as a calculator, etc... clickhouse-local --input-format Parquet --output-format JSONEachRow --query "SE
700.
by zX41ZdbW 4y ago
We have tested it on a few queries with aggregation (with low cardinality results). For MySQL, fetching the raw data and doing aggregation in ClickHouse appeared to be faster. For PostgreSQL it was identical (at least Postgres did not do si
701.
by zX41ZdbW 4y ago
For comparison, the implementation of the same feature in ClickHouse: https://github.com/ClickHouse/ClickHouse/blob/master/src/Sto... https://github.com/ClickHouse/ClickHouse&#x
702.
by zX41ZdbW 4y ago
memcpy implementation is a rabbit hole. The main trap is trying to benchmark and profile it without the context of the whole application. I recommend to read a comment in ClickHouse source code (disclaimer: written by me) https:/&#x2F
703.
by zX41ZdbW 4y ago
The author of the article underestimates the capabilities of the modern OLAP DBMS. The old beliefs are becoming less and less relevant. > When to use a data warehouse > Data warehouses are good for OLAP (online analytical processing)
704.
by zX41ZdbW 4y ago
The advantage of OLAP vs. OLTP on analytical queries is clearly visible on the benchmark: https://benchmark.clickhouse.com/ If we take the extreme - ClickHouse as the most thoroughly optimized column-oriented DBMS and compa
705.
by zX41ZdbW 4y ago
Yes, I'm working on ClickHouse, here is my GitHub profile: https://github.com/alexey-milovidov I'm also trying to follow every existing technology in the data engineering space :)
706.
by zX41ZdbW 4y ago
Interesting - the database file looks ok, but the data is lost (the table is empty): ubuntu@ip-172-31-3-138:~$ ls -l total 69561648 -rw-rw-r-- 1 ubuntu ubuntu 17631031296 Dec 16 23:57 my-db.duckdb -rw-rw-r-- 1 ubuntu ubuntu
707.
by zX41ZdbW 4y ago
I cannot even ssh into the server after trying to use DuckDB. It is completely dead (with all the ducks, what a misery). The reason is probably that it's using a full index, in contrast with the sparse index in ClickHouse, and maybe it
708.
by zX41ZdbW 4y ago
I checked DuckDB and your statement appears to be untrue. >>> con.execute("CREATE TABLE passwords (hash TEXT, count INT)") <duckdb.DuckDBPyConnection object at 0x7fc7bceb55f0> >>> con.execute(
709.
by zX41ZdbW 4y ago
But it's not necessary to keep data in Memory, because MergeTree works perfectly. Here is a slide deck showing why MergeTree is faster than in-memory tables: https://presentations.clickhouse.com/meetup53/optimizati
710.
by zX41ZdbW 4y ago
These days you can simply load it into a database (ClickHouse) - it will be compressed (about the same as the 7z file), and the search will take 3 ms: https://github.com/ClickHouse/ClickHouse/issues/42363
711.
by zX41ZdbW 4y ago
You can get the stats with a simple query to ClickHouse Playground. Example: https://play.clickhouse.com/play?user=play#U0VMRUNUCiByZXBvX... No need for a tool with a sign-in requirement.
712.
by zX41ZdbW 4y ago
PS. I closed the curtains and start wathing the video... It looks really impressive!
713.
by zX41ZdbW 4y ago
How to switch to light theme? The font is almost unreadable if you are reading under sunlight.
714.
by zX41ZdbW 4y ago
> a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations Sounds like ClickHouse. And we see kdb slowly start being repla
715.
by zX41ZdbW 4y ago
What is the relationship between DuckDB, DuckDB Labs, DuckDB Foundation, and Motherduck? Sorry for asking, but I'm really confused looking at all the websites.
716.
ClickHouse removes its own code from memory and switches to using HugePages(prog.world)
19 points by zX41ZdbW 4y ago|0 comments
717.
by zX41ZdbW 4y ago
Here is a comparison of various SQL engines in command line: https://github.com/dcmoura/spyql/blob/master/notebooks/json_...
718.
by zX41ZdbW 4y ago
Does SPyQL have any advantages over clickhouse-local?
719.
More Than 2x Faster Hashing in ClickHouse Using Rust(clickhouse.com)
4 points by zX41ZdbW 4y ago|0 comments
720.
by zX41ZdbW 4y ago
Here is a recent comparison: https://arxiv.org/pdf/2204.09795.pdf Although I'm not sure if it is using the new IOx engine or not.

More ›