4 ms·
ClickHouse already has embedded tool for data obfuscation and generation: https://clickhouse.com/docs/en/operations/utilities/clickhouse-obfuscator/ https://cli
by zX41ZdbW 5y ago
ClickHouse already has embedded tool for data obfuscation and generation:
https://clickhouse.com/docs/en/operations/utilities/clickhouse-obfuscator/ https://clickhouse.com/docs/en/operations/utilities/clickhou...
And here is an article about possible methods to solve this task and what method was selected: https://habr.com/en/company/yandex/blog/485096/ https://habr.com/en/company/yandex/blog/485096/
- rixed 5y agoInteresting, I was unaware of this. Not usable in our use case though: IIUC, clickhouse-obfuscator anonymise existing data, whereas we need to produce a giant volume of data (~1M large rows per second) in real time to test data ingestion speed of a data pipeline (that involves Kafka, ClickHouse and other tools in the way).
- zX41ZdbW 5y agoYes, that's true. Although there is an option to run it in a loop over a smaller dataset as a seed.
- zX41ZdbW 5y agoPS. The task of random dataset generation is surprisingly deep and interesting, I welcome more tools in this area. Already digging into your repository...
- rixed 5y agoAnother important difference that I can see between clickhouse-obfuscator and datasino is that obfuscator takes a long way to make sure data, although anonimised, remains statistically realistic, whereas datasino is a dumb data filler/fuzzer: the output values will just be random bits. It's OK if you care about the data structure only but not so much when you need more "realistic" data. Something I'd like to add to datasino is indeed a way to specify constraints on some fields, beside their type, so for instance a column that's supposed to store a time actually look like a time and not a random integer. I just haven't found a nice and simple way to specify those constraints yet.