3 ms·
Interesting, I was unaware of this. Not usable in our use case though: IIUC, clickhouse-obfuscator anonymise existing data, whereas we need to produce a giant
by rixed 5y ago
Interesting, I was unaware of this.
Not usable in our use case though: IIUC, clickhouse-obfuscator anonymise existing data, whereas we need to produce a giant volume of data (~1M large rows per second) in real time to test data ingestion speed of a data pipeline (that involves Kafka, ClickHouse and other tools in the way).
- zX41ZdbW 5y agoYes, that's true. Although there is an option to run it in a loop over a smaller dataset as a seed.
- zX41ZdbW 5y agoPS. The task of random dataset generation is surprisingly deep and interesting, I welcome more tools in this area. Already digging into your repository...
- rixed 5y agoAnother important difference that I can see between clickhouse-obfuscator and datasino is that obfuscator takes a long way to make sure data, although anonimised, remains statistically realistic, whereas datasino is a dumb data filler/fuzzer: the output values will just be random bits. It's OK if you care about the data structure only but not so much when you need more "realistic" data. Something I'd like to add to datasino is indeed a way to specify constraints on some fields, beside their type, so for instance a column that's supposed to store a time actually look like a time and not a random integer. I just haven't found a nice and simple way to specify those constraints yet.