3 ms·

Judging from the documentation datasino is easily the best data generator I ever came across! I will definitively give it a spin. Love your tech-stack: S-Expr,

Judging from the documentation datasino is easily the best data generator I ever came across! I will definitively give it a spin. Love your tech-stack: S-Expr, ClickHouse RowBinary, Kafka, nix and Ocaml. A man of culture!

rixed 5y ago
Haha thank you, but how many data generators did you come across? It's not a very widespread genre, although it should be IMHO. Seriously, any open-source references for command-line tools to generate data? In general people just build their own makeshift special purpose generators in my experience.
- stergiotis 5y ago
  You are exactly right! There does not seem to be much out there. I usually do hash or otherwise transform real-world (e.g., https://github.com/awesomedata/awesome-public-datasets https://github.com/awesomedata/awesome-public-datasets) data to retain basic properties like skewness utf8 well-formedness, cardinality, and such. Like many others, I often use the Enron eMail dataset, NYC Taxi, or Twitter data (for coherent use of Unicode). And then scale it either by combining attributes randomly or otherwise augmenting it. Earlier in my career, I wrote a regular expression inversion data generator; it quickly spiraled out of control complexity-wise ;-).
- mattewong 5y ago
  For relatively small data sets in CSV format, you can use the in-browser utility-- which is quite convenient (no installation, no security concerns) at https://zsvhub.com/playground/ https://zsvhub.com/playground/. A subset of the features there are available in a native command-line utility (all open-source-- see https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv), to which the "obfuscate" feature that is currently available in the playground will soon be available and is anticipated to be extremely fast (in keeping with the project's current status as the fastest CSV parser). Full disclosure: I am one of the contributors / authors of zsv/lib and zsvhub

rixed 5y ago

Haha thank you, but how many data generators did you come across? It's not a very widespread genre, although it should be IMHO. Seriously, any open-source references for command-line tools to generate data? In general people just build their own makeshift special purpose generators in my experience.

stergiotis 5y ago
You are exactly right! There does not seem to be much out there. I usually do hash or otherwise transform real-world (e.g., https://github.com/awesomedata/awesome-public-datasets https://github.com/awesomedata/awesome-public-datasets) data to retain basic properties like skewness utf8 well-formedness, cardinality, and such. Like many others, I often use the Enron eMail dataset, NYC Taxi, or Twitter data (for coherent use of Unicode). And then scale it either by combining attributes randomly or otherwise augmenting it. Earlier in my career, I wrote a regular expression inversion data generator; it quickly spiraled out of control complexity-wise ;-).
mattewong 5y ago
For relatively small data sets in CSV format, you can use the in-browser utility-- which is quite convenient (no installation, no security concerns) at https://zsvhub.com/playground/ https://zsvhub.com/playground/. A subset of the features there are available in a native command-line utility (all open-source-- see https://github.com/liquidaty/zsv https://github.com/liquidaty/zsv), to which the "obfuscate" feature that is currently available in the playground will soon be available and is anticipated to be extremely fast (in keeping with the project's current status as the fastest CSV parser). Full disclosure: I am one of the contributors / authors of zsv/lib and zsvhub