3 ms·

I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for C

I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns. On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

p33p 5y ago
> On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it. I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed? I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.