3 ms·
You normally develop the tools and scripts on a much smaller data sets. So you export 1 minute of data, examine it manually or with simple tools, write your scr
by theamk 3y ago
You normally develop the tools and scripts on a much smaller data sets. So you export 1 minute of data, examine it manually or with simple tools, write your scripts and once they work, switch to processing months worth of it.
Human-readable comes handy here.
- Dylan16807 3y agoA pretty printer is almost as handy and with significantly fewer compromises.
- jitl 3y agoI prefer SQL for tabular data larger than 3 screens high, for me it’s easier for basically any analysis compared to a grep/wc/count/cut/paste bash pipeline. I use sqlite for CSV pretty regularly, if I needed columnar I’d use duckdb for parquet