3 ms·
I have been doing plenty of analysis of ~400GB of json on a machine with 32GB of ram using jq, tr, sort and grep. So far I haven't found another solution that
by placeybordeaux 11y ago
I have been doing plenty of analysis of ~400GB of json on a machine with 32GB of ram using jq, tr, sort and grep.
So far I haven't found another solution that allows for creating 'queries' so easily. Anyone that knows a way to arbitrarily cut up more json that can easily fit in RAM let me know
- dangerlibrary 11y agoI have long been an advocate of bashreduce. I even worked for a company that ran a cluster of storage machines that each had tiny amounts of ram and contained enormous (many 100GB+) csv text files - this was right around the time hadoop was starting to be a thing, but the cluster was air-gapped for security and updates to external software were incredibly tedious. We'd write small 'cut | grep | sort | awk' sh commands and run them using a tiny perl script called 'prsh' - parallel rsh - to aggregate the results.
- gcr 11y agoWhy not just dump it all into a SQLite database?