4 ms·
Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
- john_kerl 5y agoSee also https://miller.readthedocs.io/en/latest/new-in-miller-6/ https://miller.readthedocs.io/en/latest/new-in-miller-6/
- omarhaneef 5y agoI'll give this the second best compliment: I can't believe amongst SED, AWK and so on, this doesn't already exist.
- CodeIsTheEnd 5y agoPreviously discussed (66 comments) just a few months ago: https://news.ycombinator.com/item?id=28298729 https://news.ycombinator.com/item?id=28298729
- dang 5y agoThanks! Macroexpanded: Miller CLI – Like Awk, sed, cut, join, and sort for CSV, TSV and JSON - https://news.ycombinator.com/item?id=28298729 https://news.ycombinator.com/item?id=28298729 - Aug 2021 (66 comments) Miller v5.0.0: Autodetected line-endings, in-place mode, user-defined functions - https://news.ycombinator.com/item?id=13751389 https://news.ycombinator.com/item?id=13751389 - Feb 2017 (20 comments) Miller = sed, awk, cut, join, sort for CSV and tabular JSON - https://news.ycombinator.com/item?id=11674304 https://news.ycombinator.com/item?id=11674304 - May 2016 (1 comment) Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV - https://news.ycombinator.com/item?id=10073040 https://news.ycombinator.com/item?id=10073040 - Aug 2015 (2 comments) Miller is like sed, awk, cut, join, and sort for name-indexed data such as CSV - https://news.ycombinator.com/item?id=10066742 https://news.ycombinator.com/item?id=10066742 - Aug 2015 (75 comments)
- vcdimension 5y agoThis was a great tool written in c, for handling large data files fast, unfortunately the new golang rewrite (version 6.0) is much slower (around 10x in a quick test I did).
- svnpenn 5y agoFine by me. The C version was a pain in the ass to compile. The Go version compiles in less than one minute, with a simple `go build`, on Windows, Linux or any Go platform. I will take that over the faster speed every time.
- tinix 5y agoThat's a fascinating perspective... This mentality is why modern software seems slower... Because it is. So many technological advances, computers are exponentially faster in so many ways, yet we regress in performance. It's a shame, really...
- svnpenn 5y agoWell, compared to Go, Rust, PHP, or many other modern tools, C is garbage in terms of build experience. Speed is important, but its not the only factor.
- marginalia_nu 5y agoIt's basically Parkinson's law. If you give someone 10x the performance, they'll find a way of doing the same things they did before using 10x more resources. It's why you have user interfaces today that are often as slow as, or slower than the equivalent Windows 3.11 interfaces. In 1993 things took time because the processor was an actual potato and the data fetched off a floppy disk, today things are slow because the code being run is 1-10k times slower and messages need to be continuously sent to a datacenter half way around the world to inform an analytics server about your every input.
- john_kerl 5y agoThe 10x number was before improvements on https://github.com/johnkerl/miller/pull/786 https://github.com/johnkerl/miller/pull/786 et al. The earlier negative perf results were my fault, not Go's -- I was focusing initially on the port and feature development, leaving benchmarking and optimization until the end. That said, Go is a bit slower than C line for line; however, Miller 5 (in C) was single-threaded and Miller 6 (in Go) actively uses multicore. This is why complex processing chains now run much quicker in Go than in C -- due to multicore and pipelining which are much easier to do in Go.
- ActorNightly 5y agoIve wrote my own tool for doing this, just with cli style commands. However I found that at a certain level of complexity, python is generally easier. With VScode you can run the script in the terminal and view the output file change in a tab immediately.
- jasonpeacock 5y agoIt's interesting watching these types of tools get re-invented periodically: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.
- dima55 5y agoLots of them out there. Here's one that repurposes the extends the standard unix tools to know about comments and column labels: https://github.com/dkogan/vnlog/ https://github.com/dkogan/vnlog/
- pdimitar 5y agoIf have any more links, please post them! I am a fan of collecting such tools.
- eatonphil 5y agoI just published dsq [0] for running SQL queries against CSV/JSON/Excel/Parquet/etc or just converting those files to JSON. [0] https://github.com/multiprocessio/datastation/tree/main/runner/cmd/dsq https://github.com/multiprocessio/datastation/tree/main/runn...
- loudmax 5y agoPretty sure JSON isn't part of POSIX.
- jasonpeacock 5y agoCorrect, but the point is that POSIX is missing something - maybe a standardized text format? These tools all first convert from various formats into a single, queryable format, then build further tools on top of that queryable format.
- lettergram 5y agoMy team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command: https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler Effectively loads anything into a dataframe. After that you can profile the data, merge, save, load, and take differences between profiles before generating a report.
- vjust 5y agoCool! I will give this a spin.
- vjust 5y agoI am willing to try any alternative to JQ, my main stumbling block being the jq syntax.