> And initial experiments with Rust are intriguing. Yet with none of them could I obtain the throughput I get in C: see the Performance page. I'd be curious to

> And initial experiments with Rust are intriguing. Yet with none of them could I obtain the throughput I get in C: see the Performance page. I'd be curious to see how `xsv` (written in Rust) compares performance wise to Miller. (I can't compile Miller---I commented on the issue tracker.) > Thus the absence of in-process multiprocessing is only a slight penalty in this particular application domain — parallelism here is more easily achieved by running multiple single-threaded processes, each handling its own input files, either on a single host or split across multiple hosts. I disagree. CSV files can be indexed[1] very simply, providing random access. This makes it trivial to split a large CSV file into chunks which can be operated on in parallel. This lifts the burden off the user to do the splitting and merging manually, which can be become quite tedious when you want to do frequency analysis on the fields. This type of indexing is not usually seen because it requires support from the underlying CSV parser to produce correct byte offsets. But you've written your own parser here in Miller, so you should be able to do something similar to `xsv`. Moreover, random access gets you slicing proportional to the slice. i.e., extract the last 10 records in a 4M row CSV file is instanteous. This opens up other doors too, like potentially faster joining and fast random sampling. P.S. I love the name. :-) [1] - http://burntsushi.net/rustdoc/csv/index/index.html http://burntsushi.net/rustdoc/csv/index/index.html