3 ms·

Equivalent sort(1) invocations for your examples: sort -k2 -k1 -k3 contacts.tsv sort -k1 -k2 -k3 contacts.tsv This assumes TSV input, but there are pl

by sigil 9y ago

Equivalent sort(1) invocations for your examples: sort -k2 -k1 -k3 contacts.tsv sort -k1 -k2 -k3 contacts.tsv This assumes TSV input, but there are plenty of reasons to prefer that to CSV. If I'm working from CSV sources I usually convert to TSV first thing in my shell pipeline.

johnweldon 9y ago
Thanks - I've used sort quite a bit, and I like it. I wrote this partly to just fulfil my desire to sort by named fields rather than column indexes.
- sigil 9y ago
  I know the annoyance you're talking about, but I think you're better off wrapping sort(1) with something translates from column names to indexes. Among the reasons: sortcsv buffers all input into memory [1], while sort(1) uses a divide-and-conquer merge sort to avoid this. [1] https://github.com/johnweldon/sortcsv/blob/55818bd8e5f9feeccc242b78acc3f23fd0613b44/main.go#L68 https://github.com/johnweldon/sortcsv/blob/55818bd8e5f9feecc...
  - johnweldon 9y ago
    That's a good point. I may consider a stand alone utility to wrap this[1] code that converts names to indexes. [1] https://github.com/johnweldon/sortcsv/blob/master/main.go#L104-L118 https://github.com/johnweldon/sortcsv/blob/master/main.go#L1...
feelin_googley 9y ago
When sort is used on really large files, it will automatically attempt to use disk, putting temp files in TMPDIR. This can be really slow. To overcome the slowdown of disk I/O, perhaps a workaround could be to use mfs or tmpfs, maybe something like: mkdir /dir mount -t tmpfs tmpfs /dir TMPDIR=/dir sort -k2 -k1 -k3 contacts.tsv TMPDIR=/dir sort -k1 -k2 -k3 contacts.tsv Personally, I gave up on sort for large files and use k/kdb+. I suspect it is faster for sorting than sort or the Go libraries, but I could be wrong.
- sigil 9y ago
  For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|). Instead, your best bet in that case is to give sort as much physical memory as you can spare: sort -S 95% -k1 huge.tsv Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered. Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster. Other things to check out if you need Very Fast Large Sorts: - Use `sort --parallel=N` to use multiple cores. By default it only uses 1. - Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.

by sigil 9y ago

johnweldon 9y ago

Thanks - I've used sort quite a bit, and I like it. I wrote this partly to just fulfil my desire to sort by named fields rather than column indexes.

sigil 9y ago
I know the annoyance you're talking about, but I think you're better off wrapping sort(1) with something translates from column names to indexes. Among the reasons: sortcsv buffers all input into memory [1], while sort(1) uses a divide-and-conquer merge sort to avoid this. [1] https://github.com/johnweldon/sortcsv/blob/55818bd8e5f9feeccc242b78acc3f23fd0613b44/main.go#L68 https://github.com/johnweldon/sortcsv/blob/55818bd8e5f9feecc...
- johnweldon 9y ago
  That's a good point. I may consider a stand alone utility to wrap this[1] code that converts names to indexes. [1] https://github.com/johnweldon/sortcsv/blob/master/main.go#L104-L118 https://github.com/johnweldon/sortcsv/blob/master/main.go#L1...

feelin_googley 9y ago

When sort is used on really large files, it will automatically attempt to use disk, putting temp files in TMPDIR. This can be really slow. To overcome the slowdown of disk I/O, perhaps a workaround could be to use mfs or tmpfs, maybe something like: mkdir /dir mount -t tmpfs tmpfs /dir TMPDIR=/dir sort -k2 -k1 -k3 contacts.tsv TMPDIR=/dir sort -k1 -k2 -k3 contacts.tsv Personally, I gave up on sort for large files and use k/kdb+. I suspect it is faster for sorting than sort or the Go libraries, but I could be wrong.

sigil 9y ago
For a dataset larger than physical memory, using a memory filesystem like tmpfs for the merge stage will either swap (|tmpfs| < |ram|) or deadlock (|tmpfs| >= |ram|). Instead, your best bet in that case is to give sort as much physical memory as you can spare: sort -S 95% -k1 huge.tsv Extra disk I/O is inevitable since your dataset doesn't fit in memory. At least during a merge sort your disk reads will be O(N) and sequentially ordered. Note: in the special case that your dataset is slightly larger than physical memory, splitting it up in advance such that one of the `sort -m` input files lives on a tmpfs should indeed be faster. Other things to check out if you need Very Fast Large Sorts: - Use `sort --parallel=N` to use multiple cores. By default it only uses 1. - Use `sort --batch-size=NMERGE` to increase the number of files merged at once. Otherwise you may be doing more mergesort stages than are necessary.