3 ms·
Its use case is usually going to be sorting the output of other commands, I'm impressed it works at all. If you want to test something meant to scale, feed eve
by BubRoss 6y ago
Its use case is usually going to be sorting the output of other commands, I'm impressed it works at all.
If you want to test something meant to scale, feed everything into an sqlite database and read it back. Sqlite should chew through this without any problem.
- fwip 6y agoA similar approach was used in this tool, whose writeup I found quite interesting: https://devblog.dnanexus.com/faster-bam-sorting-with-samtools-and-rocksdb/ https://devblog.dnanexus.com/faster-bam-sorting-with-samtool... Background: "samtools," a tool meant for processing multi-gigabyte genomic alignment data, uses a sort algorithm basically the same as `sort`. This fork instead jams the data into Facebook's RocksDB, and then extracts it back out. Speedup ranged from 1.2x to 5x, with bigger input files and slower disks seeing the most gains.