4 ms·
I found this super interesting - especially as all the data I've written code to manipulate has been small enough that I haven't needed to optimize my code, so
by bbkane 3y ago
I found this super interesting - especially as all the data I've written code to manipulate has been small enough that I haven't needed to optimize my code, so I've never had to think in this direction.
I think my favorite part was the very first section, where he got baseline measurements with `cat`, `wc`, and friends. I wouldn't have thought to do that and its such an easy way to get a perspective on what's "reasonable".
- latchkey 3y agoI hate to "me too", but you also nailed that analysis.
- yau8edq12i 3y ago[flagged]
- timetopay 3y agoA few months ago, I had to quickly bang out a script to output about 20 million lines of text, each the output of a hash function. My naive solution took more than a few minutes - simple optimizations such as writing every 10k lines cut the time significantly. Threading would have helped quite a bit as well.
- jasonwatkinspdx 3y agoIt also underscores just how insane raw disk bandwidth is on modern ssds. Most software is not designed around a world where you have gigabytes a second of sequential scan on a laptop.
- walth 3y agoI believe this test is run several times and the data set fits in page cache.
- deleted 3y ago[deleted]
- jasonwatkinspdx 3y agoYes, though on a laptop it's unlikely the entire file is being cached in dram. Also ssd's with multiple GiB/sec bandwidths are indeed common, which was my basic point in comparison vs how much slower sed/awk style processing is.