3 ms·
I tried using threads on multiple pipe reading, to centralise a logfile sorting problem (each discrete logfile is a gz which itself is only partially in order,
by ggm 9y ago
I tried using threads on multiple pipe reading, to centralise a logfile sorting problem (each discrete logfile is a gz which itself is only partially in order, and then between files a merge-sort has to be performed) It was enjoyable to try to fix, but ultimately I found the solution only marginally better than explicit processes feeding a single reader doing round-robin. I think the lesson I learned is that if the problem integrates back into a single context there isn't much you can do to avoid that bottleneck once all the other parallelism opportunities have been overcome.