4 ms·

I've done something very similar recently, for processing hundreds of GB of .csv files across thousands of records. I ended up writing a fast single-threaded Ru

by qchris 6y ago

I've done something very similar recently, for processing hundreds of GB of .csv files across thousands of records. I ended up writing a fast single-threaded Rust program for the file parsing, then fed it into a bash script running GNU Parallel. That thing flies, and watching the system monitor showing every thread on the system get pegged to 100% simultaneously is a neat experience. I'm sure I could have done the parallelization itself in Rust given enough time, but honestly I found the `parallel` command to be so easy and resulting in so little overhead that it didn't really even seem worth it to spend the time on a language-native solution.

spockz 6y ago
Did the 100% system utilisation also lead to better performance? More cpu utilisation doesn’t always mean faster or higher throughput.
- pdimitar 6y ago
  This can't be generalised but in many cases it did help me for CPU-bound tasks. Obviously there are cases where parallelisation is very superfluous. F.ex. if you have 1000 records of something, splitting the work of processing those (and we're talking really light processing like JSON encoding) among 4-8 threads is actually detrimental to performance.
- qchris 6y ago
  That's certainly true, but it does look like in this case. I did the preliminary benchmarking on an 8-thread CPU (the actual HPC cluster has higher counts), and the total execution time on the processing was just under 8x, which I imagine came from the OS-level thread management overhead. I still need to check that pattern holds once thread counts increases, but I'm fairly optimistic it'll work out as long as I don't run into thermal throttling or I/O limits.
heresie-dabord 6y ago
GNU Parallel with awk is a SAPANS, effective combination for data analysis. No faddish rube-goldberg machines. https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/ *SAPANS: as simple as possible, and no simpler Do note that awk and GNU Parallel are tools that use/contain C-like syntax, which is abhorrent to some factions. GNU Parallel is written in Perl.

by qchris 6y ago

spockz 6y ago

Did the 100% system utilisation also lead to better performance? More cpu utilisation doesn’t always mean faster or higher throughput.

pdimitar 6y ago
This can't be generalised but in many cases it did help me for CPU-bound tasks. Obviously there are cases where parallelisation is very superfluous. F.ex. if you have 1000 records of something, splitting the work of processing those (and we're talking really light processing like JSON encoding) among 4-8 threads is actually detrimental to performance.
qchris 6y ago
That's certainly true, but it does look like in this case. I did the preliminary benchmarking on an 8-thread CPU (the actual HPC cluster has higher counts), and the total execution time on the processing was just under 8x, which I imagine came from the OS-level thread management overhead. I still need to check that pattern holds once thread counts increases, but I'm fairly optimistic it'll work out as long as I don't run into thermal throttling or I/O limits.

heresie-dabord 6y ago

GNU Parallel with awk is a SAPANS, effective combination for data analysis. No faddish rube-goldberg machines. https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/ *SAPANS: as simple as possible, and no simpler Do note that awk and GNU Parallel are tools that use/contain C-like syntax, which is abhorrent to some factions. GNU Parallel is written in Perl.