4 ms·
Generally good advice, but note that in some cases 'slurping' is actually much better. The most obvious one is where you can exploit parallelism (either CPU pa
by kevingadd 13y ago
Generally good advice, but note that in some cases 'slurping' is actually much better.
The most obvious one is where you can exploit parallelism (either CPU parallelism or storage parallelism) by fetching multiple entire files at once and preparing them in memory. This allows you to start spending CPU time processing one loaded file, while other files load in the background. When you stream a file one line at a time, other than some basic optimistic lookahead, it's not really possible for the OS to do as much to help you there, so you're going to be effectively single-threaded. If the computation you're doing on the data is significant, you can end up being unable to even maximize your use of a single core on computation.