3 ms·
Chunking and streaming are different things to me. Chunking means you get to process multiple rows of data at the same time, usually useful to take advantage of
by dmurray 7y ago
Chunking and streaming are different things to me. Chunking means you get to process multiple rows of data at the same time, usually useful to take advantage of SIMD. Streaming means that the data is accessed in a single pass: once you compute the effect of a given row on your statistics, you never have to rewind to see it again.
Many modern performant solutions will use both, but they're not the same thing.
- nine_k 7y agoOne important application of chunking is efficient I/O. Mass storage is most efficient when doing large sequential reads and writes, so you normally feed your constant-space streaming algorithms from buffers with a large number of input records. Sometimes you can just tell the OS do efficient chunking prefetch for you.