4 ms·
Interesting observation, but I think the author crosses a bridge too far here. > If you’re processing “big data”, disk I/O probably isn’t the bottleneck. If i
by notacoward 4y ago
Interesting observation, but I think the author crosses a bridge too far here.
> If you’re processing “big data”, disk I/O probably isn’t the bottleneck.
If it fits on a single machine, it is by definition not big data. When you're dealing with really big data, it's likely coming from another machine, or more likely a cluster of them. Networks can also be pretty fast, but there will still be some delay associated with that plus the I/O (which might well be on spinning rust instead of flash) on the other end. Big data requires parallelism to cover those latencies. Requires. It might be true that I/O is no longer likely to be the bottleneck for a single-threaded program, but leave "big data" out of it because in that world I/O really is still a - if not the - major limiter.
- kragen 4y agothe author is evidently pretty unskilled and unaware of it in a lot of ways, this being one of them