3 ms·
> RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk. I wouldn't be so sure about it. Current SSDs can
by pkolaczk 4y ago
> RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk.
I wouldn't be so sure about it. Current SSDs can do gigabytes per second read and write. Sure, this is order of magnitude slower than theoretical sequential memory throughput, but might not be slower if you need to do some data processing on CPU or if you do memory access in an inefficient way. There are many ways you can screw it up and processing becomes the bottleneck.
For example if you need to compress data, you might be already not be able to saturate SSD (most compression algorithms don't go faster than 1 GB/s maybe except LZ4).
If you're not careful enough with heap allocations and you happen to create lot of tiny objects when deserializing, or you do lot of pointer chasing when serializing you may as well end up very far from theoretical maximum performance. I've seen systems maxing out at 10 MB/s just due to inefficient serde. Using plain text formats makes it somewhat worse.