3 ms·
I would question if the serialisation code speed actually matters. RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent w
by sjducb 4y ago
I would question if the serialisation code speed actually matters.
RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk.
To speed it up you have to reduce the file size so that it can be written to disk faster. I think the fastest way would be a gzipped JSON file.
You can do all of the converting to JSON and compression while you're waiting for the hard disk to finish writing.
- pkolaczk 4y ago> RAM and CPU are much much faster than an SSD so almost all (98%) of the time will be spent writing to disk. I wouldn't be so sure about it. Current SSDs can do gigabytes per second read and write. Sure, this is order of magnitude slower than theoretical sequential memory throughput, but might not be slower if you need to do some data processing on CPU or if you do memory access in an inefficient way. There are many ways you can screw it up and processing becomes the bottleneck. For example if you need to compress data, you might be already not be able to saturate SSD (most compression algorithms don't go faster than 1 GB/s maybe except LZ4). If you're not careful enough with heap allocations and you happen to create lot of tiny objects when deserializing, or you do lot of pointer chasing when serializing you may as well end up very far from theoretical maximum performance. I've seen systems maxing out at 10 MB/s just due to inefficient serde. Using plain text formats makes it somewhat worse.
- anonymoushn 4y agoI think this comment assumes very slow disks compared to the disks that are available.