3 ms·
"""Even with fast NVMe drives and splitting the database into two halves (important data vs. ephemeral data that we could lose on a tmpfs), things got slower an
by dekhn 6y ago
"""Even with fast NVMe drives and splitting the database into two halves (important data vs. ephemeral data that we could lose on a tmpfs), things got slower and slower. We knew the day would come. The file reached a peak size of 150MB and we were writing it as quickly as the disk I/O would let us. Ain’t that just peachy?"""
Uh, you compressed it first, right? because CPUs can compress data faster than disk I/O.
- bradfitz 6y agoYeah, I think it was zstd. But the bigger problem was the curve. Doing something O(N*M) where N (file size) and M (writes per second) were both growing was not a winning strategy, compression or not.
- jeffbee 6y agoHrmm. Even lz4 level 1 compresses at "only" about 500-1000MB/s on various CPU types, which isn't quiet as fast as NVMe devices demand.
- dekhn 6y agoyes, that's a very recent change (afaict, the big change was going from SATA interface to NVME).