4 ms·

If they have CRC accounting for 10% of CPU, they must be using these checksums a lot. At some point I'd imagine the false error rate simply due to bit flips and

If they have CRC accounting for 10% of CPU, they must be using these checksums a lot. At some point I'd imagine the false error rate simply due to bit flips and other random errors on the path from database through CRC function will outlast whatever value you are getting from the constant rechecks. Also, literature suggests a throughput of ∼2.67 bytes per cycle for the CRC32 instruction, a three fold improvement over best in class non-HW routines. I'm quite sure it would be worth it to reconvert previous checksums if you can do so in a way that minimizes downtime (think TrueCrypt doing a transparent initial encryption; not encrypting when theres IO load).

amluto 11y ago
Is the CRC32 instruction much faster than an optimized implementation using PCLMULQDQ? It's available on a wider range of CPUs, but I thought I remembered that PCLMULQDQ worked very quickly.
- mcbain 11y ago
  I think when I looked single instr crc32 was ~5 times faster than the pclmulqdq version. As an aside, if they have a server in production that doesn't support CLMUL, they should junk that machine - it appeared in Westmere and Bulldozer - everything earlier is EOLed. Crc32 was in sse4.2, so Nehalem. The crc32 instruction is hard to beat, has been around for years, and I would guess the different poly could be phased in on their systems like you would for changing a password algorithm, or just try the new poly since it is quick, and if it fails use the old poly.
- illumen 11y ago
  A parallel PCLMULQDQ version is faster than a serial CRC32. Because CRC32 has a latency of 3 cycles. A parallel CRC32 can be slightly faster with more data segment used, and more code size. Also PCLMULQDQ can use an polynomial I think. So they could make a compatible version. The lookup table version they used has to have used lots more CPU cache. Also, I would suggest they look at reducing the amount of times they call the function. Since there are two ways to speed up functions that are called a lot. One is to stop calling it so many times. Or since IO is limiting, they could consider compressing/decompressing the data as well. Using something fast like LZ4 compression. This will give them faster IO and a checksum at the same time. Something like Blosc can be faster than memcpy. Especially since the data is replicated over the network as well, and they would save space on their storage. Modern CPU performance optimization is very often about memory IO. If the data is in L2/L3 then you can do a LOT of computation on it, almost for free, compared to the time it takes to get it into L2/L3. During that time waiting for memory/disk/network IO, they might consider other things. Like indexing, better checksums (like SHA1) or even encryption. Especially if the data is already in L2/L3.

amluto 11y ago

Is the CRC32 instruction much faster than an optimized implementation using PCLMULQDQ? It's available on a wider range of CPUs, but I thought I remembered that PCLMULQDQ worked very quickly.

mcbain 11y ago
I think when I looked single instr crc32 was ~5 times faster than the pclmulqdq version. As an aside, if they have a server in production that doesn't support CLMUL, they should junk that machine - it appeared in Westmere and Bulldozer - everything earlier is EOLed. Crc32 was in sse4.2, so Nehalem. The crc32 instruction is hard to beat, has been around for years, and I would guess the different poly could be phased in on their systems like you would for changing a password algorithm, or just try the new poly since it is quick, and if it fails use the old poly.
illumen 11y ago
A parallel PCLMULQDQ version is faster than a serial CRC32. Because CRC32 has a latency of 3 cycles. A parallel CRC32 can be slightly faster with more data segment used, and more code size. Also PCLMULQDQ can use an polynomial I think. So they could make a compatible version. The lookup table version they used has to have used lots more CPU cache. Also, I would suggest they look at reducing the amount of times they call the function. Since there are two ways to speed up functions that are called a lot. One is to stop calling it so many times. Or since IO is limiting, they could consider compressing/decompressing the data as well. Using something fast like LZ4 compression. This will give them faster IO and a checksum at the same time. Something like Blosc can be faster than memcpy. Especially since the data is replicated over the network as well, and they would save space on their storage. Modern CPU performance optimization is very often about memory IO. If the data is in L2/L3 then you can do a LOT of computation on it, almost for free, compared to the time it takes to get it into L2/L3. During that time waiting for memory/disk/network IO, they might consider other things. Like indexing, better checksums (like SHA1) or even encryption. Especially if the data is already in L2/L3.