3 ms·

On my M1 mac "dd ... | cksum" takes 3 seconds while "dd | shasum" (sha1) takes 2 seconds. So cksum might not be the best tool for performance checking. There

On my M1 mac "dd ... | cksum" takes 3 seconds while "dd | shasum" (sha1) takes 2 seconds. So cksum might not be the best tool for performance checking. There is CPU specific code in the PG source in src/include/storage/checksum_impl.h It is written as a plain nested loop in C. So performance is fully dependent on the compiler being able to parallelize or vectorize the code. I would not be surprised if manually written SIMD code would be faster.

Joe_Cool 2y ago
Interesting. I guess M1 doesn't have the 'crc32' "acceleration" that is included in SSE4.2.
- loeg 2y ago
  M1 has CRC32 acceleration intrinsics. https://dougallj.wordpress.com/2022/05/22/faster-crc32-on-the-apple-m1/ https://dougallj.wordpress.com/2022/05/22/faster-crc32-on-th... https://github.com/corsix/fast-crc32?tab=readme-ov-file#apple-m1-performance-single-core https://github.com/corsix/fast-crc32?tab=readme-ov-file#appl...
  - silvestrov 2y ago
    So when using these intrinsics an Intel Core i7 can do 30 GB/s but the performance check linked above (by isosphere ) says only 300 MB/s, i.e. 1% Something is amiss here. If a CPU can do 30 GB/s then a CRC check should not have any real performance impact.
    - loeg 2y ago
      I don't know where you're getting 300 MB/s from.
      - silvestrov 2y ago
        Page 5 of https://www-staging.commandprompt.com/uploads/images/CommandPrompt_Performance_Analysis_of_PostgreSQL_Data_Checksums_2019-09-02.pdf https://www-staging.commandprompt.com/uploads/images/Command... says "This system can checksum data at about 300 MB/s per core." It lacks page numbers. Page 5 is first page with gray box at the top of the page.
        loeg 2y ago
        That's measuring 'cksum', which must have an awfully slow implementation. The document notes that this is distinct from measuring PG's checksum performance. (I think it's a pretty useless measurement.) Earlier (page 4): > How much CPU time does it take to checksum... > ...a specific amount of data? This is easy to estimate because PostgreSQL uses the crc32 algorithm which is very simple, and (GNU) Linux has a command line program that does the same thing: cksum. Yeah, using cksum as an estimate here appears to be very flawed.
        Joe_Cool 2y ago
        That is weird. Seems like crc optimization is quite a rabbit hole. https://github.com/komrad36/CRC https://github.com/komrad36/CRC has a massive section about it in the README. Really interesting.
        loeg 2y ago
        Yeah. crc32 may be simple in theory, but doing it as fast as possible utilizing the various execution units of modern hardware is challenging.
anarazel 2y ago
The bottleneck isn't at all the checksum computation itself. It's that to keep checksums valid we need to protect against the potential of torn pages even in cases where it doesn't matter without checksums (i.e. were just individual bits are flipped). That in turn means we need to WAL log changes we don't need to without checksums - which can be painful.

Joe_Cool 2y ago

Interesting. I guess M1 doesn't have the 'crc32' "acceleration" that is included in SSE4.2.

loeg 2y ago
M1 has CRC32 acceleration intrinsics. https://dougallj.wordpress.com/2022/05/22/faster-crc32-on-the-apple-m1/ https://dougallj.wordpress.com/2022/05/22/faster-crc32-on-th... https://github.com/corsix/fast-crc32?tab=readme-ov-file#apple-m1-performance-single-core https://github.com/corsix/fast-crc32?tab=readme-ov-file#appl...
- silvestrov 2y ago
  So when using these intrinsics an Intel Core i7 can do 30 GB/s but the performance check linked above (by isosphere ) says only 300 MB/s, i.e. 1% Something is amiss here. If a CPU can do 30 GB/s then a CRC check should not have any real performance impact.
  - loeg 2y ago
    I don't know where you're getting 300 MB/s from.
    - silvestrov 2y ago
      Page 5 of https://www-staging.commandprompt.com/uploads/images/CommandPrompt_Performance_Analysis_of_PostgreSQL_Data_Checksums_2019-09-02.pdf https://www-staging.commandprompt.com/uploads/images/Command... says "This system can checksum data at about 300 MB/s per core." It lacks page numbers. Page 5 is first page with gray box at the top of the page.
      - loeg 2y ago
        That's measuring 'cksum', which must have an awfully slow implementation. The document notes that this is distinct from measuring PG's checksum performance. (I think it's a pretty useless measurement.) Earlier (page 4): > How much CPU time does it take to checksum... > ...a specific amount of data? This is easy to estimate because PostgreSQL uses the crc32 algorithm which is very simple, and (GNU) Linux has a command line program that does the same thing: cksum. Yeah, using cksum as an estimate here appears to be very flawed.
        Joe_Cool 2y ago
        That is weird. Seems like crc optimization is quite a rabbit hole. https://github.com/komrad36/CRC https://github.com/komrad36/CRC has a massive section about it in the README. Really interesting.
        loeg 2y ago
        Yeah. crc32 may be simple in theory, but doing it as fast as possible utilizing the various execution units of modern hardware is challenging.

anarazel 2y ago

The bottleneck isn't at all the checksum computation itself. It's that to keep checksums valid we need to protect against the potential of torn pages even in cases where it doesn't matter without checksums (i.e. were just individual bits are flipped). That in turn means we need to WAL log changes we don't need to without checksums - which can be painful.