4 ms·
In the context of databases there are a few issues with relying on the filesystem for checksums: - The default checksum algorithms used in many filesystems are
by jandrewrogers 2y ago
In the context of databases there are a few issues with relying on the filesystem for checksums:
- The default checksum algorithms used in many filesystems are too weak (e.g. CRC32), you really want 128-bits for modern databases.
- While some filesystems give you a selection of checksum algorithms ranging from weak to very strong (e.g. SHA256), the databases like SQLite are often deployed in an environment where you have no control over the algorithm selected.
- The checksum algorithms available in most filesystems do not provide adequate performance on modern storage hardware. It is much easier to evolve these in database engines than in the filesystem.
- Filesystem suitability for database workloads varies widely. Few people are going to trade the performance and scalability of XFS for the built-in integrity mechanisms of ZFS. Implementing checksums in the database gets you the best of both worlds. Also databases are not always deployed on filesystems, so you need checksums for these cases.
From a database engineering perspective, the checksumming in filesystems to the extent it exists is often not fit for purpose. Many databases implement their own integrity mechanisms for this reason.