4 ms·

PostgreSQL, one of the most popular databases for reliable data management, also does not enable this feature by default. "By default, data pages are not prote

PostgreSQL, one of the most popular databases for reliable data management, also does not enable this feature by default. "By default, data pages are not protected by checksums, but this can optionally be enabled for a cluster. " https://www.postgresql.org/docs/current/checksums.html https://www.postgresql.org/docs/current/checksums.html I feel it's quite unfair for the title to call out sqlite on checksums, when in reality, as the very closing line of the article states most databases don't do checksum verifications.

llm_nerd 2y ago
The submission doesn't seem unfair. It's simply pointing out something for users of a specific database to be aware of, and I don't really think SQLite is maligned or impugned in any way. And FWIW, Oracle, SQL Server and DB2 enable page checksum storing/verification by default.
- deleted 2y ago
  [deleted]
- RicoElectrico 2y ago
  People are just bored of security sensationalism I guess. Too many people want to gain visibility just by reporting either something little-known (but still known) or that needs stars to align in a proper way to be exploitable at all.
  - MattPalmer1086 2y ago
    Checksums aren't used to protect integrity for security, they're about data corruption.
- Brian_K_White 2y ago
  A long time ago there was a famous / infamous ad campaign for some food product like bread or milk, and the ad merely stated the true fact that their milk didn't contain any bleach. Of course no one's milk (or whatever it was) had any bleach (or whatever it was). Specially highlighting something true, but out of context and with no equally special justification, is not an innocent act and is misleading. And yes, absolutely, very clearly, causes harm, and does so unfairly. A not-unfair version of this same article would just talk about databases, and include sqlite with others, and not only sqlite and not be titled sqlite. And then there is this: "Hey there I am v. I work at Turso Database."
  - llm_nerd 2y ago
    Your milk/bread comparison is specious and invalid. I might say "As an apple enthusiast -- the fruit kind -- I want you to know that apple seeds have arsenic so don't eat lots of them" without having to disclaim every other fruit existing and possibly toxins found in parts of them. If some true-believer apple fan felt victimized, well that would be bizarre, right? This is a SQLite guy talking about SQLite to SQLite users. They're describing a feature/possible downside of SQLite that users might want to be aware of. They don't need "balanced" coverage of every other DB because it hurts someone's bizarrely fragile feelings. And as I mentioned elsewhere, almost all "enterprise" databases do do checksums by default, if people really want to lean on this "no one does! Leave SQLite alone" argument. And Turso is literally a SQLite-based firm. This isn't the aha you think it is.
  - naveen99 2y ago
    Kind of like fake praise / avoiding constructive criticism in the interest of harmony and feelings.
  - derekperkins 2y ago
    A frozen banana that WON'T make you sick and kill you. https://x.com/bluthquotes/status/732749820348096512 https://x.com/bluthquotes/status/732749820348096512
- roncesvalles 2y ago
  And that, kids, is why companies pay money to Oracle, Microsoft, and IBM.
  - SubjectToChange 2y ago
    Major corporations are not paying exorbitant licensing fees to have checksumming enabled by default. In fact, for enterprises running things like vSAN, ECC DRAM, etc, database checksumming is probably nothing more than additional overhead. Database defaults in general are a touchy topic. Whatever set of defaults are chosen will be suboptimal for almost any serious user. A far more serious issue is figuring out the actual behavior of a database in different configurations. For instance, Oracle's SERIALIZABLE transaction isolation level only offers snapshoot isolation.
    - jandrewrogers 2y ago
      You don’t understand the critical problem that checksums solve at the I/O boundary. PCIe has weak error detection and correction. To transfer your data from ECC memory to your favorite super-robust storage technology requires transiting the PCIe bus, where for a brief moment it becomes relatively easy to corrupt data without anyone noticing. This is the problem that can’t be solved any other way and why checksums are primarily done at the I/O boundary in databases. It is an issue seen in real systems. PCIe v6 is intended to materially improve the integrity of data transfers but what we are using today is much worse.
anarazel 2y ago
FWIW, PG's default has been flipped to checksums being enabled by default in the in-development version. Personally I'm not sure it's the right call, as it comes with some increased write overhead.
- aflukasz 2y ago
  Yes, it's a balancing act. I think it's better to have safer defaults here, not prioritizing extra performance. People who are concerned with performance to the degree where checksums would matter need to consult configuration anyway (in many aspects, not only this one) and can disable this specific thing easily. For the reference, change is in https://github.com/postgres/postgres/commit/04bec894a04cb0d32533f1522ab81b7016141ff1 https://github.com/postgres/postgres/commit/04bec894a04cb0d3... (so should land in PostgreSQL 18 in a year, unless it will get reverted for some reason).
avinassh 2y ago
> I feel it's quite unfair for the title to call out sqlite on checksums, when in reality, as the very closing line of the article states most databases don't do checksum verifications. Hi, author of the post here. I work mostly with SQLite and that's the database I'm most familiar with, hence mentioning it in the title. I have also noted at the bottom: > Again, this is not a bug. Most databases (except a few) assume that the OS, filesystem, and disk are sound. Whether this matters depends on your application and the guarantees you need. now I wrote a follow up post - https://avi.im/blag/2024/databases-checksum https://avi.im/blag/2024/databases-checksum
refulgentis 2y ago
Every crumb of information you encounter doesn't have to be subjected to a lens of "but what if the entities involved were sports teams? Would this be "fair" to "them"?"

llm_nerd 2y ago

The submission doesn't seem unfair. It's simply pointing out something for users of a specific database to be aware of, and I don't really think SQLite is maligned or impugned in any way. And FWIW, Oracle, SQL Server and DB2 enable page checksum storing/verification by default.

deleted 2y ago
[deleted]
RicoElectrico 2y ago
People are just bored of security sensationalism I guess. Too many people want to gain visibility just by reporting either something little-known (but still known) or that needs stars to align in a proper way to be exploitable at all.
- MattPalmer1086 2y ago
  Checksums aren't used to protect integrity for security, they're about data corruption.
Brian_K_White 2y ago
A long time ago there was a famous / infamous ad campaign for some food product like bread or milk, and the ad merely stated the true fact that their milk didn't contain any bleach. Of course no one's milk (or whatever it was) had any bleach (or whatever it was). Specially highlighting something true, but out of context and with no equally special justification, is not an innocent act and is misleading. And yes, absolutely, very clearly, causes harm, and does so unfairly. A not-unfair version of this same article would just talk about databases, and include sqlite with others, and not only sqlite and not be titled sqlite. And then there is this: "Hey there I am v. I work at Turso Database."
- llm_nerd 2y ago
  Your milk/bread comparison is specious and invalid. I might say "As an apple enthusiast -- the fruit kind -- I want you to know that apple seeds have arsenic so don't eat lots of them" without having to disclaim every other fruit existing and possibly toxins found in parts of them. If some true-believer apple fan felt victimized, well that would be bizarre, right? This is a SQLite guy talking about SQLite to SQLite users. They're describing a feature/possible downside of SQLite that users might want to be aware of. They don't need "balanced" coverage of every other DB because it hurts someone's bizarrely fragile feelings. And as I mentioned elsewhere, almost all "enterprise" databases do do checksums by default, if people really want to lean on this "no one does! Leave SQLite alone" argument. And Turso is literally a SQLite-based firm. This isn't the aha you think it is.
- naveen99 2y ago
  Kind of like fake praise / avoiding constructive criticism in the interest of harmony and feelings.
- derekperkins 2y ago
  A frozen banana that WON'T make you sick and kill you. https://x.com/bluthquotes/status/732749820348096512 https://x.com/bluthquotes/status/732749820348096512
roncesvalles 2y ago
And that, kids, is why companies pay money to Oracle, Microsoft, and IBM.
- SubjectToChange 2y ago
  Major corporations are not paying exorbitant licensing fees to have checksumming enabled by default. In fact, for enterprises running things like vSAN, ECC DRAM, etc, database checksumming is probably nothing more than additional overhead. Database defaults in general are a touchy topic. Whatever set of defaults are chosen will be suboptimal for almost any serious user. A far more serious issue is figuring out the actual behavior of a database in different configurations. For instance, Oracle's SERIALIZABLE transaction isolation level only offers snapshoot isolation.
  - jandrewrogers 2y ago
    You don’t understand the critical problem that checksums solve at the I/O boundary. PCIe has weak error detection and correction. To transfer your data from ECC memory to your favorite super-robust storage technology requires transiting the PCIe bus, where for a brief moment it becomes relatively easy to corrupt data without anyone noticing. This is the problem that can’t be solved any other way and why checksums are primarily done at the I/O boundary in databases. It is an issue seen in real systems. PCIe v6 is intended to materially improve the integrity of data transfers but what we are using today is much worse.

anarazel 2y ago

FWIW, PG's default has been flipped to checksums being enabled by default in the in-development version. Personally I'm not sure it's the right call, as it comes with some increased write overhead.

aflukasz 2y ago
Yes, it's a balancing act. I think it's better to have safer defaults here, not prioritizing extra performance. People who are concerned with performance to the degree where checksums would matter need to consult configuration anyway (in many aspects, not only this one) and can disable this specific thing easily. For the reference, change is in https://github.com/postgres/postgres/commit/04bec894a04cb0d32533f1522ab81b7016141ff1 https://github.com/postgres/postgres/commit/04bec894a04cb0d3... (so should land in PostgreSQL 18 in a year, unless it will get reverted for some reason).

avinassh 2y ago

> I feel it's quite unfair for the title to call out sqlite on checksums, when in reality, as the very closing line of the article states most databases don't do checksum verifications. Hi, author of the post here. I work mostly with SQLite and that's the database I'm most familiar with, hence mentioning it in the title. I have also noted at the bottom: > Again, this is not a bug. Most databases (except a few) assume that the OS, filesystem, and disk are sound. Whether this matters depends on your application and the guarantees you need. now I wrote a follow up post - https://avi.im/blag/2024/databases-checksum https://avi.im/blag/2024/databases-checksum

refulgentis 2y ago

Every crumb of information you encounter doesn't have to be subjected to a lens of "but what if the entities involved were sports teams? Would this be "fair" to "them"?"