4 ms·

This comment is currently fading into downvotitude. Can someone explain why? Is it factually incorrect? Are people upset at the implication that other systems d

This comment is currently fading into downvotitude. Can someone explain why? Is it factually incorrect? Are people upset at the implication that other systems do lie about data being on stable storage when it isn’t? I haven’t spent much time with Illumos, but if Postgres is not impacted on Illumos, then why the downvotes?

tialaramex 8y ago
Ultimately the Operating System doesn't (and can't) know whether the storage is actually going to retain data you've written to it. Even if your OS goes out of its way to use all six kinds of double secret synchronization primitive, the cheap eBay clone product didn't bother implementing any of them and replies "Yup, definitely 100% synchronised" while doing nothing, then the user is disappointed because your OS promised that the data was safe and it isn't. This isn't really just about hard disks either. Over the years lots of manufacturers have sold "smart" network cards that corrupt data if they're left to their own devices to "speed up" networking by cutting corners, miscalculating checksums, copying things wrongly from RAM, and so on. But it's even possible to build a card where the most trivial scenario - where the OS lays out the entire packet and then explicitly does IO operations to write it to the card - still sometimes randomly corrupts data. Your OS can promise a "guarantee" that the network data is exactly what you sent... and then you find 0.1% of packets are corrupted before they even reach the switch, oops. Ultimately all bets are off when running on real hardware. Many years ago a friend of mine had a PC where randomly sometimes basic Unix features like "cat" would segfault or misbehave. Did disk checks, re-installed the entire OS, loads of hair-pulling. Eventually he discovered that the CPU fan failed, and overheating had damaged the on-die L2 cache...
- mmt 8y ago
  To be fair, no OS has control over the hardware, only how it interacts with it. The cheap eBay clone and random PC fan failure are vanishingly rare for a datacenter database environment. Still, the claim that merely selecting a an OS that may not suffer from this one particular issue results in a "bulletproof" setup is rather extraordinary. There may be other issues, some as a result of that OS choice (even if hardware issues are not). Additionally, the choice of a much less popular OS can, in and of itself, be a significant "bullet"-susceptibility factor, not just in terms of general supportability but, getting back to your point about hardware, how well one might expect it to behave with what's out there on the market.
- Annatar 8y ago
  “Ultimately the Operating System doesn't (and can't) know whether the storage is actually going to retain data you've written to it.” illumos and ZFS do know about it, especially since ZFS will checksum every block and will perform an explicit flush for every write. Should the drive lie about it, the checksum won’t match. To make the claims you made just because what I wrote doesn’t involve Linux is preposterous. Sun engineers spent 25 years making the OS robust and able to correctly cope with failing, flaky or otherwise unreliable hardware. Ever heard of fault management architecuture in Solaris? If you had, you wouldn’t have written what you wrote.
  - tialaramex 8y ago
    "illumos and ZFS do know about it" They literally can't, as I explained previously. Everything seems fine with such hardware, up until you find that some of the data you were absolutely sure you'd written isn't there after all. The fact you can't conceive of how this goes wrong is actually _illustrative_ of how Sun got screwed. The foolish man certainly does build his house on sand. Now, twenty five years ago when those Sun engineers apparently started making the OS robust (maybe this explains how it got the reputation for being a flaky piece of crap, they should have started sooner...) I was working for a company that I'm quite _sure_ a person who says they really care about robust systems has heard of. Tandem Computers. Tandem had a demo that gets to the heart of the thing you think you really care about in Illumos. Do a transaction, it's committed, and then we... pull out the CPU cards from the computer. It's dead. ... and our transaction is safe and indeed other users can continue to use the system, with the committed transaction, because it was replicated to other nodes far away and those weren't destroyed. Tandem Computers doesn't exist any more because it turns out that doing this correctly is expensive and nobody wants to pay extra for correctness. After all, Sun engineers said their OS is "able to cope with failing, flaky or otherwise unreliable hardware" and with the lower price tag a customer really wants to believe that's true, shame about their data when it isn't...
    - Annatar 8y ago
      “The fact you can't conceive of how this goes wrong is actually _illustrative_ of how Sun got screwed.“ I can very well concieve of it, as I’ve been preaching this to the Linux crowd for decades and they just don’t get it. By the way I know about Tandem because yes, I’m that old. Please, before you go preaching to the choir (me), read how ZFS works; if the drive totally completely lies about everything, if it silently corrupts data, first read from the drive will uncover it because the checksums won’t match. ZFS will still deliver good data if it has redundant copies and will attempt to overwrite the corrupt data automatically. 50% of the two memory chips in my Solaris storage server no longer function correctly; Solaris has automatically migrated the affected pages out of them. When I get new memory chips, I will off-line them and replace them. Tandem might have been even more robust, but its engineers didn’t continue working on it after the company’s demise like those Sun engineers did on illumos.
- Annatar 8y ago
  https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integrity https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integri... working with the said technology and understanding it goes a long way.

tialaramex 8y ago

Ultimately the Operating System doesn't (and can't) know whether the storage is actually going to retain data you've written to it. Even if your OS goes out of its way to use all six kinds of double secret synchronization primitive, the cheap eBay clone product didn't bother implementing any of them and replies "Yup, definitely 100% synchronised" while doing nothing, then the user is disappointed because your OS promised that the data was safe and it isn't. This isn't really just about hard disks either. Over the years lots of manufacturers have sold "smart" network cards that corrupt data if they're left to their own devices to "speed up" networking by cutting corners, miscalculating checksums, copying things wrongly from RAM, and so on. But it's even possible to build a card where the most trivial scenario - where the OS lays out the entire packet and then explicitly does IO operations to write it to the card - still sometimes randomly corrupts data. Your OS can promise a "guarantee" that the network data is exactly what you sent... and then you find 0.1% of packets are corrupted before they even reach the switch, oops. Ultimately all bets are off when running on real hardware. Many years ago a friend of mine had a PC where randomly sometimes basic Unix features like "cat" would segfault or misbehave. Did disk checks, re-installed the entire OS, loads of hair-pulling. Eventually he discovered that the CPU fan failed, and overheating had damaged the on-die L2 cache...

mmt 8y ago
To be fair, no OS has control over the hardware, only how it interacts with it. The cheap eBay clone and random PC fan failure are vanishingly rare for a datacenter database environment. Still, the claim that merely selecting a an OS that may not suffer from this one particular issue results in a "bulletproof" setup is rather extraordinary. There may be other issues, some as a result of that OS choice (even if hardware issues are not). Additionally, the choice of a much less popular OS can, in and of itself, be a significant "bullet"-susceptibility factor, not just in terms of general supportability but, getting back to your point about hardware, how well one might expect it to behave with what's out there on the market.
Annatar 8y ago
“Ultimately the Operating System doesn't (and can't) know whether the storage is actually going to retain data you've written to it.” illumos and ZFS do know about it, especially since ZFS will checksum every block and will perform an explicit flush for every write. Should the drive lie about it, the checksum won’t match. To make the claims you made just because what I wrote doesn’t involve Linux is preposterous. Sun engineers spent 25 years making the OS robust and able to correctly cope with failing, flaky or otherwise unreliable hardware. Ever heard of fault management architecuture in Solaris? If you had, you wouldn’t have written what you wrote.
- tialaramex 8y ago
  "illumos and ZFS do know about it" They literally can't, as I explained previously. Everything seems fine with such hardware, up until you find that some of the data you were absolutely sure you'd written isn't there after all. The fact you can't conceive of how this goes wrong is actually _illustrative_ of how Sun got screwed. The foolish man certainly does build his house on sand. Now, twenty five years ago when those Sun engineers apparently started making the OS robust (maybe this explains how it got the reputation for being a flaky piece of crap, they should have started sooner...) I was working for a company that I'm quite _sure_ a person who says they really care about robust systems has heard of. Tandem Computers. Tandem had a demo that gets to the heart of the thing you think you really care about in Illumos. Do a transaction, it's committed, and then we... pull out the CPU cards from the computer. It's dead. ... and our transaction is safe and indeed other users can continue to use the system, with the committed transaction, because it was replicated to other nodes far away and those weren't destroyed. Tandem Computers doesn't exist any more because it turns out that doing this correctly is expensive and nobody wants to pay extra for correctness. After all, Sun engineers said their OS is "able to cope with failing, flaky or otherwise unreliable hardware" and with the lower price tag a customer really wants to believe that's true, shame about their data when it isn't...
  - Annatar 8y ago
    “The fact you can't conceive of how this goes wrong is actually _illustrative_ of how Sun got screwed.“ I can very well concieve of it, as I’ve been preaching this to the Linux crowd for decades and they just don’t get it. By the way I know about Tandem because yes, I’m that old. Please, before you go preaching to the choir (me), read how ZFS works; if the drive totally completely lies about everything, if it silently corrupts data, first read from the drive will uncover it because the checksums won’t match. ZFS will still deliver good data if it has redundant copies and will attempt to overwrite the corrupt data automatically. 50% of the two memory chips in my Solaris storage server no longer function correctly; Solaris has automatically migrated the affected pages out of them. When I get new memory chips, I will off-line them and replace them. Tandem might have been even more robust, but its engineers didn’t continue working on it after the company’s demise like those Sun engineers did on illumos.
Annatar 8y ago
https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integrity https://blogs.oracle.com/bonwick/zfs-end-to-end-data-integri... working with the said technology and understanding it goes a long way.