3 ms·

> We could make some speculation about the disks possibly losing writes when Postgres flushed commits to disk, but we have no proof to determine what happened.

> We could make some speculation about the disks possibly losing writes when Postgres flushed commits to disk, but we have no proof to determine what happened. If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying.

thaumaturgy 16y ago
...I was wondering if I was going to be the only one that caught that. There's not enough information to know if that's even possible or not, or how it might be possible, but as far as I know there aren't any known postgres issues that would cause that. I would be extremely reluctant to store data on a network that sometimes didn't actually store the data without raising an error anywhere.
jrmg 16y ago
Most consumer /hard drives/ lie about the result of fsync, as a 'performance optimization'. It's generally possible to fsync, then cut the power before the data is physically on the disk.
- lsc 16y ago
  Yeah. but even I don't use consumer hard drives in production. (Honestly, I don't know 100% that the 'enterprise' drives are that much better, but I'd guess they'd lie less... I switched because consumer drives tend to hang RAIDs when they fail, while 'enterprise' stuff fails clean)
  - omh 16y ago
    Enterprise hard drives might lie about fsync with caching turned on, but they generally let you disable caching and behave correctly. Of course that completely shafts your performance, but hopefully you were expecting that.
  - wulczer 16y ago
    There's a section in the Postgres docs about fsync and lying hard drives: http://developer.postgresql.org/pgdocs/postgres/wal-reliability.html http://developer.postgresql.org/pgdocs/postgres/wal-reliabil... There's also a Postgres contrib module called pg_test_fsync that tests various fsync modes on your hard drive, and if you get too high results, you can strongly suspect that the dist is lying.
  - CrLf 16y ago
    Enterprise hardware lies about fsync, because even if you loose power, there is a little battery on the RAID controller that's enough to flush the cache to disk or to keep it for hours until the machine is powered up again. When the battery goes bad, the write cache is disabled automatically. On bigger hardware, like SAN storage arrays, the (redundant) batteries keep the whole thing running for a while after the loss of power.
    - omh 16y ago
      You could consider that the whole battery backup etc. means that it isn't lying about fsync. It says that it's been permanently written, and then it makes sure that is has. It might not have been burnt to spinning metal, but the system as a whole will ensure that it's permanent.
    - lsc 16y ago
      "enterprise" disk is just that; it's just disk. I've not seen a disk with an onboard battery. You can put a raid controller in front of your disks, with or without battery backed cache. (most raid cards have an optional battery module.) But, even many gold plated 'corporate' systems I've seen omit the battery, as it's usually not cheap. My personal opinion is that raid controllers are of little value without battery backed cache. I just use MD in front of 'enterprise sata' drives, as without doubling my total cost for storage, I can't get a raid card, as far as I can tell, which is better than MD, and really, if I was to double my storage cost, simply buying twice as many spindles would get me better bang for my buck, I think, than a hardware raid card that cost that much.
moe 16y ago
If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying. Jump to conclusions much? Do you know if reddit had fsync enabled in first place?
- khafra 16y ago
  If you read between the lines, "if you read between the lines" almost always means "if you make an educated jump to conclusions."
  - moe 16y ago
    So where is the education in jumping from "we have no proof to determine what happened" to an outrageous claim about EBS? It's very common to turn off fsync on a database for performance reasons. It's far less common to have a network block device driver (which tend to be designed with intermittent outages in mind) lie about fsync.
patrickgzill 16y ago
You can actually configure postgres to not do an fsync() after each transaction; instead it gets written to disk, which on Linux means written to the disk caches in RAM, then Linux flushes to disk when it does the rest of the disk writes (anywhere from 5 to 30 seconds are set as default on different versions of Linux).
- tedjdziuba 16y ago
  Why would you want to do that? This isn't MongoDB, the real world cares about durability.
  - patrickgzill 16y ago
    I am not sure why I was downvoted - I wasn't suggesting as a course of action... it gives a large speed increase and I was suggesting that possibly this was done as a performance measure.
_delirium 16y ago
A comment from an ex reddit employee linked elsewhere in this HN discussion (http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...) seems to confirm that: More recently we also discovered that these disks [EBS volumes experiencing performance degradation] will also frequently report that a disk transaction has been committed to hardware but are flat-out lying.

thaumaturgy 16y ago

...I was wondering if I was going to be the only one that caught that. There's not enough information to know if that's even possible or not, or how it might be possible, but as far as I know there aren't any known postgres issues that would cause that. I would be extremely reluctant to store data on a network that sometimes didn't actually store the data without raising an error anywhere.

jrmg 16y ago

Most consumer /hard drives/ lie about the result of fsync, as a 'performance optimization'. It's generally possible to fsync, then cut the power before the data is physically on the disk.

lsc 16y ago
Yeah. but even I don't use consumer hard drives in production. (Honestly, I don't know 100% that the 'enterprise' drives are that much better, but I'd guess they'd lie less... I switched because consumer drives tend to hang RAIDs when they fail, while 'enterprise' stuff fails clean)
- omh 16y ago
  Enterprise hard drives might lie about fsync with caching turned on, but they generally let you disable caching and behave correctly. Of course that completely shafts your performance, but hopefully you were expecting that.
- wulczer 16y ago
  There's a section in the Postgres docs about fsync and lying hard drives: http://developer.postgresql.org/pgdocs/postgres/wal-reliability.html http://developer.postgresql.org/pgdocs/postgres/wal-reliabil... There's also a Postgres contrib module called pg_test_fsync that tests various fsync modes on your hard drive, and if you get too high results, you can strongly suspect that the dist is lying.
- CrLf 16y ago
  Enterprise hardware lies about fsync, because even if you loose power, there is a little battery on the RAID controller that's enough to flush the cache to disk or to keep it for hours until the machine is powered up again. When the battery goes bad, the write cache is disabled automatically. On bigger hardware, like SAN storage arrays, the (redundant) batteries keep the whole thing running for a while after the loss of power.
  - omh 16y ago
    You could consider that the whole battery backup etc. means that it isn't lying about fsync. It says that it's been permanently written, and then it makes sure that is has. It might not have been burnt to spinning metal, but the system as a whole will ensure that it's permanent.
  - lsc 16y ago
    "enterprise" disk is just that; it's just disk. I've not seen a disk with an onboard battery. You can put a raid controller in front of your disks, with or without battery backed cache. (most raid cards have an optional battery module.) But, even many gold plated 'corporate' systems I've seen omit the battery, as it's usually not cheap. My personal opinion is that raid controllers are of little value without battery backed cache. I just use MD in front of 'enterprise sata' drives, as without doubling my total cost for storage, I can't get a raid card, as far as I can tell, which is better than MD, and really, if I was to double my storage cost, simply buying twice as many spindles would get me better bang for my buck, I think, than a hardware raid card that cost that much.

moe 16y ago

If you read between the lines, this says that EBS lies about the result of fsync(), which is horrifying. Jump to conclusions much? Do you know if reddit had fsync enabled in first place?

khafra 16y ago
If you read between the lines, "if you read between the lines" almost always means "if you make an educated jump to conclusions."
- moe 16y ago
  So where is the education in jumping from "we have no proof to determine what happened" to an outrageous claim about EBS? It's very common to turn off fsync on a database for performance reasons. It's far less common to have a network block device driver (which tend to be designed with intermittent outages in mind) lie about fsync.

patrickgzill 16y ago

You can actually configure postgres to not do an fsync() after each transaction; instead it gets written to disk, which on Linux means written to the disk caches in RAM, then Linux flushes to disk when it does the rest of the disk writes (anywhere from 5 to 30 seconds are set as default on different versions of Linux).

tedjdziuba 16y ago
Why would you want to do that? This isn't MongoDB, the real world cares about durability.
- patrickgzill 16y ago
  I am not sure why I was downvoted - I wasn't suggesting as a course of action... it gives a large speed increase and I was suggesting that possibly this was done as a performance measure.

_delirium 16y ago

A comment from an ex reddit employee linked elsewhere in this HN discussion (http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...) seems to confirm that: More recently we also discovered that these disks [EBS volumes experiencing performance degradation] will also frequently report that a disk transaction has been committed to hardware but are flat-out lying.