7 ms·

In another life (2007) I worked for a company that does digital signage. We had about 8000 systems deployed across the US. The software on them relied heavily o

In another life (2007) I worked for a company that does digital signage. We had about 8000 systems deployed across the US. The software on them relied heavily on SQLite. At some point we started seeing random corrupted SQLite databases. Trying to figure out what was causing this we realized our pc manufacturer was using the same sku but hardware inside could vary... and in particular the hard drive. Every system with a corrupt DB had a particular brand hard drive (WD), the ones that didn't fail was another (Seagate). We never really spent much more time on this other than to lock down the hardware config and replace all the hard drives in the failed systems with the "good" brand. Super interesting to see this. I'm assuming it was: Disk drives that do not honor sync requests We were of course using consumer grade equipment, because the company was pretty cheap, but also we had so many to deploy. I wonder how much this cost the company to fully resolve. Between systems not playing, having to send technicians, shipping systems overnight, time troubleshooting, etc.

FpUser 7y ago
In my other life (2000-2008 ish I think) I've also dealt with the large (tens of thousands) deployments all consumer grade as in your case. No SQLite though. Obviously we've had some hard drive failures. What is very interesting is that the results were exactly opposite. WD drives lived and Seagate drives failed. I am wondering if such discrepancy was caused by the usage patter.
- winrid 7y ago
  The drives weren't actually failing, the setup failed due to how the WD drives work. If you reformatted them they'd be fine.
  - cpuguy83 7y ago
    Yep, exactly this. The drives were fine, only the database was getting corrupted.
  - FpUser 7y ago
    Oops, sorry was not careful reading your message
pilsetnieks 7y ago
With how widespread both SQLite and WD are wouldn't half the world have apps crashing left and right, if just the hard drive brand was the cause of the problem?
- tyingq 7y ago
  It's somewhat implied it was one specific WD consumer drive model.
  - mirimir 7y ago
    I used WD RE3s back then, and never had problems.
- inetknght 7y ago
  See also: counterfeit SD cards.
- jmull 7y ago
  Yes, I think it's possible (likely, even) that there was at least one more factor.
- adrianN 7y ago
  Apps are in fact crashing left and right.
  - brewdad 7y ago
    My Materialistic app crashed about 30 seconds prior to opening this thread. I probably average about 5 "noisy" app crashes a week on my phone. No idea how many may be failing silently.
    - ShamelessC 7y ago
      I had to give up on Materialistic because of the frequency of crashes. It's a shame because the hn website is really difficult to navigate on mobile (collapsing threads with that tiny + is a nightmare). Do you know of any better alternatives that let you log in and post from the app?
wongarsu 7y ago
> because the company was pretty cheap, but also we had so many to deploy Isn't "deploy many systems in hard to reach places" the prime target for enterprise-grade hardware? Fixing problems on one bespoke system is a lot easier than fixing the same problem 8000 times, and of course with a large sample size the number of problems experienced goes up.
- michaelt 7y ago
  I always thought "people willing to pay five to ten times market rate per gigabyte" was the target of enterprise-grade storage...
- cpuguy83 7y ago
  Yep. This was a big problem there. Before they were delivering content to PC's they distributed content via DVD once per month. DVD player failure rate was pretty massive, especially since they just started recycling refurbed players.
magila 7y ago
Contrary to popular belief, even consumer drives will honor flush commands. The problem is operating systems generally do not send flush commands, even when you might expect it (e.g. last I checked fsync() on Linux does not send a flush). Disabling write caching resolves these problems. Unfortunately SATA drives typically ship with write caching enabled by default.
- zaarn 7y ago
  Some cheaper HDDs and third-rate drives won't honor flushes. Same for SSDs, a few very cheap ones ignore flushes in favor of cache performance.
- networkimprov 7y ago
  References?
  - aidos 7y ago
    Not quite the same thing but it turns out there are some other gotchas with fsync on Linux https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/
- tobias3 7y ago
  Of course fsync() sends a flush to the disk (unless you use non-standard mount options or "eatmydata"). They may not honor the FUA bit, though, but hopefully nobody relies on this if they expect it to work with consumer disks (See e.g. here https://devblogs.microsoft.com/oldnewthing/20170510-00/?p=95505 https://devblogs.microsoft.com/oldnewthing/20170510-00/?p=95... ). What I have heard is that some disks have problems if they are on a error recovery path. E.g. they have a corrupt sector or a sector that needs reallocation (pending sector), and then they have a power failure while the sector is reallocated.
- tinus_hn 7y ago
  The implementation of fsync is FS dependent. On the default ext4 filesystem with the default options (barrier) fsync does send flush.
da_chicken 7y ago
Reminds me of the Maxtor DiamondMax Plus 8 disk firmware problem. This was a model of hard drive back in 2004-2005 that had a known issue where it would corrupt it's own firmware. The disk would literally overwrite and corrupt portions of firmware code which would render the disk unusable. Heat seemed to exacerbate the problem, and the problem caused the disks to generate a lot of heat so it was self-perpetuating. I worked at a hospital at the time, and we were upgrading from Win2k to WinXP, and replacing a lot of old HP Celeron 733 MHz systems with newer HP dc5000 AMD desktops. We deployed roughly 1,000 desktops over the course of a year and a half, and fully 25% of the systems that had a DiamondMax Plus 8 had disk failures within the first 6 months of deployment. Symptoms were always the same. System would get slower and slower until one day it would hard reboot itself and then claim there was no disk drive installed. BIOS would show the disk drive was present but that it had no capacity or name. Built-in BIOS self test would report a code 7 error. It got bad enough that we refused to deploy systems that shipped with those drives. Our manager had to fight with the sales rep for about a month before they relented. We were ready to go to Dell just to get away from those stupid disks. You can still find disk recovery sites talking about the issue: https://vitaldata.ca/article/n40p-data-recovery.html https://vitaldata.ca/article/n40p-data-recovery.html
- philpem 7y ago
  Seagate had their problems too around that time. The Barracuda 7200.10 was infamous not just for its bug but also for Seagate's response. From what I recall, the SMART log wrapped around periodically - if the drive was reset when the pointer was at a certain value, it failed to boot with an assertion failure. 1 in 128 chance of a failure on a given power-up. Thankfully someone found out that you could send a couple of debug commands to it with an FTDI cable and get your data back.
numpad0 7y ago
WD: this comment Seagate: ST3000DM001 Another reason to stick to Toshiba or ex-HGST(if and when available)

FpUser 7y ago

In my other life (2000-2008 ish I think) I've also dealt with the large (tens of thousands) deployments all consumer grade as in your case. No SQLite though. Obviously we've had some hard drive failures. What is very interesting is that the results were exactly opposite. WD drives lived and Seagate drives failed. I am wondering if such discrepancy was caused by the usage patter.

winrid 7y ago
The drives weren't actually failing, the setup failed due to how the WD drives work. If you reformatted them they'd be fine.
- cpuguy83 7y ago
  Yep, exactly this. The drives were fine, only the database was getting corrupted.
- FpUser 7y ago
  Oops, sorry was not careful reading your message

pilsetnieks 7y ago

With how widespread both SQLite and WD are wouldn't half the world have apps crashing left and right, if just the hard drive brand was the cause of the problem?

tyingq 7y ago
It's somewhat implied it was one specific WD consumer drive model.
- mirimir 7y ago
  I used WD RE3s back then, and never had problems.
inetknght 7y ago
See also: counterfeit SD cards.
jmull 7y ago
Yes, I think it's possible (likely, even) that there was at least one more factor.
adrianN 7y ago
Apps are in fact crashing left and right.
- brewdad 7y ago
  My Materialistic app crashed about 30 seconds prior to opening this thread. I probably average about 5 "noisy" app crashes a week on my phone. No idea how many may be failing silently.
  - ShamelessC 7y ago
    I had to give up on Materialistic because of the frequency of crashes. It's a shame because the hn website is really difficult to navigate on mobile (collapsing threads with that tiny + is a nightmare). Do you know of any better alternatives that let you log in and post from the app?

wongarsu 7y ago

> because the company was pretty cheap, but also we had so many to deploy Isn't "deploy many systems in hard to reach places" the prime target for enterprise-grade hardware? Fixing problems on one bespoke system is a lot easier than fixing the same problem 8000 times, and of course with a large sample size the number of problems experienced goes up.

michaelt 7y ago
I always thought "people willing to pay five to ten times market rate per gigabyte" was the target of enterprise-grade storage...
cpuguy83 7y ago
Yep. This was a big problem there. Before they were delivering content to PC's they distributed content via DVD once per month. DVD player failure rate was pretty massive, especially since they just started recycling refurbed players.

magila 7y ago

Contrary to popular belief, even consumer drives will honor flush commands. The problem is operating systems generally do not send flush commands, even when you might expect it (e.g. last I checked fsync() on Linux does not send a flush). Disabling write caching resolves these problems. Unfortunately SATA drives typically ship with write caching enabled by default.

zaarn 7y ago
Some cheaper HDDs and third-rate drives won't honor flushes. Same for SSDs, a few very cheap ones ignore flushes in favor of cache performance.
networkimprov 7y ago
References?
- aidos 7y ago
  Not quite the same thing but it turns out there are some other gotchas with fsync on Linux https://lwn.net/Articles/752063/ https://lwn.net/Articles/752063/
tobias3 7y ago
Of course fsync() sends a flush to the disk (unless you use non-standard mount options or "eatmydata"). They may not honor the FUA bit, though, but hopefully nobody relies on this if they expect it to work with consumer disks (See e.g. here https://devblogs.microsoft.com/oldnewthing/20170510-00/?p=95505 https://devblogs.microsoft.com/oldnewthing/20170510-00/?p=95... ). What I have heard is that some disks have problems if they are on a error recovery path. E.g. they have a corrupt sector or a sector that needs reallocation (pending sector), and then they have a power failure while the sector is reallocated.
tinus_hn 7y ago
The implementation of fsync is FS dependent. On the default ext4 filesystem with the default options (barrier) fsync does send flush.

da_chicken 7y ago

Reminds me of the Maxtor DiamondMax Plus 8 disk firmware problem. This was a model of hard drive back in 2004-2005 that had a known issue where it would corrupt it's own firmware. The disk would literally overwrite and corrupt portions of firmware code which would render the disk unusable. Heat seemed to exacerbate the problem, and the problem caused the disks to generate a lot of heat so it was self-perpetuating. I worked at a hospital at the time, and we were upgrading from Win2k to WinXP, and replacing a lot of old HP Celeron 733 MHz systems with newer HP dc5000 AMD desktops. We deployed roughly 1,000 desktops over the course of a year and a half, and fully 25% of the systems that had a DiamondMax Plus 8 had disk failures within the first 6 months of deployment. Symptoms were always the same. System would get slower and slower until one day it would hard reboot itself and then claim there was no disk drive installed. BIOS would show the disk drive was present but that it had no capacity or name. Built-in BIOS self test would report a code 7 error. It got bad enough that we refused to deploy systems that shipped with those drives. Our manager had to fight with the sales rep for about a month before they relented. We were ready to go to Dell just to get away from those stupid disks. You can still find disk recovery sites talking about the issue: https://vitaldata.ca/article/n40p-data-recovery.html https://vitaldata.ca/article/n40p-data-recovery.html

philpem 7y ago
Seagate had their problems too around that time. The Barracuda 7200.10 was infamous not just for its bug but also for Seagate's response. From what I recall, the SMART log wrapped around periodically - if the drive was reset when the pointer was at a certain value, it failed to boot with an assertion failure. 1 in 128 chance of a failure on a given power-up. Thankfully someone found out that you could send a couple of debug commands to it with an FTDI cable and get your data back.

numpad0 7y ago

WD: this comment Seagate: ST3000DM001 Another reason to stick to Toshiba or ex-HGST(if and when available)