2 ms·
This reminds me of when I was a staff Consultant at an RDBMS company 25 years ago. A major telecom company was getting crashes in the server because of alleged
by DrScump 11y ago
This reminds me of when I was a staff Consultant at an RDBMS company 25 years ago.
A major telecom company was getting crashes in the server because of alleged I/O errors (mostly writes) in the raw i/o version of the server (no filesystem)... just a few times per several hundred thousand (or million) transactions. Subsequent disk and controller diagnostics were always fine. (This was before any fault tolerance was programmed into the server... my code for this problem was the first ever.) Most failures were writes, a few were reads. No obvious pattern (time of day, load, transaction type, lunar phase, sunspots, strength of coffee, etc.) This was on a DEC with Unix System V and DEC disk hardware, so only one vendor to deal with.
Anyway, seeing nothing "wrong" with the related code (other than no tolerance for error codes from the raw i/o calls), I theorized that maybe the "errors" were spurious and no actual fault or corruption was resulting, so I wrapped the i/o calls in retry loops (with the number of retry attempts tunable by the user) and logged any "failures" and results of the retry attempts. So, I did a build with my changes, had the customer run from my directory, then wait and watch for the carnage...
Turns out that every retry was successful. In fact, all but one of the dozen or two per day was successful on first try, and none needed more than two. No actual flaw in data was ever found.
Anyway, it turned out to be some spurious error specific to a specific drive type with that specific controller running that specific firmware... and apparently their filesystems code knew to work around it.
Client site personnel were really nice to me, too.