4 ms·
Weekly scrub errors are definitely not normal. There has been a few bug fixes in Ceph & the Kernel. I would check how up to date your packages are. Hardware er
by lathiat 4y ago
Weekly scrub errors are definitely not normal. There has been a few bug fixes in Ceph & the Kernel. I would check how up to date your packages are.
Hardware errors are also possible but there has been a few software bugs so worth checking.
Here's a really curly one we recently found and solved at work (only in March) that causes both scrub errors and sometimes bluefs aborts, it's a kernel patch. Likely to happen under memory pressure:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6c8e2a256915a223f6289f651d6b926cd7135c9e https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
https://tracker.ceph.com/issues/22464#note-50 https://tracker.ceph.com/issues/22464#note-50
But there have been various others.
- antongribok 4y agoI've not had a single PG inconsistency on my RPi cluster in the one year it's been running. Weekly errors are at work. Most of these are for actual disk media failures. For 100,000 disks what is a normal failure rate in your opinion?
- lathiat 4y ago@mike_d you are right I made a very bad assumption about the size of the environment :) @antongribok sounds like you have it in hand having seen your subsequent comments just didn’t want to leave someone assuming that error rate was normal in the smaller clusters that are more commonly but not universally the case. Evidently not the case for you :)
- mike_d 4y ago> Weekly scrub errors are definitely not normal That depends on the scale of your cluster. The Backblaze drive report shows they lose 1.3% of their disks per year on average. Larger operations will have one or more people who just replace disks all day.
- deleted 4y ago[deleted]