2 ms·
> I went about 30d with a DEGRADED disk before I realized it, turns out it was a bad cable but if another disk failed I would have been hosed. Skill issue. You
by znpy 3y ago
> I went about 30d with a DEGRADED disk before I realized it, turns out it was a bad cable but if another disk failed I would have been hosed.
Skill issue. You lack monitoring. The basic prometheus node exporter and alertmanager would be sufficient:
# curl -s http://localhost:9100/metrics | grep zpool_state
node_zfs_zpool_state{state="degraded",zpool="storage"} 0
node_zfs_zpool_state{state="faulted",zpool="storage"} 0
node_zfs_zpool_state{state="offline",zpool="storage"} 0
node_zfs_zpool_state{state="online",zpool="storage"} 1
node_zfs_zpool_state{state="removed",zpool="storage"} 0
node_zfs_zpool_state{state="unavail",zpool="storage"} 0
You can easily alert on such metrics.