2 ms·
Managers making tech decisions without the input of the people who work with systems every day who are in the best position to determine whether that's a good d
by Karunamon 9y ago
Managers making tech decisions without the input of the people who work with systems every day who are in the best position to determine whether that's a good decision or not.
True story:
A deployment of a distributed application at $dayjob a long while ago had "must have RAID6 or equivalent fault tolerance" added to its project requirements by clueless managers. The engineer who specced out the system, who was responsible for its POC, and had spent many, many hours setting up the environment and tweaking it for performance protested, since the application had fault tolerance built in. The idea being that if a node loses a hard drive, we don't care. We just throw a spare drive (that's kept in stock) in, rebuild the node (a literal 10 minute process), and no data is lost.
Protests, backed up with data, that long RAID6 rebuild times on TB+ drives increased the likelihood of data loss (a restripe takes longer than our average rebuild-the-damn-machine-from-scratch time and leaves the cluster degraded longer) went ignored.
The RAID6 requirement added about 50% to the cost of the project, and ended up having to be thrown out around ~6 month mark since the cluster write performance was unable to keep up with the RAID0 that the engineer had originally specced out. We now use the engineer's original idea, even though he long since moved somewhere where presumably, his experience is trusted rather than insulted. This incident was a contributor to his decision to leave.
To this day, RAID6 is an inside reference to management cluelessly screwing up a project against the wishes of people who know better.