This is making some very kind assumptions about basic administrative staff. For people that work in our industry, a confidence/accuracy score is something that

This is making some very kind assumptions about basic administrative staff. For people that work in our industry, a confidence/accuracy score is something that's usually well understood, and is generally treated as an assessment of a single item in context with the whole. Once you get to the general public it's another matter entirely; for instance, look at how 538 was pilloried for their "30% chance to win the presidency for the current head of state," and how many times they've put out detailed information on how 30% =/= 0%. Your general person running review of the data (not the engineer hiding in the back office tuning the results, who should have access to the information) is going to treat most things above 50% as inherently true and anything below 50% as inherently false (within some margin of error, i.e. 50% +/- 5 points). It would be better to go the 'dumb' machine route of setting up some standard escalation flags; which looks like it may be what happened with this (albeit in a bad way), "person self filing and < 18 years of age; raise flag." The system being bad at differentiating 1919 & 2019 and then sending out an automated response seems more like a bad UX problem at this juncture, since it's something that could have been avoided with a quiet internal flagging instead of a loud external one.