agreed, but now it's going to happen at scale. eg. in australia, they launched the derided 'robo-debt' system that sent people notices about potential benefit

agreed, but now it's going to happen at scale. eg. in australia, they launched the derided 'robo-debt' system that sent people notices about potential benefit fraud charges, and the harassment of people caused senate investigations in several states. turns out the new system is as correct/wrong as the manual process (20% error rate), but it's now able to process several order magnitude more cases, so effectively floods downstream workflows (claims, complaints, courts) that were not ready for such an increase in demand.