This looks like sloppy ML. But human judges do all those things already (susbtitute "just applying the law" for "doing math" in 4) and they can't be inspected -

This looks like sloppy ML. But human judges do all those things already (susbtitute "just applying the law" for "doing math" in 4) and they can't be inspected - their brains are "closed source". Sure, these humans can come up with wordy justifications for their decisions. But there are plenty of intelligent people who employ their intelligence not to arrive at a conclusion, but to justify the conclusion they already arrived at. Legal professionals aren't merely capable of this, they're explicitly trained to be experts at post-hoc justification. And legal professionals basically ignore all criticism not coming from their own professional class. They are rarely taught any kind of statistics in law school. Nobody wants to discuss math in debate club - answers with hard right and wrong answers are no fun to them. Your pessimism wrt. modeling may be justified, but you're not nearly pessimistic enough about people or the legal system.