Ya, that was my thinking. And likewise with the sort of hybrid, constrained neural net setup we're discussing here, you could likewise 'discuss' the constraint

Ya, that was my thinking. And likewise with the sort of hybrid, constrained neural net setup we're discussing here, you could likewise 'discuss' the constraints, inputs, perhaps even the thought process to some extent. But like a human, it couldn't tell you the exact causal path taken to arrive at the decision.