i developed a system like this (with reconciliation loop, as you call it) some years ago. there is most definitely failed state (for multiple reasons). but as p

i developed a system like this (with reconciliation loop, as you call it) some years ago. there is most definitely failed state (for multiple reasons). but as part of "loop" you can have logic to fix it up in order to bring it to desired state. we had integrated monitoring/log analysis to correlate failures with "things that happen"