This sort-of feedbacks on itself. I use estimators, so type errors occur far into a run, but not during training itself, and checkpoints mean very rapid recover

This sort-of feedbacks on itself. I use estimators, so type errors occur far into a run, but not during training itself, and checkpoints mean very rapid recovery, so ... I think I actually stopped caring so much because the recovery is so fast. Big help for me was coding a little script that means I can just send a notification to my android from from cli and/or python, so I know when these things need attention.