For situations like this, it's extremely useful to have a human-review queue that the automated system can drop jobs into. It keeps the rest of the system runn

For situations like this, it's extremely useful to have a human-review queue that the automated system can drop jobs into. It keeps the rest of the system running, and lets the support staff (you) fix the problem during normal business hours. This is how we handled errors in our video archiving system at Justin.tv, and to my knowledge we never lost a single frame that made it to the broadcast servers. The raw bits were streamed to disk as they came in, and only got removed once the VOD servers had the final version— any errors would retry a few times and then get flagged for manual processing. We did have a few close calls where something broke the whole archiving system, and the broadcast servers came dangerously close to shutting down due to full disks, though.