Code-wise: Delayed::Job https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job Design-wise: Two queue worker processes ind

Code-wise: Delayed::Job https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job Design-wise: Two queue worker processes independent of web server processes. Delayed::Job lets you assign tasks a priority and lets you assign queue workers to priority levels it is allowed to look at. By convention, priority 0 (highest) is interactive tasks in my applications (a user is at their keyboard waiting for an answer) and priority 10 (least priority) is, well, Mixpanel. Not because I don't love them, just because if Mixpanel blocks for an entire month my rent still gets paid and that isn't true of any other priority level. Queue worker A only works on priorities 0 through 9. Queue worker B works on 0 through 10. This ensures that even if Mixpanel (or Kissmetrics, also on 10) perpetually times out the higher-priority levels will never be totally blocked. Delayed::Job worker processes are monitored by god, which resets them if they fail, become bloated, etc. They're also separately monitored by Scout monitoring's DJ plugin, which fires a yellow alert if a job is ever older than an hour (shouldn't happen but can in event of e.g. an API outage) or a red alert if there's ever more than X jobs in the queue (high probability that this represents a crash god can't recover from). Because even that setup was letting through about one outage every six months to a year, I have one other ace-in-the-hole: my ajax polling actions which check whether particular jobs are complete will, if they fail Y times in a row, a) instantiate a queue worker within the web server process (this degrades request processing but can't totally break the site) and b) fire off an independent-from-everything-else phone call to my cell phone saying "Your first, second, and third line of defense have failed. Get ye to an SSH terminal."