4 ms·

As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should

by patio11 14y ago

As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should not block the application. Think very, very carefully before you ever block the request/response cycle on an external API. (I'd say "Never do it" but I think I could come up with conceivable apps and APIs where that makes sense if I was fully awake.) Since analytics callbacks don't generate immediate customer value and can fail totally without discomfitting anyone, you should never block a request for them. Instead, you deal with them asynchronously. There's a few ways to do it mechanically. I offload them to my job queue and set the priority to "lowest possible." When the job queue is otherwise empty, a worker process (that no actual human is waiting on) slurps up a few of the events and fires them off. Only downside, which I never bothered fixing: I have an "OMG the queue is stuffed to overflowing... the queue worker must be non-responsive!" monitoring test which, since that symptom has featured in most of my customer-visible downtime, generates a red alert. (i.e. Immediate SMS followed by phone call escalation, as opposed to an "FYI check this" email.) Any downtime at Mixpanel or KissMetrics lasting longer than a few minutes reliably triggers this alert. On the plus side, this means that when I tell you "Mixpanel really doesn't fail at 2 AM Japan time all that often" you should trust that I'd have noticed.

iheartmemcache 14y ago
I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.
- jhuckestein 14y ago
  I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
- patio11 14y ago
  Code-wise: Delayed::Job https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job Design-wise: Two queue worker processes independent of web server processes. Delayed::Job lets you assign tasks a priority and lets you assign queue workers to priority levels it is allowed to look at. By convention, priority 0 (highest) is interactive tasks in my applications (a user is at their keyboard waiting for an answer) and priority 10 (least priority) is, well, Mixpanel. Not because I don't love them, just because if Mixpanel blocks for an entire month my rent still gets paid and that isn't true of any other priority level. Queue worker A only works on priorities 0 through 9. Queue worker B works on 0 through 10. This ensures that even if Mixpanel (or Kissmetrics, also on 10) perpetually times out the higher-priority levels will never be totally blocked. Delayed::Job worker processes are monitored by god, which resets them if they fail, become bloated, etc. They're also separately monitored by Scout monitoring's DJ plugin, which fires a yellow alert if a job is ever older than an hour (shouldn't happen but can in event of e.g. an API outage) or a red alert if there's ever more than X jobs in the queue (high probability that this represents a crash god can't recover from). Because even that setup was letting through about one outage every six months to a year, I have one other ace-in-the-hole: my ajax polling actions which check whether particular jobs are complete will, if they fail Y times in a row, a) instantiate a queue worker within the web server process (this degrades request processing but can't totally break the site) and b) fire off an independent-from-everything-else phone call to my cell phone saying "Your first, second, and third line of defense have failed. Get ye to an SSH terminal."

by patio11 14y ago

iheartmemcache 14y ago

I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.

jhuckestein 14y ago
I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
patio11 14y ago
Code-wise: Delayed::Job https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job Design-wise: Two queue worker processes independent of web server processes. Delayed::Job lets you assign tasks a priority and lets you assign queue workers to priority levels it is allowed to look at. By convention, priority 0 (highest) is interactive tasks in my applications (a user is at their keyboard waiting for an answer) and priority 10 (least priority) is, well, Mixpanel. Not because I don't love them, just because if Mixpanel blocks for an entire month my rent still gets paid and that isn't true of any other priority level. Queue worker A only works on priorities 0 through 9. Queue worker B works on 0 through 10. This ensures that even if Mixpanel (or Kissmetrics, also on 10) perpetually times out the higher-priority levels will never be totally blocked. Delayed::Job worker processes are monitored by god, which resets them if they fail, become bloated, etc. They're also separately monitored by Scout monitoring's DJ plugin, which fires a yellow alert if a job is ever older than an hour (shouldn't happen but can in event of e.g. an API outage) or a red alert if there's ever more than X jobs in the queue (high probability that this represents a crash god can't recover from). Because even that setup was letting through about one outage every six months to a year, I have one other ace-in-the-hole: my ajax polling actions which check whether particular jobs are complete will, if they fail Y times in a row, a) instantiate a queue worker within the web server process (this degrades request processing but can't totally break the site) and b) fire off an independent-from-everything-else phone call to my cell phone saying "Your first, second, and third line of defense have failed. Get ye to an SSH terminal."