3 ms·

This is unfortunate. I was about to implement MixPanel on my startups site (it already laced in our development code), but their API being down completely locks

by mrchess 14y ago

This is unfortunate. I was about to implement MixPanel on my startups site (it already laced in our development code), but their API being down completely locks our application ... actually shipping MixPanel in production not looking so hot right now. EDIT: As others have voiced my issue was that it was taking up to 30 seconds for requests to timeout disrupting various things in my JS code (it is a Backbone app) -- it didn't actually "lock" it. I was using the JS API.

taf2 14y ago
shouldn't the API calls you make to a tracking service be done in such away that they would not effect your application whether they're available or not available? Analytics is nice, but not having it shouldn't cause your whole app to lock up right? Kind of similar to say a caching infrastructure - if memcached goes down your app just slows down, but doesn't crash right?
- blazingfrog2 14y ago
  Their documentation is either wrong or misleading: Basic Javascript integration We load the library onto the page asynchronously, which keeps your website loading quickly even if placed in the <head> of the page. In our case just commenting out the library load solved all our issues.
  - kcbanner 14y ago
    Yea, the problem is that making actual calls to their API servers is taking a while.
emmett 14y ago
Speaking as someone who uses a ton of mixpanel in production (http://www.twitch.tv/ http://www.twitch.tv/) the way you handle this is to make all your calls asynchronous. I am bummed that we are going to have missing data though :-(
- latchkey 14y ago
  This works fine for code where you are just firing a /track 'ping' back to the server on a page load. But, if you are trying to do things like track login's where you send a http redirect right after /track, then you need to use the optional callback parameter to /track and do the redirect in the callback. The problem with this is that if their JS doesn't load at all (which is the problem right now), then /track is never called and thus the callback is never fired. So because you can't depend on MP to reliably serve up a simple .js file, you have to write your own wrapper around all of that code to do a timeout incase the callback never fires. What a mess.
  - Skywing 14y ago
    I don't use Mixpanel frequently, but could you not just put the Javascript file somewhere more reliable? Why not pull their code down and put it on your own CDN? I've done this with Mixpanel in the past, I believe. This wouldn't solve their API endpoints not responding obviously, but it's an easy way to take the issue into your own hands for the time being. Also, I don't think I would ever make your core logic depend on events from something as dispensable as a third party analytics service. It just sounds like a poor design choice to have some core site functionality occur after receiving a response from something you do not control. I'd try to make those Mixpanel requests more "fire and forget", if you will. Edit in response to you response below: I think you're missing the point here. The scope of their documentation probably does not include best Javascript practices. You made some poor implementation decisions and now you're getting burned. At least now you have experience using third party services and probably won't be so trusting of them, again. Trivializing service stability and other things does not make your point any more credible, though. Personally, I think it makes people overlook your point because it makes it sound as if you have no idea what you're talking about.
    - latchkey 14y ago
      Yup, lesson learned. I definitely won't ever depend on Mixpanel again to do something as simple a reliably serve up a single .js file. I don't want to include their JS file because what happens if they make a change that I need? That said, their documentation should account for this better imho. There should be sample code which clearly illustrates that if you are going to rely on callbacks being fired, that you better also setup a timer to make sure that they actually get fired. Edit in response to your response above: Mixpanel sells themselves on being simple to use, from their homepage: "It takes less than 10 minutes and is incredibly simple." The reality is far from that. Yes, I made a mistake of assuming that their JS would always get loaded and their callback would always get fired. I fully own up to that and I've learned my lesson. That said, their documentation should also reflect that the callback may not fire and you should be prepared for that by doing XYZ. It is two lines in bold red text and I would have thought harder about depending on that callback to fire. It honestly didn't cross my mind that their js wouldn't load and it hasn't been an issue until now. I'm also not trivializing service stability, yes, things go down and that is a fact of life. That said, when you've got $10+ million in the bank, you can prioritize a bit of the money towards setting up reliably serving of a single file, that doesn't go down for hours on end. There are services, such as CloudFlare and CloudFront which are pretty damn reliable for exactly this purpose and yes, are trivial to implement.
patio11 14y ago
As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should not block the application. Think very, very carefully before you ever block the request/response cycle on an external API. (I'd say "Never do it" but I think I could come up with conceivable apps and APIs where that makes sense if I was fully awake.) Since analytics callbacks don't generate immediate customer value and can fail totally without discomfitting anyone, you should never block a request for them. Instead, you deal with them asynchronously. There's a few ways to do it mechanically. I offload them to my job queue and set the priority to "lowest possible." When the job queue is otherwise empty, a worker process (that no actual human is waiting on) slurps up a few of the events and fires them off. Only downside, which I never bothered fixing: I have an "OMG the queue is stuffed to overflowing... the queue worker must be non-responsive!" monitoring test which, since that symptom has featured in most of my customer-visible downtime, generates a red alert. (i.e. Immediate SMS followed by phone call escalation, as opposed to an "FYI check this" email.) Any downtime at Mixpanel or KissMetrics lasting longer than a few minutes reliably triggers this alert. On the plus side, this means that when I tell you "Mixpanel really doesn't fail at 2 AM Japan time all that often" you should trust that I'd have noticed.
- iheartmemcache 14y ago
  I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.
  - jhuckestein 14y ago
    I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
  - patio11 14y ago
lucisferre 14y ago
As others have pointed out, no SaaS provider will guarantee 100% uptime. You are blaming a very poor design on your part on a 3rd party service. Ironically one should be far more concerned about using your startup for anything than using MixPanel.
- latchkey 14y ago
  There is no sane reason why Mixpanel can't give 100% uptime for a single .js file. https://api.mixpanel.com/site_media/js/api/mixpanel.2.js https://api.mixpanel.com/site_media/js/api/mixpanel.2.js is down right now and that is the biggest part of the issue.
  - le_isms 14y ago
    There are plenty of sane reasons, DDOS included (which seems like what happened here), since our digital world is built on physical systems that have limitations and suffer from external influences that can't be contained.
    - latchkey 14y ago
      This is exactly why services like CloudFlare exist. You can get pretty much the same behavior using CloudFront too. You may get a bit of downtime, but nothing like the hour+ that is going on now. You can even chain them together... CloudFlare in front of your CloudFront. CloudFlare screws up, just turn it off, now everything goes to CloudFront.

by mrchess 14y ago

taf2 14y ago

shouldn't the API calls you make to a tracking service be done in such away that they would not effect your application whether they're available or not available? Analytics is nice, but not having it shouldn't cause your whole app to lock up right? Kind of similar to say a caching infrastructure - if memcached goes down your app just slows down, but doesn't crash right?

blazingfrog2 14y ago
Their documentation is either wrong or misleading: Basic Javascript integration We load the library onto the page asynchronously, which keeps your website loading quickly even if placed in the <head> of the page. In our case just commenting out the library load solved all our issues.
- kcbanner 14y ago
  Yea, the problem is that making actual calls to their API servers is taking a while.

emmett 14y ago

Speaking as someone who uses a ton of mixpanel in production (http://www.twitch.tv/ http://www.twitch.tv/) the way you handle this is to make all your calls asynchronous. I am bummed that we are going to have missing data though :-(

latchkey 14y ago
This works fine for code where you are just firing a /track 'ping' back to the server on a page load. But, if you are trying to do things like track login's where you send a http redirect right after /track, then you need to use the optional callback parameter to /track and do the redirect in the callback. The problem with this is that if their JS doesn't load at all (which is the problem right now), then /track is never called and thus the callback is never fired. So because you can't depend on MP to reliably serve up a simple .js file, you have to write your own wrapper around all of that code to do a timeout incase the callback never fires. What a mess.
- Skywing 14y ago
  I don't use Mixpanel frequently, but could you not just put the Javascript file somewhere more reliable? Why not pull their code down and put it on your own CDN? I've done this with Mixpanel in the past, I believe. This wouldn't solve their API endpoints not responding obviously, but it's an easy way to take the issue into your own hands for the time being. Also, I don't think I would ever make your core logic depend on events from something as dispensable as a third party analytics service. It just sounds like a poor design choice to have some core site functionality occur after receiving a response from something you do not control. I'd try to make those Mixpanel requests more "fire and forget", if you will. Edit in response to you response below: I think you're missing the point here. The scope of their documentation probably does not include best Javascript practices. You made some poor implementation decisions and now you're getting burned. At least now you have experience using third party services and probably won't be so trusting of them, again. Trivializing service stability and other things does not make your point any more credible, though. Personally, I think it makes people overlook your point because it makes it sound as if you have no idea what you're talking about.
  - latchkey 14y ago
    Yup, lesson learned. I definitely won't ever depend on Mixpanel again to do something as simple a reliably serve up a single .js file. I don't want to include their JS file because what happens if they make a change that I need? That said, their documentation should account for this better imho. There should be sample code which clearly illustrates that if you are going to rely on callbacks being fired, that you better also setup a timer to make sure that they actually get fired. Edit in response to your response above: Mixpanel sells themselves on being simple to use, from their homepage: "It takes less than 10 minutes and is incredibly simple." The reality is far from that. Yes, I made a mistake of assuming that their JS would always get loaded and their callback would always get fired. I fully own up to that and I've learned my lesson. That said, their documentation should also reflect that the callback may not fire and you should be prepared for that by doing XYZ. It is two lines in bold red text and I would have thought harder about depending on that callback to fire. It honestly didn't cross my mind that their js wouldn't load and it hasn't been an issue until now. I'm also not trivializing service stability, yes, things go down and that is a fact of life. That said, when you've got $10+ million in the bank, you can prioritize a bit of the money towards setting up reliably serving of a single file, that doesn't go down for hours on end. There are services, such as CloudFlare and CloudFront which are pretty damn reliable for exactly this purpose and yes, are trivial to implement.

patio11 14y ago

As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should not block the application. Think very, very carefully before you ever block the request/response cycle on an external API. (I'd say "Never do it" but I think I could come up with conceivable apps and APIs where that makes sense if I was fully awake.) Since analytics callbacks don't generate immediate customer value and can fail totally without discomfitting anyone, you should never block a request for them. Instead, you deal with them asynchronously. There's a few ways to do it mechanically. I offload them to my job queue and set the priority to "lowest possible." When the job queue is otherwise empty, a worker process (that no actual human is waiting on) slurps up a few of the events and fires them off. Only downside, which I never bothered fixing: I have an "OMG the queue is stuffed to overflowing... the queue worker must be non-responsive!" monitoring test which, since that symptom has featured in most of my customer-visible downtime, generates a red alert. (i.e. Immediate SMS followed by phone call escalation, as opposed to an "FYI check this" email.) Any downtime at Mixpanel or KissMetrics lasting longer than a few minutes reliably triggers this alert. On the plus side, this means that when I tell you "Mixpanel really doesn't fail at 2 AM Japan time all that often" you should trust that I'd have noticed.

iheartmemcache 14y ago
I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.
- jhuckestein 14y ago
  I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
- patio11 14y ago

lucisferre 14y ago

As others have pointed out, no SaaS provider will guarantee 100% uptime. You are blaming a very poor design on your part on a 3rd party service. Ironically one should be far more concerned about using your startup for anything than using MixPanel.

latchkey 14y ago
There is no sane reason why Mixpanel can't give 100% uptime for a single .js file. https://api.mixpanel.com/site_media/js/api/mixpanel.2.js https://api.mixpanel.com/site_media/js/api/mixpanel.2.js is down right now and that is the biggest part of the issue.
- le_isms 14y ago
  There are plenty of sane reasons, DDOS included (which seems like what happened here), since our digital world is built on physical systems that have limitations and suffer from external influences that can't be contained.
  - latchkey 14y ago
    This is exactly why services like CloudFlare exist. You can get pretty much the same behavior using CloudFront too. You may get a bit of downtime, but nothing like the hour+ that is going on now. You can even chain them together... CloudFlare in front of your CloudFront. CloudFlare screws up, just turn it off, now everything goes to CloudFront.