6 ms·

MixPanel tracking API down

48 points by mef 14y ago

deleted 14y ago
[deleted]
mrchess 14y ago
This is unfortunate. I was about to implement MixPanel on my startups site (it already laced in our development code), but their API being down completely locks our application ... actually shipping MixPanel in production not looking so hot right now. EDIT: As others have voiced my issue was that it was taking up to 30 seconds for requests to timeout disrupting various things in my JS code (it is a Backbone app) -- it didn't actually "lock" it. I was using the JS API.
- taf2 14y ago
  shouldn't the API calls you make to a tracking service be done in such away that they would not effect your application whether they're available or not available? Analytics is nice, but not having it shouldn't cause your whole app to lock up right? Kind of similar to say a caching infrastructure - if memcached goes down your app just slows down, but doesn't crash right?
  - blazingfrog2 14y ago
    Their documentation is either wrong or misleading: Basic Javascript integration We load the library onto the page asynchronously, which keeps your website loading quickly even if placed in the <head> of the page. In our case just commenting out the library load solved all our issues.
    - kcbanner 14y ago
      Yea, the problem is that making actual calls to their API servers is taking a while.
- emmett 14y ago
  Speaking as someone who uses a ton of mixpanel in production (http://www.twitch.tv/ http://www.twitch.tv/) the way you handle this is to make all your calls asynchronous. I am bummed that we are going to have missing data though :-(
  - latchkey 14y ago
    This works fine for code where you are just firing a /track 'ping' back to the server on a page load. But, if you are trying to do things like track login's where you send a http redirect right after /track, then you need to use the optional callback parameter to /track and do the redirect in the callback. The problem with this is that if their JS doesn't load at all (which is the problem right now), then /track is never called and thus the callback is never fired. So because you can't depend on MP to reliably serve up a simple .js file, you have to write your own wrapper around all of that code to do a timeout incase the callback never fires. What a mess.
    - Skywing 14y ago
      I don't use Mixpanel frequently, but could you not just put the Javascript file somewhere more reliable? Why not pull their code down and put it on your own CDN? I've done this with Mixpanel in the past, I believe. This wouldn't solve their API endpoints not responding obviously, but it's an easy way to take the issue into your own hands for the time being. Also, I don't think I would ever make your core logic depend on events from something as dispensable as a third party analytics service. It just sounds like a poor design choice to have some core site functionality occur after receiving a response from something you do not control. I'd try to make those Mixpanel requests more "fire and forget", if you will. Edit in response to you response below: I think you're missing the point here. The scope of their documentation probably does not include best Javascript practices. You made some poor implementation decisions and now you're getting burned. At least now you have experience using third party services and probably won't be so trusting of them, again. Trivializing service stability and other things does not make your point any more credible, though. Personally, I think it makes people overlook your point because it makes it sound as if you have no idea what you're talking about.
- patio11 14y ago
  As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should not block the application. Think very, very carefully before you ever block the request/response cycle on an external API. (I'd say "Never do it" but I think I could come up with conceivable apps and APIs where that makes sense if I was fully awake.) Since analytics callbacks don't generate immediate customer value and can fail totally without discomfitting anyone, you should never block a request for them. Instead, you deal with them asynchronously. There's a few ways to do it mechanically. I offload them to my job queue and set the priority to "lowest possible." When the job queue is otherwise empty, a worker process (that no actual human is waiting on) slurps up a few of the events and fires them off. Only downside, which I never bothered fixing: I have an "OMG the queue is stuffed to overflowing... the queue worker must be non-responsive!" monitoring test which, since that symptom has featured in most of my customer-visible downtime, generates a red alert. (i.e. Immediate SMS followed by phone call escalation, as opposed to an "FYI check this" email.) Any downtime at Mixpanel or KissMetrics lasting longer than a few minutes reliably triggers this alert. On the plus side, this means that when I tell you "Mixpanel really doesn't fail at 2 AM Japan time all that often" you should trust that I'd have noticed.
  - iheartmemcache 14y ago
    I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.
    - jhuckestein 14y ago
      I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
    - patio11 14y ago
- lucisferre 14y ago
  As others have pointed out, no SaaS provider will guarantee 100% uptime. You are blaming a very poor design on your part on a 3rd party service. Ironically one should be far more concerned about using your startup for anything than using MixPanel.
  - latchkey 14y ago
    There is no sane reason why Mixpanel can't give 100% uptime for a single .js file. https://api.mixpanel.com/site_media/js/api/mixpanel.2.js https://api.mixpanel.com/site_media/js/api/mixpanel.2.js is down right now and that is the biggest part of the issue.
    - le_isms 14y ago
      There are plenty of sane reasons, DDOS included (which seems like what happened here), since our digital world is built on physical systems that have limitations and suffer from external influences that can't be contained.
      - latchkey 14y ago
        This is exactly why services like CloudFlare exist. You can get pretty much the same behavior using CloudFront too. You may get a bit of downtime, but nothing like the hour+ that is going on now. You can even chain them together... CloudFlare in front of your CloudFront. CloudFlare screws up, just turn it off, now everything goes to CloudFront.
suhail 14y ago
Hi everyone, We believe we have figured out the problem and have already gone ahead with a solution. The issue, at this time, appears to be that the traffic we normally get to our API has increased substantially. Whether it's legitimate or not is unknown but we're unfortunately at the mercy of waiting for DNS to update. It's also possible we may see bottlenecks further down our infrastructure path but we're actively thinking through what's next - fortunately, we have more control there. We'll have a full transparent write up once things are back to normal. If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost. We're deeply sorry and disappointed ourselves. Our support team is happy to talk to you at status.mixpanel.com to give you updates as they get them. Suhail
- rsbrown 14y ago
  Suhail, with regard to impact on installed sites I agree that lost event data is not the end of the world. However, it's currently taking 20-30 seconds for requests (inserting the mixpanel js) to timeout. During the outage, it would be much less painful if the requests were failing more immediately. I just sent this same message to support (at) mixpanel.com. EDIT: just to reiterate -- the major problem isn't the API being down. It's the extreme timeouts being experienced trying to load a static JS file.
  - stevenou 14y ago
    I completely agree. I woke up to some crazy long page rendering time on NewRelic and couldn't figure out what was going on... until I saw the huge backlog of mixpanel events on my db. Not only is it making page rendering slow, even worse, it's tying up my background workers for 30 seconds at a time, causing a huge backlog on ALL my background tasks.
    - ksowocki 14y ago
      Same here. I wish we had an MP widget that insulated us from downtime from the mixpanel API.
  - 14y ago
jhuckestein 14y ago
This is why I queue all my mixpanel requests on the server. I'll just re-run them once it's back up.
- le_isms 14y ago
  This is a great idea! Do you mind letting us know what your setup's like?
  - wulczer 14y ago
    We just issue Celery jobs to track events in Mixpanel, so if their API is failing, the jobs fail and get retried later on. If we get too many messages accumulated in the queue where these jobs go, we just purge it (better lose that data than let RabbitMQ die because of memory exhaustion). Although there have to be a whole damn lot of events there for us to actually notice
    - le_isms 14y ago
      Really neat! I might be a newbie to this, but how do you get those client-side javascript function calls over to the serverside? I am assuming the client's browser is running the Mixpanel Javascript and that your events are serverside.
      - rprime 14y ago
        I think he is not talking about the javascript events. Probably they fire the events in the backend. [1] [1] https://mixpanel.com/docs/integration-libraries/ https://mixpanel.com/docs/integration-libraries/
      - wulczer 14y ago
        Yep, like rprime said, we're sending tracking events from the server (let me plug http://libsaas.net http://libsaas.net here). So instead of using mixpanel.track from Javascript, you do an AJAX call to your own server and schedule a Celery job there. There's an issue with passing Mixpanel's super properties to your server-side handler, but that's the general idea.

48 points by mef 14y ago

deleted 14y ago

[deleted]

This is unfortunate. I was about to implement MixPanel on my startups site (it already laced in our development code), but their API being down completely locks our application ... actually shipping MixPanel in production not looking so hot right now. EDIT: As others have voiced my issue was that it was taking up to 30 seconds for requests to timeout disrupting various things in my JS code (it is a Backbone app) -- it didn't actually "lock" it. I was using the JS API.

taf2 14y ago
shouldn't the API calls you make to a tracking service be done in such away that they would not effect your application whether they're available or not available? Analytics is nice, but not having it shouldn't cause your whole app to lock up right? Kind of similar to say a caching infrastructure - if memcached goes down your app just slows down, but doesn't crash right?
- blazingfrog2 14y ago
  Their documentation is either wrong or misleading: Basic Javascript integration We load the library onto the page asynchronously, which keeps your website loading quickly even if placed in the <head> of the page. In our case just commenting out the library load solved all our issues.
  - kcbanner 14y ago
    Yea, the problem is that making actual calls to their API servers is taking a while.
emmett 14y ago
Speaking as someone who uses a ton of mixpanel in production (http://www.twitch.tv/ http://www.twitch.tv/) the way you handle this is to make all your calls asynchronous. I am bummed that we are going to have missing data though :-(
- latchkey 14y ago
  This works fine for code where you are just firing a /track 'ping' back to the server on a page load. But, if you are trying to do things like track login's where you send a http redirect right after /track, then you need to use the optional callback parameter to /track and do the redirect in the callback. The problem with this is that if their JS doesn't load at all (which is the problem right now), then /track is never called and thus the callback is never fired. So because you can't depend on MP to reliably serve up a simple .js file, you have to write your own wrapper around all of that code to do a timeout incase the callback never fires. What a mess.
  - Skywing 14y ago
    I don't use Mixpanel frequently, but could you not just put the Javascript file somewhere more reliable? Why not pull their code down and put it on your own CDN? I've done this with Mixpanel in the past, I believe. This wouldn't solve their API endpoints not responding obviously, but it's an easy way to take the issue into your own hands for the time being. Also, I don't think I would ever make your core logic depend on events from something as dispensable as a third party analytics service. It just sounds like a poor design choice to have some core site functionality occur after receiving a response from something you do not control. I'd try to make those Mixpanel requests more "fire and forget", if you will. Edit in response to you response below: I think you're missing the point here. The scope of their documentation probably does not include best Javascript practices. You made some poor implementation decisions and now you're getting burned. At least now you have experience using third party services and probably won't be so trusting of them, again. Trivializing service stability and other things does not make your point any more credible, though. Personally, I think it makes people overlook your point because it makes it sound as if you have no idea what you're talking about.
patio11 14y ago
As somebody who consumes a few of these things, and got a pager about it prior to it being on HN, this is something that a) will inevitably happen and b) should not block the application. Think very, very carefully before you ever block the request/response cycle on an external API. (I'd say "Never do it" but I think I could come up with conceivable apps and APIs where that makes sense if I was fully awake.) Since analytics callbacks don't generate immediate customer value and can fail totally without discomfitting anyone, you should never block a request for them. Instead, you deal with them asynchronously. There's a few ways to do it mechanically. I offload them to my job queue and set the priority to "lowest possible." When the job queue is otherwise empty, a worker process (that no actual human is waiting on) slurps up a few of the events and fires them off. Only downside, which I never bothered fixing: I have an "OMG the queue is stuffed to overflowing... the queue worker must be non-responsive!" monitoring test which, since that symptom has featured in most of my customer-visible downtime, generates a red alert. (i.e. Immediate SMS followed by phone call escalation, as opposed to an "FYI check this" email.) Any downtime at Mixpanel or KissMetrics lasting longer than a few minutes reliably triggers this alert. On the plus side, this means that when I tell you "Mixpanel really doesn't fail at 2 AM Japan time all that often" you should trust that I'd have noticed.
- iheartmemcache 14y ago
  I (and I'm sure other HNers) would love it if you couldgo into a little bit more detail as to how your job queue is constructed.
  - jhuckestein 14y ago
    I'm not the GP but in my startup we just use Resque, it works wonderfully. I've also had semi-good experiences with RabbitMQ via AMQP. There's nothing special about the structure. Depending on the system you'll use channels/queue names/routing keys/whatever when enqueuing a job and then workers will pull the jobs they're in charge off out of the queue. In the case of mixpanel jobs you should set it up so they stick around when they fail
  - patio11 14y ago
lucisferre 14y ago
As others have pointed out, no SaaS provider will guarantee 100% uptime. You are blaming a very poor design on your part on a 3rd party service. Ironically one should be far more concerned about using your startup for anything than using MixPanel.
- latchkey 14y ago
  There is no sane reason why Mixpanel can't give 100% uptime for a single .js file. https://api.mixpanel.com/site_media/js/api/mixpanel.2.js https://api.mixpanel.com/site_media/js/api/mixpanel.2.js is down right now and that is the biggest part of the issue.
  - le_isms 14y ago
    There are plenty of sane reasons, DDOS included (which seems like what happened here), since our digital world is built on physical systems that have limitations and suffer from external influences that can't be contained.
    - latchkey 14y ago
      This is exactly why services like CloudFlare exist. You can get pretty much the same behavior using CloudFront too. You may get a bit of downtime, but nothing like the hour+ that is going on now. You can even chain them together... CloudFlare in front of your CloudFront. CloudFlare screws up, just turn it off, now everything goes to CloudFront.

suhail 14y ago

Hi everyone, We believe we have figured out the problem and have already gone ahead with a solution. The issue, at this time, appears to be that the traffic we normally get to our API has increased substantially. Whether it's legitimate or not is unknown but we're unfortunately at the mercy of waiting for DNS to update. It's also possible we may see bottlenecks further down our infrastructure path but we're actively thinking through what's next - fortunately, we have more control there. We'll have a full transparent write up once things are back to normal. If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost. We're deeply sorry and disappointed ourselves. Our support team is happy to talk to you at status.mixpanel.com to give you updates as they get them. Suhail

rsbrown 14y ago
Suhail, with regard to impact on installed sites I agree that lost event data is not the end of the world. However, it's currently taking 20-30 seconds for requests (inserting the mixpanel js) to timeout. During the outage, it would be much less painful if the requests were failing more immediately. I just sent this same message to support (at) mixpanel.com. EDIT: just to reiterate -- the major problem isn't the API being down. It's the extreme timeouts being experienced trying to load a static JS file.
- stevenou 14y ago
  I completely agree. I woke up to some crazy long page rendering time on NewRelic and couldn't figure out what was going on... until I saw the huge backlog of mixpanel events on my db. Not only is it making page rendering slow, even worse, it's tying up my background workers for 30 seconds at a time, causing a huge backlog on ALL my background tasks.
  - ksowocki 14y ago
    Same here. I wish we had an MP widget that insulated us from downtime from the mixpanel API.
- 14y ago

jhuckestein 14y ago

This is why I queue all my mixpanel requests on the server. I'll just re-run them once it's back up.

le_isms 14y ago
This is a great idea! Do you mind letting us know what your setup's like?
- wulczer 14y ago
  We just issue Celery jobs to track events in Mixpanel, so if their API is failing, the jobs fail and get retried later on. If we get too many messages accumulated in the queue where these jobs go, we just purge it (better lose that data than let RabbitMQ die because of memory exhaustion). Although there have to be a whole damn lot of events there for us to actually notice
  - le_isms 14y ago
    Really neat! I might be a newbie to this, but how do you get those client-side javascript function calls over to the serverside? I am assuming the client's browser is running the Mixpanel Javascript and that your events are serverside.
    - rprime 14y ago
      I think he is not talking about the javascript events. Probably they fire the events in the backend. [1] [1] https://mixpanel.com/docs/integration-libraries/ https://mixpanel.com/docs/integration-libraries/
    - wulczer 14y ago
      Yep, like rprime said, we're sending tracking events from the server (let me plug http://libsaas.net http://libsaas.net here). So instead of using mixpanel.track from Javascript, you do an AJAX call to your own server and schedule a Celery job there. There's an issue with passing Mixpanel's super properties to your server-side handler, but that's the general idea.