4 ms·

Hi everyone, We believe we have figured out the problem and have already gone ahead with a solution. The issue, at this time, appears to be that the traffic we

by suhail 14y ago

Hi everyone, We believe we have figured out the problem and have already gone ahead with a solution. The issue, at this time, appears to be that the traffic we normally get to our API has increased substantially. Whether it's legitimate or not is unknown but we're unfortunately at the mercy of waiting for DNS to update. It's also possible we may see bottlenecks further down our infrastructure path but we're actively thinking through what's next - fortunately, we have more control there. We'll have a full transparent write up once things are back to normal. If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost. We're deeply sorry and disappointed ourselves. Our support team is happy to talk to you at status.mixpanel.com to give you updates as they get them. Suhail

rsbrown 14y ago
Suhail, with regard to impact on installed sites I agree that lost event data is not the end of the world. However, it's currently taking 20-30 seconds for requests (inserting the mixpanel js) to timeout. During the outage, it would be much less painful if the requests were failing more immediately. I just sent this same message to support (at) mixpanel.com. EDIT: just to reiterate -- the major problem isn't the API being down. It's the extreme timeouts being experienced trying to load a static JS file.
- stevenou 14y ago
  I completely agree. I woke up to some crazy long page rendering time on NewRelic and couldn't figure out what was going on... until I saw the huge backlog of mixpanel events on my db. Not only is it making page rendering slow, even worse, it's tying up my background workers for 30 seconds at a time, causing a huge backlog on ALL my background tasks.
  - ksowocki 14y ago
    Same here. I wish we had an MP widget that insulated us from downtime from the mixpanel API.
- rhizome 14y ago
  Which might also be a factor in the apparent request load if stale requests are ballooning the numbers, if not consuming resources that cause the apparent problem to appear larger. A bunch of elephants standing around blocking things is quite a different problem than a bunch of pigs running down the road.
latchkey 14y ago
I do use the JS api's and it was affecting my website. I was seeing a high number of JS errors and even worse, my users were seeing odd behavior because I was depending on your /track callbacks to fire and they weren't. This is the second time you guys have had a major outage in recent memory and your status page looks like a christmas tree. I really do love mixpanel, but unfortunately I'm going to relegate your service in my mind to 'untrustworthy third party service' every time I write code against your api. Also the fact that I found a pretty major bug in your loader code which prevents it from working correctly with IE browsers (https://mixpanel.com/docs/integration-libraries/javascript https://mixpanel.com/docs/integration-libraries/javascript) last week and today you are still distributing the broken code has me worried. Do you not take this stuff seriously?
- raylu 14y ago
  Could you elaborate on the loader code bug?
  - latchkey 14y ago
    Pretty print their loader JS that is quoted in the url above. I like using the Developer Tools in chrome to do it for me: Load up that JS, go to the Sources tab and click on the {} near the bottom of the screen to auto format it for you. The issue is that you can't insert b (the <script> tag) into the DOM until you've declared window.mixpanel because the code within mixpanel.2.js depends on window.mixpanel to be defined. It is a subtle race condition that only seems to happen with IE, but when it would fail, it would cause one of those ugly IE dialogs to popup to the end user. Not good. The correct code should look like this near the end... just move the insertBefore to be after the window.mixpanel: a.__SV = 1.1; window.mixpanel = a; d.parentNode.insertBefore(b, d);
- schammy 14y ago
  > I found a pretty major bug in your loader code ... last week and today you are still distributing the broken code has me worried. Do you not take this stuff seriously? Clearly you have never run an actual company before.
alexatkeplar 14y ago
Just an idea - but when your event collectors start to become unresponsive, why not redirect your load balancer to a SnowPlow-style "no moving parts" collector, e.g. CloudFront + S3? You would have to merge the S3 log data into your main data store after the outage was over, but at least there would be no data loss (at least, not for events logged by GETs - POSTs are not supported by CloudFront), and webpages wouldn't be hanging. The SnowPlow architecture diagram is here: https://github.com/snowplow/snowplow/wiki/Technical-architecture https://github.com/snowplow/snowplow/wiki/Technical-architec...
newhouseb 14y ago
In case you're not doing this already, it's generally a good idea to set a low TTL like 60 seconds on your DNS so that you can make huge configuration changes relatively quickly.
datahack 14y ago
Nice response, and glad to see a good one. We know how it goes, but I agree with timeout length being an issue for us as well. It would almost be better if we were 'errored out' so our app performance wouldn't lag. Might I suggest something like Chaos Monkey? Real-time APIs are highly vulnerable to DDOS and other traffic conditions because they'll so readily accept new connections, might be something worth throwing on the crew regular-like. :)
dorkusprime 14y ago
"If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost." I wish that was true. But since the library doesn't handle service downtime at all gracefully, we're stuck with the callbacks not firing -- something we depended on.
reinhardt 14y ago
Looking forward to the postmortem.

by suhail 14y ago

rsbrown 14y ago

Suhail, with regard to impact on installed sites I agree that lost event data is not the end of the world. However, it's currently taking 20-30 seconds for requests (inserting the mixpanel js) to timeout. During the outage, it would be much less painful if the requests were failing more immediately. I just sent this same message to support (at) mixpanel.com. EDIT: just to reiterate -- the major problem isn't the API being down. It's the extreme timeouts being experienced trying to load a static JS file.

stevenou 14y ago
I completely agree. I woke up to some crazy long page rendering time on NewRelic and couldn't figure out what was going on... until I saw the huge backlog of mixpanel events on my db. Not only is it making page rendering slow, even worse, it's tying up my background workers for 30 seconds at a time, causing a huge backlog on ALL my background tasks.
- ksowocki 14y ago
  Same here. I wish we had an MP widget that insulated us from downtime from the mixpanel API.
rhizome 14y ago
Which might also be a factor in the apparent request load if stale requests are ballooning the numbers, if not consuming resources that cause the apparent problem to appear larger. A bunch of elephants standing around blocking things is quite a different problem than a bunch of pigs running down the road.

latchkey 14y ago

I do use the JS api's and it was affecting my website. I was seeing a high number of JS errors and even worse, my users were seeing odd behavior because I was depending on your /track callbacks to fire and they weren't. This is the second time you guys have had a major outage in recent memory and your status page looks like a christmas tree. I really do love mixpanel, but unfortunately I'm going to relegate your service in my mind to 'untrustworthy third party service' every time I write code against your api. Also the fact that I found a pretty major bug in your loader code which prevents it from working correctly with IE browsers (https://mixpanel.com/docs/integration-libraries/javascript https://mixpanel.com/docs/integration-libraries/javascript) last week and today you are still distributing the broken code has me worried. Do you not take this stuff seriously?

raylu 14y ago
Could you elaborate on the loader code bug?
- latchkey 14y ago
  Pretty print their loader JS that is quoted in the url above. I like using the Developer Tools in chrome to do it for me: Load up that JS, go to the Sources tab and click on the {} near the bottom of the screen to auto format it for you. The issue is that you can't insert b (the <script> tag) into the DOM until you've declared window.mixpanel because the code within mixpanel.2.js depends on window.mixpanel to be defined. It is a subtle race condition that only seems to happen with IE, but when it would fail, it would cause one of those ugly IE dialogs to popup to the end user. Not good. The correct code should look like this near the end... just move the insertBefore to be after the window.mixpanel: a.__SV = 1.1; window.mixpanel = a; d.parentNode.insertBefore(b, d);
schammy 14y ago
> I found a pretty major bug in your loader code ... last week and today you are still distributing the broken code has me worried. Do you not take this stuff seriously? Clearly you have never run an actual company before.

alexatkeplar 14y ago

Just an idea - but when your event collectors start to become unresponsive, why not redirect your load balancer to a SnowPlow-style "no moving parts" collector, e.g. CloudFront + S3? You would have to merge the S3 log data into your main data store after the outage was over, but at least there would be no data loss (at least, not for events logged by GETs - POSTs are not supported by CloudFront), and webpages wouldn't be hanging. The SnowPlow architecture diagram is here: https://github.com/snowplow/snowplow/wiki/Technical-architecture https://github.com/snowplow/snowplow/wiki/Technical-architec...

newhouseb 14y ago

In case you're not doing this already, it's generally a good idea to set a low TTL like 60 seconds on your DNS so that you can make huge configuration changes relatively quickly.

datahack 14y ago

Nice response, and glad to see a good one. We know how it goes, but I agree with timeout length being an issue for us as well. It would almost be better if we were 'errored out' so our app performance wouldn't lag. Might I suggest something like Chaos Monkey? Real-time APIs are highly vulnerable to DDOS and other traffic conditions because they'll so readily accept new connections, might be something worth throwing on the crew regular-like. :)

dorkusprime 14y ago

"If you're using our JavaScript libraries, your website should not be greatly affected by our downtime other than the data that will be lost." I wish that was true. But since the library doesn't handle service downtime at all gracefully, we're stuck with the callbacks not firing -- something we depended on.

reinhardt 14y ago

Looking forward to the postmortem.