9 ms·

Mixpanel analytics accidentally slurped up passwords

105 points by tareqak 9y ago

tekkk 9y ago
One more reason to use adblocker. The beauty of 3rd party scripts running on your website, hijacking people's passwords like it's no big deal. I've used Mixpanel as a developer and also been disturbed how they combine analytics data with the user profiles creating really detailed profiles what they do on their website. Even that wouldn't be such a big deal if it was obvious to the user but really there is no law stopping companies from doing it. I hope GDPR will clear this thing out in the future and companies can't operate without user's direct consent and opt-in becomes the standard instead of opt-out.
- ryanwaggoner 9y ago
  That seems crazy to me. It should be illegal for a website to track what you do on that site? We’re not talking about tracking activity across many sites, and you don’t need a third-party tool to do this at all. If it were illegal, a huge number of sites wouldn’t even function without fundamentally changing the way they work. You know that the HN database has records of all your votes, comments, etc, right? I get the outrage at being tracked by third parties across the web, but that’s different from making it illegal for any website to track any of your activity on their site.
  - toomuchtodo 9y ago
    It should be illegal for a site to store your personal data without your consent, which is what the GDPR dictates for EU residents. What is considered personal data is for governments, regulators, and courts to decide; not website owners. The days of the Internet being the Wild West are coming to a close, and not a moment too soon. Side note: it’s disturbing it took Mixpanel nine months to notice they were ingesting sensitive data. Turn on All The Ad Blocking.
    - ryanwaggoner 9y ago
      I think your view of the GDPR is naive. 99% of people will just consent and that’ll be that. The second they want something the site offers and they have to register (the point at which they’re handing over PII), they’ll be forced to consent, and just like everyone does for cookies and TOS and privacy policy and whatever else, they’ll quickly be trained to consent and move on. I could be wrong, but that’s what I expect. The assymetry here is incredible. That data and the consent has a perceptual value of approximately zero to the consumer, and millions or billions in aggregate to these companies. They’ll figure out a way to get people to consent. Or most of the web just won’t be usable to you.
      - deleted 9y ago
        [deleted]
      - toomuchtodo 9y ago
        I think you’re underestimating the intent of the EU and the aggressive enforcement available to them. If we have to break the web, we break the web. Digital rights supercede tech profits.
        stryk 9y ago
        I have not much more than a cursory understanding of this new law, but holy shit do I hope somebody breaks the web. Fast. This is out of control with the tracking and the monitoring and every other website seems to pull in 14 or so different javascript analytics with who-knows how many 3rd party companies that you had no idea you were interacting with when you went to the original site. I still don't understand how they plan to enforce this law worldwide, though. The EU is powerful, sure, but if I was China I might just tell them to piss off when they come knocking.
        beojan 9y ago
        > The EU is powerful, sure, but if I was China I might just tell them to piss off when they come knocking. It's not really Chinese companies doing the tracking, it's American ones. The EU is simply too big a market for them to ignore.
  - kangoo1707 9y ago
    How would startups gather BI information then?
    - tzahola 9y ago
      Not my problem.
    - jacquesm 9y ago
      By asking politely?
    - templar_muse 9y ago
      Why do they need it and are they gathering purely for the sake of gathering?
- bluesign 9y ago
  ‘One more reason to use adblocker. The beauty of 3rd party scripts running on your website, hijacking people's passwords like it's no big deal.’ Yeah but soon adblock and 3rd party adaptation will reach critical mass, we will start to see, 3rd party tools integrated as 1st party (with subdomains etc) then adblocks will be not much efficient
  - TeMPOraL 9y ago
    We can't win this war with technology alone. It's good the GDPR is coming on the data collection front, because the only way you can stop this fuckery is through out-of-band threat of legal problems.
  - chopin 9y ago
    This could backfire badly. I don't use an explicit adblocker but NoScript, which blocks almost as efficient (and makes tracking more difficult). For the occasional site I allow 1st party script. If sites start to serve me malware this way, I will block them right away, too. I can't do business with you without Javascript enabled? I just move on to the next one. Or I go back to brick and mortar. As well, serving malware 1st party may have some legal consequences if got caught.
RoboTeddy 9y ago
Looks like Mixpanel is handling this well: they deleted the data, notified affected customers, and then are making sure it can't happen again. +1 for admitting responsibility instead of deflecting or minimizing.
- spenvo 9y ago
  Ok, they took responsibility... But they took about a month to notify clients via email and didn't notify the public until Techcrunch inquired for the purposes of writing this article. So -1 for not being upfront with end users and clients.
  - ajeet_dhaliwal 9y ago
    Take away a few more for not noticing the problem for nine months.
    - snissn 9y ago
      Would be a good move to put together a common password list and regularly check their data against it.
  - perfectstorm 9y ago
    They were notified on Jan 5th. They destroyed the stored passwords on the 9th and informed the customers on the 1st of February. It seems reasonable. Why would they inform the end user though ? The end user doesn't even know what Mixpanel is and would be confused if they emailed them directly. Informing the clients (BMW, Samsung etc.) is the right thing to do which they did. I'm sure they had to do a postmortem and make sure a fix is in place before informing the clients and urging them to update the SDK.
    - anc84 9y ago
      > Why would they inform the end user though ? The end user doesn't even know what Mixpanel is and would be confused if they emailed them directly. So that the end users can grasp to which extent their privacy was violated? If a third-party I never heard of, contacted me and told me they got my login details, I would be bloody furious. And that is a good thing if people are enabled to this.
nvr219 9y ago
Always do ublock origin + privacy badger (and pi hole when you're at home)
- tylersmith 9y ago
  I set up a network-wide pi hole recently and it took less than 15 minutes once I had the rpi and sd card in hand. It's a fun and easy experience that I highly recommend it to anybody. I check the dashboard every morning to see how much I've blocked, and to what. And my wife, who doesn't care about privacy for some insane reason, now has more privacy than she expected without any work.
- bonestamp2 9y ago
  uMatrix is well worth the time it takes to use too. It shows you a matrix of exactly what each site is requesting. I've become so much more aware of what is happening on websites since installing the plugin about a year ago. The great thing is that it's so quick and easy to enable/disable/customize too. For example, I've identified two scripts that block websites from loading when it doesn't get loaded... so I allow those to load on every website -- no ads, but the site still loads. Mixpanel is blocked by default in the blacklist that I subscribe to.
  - faitswulff 9y ago
    Are those two scripts widespread or specific to a site? And if they're quite widespread, would you be able to share how to identify them?
    - bonestamp2 9y ago
      Just saw one a few minutes ago. It's actually not even the script you need to allow, it's just an XHR request to adiode.com. I can't remember the other one, I'll report back here when I see it again (usually every couple days). I saw the adiode.com one on https://www.merriam-webster.com https://www.merriam-webster.com if you want to have a look for yourself.
wcr3 9y ago
Staff photo shouldn't be in the article.
btown 9y ago
> We immediately began investigating further and learned that the behavior the customer was observing was due to a change to the React JavaScript library made in March 2017. This change placed copies of the values of hidden and password fields into the input elements’ attributes, which Autotrack then inadvertently received. Upon investigating further, we realized that, because of the way we had implemented Autotrack when it launched in August 2016, this could happen in other scenarios where browser plugins (such as the 1Password password manager) and website frameworks place sensitive data into form element attributes. Here's the commit in question: https://github.com/mixpanel/mixpanel-js/commit/98a1845c5c55f8b362847517860cc97c35f1c08e https://github.com/mixpanel/mixpanel-js/commit/98a1845c5c55f... - as referenced here: https://github.com/mixpanel/mixpanel-js/issues/164 https://github.com/mixpanel/mixpanel-js/issues/164 . The meat of it is that the check of the type of the <input> node to ensure it's not "password" or "hidden," now wraps the iteration over attributes as well as inclusion of the value. That heuristic, though, is far from perfect: see, for instance, https://www.troyhunt.com/bypassing-browser-security-warnings-with-pseudo-password-fields/ https://www.troyhunt.com/bypassing-browser-security-warnings... . And even if you're not doing something funky like that, you're not out of the woods. For instance, it's highly likely that a site collecting "secret answers" in plaintext input fields (for password resets) would leak that information to Mixpanel if Autotrack were turned on. This commit does nothing to change that scenario. And while Autotrack is now opt-in, it can be done entirely by a business team with zero interactions with engineering, if (for example) a tag management solution is set up: see https://mixpanel.com/blog/2015/03/27/community-tip-implementing-mixpanel-via-google-tag-manager/ https://mixpanel.com/blog/2015/03/27/community-tip-implement... and https://help.mixpanel.com/hc/en-us/articles/115004613366-What-is-Autotrack-and-how-do-I-get-started-using-it- https://help.mixpanel.com/hc/en-us/articles/115004613366-Wha... SaaS analytics companies, especially those able to slurp up everything into a managed data lake and provide perfect retroactive analysis capabilities, provide a great experience compared to self-hosted solutions, but it's inevitable that you'll end up giving them more information than you'd originally planned. For many businesses, that's the right decision, but it should be one made with eyes wide open.
colmvp 9y ago
On a side note, Mixpanel used to host conferences but stopped after 2014. Anyone know why?
dylanhassinger 9y ago
sounds to me like the problem was with React. which is owned by Facebook. Mark Z helping out the hackers and bots once again
BuildTheRobots 9y ago
Call me thick and naive if you will, but how does an advert manage to hoover up all typed inputs on a page including (somehow accidentally) passwords. Isn't the underlying problem with the web browser that allows this, or are browsers just fundamentally broken when it comes to protecting users?
- TeMPOraL 9y ago
  Analytics analyze. They read data. Mixpanel decided to read more than they should, and they unintentionally slurped some passwords. The core technical problem is that scripts can read data. But that's kind of their job. "Solving" that would require to reduce the functionality of webpages. The core social problem is that people include third-party analytics and advertisements. Solving that would require to destroy the entire adtech industry. I'm all for the latter solution.
  - bluesign 9y ago
    Where you will draw the line, then main ad industries will start to serve ads as 1st party as custom subdomains. There is no way to win this fight except regulations and oversight
    - TeMPOraL 9y ago
      Yes, I strongly agree. That's why I can't wait for GDPR to come to life this May.
    - chopin 9y ago
      In this case, they can be made fully responsible. You'd better vet the Javascript you're going to deliver.
NicoJuicy 9y ago
A lot of people talk here about GDPR, but does anyone know how to "handle" this in my ecommerce? I haven't found any indication on how to do it "legally" or a small "howto". I'm also not interested in an IBM solution that would cost me 10 times my earnings
- jmickey 9y ago
  In essence the GDPR requires you to be accountable about all personal data you collect and use in your daily operations. Compiling a list of all personal data currently located in your systems and making sure you have a legal basis for each item, goes a long way towards compliance (though of course this is not all you need to do)
  - NicoJuicy 9y ago
    So, let's say: - WooCommerce on url xxx: phone number, email,name & address. Google Analytics & facebook Pixel. Requirement for e-commerce fullfillment and analysing website performance / ad performance. - Mailchimp : email and name, when accepting WooCommerce "Terms and conditions" n°2. Requirement for recurring ecommerce updates/changes of new products. - OpenERP: ( invoicing - local network) - Firstname, lastname, address, email, phone, orders. Requirement for invoicing Somehow i can't believe that would be sufficient.
    - ryanwaggoner 9y ago
      It’s not. GDPR is so overly broad that it’s almost impossible to be in full compliance. Not only does it count IP addresses as personal information, but it covers all EU residents, apparently no matter where they are in the world, even if you have no way of knowing they’re EU residents. I really doubt most small-medium businesses without ties to the EU are going to pay any attention. Just like VAT actually.
      - jmickey 9y ago
        Mind you an IP address is personal data only if it identifies an individual. Same goes with any other information.

105 points by tareqak 9y ago

One more reason to use adblocker. The beauty of 3rd party scripts running on your website, hijacking people's passwords like it's no big deal. I've used Mixpanel as a developer and also been disturbed how they combine analytics data with the user profiles creating really detailed profiles what they do on their website. Even that wouldn't be such a big deal if it was obvious to the user but really there is no law stopping companies from doing it. I hope GDPR will clear this thing out in the future and companies can't operate without user's direct consent and opt-in becomes the standard instead of opt-out.

ryanwaggoner 9y ago
That seems crazy to me. It should be illegal for a website to track what you do on that site? We’re not talking about tracking activity across many sites, and you don’t need a third-party tool to do this at all. If it were illegal, a huge number of sites wouldn’t even function without fundamentally changing the way they work. You know that the HN database has records of all your votes, comments, etc, right? I get the outrage at being tracked by third parties across the web, but that’s different from making it illegal for any website to track any of your activity on their site.
- toomuchtodo 9y ago
  It should be illegal for a site to store your personal data without your consent, which is what the GDPR dictates for EU residents. What is considered personal data is for governments, regulators, and courts to decide; not website owners. The days of the Internet being the Wild West are coming to a close, and not a moment too soon. Side note: it’s disturbing it took Mixpanel nine months to notice they were ingesting sensitive data. Turn on All The Ad Blocking.
  - ryanwaggoner 9y ago
    I think your view of the GDPR is naive. 99% of people will just consent and that’ll be that. The second they want something the site offers and they have to register (the point at which they’re handing over PII), they’ll be forced to consent, and just like everyone does for cookies and TOS and privacy policy and whatever else, they’ll quickly be trained to consent and move on. I could be wrong, but that’s what I expect. The assymetry here is incredible. That data and the consent has a perceptual value of approximately zero to the consumer, and millions or billions in aggregate to these companies. They’ll figure out a way to get people to consent. Or most of the web just won’t be usable to you.
    - deleted 9y ago
      [deleted]
    - toomuchtodo 9y ago
      I think you’re underestimating the intent of the EU and the aggressive enforcement available to them. If we have to break the web, we break the web. Digital rights supercede tech profits.
      - stryk 9y ago
        I have not much more than a cursory understanding of this new law, but holy shit do I hope somebody breaks the web. Fast. This is out of control with the tracking and the monitoring and every other website seems to pull in 14 or so different javascript analytics with who-knows how many 3rd party companies that you had no idea you were interacting with when you went to the original site. I still don't understand how they plan to enforce this law worldwide, though. The EU is powerful, sure, but if I was China I might just tell them to piss off when they come knocking.
        beojan 9y ago
        > The EU is powerful, sure, but if I was China I might just tell them to piss off when they come knocking. It's not really Chinese companies doing the tracking, it's American ones. The EU is simply too big a market for them to ignore.
- kangoo1707 9y ago
  How would startups gather BI information then?
  - tzahola 9y ago
    Not my problem.
  - jacquesm 9y ago
    By asking politely?
  - templar_muse 9y ago
    Why do they need it and are they gathering purely for the sake of gathering?
bluesign 9y ago
‘One more reason to use adblocker. The beauty of 3rd party scripts running on your website, hijacking people's passwords like it's no big deal.’ Yeah but soon adblock and 3rd party adaptation will reach critical mass, we will start to see, 3rd party tools integrated as 1st party (with subdomains etc) then adblocks will be not much efficient
- TeMPOraL 9y ago
  We can't win this war with technology alone. It's good the GDPR is coming on the data collection front, because the only way you can stop this fuckery is through out-of-band threat of legal problems.
- chopin 9y ago
  This could backfire badly. I don't use an explicit adblocker but NoScript, which blocks almost as efficient (and makes tracking more difficult). For the occasional site I allow 1st party script. If sites start to serve me malware this way, I will block them right away, too. I can't do business with you without Javascript enabled? I just move on to the next one. Or I go back to brick and mortar. As well, serving malware 1st party may have some legal consequences if got caught.

RoboTeddy 9y ago

Looks like Mixpanel is handling this well: they deleted the data, notified affected customers, and then are making sure it can't happen again. +1 for admitting responsibility instead of deflecting or minimizing.

spenvo 9y ago
Ok, they took responsibility... But they took about a month to notify clients via email and didn't notify the public until Techcrunch inquired for the purposes of writing this article. So -1 for not being upfront with end users and clients.
- ajeet_dhaliwal 9y ago
  Take away a few more for not noticing the problem for nine months.
  - snissn 9y ago
    Would be a good move to put together a common password list and regularly check their data against it.
- perfectstorm 9y ago
  They were notified on Jan 5th. They destroyed the stored passwords on the 9th and informed the customers on the 1st of February. It seems reasonable. Why would they inform the end user though ? The end user doesn't even know what Mixpanel is and would be confused if they emailed them directly. Informing the clients (BMW, Samsung etc.) is the right thing to do which they did. I'm sure they had to do a postmortem and make sure a fix is in place before informing the clients and urging them to update the SDK.
  - anc84 9y ago
    > Why would they inform the end user though ? The end user doesn't even know what Mixpanel is and would be confused if they emailed them directly. So that the end users can grasp to which extent their privacy was violated? If a third-party I never heard of, contacted me and told me they got my login details, I would be bloody furious. And that is a good thing if people are enabled to this.

nvr219 9y ago

Always do ublock origin + privacy badger (and pi hole when you're at home)

tylersmith 9y ago
I set up a network-wide pi hole recently and it took less than 15 minutes once I had the rpi and sd card in hand. It's a fun and easy experience that I highly recommend it to anybody. I check the dashboard every morning to see how much I've blocked, and to what. And my wife, who doesn't care about privacy for some insane reason, now has more privacy than she expected without any work.
bonestamp2 9y ago
uMatrix is well worth the time it takes to use too. It shows you a matrix of exactly what each site is requesting. I've become so much more aware of what is happening on websites since installing the plugin about a year ago. The great thing is that it's so quick and easy to enable/disable/customize too. For example, I've identified two scripts that block websites from loading when it doesn't get loaded... so I allow those to load on every website -- no ads, but the site still loads. Mixpanel is blocked by default in the blacklist that I subscribe to.
- faitswulff 9y ago
  Are those two scripts widespread or specific to a site? And if they're quite widespread, would you be able to share how to identify them?
  - bonestamp2 9y ago
    Just saw one a few minutes ago. It's actually not even the script you need to allow, it's just an XHR request to adiode.com. I can't remember the other one, I'll report back here when I see it again (usually every couple days). I saw the adiode.com one on https://www.merriam-webster.com https://www.merriam-webster.com if you want to have a look for yourself.

wcr3 9y ago

Staff photo shouldn't be in the article.

btown 9y ago

> We immediately began investigating further and learned that the behavior the customer was observing was due to a change to the React JavaScript library made in March 2017. This change placed copies of the values of hidden and password fields into the input elements’ attributes, which Autotrack then inadvertently received. Upon investigating further, we realized that, because of the way we had implemented Autotrack when it launched in August 2016, this could happen in other scenarios where browser plugins (such as the 1Password password manager) and website frameworks place sensitive data into form element attributes. Here's the commit in question: https://github.com/mixpanel/mixpanel-js/commit/98a1845c5c55f8b362847517860cc97c35f1c08e https://github.com/mixpanel/mixpanel-js/commit/98a1845c5c55f... - as referenced here: https://github.com/mixpanel/mixpanel-js/issues/164 https://github.com/mixpanel/mixpanel-js/issues/164 . The meat of it is that the check of the type of the <input> node to ensure it's not "password" or "hidden," now wraps the iteration over attributes as well as inclusion of the value. That heuristic, though, is far from perfect: see, for instance, https://www.troyhunt.com/bypassing-browser-security-warnings-with-pseudo-password-fields/ https://www.troyhunt.com/bypassing-browser-security-warnings... . And even if you're not doing something funky like that, you're not out of the woods. For instance, it's highly likely that a site collecting "secret answers" in plaintext input fields (for password resets) would leak that information to Mixpanel if Autotrack were turned on. This commit does nothing to change that scenario. And while Autotrack is now opt-in, it can be done entirely by a business team with zero interactions with engineering, if (for example) a tag management solution is set up: see https://mixpanel.com/blog/2015/03/27/community-tip-implementing-mixpanel-via-google-tag-manager/ https://mixpanel.com/blog/2015/03/27/community-tip-implement... and https://help.mixpanel.com/hc/en-us/articles/115004613366-What-is-Autotrack-and-how-do-I-get-started-using-it- https://help.mixpanel.com/hc/en-us/articles/115004613366-Wha... SaaS analytics companies, especially those able to slurp up everything into a managed data lake and provide perfect retroactive analysis capabilities, provide a great experience compared to self-hosted solutions, but it's inevitable that you'll end up giving them more information than you'd originally planned. For many businesses, that's the right decision, but it should be one made with eyes wide open.

colmvp 9y ago

On a side note, Mixpanel used to host conferences but stopped after 2014. Anyone know why?

dylanhassinger 9y ago

sounds to me like the problem was with React. which is owned by Facebook. Mark Z helping out the hackers and bots once again

BuildTheRobots 9y ago

Call me thick and naive if you will, but how does an advert manage to hoover up all typed inputs on a page including (somehow accidentally) passwords. Isn't the underlying problem with the web browser that allows this, or are browsers just fundamentally broken when it comes to protecting users?

TeMPOraL 9y ago
Analytics analyze. They read data. Mixpanel decided to read more than they should, and they unintentionally slurped some passwords. The core technical problem is that scripts can read data. But that's kind of their job. "Solving" that would require to reduce the functionality of webpages. The core social problem is that people include third-party analytics and advertisements. Solving that would require to destroy the entire adtech industry. I'm all for the latter solution.
- bluesign 9y ago
  Where you will draw the line, then main ad industries will start to serve ads as 1st party as custom subdomains. There is no way to win this fight except regulations and oversight
  - TeMPOraL 9y ago
    Yes, I strongly agree. That's why I can't wait for GDPR to come to life this May.
  - chopin 9y ago
    In this case, they can be made fully responsible. You'd better vet the Javascript you're going to deliver.

NicoJuicy 9y ago

A lot of people talk here about GDPR, but does anyone know how to "handle" this in my ecommerce? I haven't found any indication on how to do it "legally" or a small "howto". I'm also not interested in an IBM solution that would cost me 10 times my earnings

jmickey 9y ago
In essence the GDPR requires you to be accountable about all personal data you collect and use in your daily operations. Compiling a list of all personal data currently located in your systems and making sure you have a legal basis for each item, goes a long way towards compliance (though of course this is not all you need to do)
- NicoJuicy 9y ago
  So, let's say: - WooCommerce on url xxx: phone number, email,name & address. Google Analytics & facebook Pixel. Requirement for e-commerce fullfillment and analysing website performance / ad performance. - Mailchimp : email and name, when accepting WooCommerce "Terms and conditions" n°2. Requirement for recurring ecommerce updates/changes of new products. - OpenERP: ( invoicing - local network) - Firstname, lastname, address, email, phone, orders. Requirement for invoicing Somehow i can't believe that would be sufficient.
  - ryanwaggoner 9y ago
    It’s not. GDPR is so overly broad that it’s almost impossible to be in full compliance. Not only does it count IP addresses as personal information, but it covers all EU residents, apparently no matter where they are in the world, even if you have no way of knowing they’re EU residents. I really doubt most small-medium businesses without ties to the EU are going to pay any attention. Just like VAT actually.
    - jmickey 9y ago
      Mind you an IP address is personal data only if it identifies an individual. Same goes with any other information.