3 ms·

I have to confess I didn't see the link to the whitepaper in my first read through - shame on me for commenting before I did :) I think you all did a pretty goo

by sjbase 10y ago

I have to confess I didn't see the link to the whitepaper in my first read through - shame on me for commenting before I did :) I think you all did a pretty good job putting this together overall after reading through it, and I applaud the effort. That said, the critical HNer in me sees some bias in the questions and answers. Specifics: * The results on page 17 jumped out at me the most. There are 4 specific options for negative emotional reactions, and one generic positive option. It's easy to underestimate the powerful effect this has on pushing the responder to a specific answer. Imagine if your answer options were only "content, secure, informed, protected, neutral, misled" and how different your results might be. * The question for the above, specifically the tone of "[...] does not offer any additional protection [...]" is also a form of leading the responder to an answer. While the statement is mostly true, there's some contention on the point esp. regarding cookies (see the other comments on this thread). Either way, you'd get a more honest answer if you strictly list the things private browsing does, and the things it does not do in a way that's as objective and incontrovertible as possible. * It would be useful to see the benefits underlying the "correctly identified" category on page 13. I couldn't find those anywhere which was a bit suspicious. In consulting, I used to survey users on their personal security practices (password strength, adherence to policies, secure disposal, etc.) and gained a huge appreciation for how powerful bias and misinterpretation can be in surveys, and how hard it is to control. But to reiterate, I think this is overall pretty well done and has some really interesting results, esp. the usage data in the first few sections.

AdamSC1 10y ago
Great questions and valid points. Every bit of data should be taken with a great of salt as no study is perfect. We've done the best possible to be statistically sound and disclose anything that may bias that. Initially we started this survey out of our own interest so a few of those decisions are rooted from there. 1) Agreed, however we've already divided out the people who previously learned about private browsing. This leaves us with a cohort of users who have used private browsing, thought it protected them more and learned that it was wrong. It's very unlikely in any case that there would be positive emotions experienced. In our background research before the final survey we were only focusing on if users had a positive or negative experience. In this final major survey we wanted to get a better understanding of what those negative emotions were and so teased it out more. I agree that a skewed set of options can bias a question but it is case-by-case. Example if I ask "how do you feel when your ice cream falls on the floor?" and the options are "Happy, Joyful, Amazing, Wonderful, Excited, Sad" you'll still likely pick sad. 2) A good point. This question is one that is linguistically challenging. We want to use phrases like "it only does X" to stress limits, but those can be misinterpreted. I agree you may see some shift if you ask the question as a feature list, but I think in the question we're looking at it would still fall well within the margin of error. I'm curious to run some tests on it either way! 3) We didn't release those in the whitepaper as they were broken down by browser since each browser has different feature sets. We wanted to keep this report agnostic of browser. We're not looking to shame anyone, we're trying to support the conversation around the education gap for the average non-technical internet user. They are telling us they want more privacy, and don't feel that it is being provided or explained well enough. Overall, I really appreciate the feedback - it is more important than ever to question and discuss the things we see online. Hope this sheds some light on why we made the various decisions we did! :)

by sjbase 10y ago

AdamSC1 10y ago

Great questions and valid points. Every bit of data should be taken with a great of salt as no study is perfect. We've done the best possible to be statistically sound and disclose anything that may bias that. Initially we started this survey out of our own interest so a few of those decisions are rooted from there. 1) Agreed, however we've already divided out the people who previously learned about private browsing. This leaves us with a cohort of users who have used private browsing, thought it protected them more and learned that it was wrong. It's very unlikely in any case that there would be positive emotions experienced. In our background research before the final survey we were only focusing on if users had a positive or negative experience. In this final major survey we wanted to get a better understanding of what those negative emotions were and so teased it out more. I agree that a skewed set of options can bias a question but it is case-by-case. Example if I ask "how do you feel when your ice cream falls on the floor?" and the options are "Happy, Joyful, Amazing, Wonderful, Excited, Sad" you'll still likely pick sad. 2) A good point. This question is one that is linguistically challenging. We want to use phrases like "it only does X" to stress limits, but those can be misinterpreted. I agree you may see some shift if you ask the question as a feature list, but I think in the question we're looking at it would still fall well within the margin of error. I'm curious to run some tests on it either way! 3) We didn't release those in the whitepaper as they were broken down by browser since each browser has different feature sets. We wanted to keep this report agnostic of browser. We're not looking to shame anyone, we're trying to support the conversation around the education gap for the average non-technical internet user. They are telling us they want more privacy, and don't feel that it is being provided or explained well enough. Overall, I really appreciate the feedback - it is more important than ever to question and discuss the things we see online. Hope this sheds some light on why we made the various decisions we did! :)