6 ms·

Common research methods have significant limitations. Web crawls, for instance, usually don't realistically simulate user activity and experiences. Lab studies

Common research methods have significant limitations. Web crawls, for instance, usually don't realistically simulate user activity and experiences. Lab studies often involve simplified systems that don't generalize to the real world. Surveys yield self-reported data, which can be very unreliable. Rally studies, by contrast, reflect real-world user activity and experiences. In science jargon, Rally enables field studies and intervention experiments with excellent ecological validity.

ysavir 5y ago
Thanks for clarifying! Makes sense. A few follow up questions: 1. Do you expect the opt-in nature of these studies to impact their findings? 2. To compensate for the voluntary nature of the studies, do you think researchers in general will still be incentivized to find data sources that are less respectful of people's privacy and don't require an opt-in to the study?
- jonathanmayer 5y ago
  > 1. Do you expect the opt-in nature of these studies to impact their findings? The Rally participant population is not representative of the U.S. population—these are users who run Firefox (other browsers coming soon), choose to join Rally, and choose to join a study. In research jargon, there's significant sampling bias. For some studies, that's OK, because the research doesn't depend on a representative sample. For other studies, researchers can approximate U.S. population demographics. When a user joins Rally, they can optionally provide demographic information. Researchers can then use the demographics with reweighting, matching, subsampling, and similar methods to approximate a representative population. Those methods already appear throughout social science; whether they're sufficient also depends on the study. > 2. To compensate for the voluntary nature of the studies, do you think researchers in general will still be incentivized to find data sources that are less respectful of people's privacy and don't require an opt-in to the study? Rally is designed to provide a new research capability that didn't exist before. I don't expect a substitution effect like that.
  - ysavir 5y ago
    Got it. Thanks Jonathan!
- cycomanic 5y ago
  Regarding 2. that would run afoul of many ethics boards at universities. Generally they require that (informed) consent has been given to take part in the study.
cpeterso 5y ago
> Rally studies, by contrast, reflect real-world user activity and experiences. In science jargon, Rally enables field studies and intervention experiments with excellent ecological validity. Rally users are all opt-in. How does that impact the design of a Rally study and the conclusions you can draw from it?
- 9wzYQbTYsAIc 5y ago
  Academic research in the social sciences is rigorously based on the concept of informed consent (i.e., opt-in), in the first place. There would be no change in terms of research design and the ability to draw scientific conclusions. edit: also, see https://news.ycombinator.com/item?id=27633212 https://news.ycombinator.com/item?id=27633212 for details on research design considerations when conducting social science.
  - analognoise 5y ago
    Except as noted elsewhere, Mozilla also gets the data to "improve products and services" right? So it sounds like a nice shiny cloak for...exactly the kind of data collection nobody actually likes. Yay for extra steps?
- yellowfish 5y ago
  Mozilla has been known to be pretty iffy when it comes to 'opt in' ( the mr. robot tie in .. etc )
  - Nicksil 5y ago
    >Mozilla has been known to be pretty iffy when it comes to 'opt in' ( the mr. robot tie in .. etc ) Did the instance you're referencing state it was opt-in then turn out to not be opt-in?

ysavir 5y ago

Thanks for clarifying! Makes sense. A few follow up questions: 1. Do you expect the opt-in nature of these studies to impact their findings? 2. To compensate for the voluntary nature of the studies, do you think researchers in general will still be incentivized to find data sources that are less respectful of people's privacy and don't require an opt-in to the study?

jonathanmayer 5y ago
> 1. Do you expect the opt-in nature of these studies to impact their findings? The Rally participant population is not representative of the U.S. population—these are users who run Firefox (other browsers coming soon), choose to join Rally, and choose to join a study. In research jargon, there's significant sampling bias. For some studies, that's OK, because the research doesn't depend on a representative sample. For other studies, researchers can approximate U.S. population demographics. When a user joins Rally, they can optionally provide demographic information. Researchers can then use the demographics with reweighting, matching, subsampling, and similar methods to approximate a representative population. Those methods already appear throughout social science; whether they're sufficient also depends on the study. > 2. To compensate for the voluntary nature of the studies, do you think researchers in general will still be incentivized to find data sources that are less respectful of people's privacy and don't require an opt-in to the study? Rally is designed to provide a new research capability that didn't exist before. I don't expect a substitution effect like that.
- ysavir 5y ago
  Got it. Thanks Jonathan!
cycomanic 5y ago
Regarding 2. that would run afoul of many ethics boards at universities. Generally they require that (informed) consent has been given to take part in the study.

cpeterso 5y ago

> Rally studies, by contrast, reflect real-world user activity and experiences. In science jargon, Rally enables field studies and intervention experiments with excellent ecological validity. Rally users are all opt-in. How does that impact the design of a Rally study and the conclusions you can draw from it?

9wzYQbTYsAIc 5y ago
Academic research in the social sciences is rigorously based on the concept of informed consent (i.e., opt-in), in the first place. There would be no change in terms of research design and the ability to draw scientific conclusions. edit: also, see https://news.ycombinator.com/item?id=27633212 https://news.ycombinator.com/item?id=27633212 for details on research design considerations when conducting social science.
- analognoise 5y ago
  Except as noted elsewhere, Mozilla also gets the data to "improve products and services" right? So it sounds like a nice shiny cloak for...exactly the kind of data collection nobody actually likes. Yay for extra steps?
yellowfish 5y ago
Mozilla has been known to be pretty iffy when it comes to 'opt in' ( the mr. robot tie in .. etc )
- Nicksil 5y ago
  >Mozilla has been known to be pretty iffy when it comes to 'opt in' ( the mr. robot tie in .. etc ) Did the instance you're referencing state it was opt-in then turn out to not be opt-in?