3 ms·
> meanwhile Amazon generates petabytes of incredibled detailed and realtime sales data The problem is that you can't use Amazon's sales data to answer research
by nequo 4y ago
> meanwhile Amazon generates petabytes of incredibled detailed and realtime sales data
The problem is that you can't use Amazon's sales data to answer research questions about maternity leave policies, UBI, or the criminal justice system. All of that data is great but it benefits a relatively small part of social science.
- eigenrobot 4y agothis deserves its own post, but the other advantage working with tech datasets on the regular is developing perspective on the methodological shortcomings of what's done at universities. what i mean is that once i started running product experiments at scale in really hyperoptimized environments, my sense of what's realistically possible with small data sets collected by humans describing meatspace circumstances contracted fairly dramatically. I don't just mean replication crisis issues, although those are real; im instead skeptical of nearly all of the empirical work being done in nearly every social science field. that is, im not arguing that tech does a great job facilitating research into eg maternity leave policies. rather, im saying i dont think anyone is learning much about them at all. im happy to acknowledge that this is a minority view.
- nequo 4y agoAre you mainly concerned about the quality of the data that is collected, or the econometric methods that are applied to them, or the way in which respondents are sampled? I think that the direction that is promising is combining data sets like those that are built up at Amazon with administrative data that come from other sources. It is a combination of such data sets that opens up answers to important research questions. But I am doubtful that Amazon wants to fund the pursuit of such questions. That is not what will increase shareholder value.