7 ms·

Not a researcher, but I built a data governance system for sharing medical research data. When interviewing people, one thing that surprised me initially (but w

• xyzzyz 4y ago
  Most researchers are not super smart, and so they cannot get out of the data more than is relatively obvious. If they work hard to obtain the data, but then share it freely, someone else will just steal their lunch.
• yawnxyz 4y ago
  I'm an RSE building such a tool right now for a clinical trial, and came in knowing this is the norm. We're experimenting with building a data collection system that uses adversarial/multiplayer game mechanics incentivizes researchers to share (gain co-authorship) and excludes them from the benefits (e.g. co-authorship) if the minimum submitted data threshold isn't met. Still early days, but we figured this is the only way to get people to share data, even within other people within the same project / lab. Similarly, I'm getting a lot of ideas from HN on event-driven data collection and immutable data storage to help make the data more tamper proof.
  • Neurocynic 4y ago
    I would love to talk to you about this. I am currently doing something similar but accentuating the dataset by adding info on publications etc. thereby making them more "legit" (for the lack of a better word)
  • jrochkind1 4y ago
    This is really interesting! I'm interested in knowing more and sharing with colleagues in academic library/publishing tech, if you want to drop a link! (and are interested in sharing your work :) )
  • deleted 4y ago
    [deleted]
• axg11 4y ago
  As a former (PhD) researcher, this is spot on. Here are the factors I've observed: - fear of getting scooped: there's a general fear that sharing data makes it easier for competing labs to publish on the same topics, leading to the publishing lab "missing out" on publications - fear of being wrong: sharing the data leads to more scrutiny, leading to a higher chance of a fatal flaw being found - extra work: publishing data often involves jumping through extra hoops with little to no personal gain
  • Hermitian909 4y ago
    Not a researcher, but have worked closely with them on data sharing. I think there's a variant of getting scooped worth pointing out, which is that in some fields people are scared of being scooped by a low quality lab that fakes the results. I had a very senior materials science researcher describe what he thought was a promising line of study get shut down because a low quality lab in china scooped him with results good enough for publication but not good enough that people would want to replicate. He was 100% confident their data was faked but getting grant money once that experiment was done was like pulling teeth and so he abandoned the route for other ideas.
    • axg11 4y ago
      This is interesting. Not something that I ever came across but can very much believe this happens. Most problems in academia come down to bad incentives. Researchers are incentivised to publish for prestige and citations, which are poor proxies for improving our understanding of the world around us.
      • alexb_ 4y ago
        All problems, and really all actions, can be explained by incentives. Once you understand that people will never under any circumstances do something that both takes effort and is opposite to their incentives, you understand why nearly every problem occurs and why every action is taken.
        • 4y ago
• jltsiren 4y ago
  It's about the incentives, not the mentality. Collecting the data is often the bulk of the work, but the rewards mostly come from publishing papers that make discoveries from the data. If you want to base science on experiments that are properly planned and executed, the scientists doing that work must get the recognition they need to advance their careers.
  • Helmut10001 4y ago
    Cannot agree more. My experience: 50 to 60% of experiments are collecting data, 20 % doing the actual experiment (data processing) and 20% writing the paper (and going through peer review).
    • kilburn 4y ago
      Now think about what "collecting the data" means in the medical field: - Preparing protocols and getting them through ethical review committees - Finding suitable patients and getting them on board - 1 on 1 patient consultations, oftentimes multiple of those for each patient and spread through a few years. I'd say the balance can easily turn to 90% data collection, 10% everything else in many cases...
      • MichaelZuo 4y ago
        I don't see how there's any way 90% of the recognition and reward of published research can go to the data collectors over everybody else combined, even in the cases where data collection is really that onerous. Which I guess is why all efforts to try to split up the work in a more efficient manner have failed and the current model persists. This means that open academic research effectively remains a cottage industry compared to industrial, military, and other research types that can be kept secret.
  • nerdponx 4y ago
    This is a great suggestion. Data collection is itself a (sometimes big) contribution to science and humanity's state of knowledge, and we should reward it accordingly.
• hooloovoo_zoo 4y ago
  Interestingly, one of the best ways to rack up citations in ML is to publish a benchmark dataset.
• gww 4y ago
  I can't speak for all research datasets but I can speak for sequencing based datasets such as whole genome sequencing, RNA sequencing etc. These projects can cost hundreds of thousands to millions of dollars. Academic and industrial research projects tend to do everything they can to avoid sharing this data for as long as possible. The academic researchers that generate these datasets and their funding sources such as granting agencies and donors want to see the money being used to generate something important. In this case it certainly in the researchers interest to avoid sharing the data for as long as possible to get as many publications as possible. This makes it easier to get more funding. There is a push by journals now for researchers to include the data in publicly accessible repositories such as the NCBI SRA or European Genome-phenome Archive. These sites archive the data and make it available for researchers. However, in most cases they require a comprehensive data sharing agreement between your institution and theirs. Most of these agreements have extremely demanding requirements that make it very difficult if not impossible for a requesting institutes legal team to agree to. For example, the institute that owns the data may make demands such as "the right to approve any publications or research work created with this data", which entails sending any publication to them prior to submission and giving them veto power. I understand the reasoning for the agreements their intent was to protect private information from being shared publicly while allowing researchers to use the data. But I think the system is being abused now. On the other hand, it can be very difficult to get past institutes to make data public. For example, sequencing data contains personally identifiable information and getting approval to share the raw data can be difficult. For prospective studies, Human patients need to consent to their data being shared and may not consent to it being shared publicly. I can't comment on industry as much. However, I have colleagues who work at companies and publish numerous papers on the same proprietary datasets but refuse to share it with anyone. This is particularly challenging in fields such as cancer research where they may publish some fancy/superior new model for disease risk stratification on their own data but without sharing it's impossible for researchers to independently validate it.
• Neurocynic 4y ago
  Oh I would love to talk to you about this. I am working on building a similar system and encountered this so much and other issues. Is there a way I could contact you?
• pabs3 4y ago
  Perhaps the incentives need to be re-aligned to separate the collection of data and the publishing of studies based on that data.
• Fomite 4y ago
  Am a researcher, and also work in data governance, data sharing, etc. for medical research data. It's not about an "open source mentality" - almost all of my analytical code is online, and when I work with simulated data sets (the bulk of my work) it's freely available. It's about incentives, and effort. A couple examples: 1) Primary data collection is hugely time and resource intensive. It's hard to be okay with all of that, getting a single paper out, and then immediately having that be expected to be freely open and available. The argument that it's necessary for reproducibility is a compelling one, but also one that's much rare than what most people are looking for, which is to do new and novel research using your primary data. 2) Research has a potentially long time from data collection to publication. It's entirely possibly the reason isn't "I may want to work on that someday" but "A graduate student is working on that right now". I have been in the position of potentially having a graduate student be scooped on something, and it is terrible (it ended up being a false alarm). They're potentially also the person who put in a lot of the labor on publishing the paper in the first place. 3) Sharing comes with an unknown expectation of support, documentation, etc. wherein there's absolutely no incentive structure in place to support the work that takes. I write in time and budget to do so in my grants, but that also mean my grants have less science in them than other people's, and I have to rely on that mattering to someone enough to make up the difference. 4) Genuine data ownership issues. Health data is especially complex in this space. Do you want a simulated data set that will give you the same broad patterns? That's easy enough. But if you want the real data - that's potentially a very complex issue wherein even if I say "Sure, let's get that process started..." you're potentially looking at a year or more before you get the data.
  • mattkrause 4y ago
    This x100. I'd also add that primary data collection is risky. You can spend months to years working to get data, only to end up with nothing---not even a null result---if the experiment itself goes south. I lost over a year of my PhD when an animal I had spent ages training fell while playing. A classmate had a virus wipe out her colony of genetically-engineered mice. Using someone else's data obviates these sorts of risks--while doing a better job preparing you for non-academic careers too! As a result, I think it's not totally nuts for the original data collectors to maybe get some kind of "risk premium." I can imagine a lot of ways to do it, but early/exclusive access to data has been one of them.
• BackBlast 4y ago
  Sounds like it's time to split the industry. One side collects the data and puts it into common repositories. Another group does research using the primary data.
  • speedgoose 4y ago
    That sounds very inefficient though.
• radu_floricica 4y ago
  Couldn't this be fixed by having the creator of the data as a co-author on any paper that uses the data? The basic conflict is that it's hard/expensive/time-consuming to make data, and easier to write papers on existing data sets. So just split the work, and make it a legitimate occupation to create good data sets. As of now, generating the data by itself has no benefits for scientists, other than the papers they can subsequently write using it. Feels a bit like having to make a screwdriver every time you want to work on your car.
  • IanCal 4y ago
    A better solution is to properly credit researchers for releasing data that then gets used by caring about citations to data. That needs to be tracked better, things are improving but until there are metrics and tracking as a common part of evaluation I don't think it will change.
  • ISL 4y ago
    As a sibling says, it is probably better to improve citation and culture around the importance of datasets. As an experimentalist, I really want people to make use of my measurements, but I really don't want to be a co-author on a paper that makes wild and incorrect conclusions using my data.
• UncleMeat 4y ago
  I've heard one even worse. I know people who were told by their PI to tell other researchers at conferences that they were working on the stuff that they already tried that didn't work. This would encourage competing labs to try out these techniques and waste their time. Being scooped is a huge blow to a career, for faculty but more intensely for graduate students. With arXiv, you can be scooped at literally any moment rather than just on the known conference schedule. Being scooped was personally traumatic for me and I still have nightmares about the experience. This leads to a massively toxic ecosystem of over-competition where individual labs win at the detriment of scientific progress.
• xhkkffbf 4y ago
  It's not just publishing it before them, it's opening themselves up to criticism. If someone wants to look for mistakes or flaws in your data, it's easier with access. Of course, arguing about the right answer is supposed to be what science is about. That doesn't make it any more pleasant.