12 ms·

Many researchers were not compliant with their published data sharing statement

205 points by miohtama 4y ago

joebob42 4y ago
Why do researchers add these statements in the first place if they are pretty broadly not complying with them? Does it somehow help them get published just to have the statement there?
- Isamu 4y ago
  Many researchers are just cranking out papers to meet some metric, and a Data Availability Statement (DAS) is just a bit of boilerplate to make your paper seem more plausible. I think this paper shows that it is just more research theater, meant to distract.
- nneonneo 4y ago
  Some journals make these statements mandatory; for others, not promising to share your data can be a significant obstacle to publication (e.g. reviewers may ask why the data is not being made available).
- joshvm 4y ago
  Journals are increasingly requesting that authors provide mandatory data availability statements. It doesn't necessarily mean that the data should be open, just that the authors have to declare if it is or not. Trouble is it's very difficult to prove a statement like "We intend to release our code and data as open source at <repository>", unless you hard-enforce that things are provided at publication time (and then you're assuming the reviewers will actually check, compile, etc.)
- frou_dh 4y ago
  There's an overused term, but in this case it could quite literally be classed as virtue-signalling?
pythonguython 4y ago
I only have a bachelor’s degree, but I’ve worked in research labs and read a lot of papers. I’ve always thought it was silly that all data, design documents, and code is not published as a standard. Is it just protocol from a time when it was more difficult to share data and designs? In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code?
- bsder 4y ago
  > In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code? Because the incentive in academia is to take a single idea that kinda works and parcel it out into as many, generally low quality, published papers as you can. The moment you release the data somebody else can start parceling out those papers instead of you.
- DantesKite 4y ago
  Makes me curious whether researchers would use software that integrates all these features for them automatically, making it easy to publish and share data sets in a consistent format online.
  - anitil 4y ago
    This is a niche that datasette (by simonw on here) could fill very nicely. If you can wrangle it in to a csv, then datasette can publish it as an api with an sql interface. I'm not sure how this could work with things like photos etc (though there are plugins for some of this)
  - gpm 4y ago
    I think you've got to be asking - What benefit does this give to the researchers who are publishing the data? - Who is paying for storing the data - frequently in the TB? When the answers are "approximately none", and "the researchers" you're not going to get many takers. And that's assuming it's easy, it reality it often won't be. If there's a lot of data, it's literally just a pain to upload it (network bandwidth). If there's sensitive data (PII), it's a pain to redact it and make sure you aren't leaking any. Data is frequently in strange formats, and it's a pain to translate it to a standard one. Etc. --- I've worked with 3 university labs as a contract programmer. In all of them I worked with data with one of the issues mentioned above. Health information in one, TBs of photonics data in another (which was being parsed by extremely janky code too), and 4 16-bit channel images in the last (hundreds of GB of them too). Admittedly for the last it would have been easy to upload it as long as you didn't want people to be able to actually view it (on the other hand I wrote some software to let people false color them live in a browser for the lab, so that the researchers could view them).
frozenport 4y ago
An author threatened to got to the editor and then my PI became compliant.
karaterobot 4y ago
Not a researcher, but I built a data governance system for sharing medical research data. When interviewing people, one thing that surprised me initially (but which makes total sense in retrospect) is that most researchers do not want to share their data. They want to keep it proprietary until they have published as much as possible on it. In many cases, their funders require the data to be shared, so they go through the motions, but if you give them a chance they will almost always try to deny access to anybody else. They're very afraid of someone coming along and publishing on their data before they do. It's very much not an open source mentality. Everybody wants other scientists to share their data, but nobody wants to share their own.
- xyzzyz 4y ago
  Most researchers are not super smart, and so they cannot get out of the data more than is relatively obvious. If they work hard to obtain the data, but then share it freely, someone else will just steal their lunch.
- yawnxyz 4y ago
  I'm an RSE building such a tool right now for a clinical trial, and came in knowing this is the norm. We're experimenting with building a data collection system that uses adversarial/multiplayer game mechanics incentivizes researchers to share (gain co-authorship) and excludes them from the benefits (e.g. co-authorship) if the minimum submitted data threshold isn't met. Still early days, but we figured this is the only way to get people to share data, even within other people within the same project / lab. Similarly, I'm getting a lot of ideas from HN on event-driven data collection and immutable data storage to help make the data more tamper proof.
  - Neurocynic 4y ago
    I would love to talk to you about this. I am currently doing something similar but accentuating the dataset by adding info on publications etc. thereby making them more "legit" (for the lack of a better word)
  - jrochkind1 4y ago
haihaibye 4y ago
I work as a bioinformatician (programmer in genomics), a typical paper may involve a dozen sequencing samples (~25G each), and a few thousand lines of scripts to produce CSVs/graphs When I started my lab head got a request for an old paper (before my time) and I had to hunt down the data, put it on the web, find the code, send an email etc - it was a lot of work, almost as much as just sharing it in the first place. So now all the code goes up on GitHub, all the sequencing reads to public archives, and we fill the supps full of any intermediate CSVs etc we think people might find useful. I saw it as "people bothering me so I would do admin" but some lab heads see being requested as "strategic networking and being informed about other research in the field" where they can see what others are doing and gain collaborations / be put onto papers etc I guess it's just expectations: I've met surgeons who have the expectation that because they kept a piece of tumor from a surgery and gave it to a lab, they will be on any paper that ever uses data from that tissue until the end of time (even on labs on the other side of the world re-using the same data). When I'm collecting public data, I usually don't bother to ask due to the hassle, and move straight on to the actually public stuff. My bet is that papers with public code/data are used/cited way more often, but the tradeoff is probably their authors get less total papers over all. Do you want a larger number of people slightly happy with you or a fewer number of "close friends" in your area who may look kindly on you for grants or paper reviews.
- aeonik 4y ago
  I think paper citations are an archaic method of tracking contributions to research. We have all this technology, and Git is a great example, that allows us to more granularly give credit to contributors. In fact, even the very idea of a published paper feels out of date. Why have one model that is set in stone that is published as one snapshot in time? In the DevOps and Cybersecurity world our alerting mechanisms are essentially simple data models (sometimes not so simple) that are tapped into the raw data feeds, and can be updated and refined on an ongoing basis. While instrumenting the physical world can have more challenges, I still feel that the sciences have a lot to be gained from some of the more "pedestrian" operational processes that we use in the technology industry.
vedtopkar 4y ago
I'll echo that it's a tremendous amount of work to appropriately format data and metadata for upload to a public repository. It's not as simple as just mass uploading raw data files. It's no surprise that people skirt around this requirement when possible, especially as the size of files and number of samples per paper continues to balloon. That's not to say that it's not important, but the labor required is vastly underappreciated. I say this as someone who has completed the task for many papers.
- Gatsky 4y ago
  Yeah this is the main reason, as anyone actually doing research knows. I don’t think civillians realise what an incredible amount of work it is to publish a paper.
  - jonnycomputer 4y ago
    yeah. holy hell is it a lot of work. hate doing it even when I know I'll get a paper out of it.
  - Fomite 4y ago
    Hell, even things like journals requiring formatting after acceptance being taken as a godsend.
- TaylorAlexander 4y ago
  Is there some way to work this in to the system? Like by having a different person do this work? At a very high level, all of science would benefit from more data being properly formatted and published, so maybe this is something that could be budgeted for in the planning phase? Or the work could be done by undergraduates studying for the field?
  - Fomite 4y ago
    The amount for a non-modular (i.e. the default, you don't have to have a super detailed budget) NIH R01, which is the basic unit of biomedical science funding, has been stagnant since 1999. Budgeting for a separate person to do this would eat up a tremendous amount of said budget. Which is fine, except if we think about this from a productivity standpoint, the person who budgeted for that person is probably short a graduate student compared to the person who didn't, on the outside chance that someone cares about their stuff. For example, I looked at my lab's publicly available repositories linked to papers on Github. Watches: 15 (probably half of these are people involved in the project) Forks: 3 Stars: 4 Visitors in the Past Month: 4 That's it. For all of them. While I keep doing it based on principle, if I stopped tomorrow, it would impact me not at all. And undergraduates, candidly, are not usually time savers.
javert 4y ago
If I were a billionaire, I'd fund a research institute whose only purpose would be to detect and call out fraud and shoddy work in academia. I'd hire the very best people I could in each field and put them onto this task. I imagine it would be easy to find excellent but disgruntled scientists in every niche. You'd need to hire the best people with niche expertise. I know from my years in academia, nobody can understand and scrutinize your bullshit unless they work in the same sub-sub-sub-field.
- Qem 4y ago
  Just create a glassdoor-like site, but for ex-students tô tell their experiences with research groups and advisors.
  - photochemsyn 4y ago
    That's not going to work very well, you'll get a mixed cohort of people reporting actual fraud and deceptive practices (the desired group) and people who were just pissed off because they didn't get the support they needed (a legitimate complaint, but not as relevant to the issue of fraud) or had negative personal interactions (not such a legitimate complaint, as they might have been the problematic person).
    - Qem 4y ago
      I think the same issues affect Glassdoor, but it didn't stop the site being a hit. Just let people speak, and the audience judge by themselves and filter the outlier report from the problematic person.
- hoseja 4y ago
  You'd probably suffer a fatal accident with chinese characteristics very soon.
  - javert 4y ago
    That made me laugh. Much of the fraudulent work isn't coming from Chinese researchers, but much of it is.
deleted 4y ago
[deleted]
Ansil849 4y ago
Does this paper come with its own Data Availability Statement?
forum_ghost 4y ago
- agucova 4y ago
  They've explicitly justified not sharing it, for good ethical reasons.
  - mike_hearn 4y ago
    The reasons aren't good. If the DAS posted elsewhere in this thread is accurate then they don't want to be seen as "shaming" people, but the behavior they detected is simple promise breaking. The researchers stated they'd share their data, benefited from the aura of respectability that claim created, and later refused. And this behaviour was absolutely widespread: more than 90% of the papers selected didn't provide data as promised. That's the sort of behaviour that actually should be shamed, right? Why shouldn't people know whose word can be trusted and whose shouldn't? It's a very silly form of ethics that defends the honor of those who deliberately lie for profit. It also totally devalues the efforts and integrity of the tiny minority who did do things by the book. The actual reason they refused to share their own data almost certainly has nothing to do with ethics. It's because they don't want to make 3500+ enemies within their field.
    - forum_ghost 4y ago
      >The actual reason they refused to share their own data almost certainly has nothing to do with ethics. It's because they don't want to make 3500+ enemies within their field. This is exactly why peer review is nonsense.
Prcmaker 4y ago
When I was a university researcher, after publishing my first paper in the role I carefully read the policy requiring making data available, as I wanted to be correct by the book, but also have my data (which I took pride in) publicly available for critical review. The process as described was simple, though I soon found there was no mechanism implemented to actually achieve this. URLs and systems no longer existed, no alternatives available. I requested to publish via an external provider and was informed I would be fired if I did so. Disappointing to say the least.
andrewljohnson 4y ago
Tangential to topic, but I have been reading a Newton biography (Never at Rest) recently, and I found it interesting how reliant Newton was on the observations of other scientists. He had correspondence with scientists like Halley and Flamsteed on the basis of getting their observations of comets or observations of the moon, to help him work out orbits. At one point, Newton was unhappy with Flamsteed, because Flamsteed wanted to work on his star map instead of Newton's moon observations, but Flamsteed said he needed an accurate star map to produce accurate moon observations. Data sharing has been an issue in science for hundreds of years.
nomilk 4y ago
The cost-benefit ratio of open sourcing data has been known for long enough now, it's time to stop excusing this practice and start treating studies with unavailable datasets as heresay. When scientists don't publish datasets, it contributes to a culture that allows a small (or not small, depending on who you talk to) portion of scientists can p-hack their way to a publication and ruin the reputation of research for everyone.
xiaodai 4y ago
Some of research are fake as well. I tried to email an author who claimed some extraordinary results with their sorting algorithm and got crickets. Suspect they dont really care beyond having it "published".
gxs 4y ago
This is simple economics. If benefits of non compliance > consequences, don’t comply, else, comply. That simple, and applies to tech companies as well. At the end of the day who’s policing this stuff anyway? People only find out when there’s a leak.
dontbenebby 4y ago
Maybe someone should look them up in popular search engines using simple OSINT techniques and see that else they were reckless about. For many years, scientific computing facilities were "off limits" to honorable hackers, but I've noticed a persistent air of entitlement from the biological sciences, paired with a deep disrespect for the field of information science. I have a bachelor's in information science with a concentrations in information security and related areas in psychology and English literature from one of the highest ranked schools of information science in the world[0], followed by a PhD I left "ABD" after numerous peer reviewed publications and scores of conference talks and other informal lectures and volunteer work. I once saw an entire zoo aquarium's exhibits fog up in my hometown as it switched to brownouts, as a conference's attendees went on to obstruct policy initatives that would prevent the entire state they claim to want to be permanent residents from one day losing power in a similar manner to what happened in Texas.[1] It's why I started one of my pet phrases when I'd do volunteer work: Voting is a right, whereas electricity, like driving, is a privilege. Anyways, sorry for the stern response, but I hate being the guy who just goes "Oh, those results are unsurprising", but some nights I feel like I really need to mansplain what can happen if folks ignore my "crazy" left-libertarian advice. [1] After a decade of warnings, a weak mandate to "winterize" https://www.npr.org/2021/06/02/1002277720/texas-lawmakers-passed-changes-to-prevent-more-blackouts-experts-say-its-not-eno https://www.npr.org/2021/06/02/1002277720/texas-lawmakers-pa...
- Edman274 4y ago
  I don't understand the connection between the zoo's aquarium "fogging up" in your home town because of brownouts and the unwillingness of scientists to share data. I don't understand the connection between scientific computing facilities being unwilling to host "honorable hackers" in "information science" to use their computational power and data being closed. I don't understand what your "stern response" is to. I don't understand why you used the word "mansplain" to describe what you're doing when it has nothing to do with identity politics. I don't know what your "highest ranked school" is. You added it as a footnote but it doesn't seem to appear later in the comment.
  - dontbenebby 4y ago
photochemsyn 4y ago
The only way I'd expect raw datasets to be published online immediately in a useful, curated manner by a research group is if that was stipulated in their research grant, period, i.e. that's what the goal of the project was, at least in part. Now this is not entirely uncommon in many research areas; you can get all the current weather and climate datasets with little difficulty as these are shared globally (as is necessary for effective short-term weather forecasting). Anyone publishing DNA/RNA/protein work generally uploads those sequences to GenBank, and that gets updated every two months, so is pretty current relative to the published literature: https://www.ncbi.nlm.nih.gov/genbank/ https://www.ncbi.nlm.nih.gov/genbank/ However this article seems to focus on "Biomed" and that whole world is where pharmaceutical/corporate influence in academics is off the charts. I'd guess well over half of the researchers (if not over 90%) have some dream of getting their work patented, setting up a biomed startup, and having Pfizer or J&J swoop in and buy them out (they then get a percentage and become wealthy), and their university patent office is likely pushing for that as well (they get another percentage). No way are those people going to share their data, which might very well be slanted or cooked in ways that will make their little entrepreneurial effort more attractive to a big buyer. It's way more like pitching to a VC than doing actual rigorous scientific research in the tradition of open sharing of results and ideas - at least not until the patent is granted. Then you're into whether or not the clinical trial data will be released to the public, and corporations don't like to do that either (as someone might find that their claimed success story is riddled with statistical errors, etc.). Business and science have orthogonal goals, and mixing the two is almost always a recipe for the rise of fraudulent con artists pushing bogus BS in the name of turning a quick profit.
chrismartin 4y ago
Is there irony in asking for other researchers' data (presumably free of charge) and then publishing the results of that effort behind a paywall?
sinab 4y ago
The data availability statement in this article undermines its entire premise. > We did not publish our raw data along with the manuscript because it could be understood that we are publicly shaming authors who did not want to share their data. As for the raw data that were received during this study, we informed our study participants that those raw data will be deleted after being examined and that all data and communication will be treated with strict confidence.
- agucova 4y ago
  There's a big difference between explicitly denying to share data for justified ethical reasons, and saying you will share the data and then failing to follow through. There is absolutely no contradiction here.
  - sinab 4y ago
    The concern of “shaming” is not a justified ethical reason.
type0 4y ago
Some do share it but even then manage to use their own proprietary software to do the analysis, it boggles my mind how such people can receive grants and publish in prestigious journals, corruption - plain and simple.
tgbugs 4y ago
The NIH has mandated that starting in 2023 all grants that it funds must include a plan for data sharing. A serious lingering question is whether there is actually going to be any enforcement, given that it is already a monumental challenge for investigators to even account for the data that has been submitted/shared/archived by their lab, forget some 3rd party trying to do it. The tools for doing this across a whole diverse ecosystem are shall we say not user friendly (if they exist at all), and that is being charitable (and I say this as someone who has written some of those less than user friendly tools). Most of the systems, archives, etc. are not interoperable, they don't share common APIs, much less have some common interface that would allow someone to track all their published data, forget integrate it. Program officers are going to be desperate for ways to account for this stuff, but I worry that at least initially they are not going to have the leverage needed to get researchers to engage in even the most basic data management practices which would allow them to start solving the problem down the line. How many files did you upload to archive X? Which project were they for? How many datasets are you expecting to produce for this grant? You have published 10 papers associated with this grant number, where are the datasets that correspond to those papers? Some investigators can answer these immediately and easily, others will throw a screaming fit that they were even asked for such a thing.
- mike_hearn 4y ago
  The DAT protocol originally came out of this need. It's a very neat data sharing system that's designed to work with huge files, and it also came with a user friendly browser called Beaker Browser. Unfortunately the project seems to have split or evolved into a different project or something. They went through some rebrands, a rewrite, Beaker got abandoned and I lost track of what they're doing now. I think their grant funding ran out. But it was a pretty good attempt at making this stuff easy or easier.
isaacfrond 4y ago
I get that you don't want to share the data. Collection data is most of the work, people can steal it. God forbid people might find your wrong etc. etc. But maybe just maybe you shouldn't say that are going to share the data if you are not. How trustworthy is the rest in the article if your Data Availability Statement (DAS) is a lie?
academia_hack 4y ago
I left academia for industry a few years ago and still get hassled every week or so for help with open source code I published in conjunction with one of my papers. My other papers, where I provided no artifacts, generate an average of zero support emails a year but are cited just as much. The emails are often from first year PhD. students and are vaguely accusatory, along the lines of: "We're having issues replicating experiment X and wanted to give you a chance to correct/review before we publish on it." The cause of the issues is, 100% of the time to date, people not reading documentation or having more than a superficial understanding of the topic. However, I feel compelled to respond so that my old lab doesn't get dragged on twitter or at some conference over something that isn't true. I barely got paid to do this sort of work when I was in academia and certainly don't now. If I could go back in time, I would have kept my research closed source. The personal benefits are essentially zero and the world would get along just fine if I had never written any papers, so the altruistic reasons aren't terribly compelling either.
- bell-cot 4y ago
  If it's really "ever week or so", for years, I'd be tempted to compile data on the support e-mails and their senders. At some point, you've got a meta-paper on the people who contacted you for support, breakdowns of where they are in their careers, what topics they don't seem to have mastered yet, which parts of the docs they failed to read, etc. The fun could start when you let drafts of that meta-paper start circulating.

205 points by miohtama 4y ago

Why do researchers add these statements in the first place if they are pretty broadly not complying with them? Does it somehow help them get published just to have the statement there?

Isamu 4y ago
Many researchers are just cranking out papers to meet some metric, and a Data Availability Statement (DAS) is just a bit of boilerplate to make your paper seem more plausible. I think this paper shows that it is just more research theater, meant to distract.
nneonneo 4y ago
Some journals make these statements mandatory; for others, not promising to share your data can be a significant obstacle to publication (e.g. reviewers may ask why the data is not being made available).
joshvm 4y ago
Journals are increasingly requesting that authors provide mandatory data availability statements. It doesn't necessarily mean that the data should be open, just that the authors have to declare if it is or not. Trouble is it's very difficult to prove a statement like "We intend to release our code and data as open source at <repository>", unless you hard-enforce that things are provided at publication time (and then you're assuming the reviewers will actually check, compile, etc.)
frou_dh 4y ago
There's an overused term, but in this case it could quite literally be classed as virtue-signalling?

pythonguython 4y ago

I only have a bachelor’s degree, but I’ve worked in research labs and read a lot of papers. I’ve always thought it was silly that all data, design documents, and code is not published as a standard. Is it just protocol from a time when it was more difficult to share data and designs? In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code?

bsder 4y ago
> In the modern era, why shouldn’t research colleges and journals maintain a digital database of data, design documents, and code? Because the incentive in academia is to take a single idea that kinda works and parcel it out into as many, generally low quality, published papers as you can. The moment you release the data somebody else can start parceling out those papers instead of you.
DantesKite 4y ago
Makes me curious whether researchers would use software that integrates all these features for them automatically, making it easy to publish and share data sets in a consistent format online.
- anitil 4y ago
  This is a niche that datasette (by simonw on here) could fill very nicely. If you can wrangle it in to a csv, then datasette can publish it as an api with an sql interface. I'm not sure how this could work with things like photos etc (though there are plugins for some of this)
- gpm 4y ago
  I think you've got to be asking - What benefit does this give to the researchers who are publishing the data? - Who is paying for storing the data - frequently in the TB? When the answers are "approximately none", and "the researchers" you're not going to get many takers. And that's assuming it's easy, it reality it often won't be. If there's a lot of data, it's literally just a pain to upload it (network bandwidth). If there's sensitive data (PII), it's a pain to redact it and make sure you aren't leaking any. Data is frequently in strange formats, and it's a pain to translate it to a standard one. Etc. --- I've worked with 3 university labs as a contract programmer. In all of them I worked with data with one of the issues mentioned above. Health information in one, TBs of photonics data in another (which was being parsed by extremely janky code too), and 4 16-bit channel images in the last (hundreds of GB of them too). Admittedly for the last it would have been easy to upload it as long as you didn't want people to be able to actually view it (on the other hand I wrote some software to let people false color them live in a browser for the lab, so that the researchers could view them).

frozenport 4y ago

An author threatened to got to the editor and then my PI became compliant.

karaterobot 4y ago

Not a researcher, but I built a data governance system for sharing medical research data. When interviewing people, one thing that surprised me initially (but which makes total sense in retrospect) is that most researchers do not want to share their data. They want to keep it proprietary until they have published as much as possible on it. In many cases, their funders require the data to be shared, so they go through the motions, but if you give them a chance they will almost always try to deny access to anybody else. They're very afraid of someone coming along and publishing on their data before they do. It's very much not an open source mentality. Everybody wants other scientists to share their data, but nobody wants to share their own.

xyzzyz 4y ago
Most researchers are not super smart, and so they cannot get out of the data more than is relatively obvious. If they work hard to obtain the data, but then share it freely, someone else will just steal their lunch.
yawnxyz 4y ago
I'm an RSE building such a tool right now for a clinical trial, and came in knowing this is the norm. We're experimenting with building a data collection system that uses adversarial/multiplayer game mechanics incentivizes researchers to share (gain co-authorship) and excludes them from the benefits (e.g. co-authorship) if the minimum submitted data threshold isn't met. Still early days, but we figured this is the only way to get people to share data, even within other people within the same project / lab. Similarly, I'm getting a lot of ideas from HN on event-driven data collection and immutable data storage to help make the data more tamper proof.
- Neurocynic 4y ago
  I would love to talk to you about this. I am currently doing something similar but accentuating the dataset by adding info on publications etc. thereby making them more "legit" (for the lack of a better word)
- jrochkind1 4y ago

haihaibye 4y ago

I work as a bioinformatician (programmer in genomics), a typical paper may involve a dozen sequencing samples (~25G each), and a few thousand lines of scripts to produce CSVs/graphs When I started my lab head got a request for an old paper (before my time) and I had to hunt down the data, put it on the web, find the code, send an email etc - it was a lot of work, almost as much as just sharing it in the first place. So now all the code goes up on GitHub, all the sequencing reads to public archives, and we fill the supps full of any intermediate CSVs etc we think people might find useful. I saw it as "people bothering me so I would do admin" but some lab heads see being requested as "strategic networking and being informed about other research in the field" where they can see what others are doing and gain collaborations / be put onto papers etc I guess it's just expectations: I've met surgeons who have the expectation that because they kept a piece of tumor from a surgery and gave it to a lab, they will be on any paper that ever uses data from that tissue until the end of time (even on labs on the other side of the world re-using the same data). When I'm collecting public data, I usually don't bother to ask due to the hassle, and move straight on to the actually public stuff. My bet is that papers with public code/data are used/cited way more often, but the tradeoff is probably their authors get less total papers over all. Do you want a larger number of people slightly happy with you or a fewer number of "close friends" in your area who may look kindly on you for grants or paper reviews.

aeonik 4y ago
I think paper citations are an archaic method of tracking contributions to research. We have all this technology, and Git is a great example, that allows us to more granularly give credit to contributors. In fact, even the very idea of a published paper feels out of date. Why have one model that is set in stone that is published as one snapshot in time? In the DevOps and Cybersecurity world our alerting mechanisms are essentially simple data models (sometimes not so simple) that are tapped into the raw data feeds, and can be updated and refined on an ongoing basis. While instrumenting the physical world can have more challenges, I still feel that the sciences have a lot to be gained from some of the more "pedestrian" operational processes that we use in the technology industry.

vedtopkar 4y ago

I'll echo that it's a tremendous amount of work to appropriately format data and metadata for upload to a public repository. It's not as simple as just mass uploading raw data files. It's no surprise that people skirt around this requirement when possible, especially as the size of files and number of samples per paper continues to balloon. That's not to say that it's not important, but the labor required is vastly underappreciated. I say this as someone who has completed the task for many papers.

Gatsky 4y ago
Yeah this is the main reason, as anyone actually doing research knows. I don’t think civillians realise what an incredible amount of work it is to publish a paper.
- jonnycomputer 4y ago
  yeah. holy hell is it a lot of work. hate doing it even when I know I'll get a paper out of it.
- Fomite 4y ago
  Hell, even things like journals requiring formatting after acceptance being taken as a godsend.
TaylorAlexander 4y ago
Is there some way to work this in to the system? Like by having a different person do this work? At a very high level, all of science would benefit from more data being properly formatted and published, so maybe this is something that could be budgeted for in the planning phase? Or the work could be done by undergraduates studying for the field?
- Fomite 4y ago
  The amount for a non-modular (i.e. the default, you don't have to have a super detailed budget) NIH R01, which is the basic unit of biomedical science funding, has been stagnant since 1999. Budgeting for a separate person to do this would eat up a tremendous amount of said budget. Which is fine, except if we think about this from a productivity standpoint, the person who budgeted for that person is probably short a graduate student compared to the person who didn't, on the outside chance that someone cares about their stuff. For example, I looked at my lab's publicly available repositories linked to papers on Github. Watches: 15 (probably half of these are people involved in the project) Forks: 3 Stars: 4 Visitors in the Past Month: 4 That's it. For all of them. While I keep doing it based on principle, if I stopped tomorrow, it would impact me not at all. And undergraduates, candidly, are not usually time savers.

javert 4y ago

If I were a billionaire, I'd fund a research institute whose only purpose would be to detect and call out fraud and shoddy work in academia. I'd hire the very best people I could in each field and put them onto this task. I imagine it would be easy to find excellent but disgruntled scientists in every niche. You'd need to hire the best people with niche expertise. I know from my years in academia, nobody can understand and scrutinize your bullshit unless they work in the same sub-sub-sub-field.

Qem 4y ago
Just create a glassdoor-like site, but for ex-students tô tell their experiences with research groups and advisors.
- photochemsyn 4y ago
  That's not going to work very well, you'll get a mixed cohort of people reporting actual fraud and deceptive practices (the desired group) and people who were just pissed off because they didn't get the support they needed (a legitimate complaint, but not as relevant to the issue of fraud) or had negative personal interactions (not such a legitimate complaint, as they might have been the problematic person).
  - Qem 4y ago
    I think the same issues affect Glassdoor, but it didn't stop the site being a hit. Just let people speak, and the audience judge by themselves and filter the outlier report from the problematic person.
hoseja 4y ago
You'd probably suffer a fatal accident with chinese characteristics very soon.
- javert 4y ago
  That made me laugh. Much of the fraudulent work isn't coming from Chinese researchers, but much of it is.

deleted 4y ago

[deleted]

Ansil849 4y ago

Does this paper come with its own Data Availability Statement?

forum_ghost 4y ago

agucova 4y ago
They've explicitly justified not sharing it, for good ethical reasons.
- mike_hearn 4y ago
  The reasons aren't good. If the DAS posted elsewhere in this thread is accurate then they don't want to be seen as "shaming" people, but the behavior they detected is simple promise breaking. The researchers stated they'd share their data, benefited from the aura of respectability that claim created, and later refused. And this behaviour was absolutely widespread: more than 90% of the papers selected didn't provide data as promised. That's the sort of behaviour that actually should be shamed, right? Why shouldn't people know whose word can be trusted and whose shouldn't? It's a very silly form of ethics that defends the honor of those who deliberately lie for profit. It also totally devalues the efforts and integrity of the tiny minority who did do things by the book. The actual reason they refused to share their own data almost certainly has nothing to do with ethics. It's because they don't want to make 3500+ enemies within their field.
  - forum_ghost 4y ago
    >The actual reason they refused to share their own data almost certainly has nothing to do with ethics. It's because they don't want to make 3500+ enemies within their field. This is exactly why peer review is nonsense.

Prcmaker 4y ago

When I was a university researcher, after publishing my first paper in the role I carefully read the policy requiring making data available, as I wanted to be correct by the book, but also have my data (which I took pride in) publicly available for critical review. The process as described was simple, though I soon found there was no mechanism implemented to actually achieve this. URLs and systems no longer existed, no alternatives available. I requested to publish via an external provider and was informed I would be fired if I did so. Disappointing to say the least.

andrewljohnson 4y ago

Tangential to topic, but I have been reading a Newton biography (Never at Rest) recently, and I found it interesting how reliant Newton was on the observations of other scientists. He had correspondence with scientists like Halley and Flamsteed on the basis of getting their observations of comets or observations of the moon, to help him work out orbits. At one point, Newton was unhappy with Flamsteed, because Flamsteed wanted to work on his star map instead of Newton's moon observations, but Flamsteed said he needed an accurate star map to produce accurate moon observations. Data sharing has been an issue in science for hundreds of years.

nomilk 4y ago

The cost-benefit ratio of open sourcing data has been known for long enough now, it's time to stop excusing this practice and start treating studies with unavailable datasets as heresay. When scientists don't publish datasets, it contributes to a culture that allows a small (or not small, depending on who you talk to) portion of scientists can p-hack their way to a publication and ruin the reputation of research for everyone.

xiaodai 4y ago

Some of research are fake as well. I tried to email an author who claimed some extraordinary results with their sorting algorithm and got crickets. Suspect they dont really care beyond having it "published".

gxs 4y ago

This is simple economics. If benefits of non compliance > consequences, don’t comply, else, comply. That simple, and applies to tech companies as well. At the end of the day who’s policing this stuff anyway? People only find out when there’s a leak.

dontbenebby 4y ago

Maybe someone should look them up in popular search engines using simple OSINT techniques and see that else they were reckless about. For many years, scientific computing facilities were "off limits" to honorable hackers, but I've noticed a persistent air of entitlement from the biological sciences, paired with a deep disrespect for the field of information science. I have a bachelor's in information science with a concentrations in information security and related areas in psychology and English literature from one of the highest ranked schools of information science in the world[0], followed by a PhD I left "ABD" after numerous peer reviewed publications and scores of conference talks and other informal lectures and volunteer work. I once saw an entire zoo aquarium's exhibits fog up in my hometown as it switched to brownouts, as a conference's attendees went on to obstruct policy initatives that would prevent the entire state they claim to want to be permanent residents from one day losing power in a similar manner to what happened in Texas.[1] It's why I started one of my pet phrases when I'd do volunteer work: Voting is a right, whereas electricity, like driving, is a privilege. Anyways, sorry for the stern response, but I hate being the guy who just goes "Oh, those results are unsurprising", but some nights I feel like I really need to mansplain what can happen if folks ignore my "crazy" left-libertarian advice. [1] After a decade of warnings, a weak mandate to "winterize" https://www.npr.org/2021/06/02/1002277720/texas-lawmakers-passed-changes-to-prevent-more-blackouts-experts-say-its-not-eno https://www.npr.org/2021/06/02/1002277720/texas-lawmakers-pa...

Edman274 4y ago
I don't understand the connection between the zoo's aquarium "fogging up" in your home town because of brownouts and the unwillingness of scientists to share data. I don't understand the connection between scientific computing facilities being unwilling to host "honorable hackers" in "information science" to use their computational power and data being closed. I don't understand what your "stern response" is to. I don't understand why you used the word "mansplain" to describe what you're doing when it has nothing to do with identity politics. I don't know what your "highest ranked school" is. You added it as a footnote but it doesn't seem to appear later in the comment.
- dontbenebby 4y ago

photochemsyn 4y ago

The only way I'd expect raw datasets to be published online immediately in a useful, curated manner by a research group is if that was stipulated in their research grant, period, i.e. that's what the goal of the project was, at least in part. Now this is not entirely uncommon in many research areas; you can get all the current weather and climate datasets with little difficulty as these are shared globally (as is necessary for effective short-term weather forecasting). Anyone publishing DNA/RNA/protein work generally uploads those sequences to GenBank, and that gets updated every two months, so is pretty current relative to the published literature: https://www.ncbi.nlm.nih.gov/genbank/ https://www.ncbi.nlm.nih.gov/genbank/ However this article seems to focus on "Biomed" and that whole world is where pharmaceutical/corporate influence in academics is off the charts. I'd guess well over half of the researchers (if not over 90%) have some dream of getting their work patented, setting up a biomed startup, and having Pfizer or J&J swoop in and buy them out (they then get a percentage and become wealthy), and their university patent office is likely pushing for that as well (they get another percentage). No way are those people going to share their data, which might very well be slanted or cooked in ways that will make their little entrepreneurial effort more attractive to a big buyer. It's way more like pitching to a VC than doing actual rigorous scientific research in the tradition of open sharing of results and ideas - at least not until the patent is granted. Then you're into whether or not the clinical trial data will be released to the public, and corporations don't like to do that either (as someone might find that their claimed success story is riddled with statistical errors, etc.). Business and science have orthogonal goals, and mixing the two is almost always a recipe for the rise of fraudulent con artists pushing bogus BS in the name of turning a quick profit.

chrismartin 4y ago

Is there irony in asking for other researchers' data (presumably free of charge) and then publishing the results of that effort behind a paywall?

sinab 4y ago

The data availability statement in this article undermines its entire premise. > We did not publish our raw data along with the manuscript because it could be understood that we are publicly shaming authors who did not want to share their data. As for the raw data that were received during this study, we informed our study participants that those raw data will be deleted after being examined and that all data and communication will be treated with strict confidence.

agucova 4y ago
There's a big difference between explicitly denying to share data for justified ethical reasons, and saying you will share the data and then failing to follow through. There is absolutely no contradiction here.
- sinab 4y ago
  The concern of “shaming” is not a justified ethical reason.

type0 4y ago

Some do share it but even then manage to use their own proprietary software to do the analysis, it boggles my mind how such people can receive grants and publish in prestigious journals, corruption - plain and simple.

tgbugs 4y ago

The NIH has mandated that starting in 2023 all grants that it funds must include a plan for data sharing. A serious lingering question is whether there is actually going to be any enforcement, given that it is already a monumental challenge for investigators to even account for the data that has been submitted/shared/archived by their lab, forget some 3rd party trying to do it. The tools for doing this across a whole diverse ecosystem are shall we say not user friendly (if they exist at all), and that is being charitable (and I say this as someone who has written some of those less than user friendly tools). Most of the systems, archives, etc. are not interoperable, they don't share common APIs, much less have some common interface that would allow someone to track all their published data, forget integrate it. Program officers are going to be desperate for ways to account for this stuff, but I worry that at least initially they are not going to have the leverage needed to get researchers to engage in even the most basic data management practices which would allow them to start solving the problem down the line. How many files did you upload to archive X? Which project were they for? How many datasets are you expecting to produce for this grant? You have published 10 papers associated with this grant number, where are the datasets that correspond to those papers? Some investigators can answer these immediately and easily, others will throw a screaming fit that they were even asked for such a thing.

mike_hearn 4y ago
The DAT protocol originally came out of this need. It's a very neat data sharing system that's designed to work with huge files, and it also came with a user friendly browser called Beaker Browser. Unfortunately the project seems to have split or evolved into a different project or something. They went through some rebrands, a rewrite, Beaker got abandoned and I lost track of what they're doing now. I think their grant funding ran out. But it was a pretty good attempt at making this stuff easy or easier.

isaacfrond 4y ago

I get that you don't want to share the data. Collection data is most of the work, people can steal it. God forbid people might find your wrong etc. etc. But maybe just maybe you shouldn't say that are going to share the data if you are not. How trustworthy is the rest in the article if your Data Availability Statement (DAS) is a lie?

academia_hack 4y ago

I left academia for industry a few years ago and still get hassled every week or so for help with open source code I published in conjunction with one of my papers. My other papers, where I provided no artifacts, generate an average of zero support emails a year but are cited just as much. The emails are often from first year PhD. students and are vaguely accusatory, along the lines of: "We're having issues replicating experiment X and wanted to give you a chance to correct/review before we publish on it." The cause of the issues is, 100% of the time to date, people not reading documentation or having more than a superficial understanding of the topic. However, I feel compelled to respond so that my old lab doesn't get dragged on twitter or at some conference over something that isn't true. I barely got paid to do this sort of work when I was in academia and certainly don't now. If I could go back in time, I would have kept my research closed source. The personal benefits are essentially zero and the world would get along just fine if I had never written any papers, so the altruistic reasons aren't terribly compelling either.

bell-cot 4y ago
If it's really "ever week or so", for years, I'd be tempted to compile data on the support e-mails and their senders. At some point, you've got a meta-paper on the people who contacted you for support, breakdowns of where they are in their careers, what topics they don't seem to have mastered yet, which parts of the docs they failed to read, etc. The fun could start when you let drafts of that meta-paper start circulating.