4 ms·

this is sort of one of the weird problems that shows up at the intersection between science in the public interest and a market driven system of production. pu

by a-dub 4y ago

this is sort of one of the weird problems that shows up at the intersection between science in the public interest and a market driven system of production. pure science that is publicly funded in the public interest would publish all raw data along with re-runnable processing pipelines that will literally reproduce the figures of interest. but, the funding is often provided by governments with the aim of producing commercializable new technology that can make life better for society. the problem is that if you do the science in the open, then it can be literally picked off by large incumbents before smaller inventors have a chance to try and spin up commercialization of their life's work. so we have this system today where science is semi-closed in order to protect the inventors, but sometimes to the detriment of the science itself.

a-dub 4y ago
...also, if a technique appears in a paper, an expert on that technique should be a reviewer and/or a standard rubric should be applied (i think nature and science have gotten much more rigorous about this in recent years in the wake of the psychology replication crisis).
adminprof 4y ago
I think you're missing two fatal problems in this "publish all raw data and code" mindset. I don't think the desire of commercialization is high on the list of fatal problems preventing people from publishing data+software. 1) How do you handle research in domains where the data is about people, so that releasing it harms their privacy? Healthcare, web activity, finances. Sure you can try to anonymize it, anonymization is imperfect, and even fully anonymized data can be joined to other data sources to de-identify people; k-anonymity only works in a closed ecosystem. If we live in a world where search engine companies don't publish their research because of this constraint, that seems worse than the current system. 2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?
- nicoco 4y ago
  From my experience in the digital health sector, concerns for privacy is always the reason given for not sharing anything valuable and/or useful to others. But it's just a convenient way of hiding the 'desire of commercialisation'.
  - a-dub 4y ago
    this is also true, and it also runs within science itself. if someone spends two years collecting some data that is very hard to collect and it has a few papers worth of insights within it, they're going to want to keep that data private until they can get those papers out themselves lest someone else come along, download their data and scoop them before they have a chance to see the fruits of their hard labor. while it's not great for science at large, i don't blame them either.
    - jmalicki 4y ago
      It's solvable if publishing the dataset counts as a paper, and citations of the dataset which should be required count as citations for e.g. tenure. For example, ImageNet for machine learning is a very expensive and difficult data set to produce that has resulted in revolutionary advances in machine learning. And people build models on it, cite their results as evidence their models are good, and cite the paper.
      - nicoco 4y ago
        This is an interesting idea. Although I am afraid that publishing a dataset, even a good one, will not be considered "real science" by our (broken) institutions.
    - nicoco 4y ago
      You have a valid point here. It's probably utopian, but to me the only reasonable answer to this is to acknowledge that science is a collective process. Of course, this goes against the (stupid) idea that some extremely deserving geniuses are the ones that make science...
majormajor 4y ago
Even in areas where commercialization isn't super relevant I've known few academics particularly interested in doing all the tedious process crap to make it easy to publish their data and their calculations in a "reproducible pipeline" sort of way. I think in large parts it's a tooling and awareness problem - like, historically awareness and use of source control has been very low in the space, but for data, even industry tools tend to be less sophisticated than the ones for code.
uoaei 4y ago
So what you're saying is, intellectual property rights and the profit potential of excluding certain actors from using certain techniques is the source of our woes. And there are those who say progress proceeds optimally in this arrangement! Hardly: only "innovation", which is distinct from progress, is optimized. Progress is the act of cleaning up the mess made by ineffective innovations (ie "minimum publishable units"), distilling them into a basis for the next paradigm shift. Progress, everyone's ostensible goal, is hampered by siloes and IP hoarding, as it multiplies the efforts made by many who should otherwise be working in tandem, by hiding them each in their own little silo so they all learn the same lessons completely separately.
daniel-cussen 4y ago
That's how it has to be. This is a fairly open system in comparison to what America's founders were objecting against, this really closed guild system where you only find out how to actually make the contraption work for real when the master [of the craft, who teaches the other journeymen and apprentices] is eg getting death threats on a fluke happenstance and finally telling you...the tricks of the trade. On his deathbed. Other times valuable knowledge was just forgotten. And from that point on, they didn't make them like they used to. That's how you get "limited protection" of the arts and sciences, and patents that were democratic ie I can file on my own for a provisional patent, it's really hard in cases like the Wright Brothers--and some others, and there's crackpots--but at least you didn't need to lobby to get one like in England, where it was an act of Parliament to get an inventor his patent, like James Watt. [Don't have visibility on how the two systems evolved when they broke apart.]

by a-dub 4y ago

a-dub 4y ago

...also, if a technique appears in a paper, an expert on that technique should be a reviewer and/or a standard rubric should be applied (i think nature and science have gotten much more rigorous about this in recent years in the wake of the psychology replication crisis).

adminprof 4y ago

I think you're missing two fatal problems in this "publish all raw data and code" mindset. I don't think the desire of commercialization is high on the list of fatal problems preventing people from publishing data+software. 1) How do you handle research in domains where the data is about people, so that releasing it harms their privacy? Healthcare, web activity, finances. Sure you can try to anonymize it, anonymization is imperfect, and even fully anonymized data can be joined to other data sources to de-identify people; k-anonymity only works in a closed ecosystem. If we live in a world where search engine companies don't publish their research because of this constraint, that seems worse than the current system. 2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?

nicoco 4y ago
From my experience in the digital health sector, concerns for privacy is always the reason given for not sharing anything valuable and/or useful to others. But it's just a convenient way of hiding the 'desire of commercialisation'.
- a-dub 4y ago
  this is also true, and it also runs within science itself. if someone spends two years collecting some data that is very hard to collect and it has a few papers worth of insights within it, they're going to want to keep that data private until they can get those papers out themselves lest someone else come along, download their data and scoop them before they have a chance to see the fruits of their hard labor. while it's not great for science at large, i don't blame them either.
  - jmalicki 4y ago
    It's solvable if publishing the dataset counts as a paper, and citations of the dataset which should be required count as citations for e.g. tenure. For example, ImageNet for machine learning is a very expensive and difficult data set to produce that has resulted in revolutionary advances in machine learning. And people build models on it, cite their results as evidence their models are good, and cite the paper.
    - nicoco 4y ago
      This is an interesting idea. Although I am afraid that publishing a dataset, even a good one, will not be considered "real science" by our (broken) institutions.
  - nicoco 4y ago
    You have a valid point here. It's probably utopian, but to me the only reasonable answer to this is to acknowledge that science is a collective process. Of course, this goes against the (stupid) idea that some extremely deserving geniuses are the ones that make science...

majormajor 4y ago

Even in areas where commercialization isn't super relevant I've known few academics particularly interested in doing all the tedious process crap to make it easy to publish their data and their calculations in a "reproducible pipeline" sort of way. I think in large parts it's a tooling and awareness problem - like, historically awareness and use of source control has been very low in the space, but for data, even industry tools tend to be less sophisticated than the ones for code.

uoaei 4y ago

So what you're saying is, intellectual property rights and the profit potential of excluding certain actors from using certain techniques is the source of our woes. And there are those who say progress proceeds optimally in this arrangement! Hardly: only "innovation", which is distinct from progress, is optimized. Progress is the act of cleaning up the mess made by ineffective innovations (ie "minimum publishable units"), distilling them into a basis for the next paradigm shift. Progress, everyone's ostensible goal, is hampered by siloes and IP hoarding, as it multiplies the efforts made by many who should otherwise be working in tandem, by hiding them each in their own little silo so they all learn the same lessons completely separately.

daniel-cussen 4y ago

That's how it has to be. This is a fairly open system in comparison to what America's founders were objecting against, this really closed guild system where you only find out how to actually make the contraption work for real when the master [of the craft, who teaches the other journeymen and apprentices] is eg getting death threats on a fluke happenstance and finally telling you...the tricks of the trade. On his deathbed. Other times valuable knowledge was just forgotten. And from that point on, they didn't make them like they used to. That's how you get "limited protection" of the arts and sciences, and patents that were democratic ie I can file on my own for a provisional patent, it's really hard in cases like the Wright Brothers--and some others, and there's crackpots--but at least you didn't need to lobby to get one like in England, where it was an act of Parliament to get an inventor his patent, like James Watt. [Don't have visibility on how the two systems evolved when they broke apart.]