6 ms·

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades,

I can confirm that in my current area of interest (how to synthesize a cello or saxophone sound), there are hundreds of academic papers published over decades, each of them says "our method sounds more realistic than others", but code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding. Reading a forum of synth hobbyists is a hundred times more useful.

jakub_g 5y ago
I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive: In opensource the attitude is "See bug? Send a PR!" Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.
- remram 5y ago
  The biggest difference IMHO is when comparing to something like Wikipedia or Stackoverflow. I wish the fabric of scholarly communication similarly allowed for browsing reviews, updating papers, commenting with new references, etc.
  - kwertyoowiyop 5y ago
    Now THAT might be a realistic technical goal & business opportunity.
    - Mezzie 5y ago
      The legal costs make this a non-starter unless it's done by a giant company. Who would, in my opinion, ruin it, and the odds of enough academics complying with a big tech company are small imo. It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them. Building this would be my dream job, but hahaha no.
  - deleted 5y ago
    [deleted]
  - Mezzie 5y ago
    I think this is a valuable idea. There are online archives that allow for paper updating for academics, like SSRN, but as a CONSUMER of academic literature, the land is pretty barren. The difficulty in such a thing would be the journals and database companies are holding on to their exclusivity and profit motives with an iron fist, so unless you want to get sued into oblivion, you'd have to stick with open source or accessible articles, so you'd need to either specialize in disciplines that have moved away from closed-source enough that the tool wouldn't have massive holes in it. Also determining which new references and reviews have relevance (like if anybody can comment with new references, who goes through to check they're actually relevant or say what the person says they say?), preventing academics/administrators from gaming the system if it DOES get popular, etc. In open source, this is crowd-sourced, but for some academic fields the number of people who are qualified to speak on a matter is extremely small. /academic librarian thoughts
- kovvy 5y ago
  Generally, anyone writing a paper about something that could benefit from bugfixes would love to accept them, but doesn't have the time or resources to actually do so - unless there's another paper in it. If they have somehow managed to find enough personal time to have a hobby project, then they probably do accept bugfixes - and you should get them in before that person burns out.
  - Fomite 5y ago
    It also doesn't happen enough to design for - I once presented a fairly open-source contributor friendly project at SciPy that I hoped would be compelling (it was about modeling the zombie epidemic), actively asked for help, had set up a couple open requests of varying levels of complexity. I think there was one pull request total? The juice just didn't end up being worth the squeeze.
- deleted 5y ago
  [deleted]
kkylin 5y ago
I'm an academic (applied math) and want to respond to this: academic papers are the way they are for lots of reasons, many of which (not so good) have been mentioned on HN. There are a couple that I do not see very often however: (1) Many academics aren't aware non-academics read their papers at all: we work with other academics, go to conferences with other academics, and on the rare occasions we hear from readers, it's from other academics. Big exception: in some fields academia and industry have much more interaction, biomedical research (the subject of the linked article) being one of them. Extracting knowledge from that literature has a large number of practical and economic implications. (2) There seems to be a perception that published papers are a repository of established or state-of-the-art knowledge. Perhaps they were meant to be that way, and perhaps more of them should be. But for many journals in many fields, publications are a form of moderated discussion. Reconstructing the state of knowledge from snippets of conversation is always going to be hard. What can help make the literature more accessible? some of the forces are structural, some are due to current limitations of technology. But one thing that can help if you find the results of a paper interesting and are able to track down the authors, write them. People like hearing their work is noticed, and they like talking to people about things they're interested in. Another is to make constructive suggestions (or even pitch in to improve code where it's open source & available). Between teaching, advising, committee work, etc (not to mention family), most of us have to prioritize, and as much as I'd like to clean up old code for release in the hopes someone finds it useful, it isn't going to get my grad students out the door with a degree or a job -- I'm generally spending more time on their research problems these days than my own. But if I know there's interest / use I might prioritize time a little differently.
- Tomte 5y ago
  > What can help make the literature more accessible? Review articles, sometimes called surveys. I've always thought that new PhDs would be excellent authors for those, having digested lots literature for their dissertation.
  - wheelinsupial 5y ago
    > Review articles, sometimes called surveys. Is this field specific? I have read survey articles in math and biology, and was told by some of my profs that they use these articles as an introduction to a new field. A quick Google search seems to show these exist in CS (along with tutorial papers), physics, and chemistry but I'm having a little difficulty finding statistics survey papers (survey methods come up instead). Is the problem that there aren't enough of them or they are behind paywalls?
deleted 5y ago
[deleted]
dekhn 5y ago
Hey, that unusability of papers is a form of job security. Seriously though, you're totally right. I got very dissatisfied with science when I realized that many people were effectively publishing unreproducible crap created by terrible code. Fortunately, more and more people are learning how to recognize the crap.
captainmuon 5y ago
I may be a bit cynical, but at least in my former field (experimental physics), the main purpose of papers seems to be to "lock in" a finished achivement. You do the actual research, pass internal reviews and peer review, and then publishing the paper is just to make it "official". Unfortunately, many papers are never expected to be read. The crucial information exists, but you usually get it from personal communication, internal wikis, or review articles. You just need the paper to copy a formula or graph, and to cite it in the end. There are papers that are well-written and useful, but there are at least as many that are just drivel (I probably contributed to both kinds). Unfortunately, the prevailing attitude is that outside people will not understand our stuff anyway, so we often make no effort to make papers understandable, or to publish data. (There is a lot of great outreach and science communication, but not so much for students or researchers from other fields who want to follow the technical details.)
- temporaryi3 5y ago
  I did my phd in experimental physics and I have to say my realisation of this large point, that papers are little more than resume padding to lock in an achievement was a significant contributor towards destroying, and I use destroying seriously here, any faith or trust that peer review or publishing has anything at all to do with the scientific method at all. Your results replicate, or they don't. Your calculations, equations, and models predict experiment. Or they don't. Writing papers about it and getting the feedback of "peers" is nothing more than an old fashioned circle jerk for padding resumes, CVs, and persuading other people in that academic hierarchy that you deserve funding. It is a game that is divorced from actually learning, researching, understanding, measuring, and predicting the world.
  - cyanydeez 5y ago
    There's probably space for a startup that properly archives the technical nature of findings.
  - orbifold 5y ago
    In academia there always is a difference between the way results are advertised and what conclusions are drawn internally. This is more true in some fields than others, I'm most familiar with it ML, Physics. Part of your skill as a researcher is to understand based on omissions, the datasets etc. the quiet part that isn't said out loud. Depending how you sell things you can get a Nature / Science paper with confusing inconsistent terminology, hand rolled C++ implementation, provided you are the first and a another method which might be 1000x times faster will only make it into PRL (yes I'm thinking of two specific papers, but won't say which).
tchalla 5y ago
> I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding. If you think the entire field of academia doesn't achieve any purpose, you may want to reconsider your position. Most likely, almost everything that you do today on a computer was an academic paper. Yes, it was without code and data. Yet, it was not unusable and achieved more than enough purpose. The average comment on HN on academia comes from a mindset where everyone wants a product. The purpose of a paper is NOT to release a software or a product. But, to test an idea under some assumptions. That's what all research does at its core - formulate a hypothesis, design an experiment to test the hypothesis and report the results and implications. Are all research papers perfect"? No. Are all of them usable? No. Your use case - sound synthesis for a specific instrument - may not be a scientific challenge. It is however an engineering challenge and hence, you found a better answer amongst hobbyists and tinkerers. Now, try looking for the a vaccine for Covid - and guess where you'd find that answer? In decades of research on mRNA with repeated failures, papers that couldn't be replicated, unavailability of "code" and samples with verbal descriptions skipping crucial details.
- tvhahn 5y ago
  Balaji Srinivasan had a good take on this recently in his conversation with Tim Ferris. I quote: "The thing is, I don’t care if something has a thousand retweets, what I care about is if it has two or three independent confirmations from economically dis-aligned actors. This is the same as academia, by the way, everybody’s optimizing citations. What you actually want to optimize is independent replication. That’s what true science is. It’s not peer review. It is physical tests."
  - czzr 5y ago
    Yes and no. Literal replications are less valuable than people think - what you really want are independent tests of different parts of the causal network of the underlying model.
ska 5y ago
> but their output in paper form is basically unusable Others have commented as well but I will reinforce: their output is basically unusable for you for the purpose you want to put it to. Which is fair, but you should also recognize that you are not the audience of the papers and for good or for ill the system is not set up to help you with this.
- cousin_it 5y ago
  I don't see how the system is very good for its intended audience either. If papers about making sound with code actually included sound and code, people in academia would be better off too. It would make it easier to search through literature, and to build on each other's work.
  - ska 5y ago
    Papers are a conversation as much as anything else. I agree that these days the tooling makes it much easier to distribute code & data somehow to match up, but there is also a cost/incentive mismatch. Basically to do a decent release of what you are working on and worse, potentially support it, costs time but has no real career value (yet). Which means it's mostly only done by people who are philosophically convinced of its value. I think this will change over time, at least in some areas, but it won't be quick.
robotresearcher 5y ago
> code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, It doesn't have to be this way. Here's the process I use in my lab: 1. Every paper that makes a claim of any kind based on code contains a link to a public Git repo. 2. The paper contains the Git hash identifying the exact commit used to justify the claim. Copy-paste it from the paper into your checkout. 3. The repo may have moved on with fixes and improvements, you can have those too. If you are using version control already, it's not much work to do this. Of course, you have to be committed to making your code public.

jakub_g 5y ago

I'm an outsider but it seems to me the difference between academia and opensource/hobby forums is massive: In opensource the attitude is "See bug? Send a PR!" Whereas academic papers are like publishing software into a blockchain (and not source but binaries, i.e. PDFs full of shortcuts): you don't want for people to easily find bugs and contribute fixes, so you handwave a lot so that no one can reproduce your exact thing.

remram 5y ago
The biggest difference IMHO is when comparing to something like Wikipedia or Stackoverflow. I wish the fabric of scholarly communication similarly allowed for browsing reviews, updating papers, commenting with new references, etc.
- kwertyoowiyop 5y ago
  Now THAT might be a realistic technical goal & business opportunity.
  - Mezzie 5y ago
    The legal costs make this a non-starter unless it's done by a giant company. Who would, in my opinion, ruin it, and the odds of enough academics complying with a big tech company are small imo. It'd be viable for fields that don't use/rely on for-profit or closed journals, but I don't know if the money to run it would be there, especially since the odds of the big Schol Comm players suing is still there, because it'd be worth it to ruin the tool/effort before it can challenge them. Building this would be my dream job, but hahaha no.
- deleted 5y ago
  [deleted]
- Mezzie 5y ago
  I think this is a valuable idea. There are online archives that allow for paper updating for academics, like SSRN, but as a CONSUMER of academic literature, the land is pretty barren. The difficulty in such a thing would be the journals and database companies are holding on to their exclusivity and profit motives with an iron fist, so unless you want to get sued into oblivion, you'd have to stick with open source or accessible articles, so you'd need to either specialize in disciplines that have moved away from closed-source enough that the tool wouldn't have massive holes in it. Also determining which new references and reviews have relevance (like if anybody can comment with new references, who goes through to check they're actually relevant or say what the person says they say?), preventing academics/administrators from gaming the system if it DOES get popular, etc. In open source, this is crowd-sourced, but for some academic fields the number of people who are qualified to speak on a matter is extremely small. /academic librarian thoughts
kovvy 5y ago
Generally, anyone writing a paper about something that could benefit from bugfixes would love to accept them, but doesn't have the time or resources to actually do so - unless there's another paper in it. If they have somehow managed to find enough personal time to have a hobby project, then they probably do accept bugfixes - and you should get them in before that person burns out.
- Fomite 5y ago
  It also doesn't happen enough to design for - I once presented a fairly open-source contributor friendly project at SciPy that I hoped would be compelling (it was about modeling the zombie epidemic), actively asked for help, had set up a couple open requests of varying levels of complexity. I think there was one pull request total? The juice just didn't end up being worth the squeeze.
deleted 5y ago
[deleted]

kkylin 5y ago

I'm an academic (applied math) and want to respond to this: academic papers are the way they are for lots of reasons, many of which (not so good) have been mentioned on HN. There are a couple that I do not see very often however: (1) Many academics aren't aware non-academics read their papers at all: we work with other academics, go to conferences with other academics, and on the rare occasions we hear from readers, it's from other academics. Big exception: in some fields academia and industry have much more interaction, biomedical research (the subject of the linked article) being one of them. Extracting knowledge from that literature has a large number of practical and economic implications. (2) There seems to be a perception that published papers are a repository of established or state-of-the-art knowledge. Perhaps they were meant to be that way, and perhaps more of them should be. But for many journals in many fields, publications are a form of moderated discussion. Reconstructing the state of knowledge from snippets of conversation is always going to be hard. What can help make the literature more accessible? some of the forces are structural, some are due to current limitations of technology. But one thing that can help if you find the results of a paper interesting and are able to track down the authors, write them. People like hearing their work is noticed, and they like talking to people about things they're interested in. Another is to make constructive suggestions (or even pitch in to improve code where it's open source & available). Between teaching, advising, committee work, etc (not to mention family), most of us have to prioritize, and as much as I'd like to clean up old code for release in the hopes someone finds it useful, it isn't going to get my grad students out the door with a degree or a job -- I'm generally spending more time on their research problems these days than my own. But if I know there's interest / use I might prioritize time a little differently.

Tomte 5y ago
> What can help make the literature more accessible? Review articles, sometimes called surveys. I've always thought that new PhDs would be excellent authors for those, having digested lots literature for their dissertation.
- wheelinsupial 5y ago
  > Review articles, sometimes called surveys. Is this field specific? I have read survey articles in math and biology, and was told by some of my profs that they use these articles as an introduction to a new field. A quick Google search seems to show these exist in CS (along with tutorial papers), physics, and chemistry but I'm having a little difficulty finding statistics survey papers (survey methods come up instead). Is the problem that there aren't enough of them or they are behind paywalls?

deleted 5y ago

[deleted]

dekhn 5y ago

Hey, that unusability of papers is a form of job security. Seriously though, you're totally right. I got very dissatisfied with science when I realized that many people were effectively publishing unreproducible crap created by terrible code. Fortunately, more and more people are learning how to recognize the crap.

captainmuon 5y ago

I may be a bit cynical, but at least in my former field (experimental physics), the main purpose of papers seems to be to "lock in" a finished achivement. You do the actual research, pass internal reviews and peer review, and then publishing the paper is just to make it "official". Unfortunately, many papers are never expected to be read. The crucial information exists, but you usually get it from personal communication, internal wikis, or review articles. You just need the paper to copy a formula or graph, and to cite it in the end. There are papers that are well-written and useful, but there are at least as many that are just drivel (I probably contributed to both kinds). Unfortunately, the prevailing attitude is that outside people will not understand our stuff anyway, so we often make no effort to make papers understandable, or to publish data. (There is a lot of great outreach and science communication, but not so much for students or researchers from other fields who want to follow the technical details.)

temporaryi3 5y ago
I did my phd in experimental physics and I have to say my realisation of this large point, that papers are little more than resume padding to lock in an achievement was a significant contributor towards destroying, and I use destroying seriously here, any faith or trust that peer review or publishing has anything at all to do with the scientific method at all. Your results replicate, or they don't. Your calculations, equations, and models predict experiment. Or they don't. Writing papers about it and getting the feedback of "peers" is nothing more than an old fashioned circle jerk for padding resumes, CVs, and persuading other people in that academic hierarchy that you deserve funding. It is a game that is divorced from actually learning, researching, understanding, measuring, and predicting the world.
- cyanydeez 5y ago
  There's probably space for a startup that properly archives the technical nature of findings.
- orbifold 5y ago
  In academia there always is a difference between the way results are advertised and what conclusions are drawn internally. This is more true in some fields than others, I'm most familiar with it ML, Physics. Part of your skill as a researcher is to understand based on omissions, the datasets etc. the quiet part that isn't said out loud. Depending how you sell things you can get a Nature / Science paper with confusing inconsistent terminology, hand rolled C++ implementation, provided you are the first and a another method which might be 1000x times faster will only make it into PRL (yes I'm thinking of two specific papers, but won't say which).

tchalla 5y ago

> I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, I'm not sure it achieves any purpose besides resume padding. If you think the entire field of academia doesn't achieve any purpose, you may want to reconsider your position. Most likely, almost everything that you do today on a computer was an academic paper. Yes, it was without code and data. Yet, it was not unusable and achieved more than enough purpose. The average comment on HN on academia comes from a mindset where everyone wants a product. The purpose of a paper is NOT to release a software or a product. But, to test an idea under some assumptions. That's what all research does at its core - formulate a hypothesis, design an experiment to test the hypothesis and report the results and implications. Are all research papers perfect"? No. Are all of them usable? No. Your use case - sound synthesis for a specific instrument - may not be a scientific challenge. It is however an engineering challenge and hence, you found a better answer amongst hobbyists and tinkerers. Now, try looking for the a vaccine for Covid - and guess where you'd find that answer? In decades of research on mRNA with repeated failures, papers that couldn't be replicated, unavailability of "code" and samples with verbal descriptions skipping crucial details.

tvhahn 5y ago
Balaji Srinivasan had a good take on this recently in his conversation with Tim Ferris. I quote: "The thing is, I don’t care if something has a thousand retweets, what I care about is if it has two or three independent confirmations from economically dis-aligned actors. This is the same as academia, by the way, everybody’s optimizing citations. What you actually want to optimize is independent replication. That’s what true science is. It’s not peer review. It is physical tests."
- czzr 5y ago
  Yes and no. Literal replications are less valuable than people think - what you really want are independent tests of different parts of the causal network of the underlying model.

ska 5y ago

> but their output in paper form is basically unusable Others have commented as well but I will reinforce: their output is basically unusable for you for the purpose you want to put it to. Which is fair, but you should also recognize that you are not the audience of the papers and for good or for ill the system is not set up to help you with this.

cousin_it 5y ago
I don't see how the system is very good for its intended audience either. If papers about making sound with code actually included sound and code, people in academia would be better off too. It would make it easier to search through literature, and to build on each other's work.
- ska 5y ago
  Papers are a conversation as much as anything else. I agree that these days the tooling makes it much easier to distribute code & data somehow to match up, but there is also a cost/incentive mismatch. Basically to do a decent release of what you are working on and worse, potentially support it, costs time but has no real career value (yet). Which means it's mostly only done by people who are philosophically convinced of its value. I think this will change over time, at least in some areas, but it won't be quick.

robotresearcher 5y ago

> code and audio samples are never available, and verbal descriptions always skip crucial details. I have no doubt that academics have a ton of expertise, but their output in paper form is basically unusable, It doesn't have to be this way. Here's the process I use in my lab: 1. Every paper that makes a claim of any kind based on code contains a link to a public Git repo. 2. The paper contains the Git hash identifying the exact commit used to justify the claim. Copy-paste it from the paper into your checkout. 3. The repo may have moved on with fixes and improvements, you can have those too. If you are using version control already, it's not much work to do this. Of course, you have to be committed to making your code public.