3 ms·

Would open publishing on the web solve this? After all, the journal system was invented to solve distribution of papers. We have the internet now, so is there

Would open publishing on the web solve this? After all, the journal system was invented to solve distribution of papers. We have the internet now, so is there any need for the journal system any more? Independent reviewers would/could easily step up to pick up the interesting papers and present a "feed" of the good stuff.

bonoboTP 6y ago
First we need to understand what we want to solve. The attention economy and PR war goes on just as much on the open internet. E.g. people only reading papers from big shots, MIT, Stanford, Harvard labs etc. The problem is way deeper than just academia. Such as, is there fairness in the world deep down, is mass-produced excellence possible? Does individual greatness actually exist or is it all just a power play? Overall, the quality of science is extremely difficult to measure, precisely because it operates on the border of the unknown and because people try their best to appear the best possible. Science is difficult to understand and is often far removed from the here and now, and may only bear fruit decades down the line. It's hard to judge for the same reason that antelopes are hard to catch for cheetahs: competition (mainly the antelope vs antelope type). In the end, science has only been this mass product for a few decades. Before that it was mostly a pastime of weird nerdy aristocrats or people paid by aristocrats for showoff purposes. Or church people with too much time on their hands. In reality, from the top down view it's a huge gamble. You try to get good people to do their honest best and then see what happens. Then at the end there will be some breakthroughs. But only a few every few years in each field. However, this does not satisfy the participants. I toiled away as well, but the reward is only paid to the lucky one. So everyone tries to be the lucky one, which perversely pushes everyone to take fewer risks, making the collective likelihood of a breakthrough lower, but their own expected reward better. ----- My grandfather used to recite the story of a farmer who had three pigs. Every morning he'd throw two apples in their confinement. He'd then grab a big stick and beat the one that didn't get any apple: why didn't it try harder? ----- My prediction is that as with all signaling spirals and treadmill effects, there will be something new to aspire to, to tell the wheat from the chaff, a signal that's harder to fake. It's a constant race. You demonstrate your fitness by being adaptive to how the system changes. Overall the "quality" of people obviously doesn't change over time, it's just that the competent/powerful drive the criteria to their benefit. As academia/publishing etc. is now flooded with "the plebs", "the elite" will move on and will perhaps use other criteria. ---- Now, going back to assuming this is about the object-level science itself. Where to find the best science? You cannot do this in general. You have to educate yourself and dive in yourself. You try to learn how to judge people's character and try to listen to and digest the assessment of those you trust. There's no other way, gather experience and become "better" yourself. Use the cognitive resources of your brain to try and outsmart your opponent: the writer of the piece of text you are reading. This cannot be standardized/metrificated in a simple way (outside human-level AGI). If your organization does not put in the cognitive power of extensively processing the content of a particular research and critically examining the motivations behind it etc., there is no way to judge it. Then you're back to credentials. Did it come from a highly cited person? Is this person endorsed by other big shots, where the "seed big shots" are the researchers at the historically most prestigious institutions. ---- Currently, to find interesting research I personally use Github recommendations, Google Scholar alerts watching for citations of landmark papers (good indicators for progress in niches) and authors. A well-curated Twitter-feed is also useful, as is arxiv-sanity. In the end, I have to make up my mind if it's good work or not, and as everyone I don't have infinite cognitive resources. So I make snap judgements based on paper gestalt, affiliations, plausibility, result tables, etc. If it clears this bar, I dive in more. Over time, you learn to trust some smart people and can follow them online and see what they say and recommend. And continuously learn and grind your brain. Cognitive work cannot be spared, just like you cannot spare physical exhaustion in sport competitions.
- kerkeslager 6y ago
  > First we need to understand what we want to solve. The attention economy and PR war goes on just as much on the open internet. E.g. people only reading papers from big shots, MIT, Stanford, Harvard labs etc. Well, I think there's two separate problems: 1. Academic journals not publishing "boring" results, which incentivizes scientists to bias toward novelty and not publish null results. I think the solution to this is for journals to accept based on methodology and subject matter of experiments/studies, and to commit to publish before the study is even done, so there's no chance that results can influence the decision to publish. 2. Science journalism being written by people unqualified to analyze the science who instead sensationalize for clicks. I'm not particularly concerned that the people are "only reading papers from big shots". The problem is that prestige is measured by impact analysis rather than replication. If you incentivize a metric which is largely a popularity contest for ideas, it should be no surprise that what you get is popular lies[1]. One might think that the focus on impact in prestigious journals means there might be higher replication in less prestigious journals, but from what I can tell, the less prestigious journals are no better: they're mostly targeting the same metrics, they're just less successful at reaching their goals. [1] My intent is not to actually accusing anyone of intent to deceive here--I just can't quite figure out the right word. Untruths doesn't quite match what I'm trying to say.
brigandish 6y ago
A standardised data format for papers would help (beyond that of introduction > methodology etc, treat it like an insurance claim or something of that ilk), that way content could be distributed and compared and discovered far more easily than having to wade through papers written in different formats, with all the flowery academic language etc. It'd probably make writing them easier too.
6510 6y ago
The w3 (failed) PICS rating system[1] has always fascinated me. In short: Everyone (including the author) gets to rate everything by whatever scale they want to use. Everyone gets to use ratings made by whoever they like. I could see it work as well for academics and intelectuals as bodybuilders and progamers. One angle of my fascination was how the [rating] baby was pretty much tossed out with the bath water. It was shouted down and denounced by journalists (as for example an adult filter which ironically ended up its only application) Some journalists described a perspective as if they had a kind of tenure. They were published for so long that the idea of a rating system was just offensive. We could argue that a good rating system would use existing talent for calibration but the real question to ask imho is: If PICS was so bad, what did we get in stead? Anon 5 star ratings? Thumbs up? HN points? Number of github saved games? To say it doesn't compete with publishing in journals is somewhat of an understatement. End of the day all we are looking for is good meta data. If note worthy people in a field want to endorse a HN topic, a blog posting, a usenet post, a tweet, a youtube video, a facebook posting or a torrent[2] real credit could go to the author. A rating system or spec therefore could simply accommodate that process. (It should for example require the author and their endorsers make backups available.) Journals are from the horse and carriage days. It is quite embarrassing how we didn't come up with something modern. [1] - https://www.w3.org/PICS/services-960303.html https://www.w3.org/PICS/services-960303.html [2] - torrents are nice to share huge data sets
- marcus_holmes 6y ago
  Totally agree about the entitlement of journalists. I hadn't heard of PICS before, it's interesting. Is there a published story about why it failed anywhere?
  - 6510 6y ago
    I cant find anything anymore but the story seems obvious enough. Content rating systems were tried. They didn't gain popularity. Eventually only adult filters and movie classification remained. It isn't just journalists apparently, I think most people hate anything that smells like scrutiny. There are probably plenty under appreciated authors. They ironically have no way of knowing. I'm convinced there are and always have been fantastic ideas out there that didn't even get written down. Why even bother having them or fleshing them out if they cant be judged? In my inner dialog I consider it the greatest puzzle of all times (and expand the scope to rating all human creations) if anyone ever truly solves it all previous revolutions will be reduced to simple events. The potential for recursive self improvement of such system is probably equal to a general AI but the results will be better. haha, with the lack of published stories I just discover I sound to myself like I have a lot of explaining to do making such fancy claims. I will have to ponder writing it into a blog. For now a business plan that everyone is specifically uninterested in is not going to work. But then again, that is like, just everyone's opinion? We have no means of judging the value of it.

bonoboTP 6y ago

First we need to understand what we want to solve. The attention economy and PR war goes on just as much on the open internet. E.g. people only reading papers from big shots, MIT, Stanford, Harvard labs etc. The problem is way deeper than just academia. Such as, is there fairness in the world deep down, is mass-produced excellence possible? Does individual greatness actually exist or is it all just a power play? Overall, the quality of science is extremely difficult to measure, precisely because it operates on the border of the unknown and because people try their best to appear the best possible. Science is difficult to understand and is often far removed from the here and now, and may only bear fruit decades down the line. It's hard to judge for the same reason that antelopes are hard to catch for cheetahs: competition (mainly the antelope vs antelope type). In the end, science has only been this mass product for a few decades. Before that it was mostly a pastime of weird nerdy aristocrats or people paid by aristocrats for showoff purposes. Or church people with too much time on their hands. In reality, from the top down view it's a huge gamble. You try to get good people to do their honest best and then see what happens. Then at the end there will be some breakthroughs. But only a few every few years in each field. However, this does not satisfy the participants. I toiled away as well, but the reward is only paid to the lucky one. So everyone tries to be the lucky one, which perversely pushes everyone to take fewer risks, making the collective likelihood of a breakthrough lower, but their own expected reward better. ----- My grandfather used to recite the story of a farmer who had three pigs. Every morning he'd throw two apples in their confinement. He'd then grab a big stick and beat the one that didn't get any apple: why didn't it try harder? ----- My prediction is that as with all signaling spirals and treadmill effects, there will be something new to aspire to, to tell the wheat from the chaff, a signal that's harder to fake. It's a constant race. You demonstrate your fitness by being adaptive to how the system changes. Overall the "quality" of people obviously doesn't change over time, it's just that the competent/powerful drive the criteria to their benefit. As academia/publishing etc. is now flooded with "the plebs", "the elite" will move on and will perhaps use other criteria. ---- Now, going back to assuming this is about the object-level science itself. Where to find the best science? You cannot do this in general. You have to educate yourself and dive in yourself. You try to learn how to judge people's character and try to listen to and digest the assessment of those you trust. There's no other way, gather experience and become "better" yourself. Use the cognitive resources of your brain to try and outsmart your opponent: the writer of the piece of text you are reading. This cannot be standardized/metrificated in a simple way (outside human-level AGI). If your organization does not put in the cognitive power of extensively processing the content of a particular research and critically examining the motivations behind it etc., there is no way to judge it. Then you're back to credentials. Did it come from a highly cited person? Is this person endorsed by other big shots, where the "seed big shots" are the researchers at the historically most prestigious institutions. ---- Currently, to find interesting research I personally use Github recommendations, Google Scholar alerts watching for citations of landmark papers (good indicators for progress in niches) and authors. A well-curated Twitter-feed is also useful, as is arxiv-sanity. In the end, I have to make up my mind if it's good work or not, and as everyone I don't have infinite cognitive resources. So I make snap judgements based on paper gestalt, affiliations, plausibility, result tables, etc. If it clears this bar, I dive in more. Over time, you learn to trust some smart people and can follow them online and see what they say and recommend. And continuously learn and grind your brain. Cognitive work cannot be spared, just like you cannot spare physical exhaustion in sport competitions.

kerkeslager 6y ago
> First we need to understand what we want to solve. The attention economy and PR war goes on just as much on the open internet. E.g. people only reading papers from big shots, MIT, Stanford, Harvard labs etc. Well, I think there's two separate problems: 1. Academic journals not publishing "boring" results, which incentivizes scientists to bias toward novelty and not publish null results. I think the solution to this is for journals to accept based on methodology and subject matter of experiments/studies, and to commit to publish before the study is even done, so there's no chance that results can influence the decision to publish. 2. Science journalism being written by people unqualified to analyze the science who instead sensationalize for clicks. I'm not particularly concerned that the people are "only reading papers from big shots". The problem is that prestige is measured by impact analysis rather than replication. If you incentivize a metric which is largely a popularity contest for ideas, it should be no surprise that what you get is popular lies[1]. One might think that the focus on impact in prestigious journals means there might be higher replication in less prestigious journals, but from what I can tell, the less prestigious journals are no better: they're mostly targeting the same metrics, they're just less successful at reaching their goals. [1] My intent is not to actually accusing anyone of intent to deceive here--I just can't quite figure out the right word. Untruths doesn't quite match what I'm trying to say.

brigandish 6y ago

A standardised data format for papers would help (beyond that of introduction > methodology etc, treat it like an insurance claim or something of that ilk), that way content could be distributed and compared and discovered far more easily than having to wade through papers written in different formats, with all the flowery academic language etc. It'd probably make writing them easier too.

6510 6y ago

The w3 (failed) PICS rating system[1] has always fascinated me. In short: Everyone (including the author) gets to rate everything by whatever scale they want to use. Everyone gets to use ratings made by whoever they like. I could see it work as well for academics and intelectuals as bodybuilders and progamers. One angle of my fascination was how the [rating] baby was pretty much tossed out with the bath water. It was shouted down and denounced by journalists (as for example an adult filter which ironically ended up its only application) Some journalists described a perspective as if they had a kind of tenure. They were published for so long that the idea of a rating system was just offensive. We could argue that a good rating system would use existing talent for calibration but the real question to ask imho is: If PICS was so bad, what did we get in stead? Anon 5 star ratings? Thumbs up? HN points? Number of github saved games? To say it doesn't compete with publishing in journals is somewhat of an understatement. End of the day all we are looking for is good meta data. If note worthy people in a field want to endorse a HN topic, a blog posting, a usenet post, a tweet, a youtube video, a facebook posting or a torrent[2] real credit could go to the author. A rating system or spec therefore could simply accommodate that process. (It should for example require the author and their endorsers make backups available.) Journals are from the horse and carriage days. It is quite embarrassing how we didn't come up with something modern. [1] - https://www.w3.org/PICS/services-960303.html https://www.w3.org/PICS/services-960303.html [2] - torrents are nice to share huge data sets

marcus_holmes 6y ago
Totally agree about the entitlement of journalists. I hadn't heard of PICS before, it's interesting. Is there a published story about why it failed anywhere?
- 6510 6y ago
  I cant find anything anymore but the story seems obvious enough. Content rating systems were tried. They didn't gain popularity. Eventually only adult filters and movie classification remained. It isn't just journalists apparently, I think most people hate anything that smells like scrutiny. There are probably plenty under appreciated authors. They ironically have no way of knowing. I'm convinced there are and always have been fantastic ideas out there that didn't even get written down. Why even bother having them or fleshing them out if they cant be judged? In my inner dialog I consider it the greatest puzzle of all times (and expand the scope to rating all human creations) if anyone ever truly solves it all previous revolutions will be reduced to simple events. The potential for recursive self improvement of such system is probably equal to a general AI but the results will be better. haha, with the lack of published stories I just discover I sound to myself like I have a lot of explaining to do making such fancy claims. I will have to ponder writing it into a blog. For now a business plan that everyone is specifically uninterested in is not going to work. But then again, that is like, just everyone's opinion? We have no means of judging the value of it.