3 ms·

> whilst keeping the precise, mechanical and quantitative spirit of science alive It's not obvious to me that measuring scientists or research quality has anyt

> whilst keeping the precise, mechanical and quantitative spirit of science alive It's not obvious to me that measuring scientists or research quality has anything to do with the "spirit of science." Just because science involves very carefully measuring the objects of study, it doesn't follow that science is best served by trying to meta-measure that process. And there is evidence that it can be harmful.

wojcech 10y ago
So how do you ensure that sciencists stay careful and try to be objective, how do you have any objectivity in allocation of funds etc. without some metric? I hate the publish or perish system, but that is due to wrong metrics. If (purely pulling this out of my as, probably a horrible idea) a flimsy study counted negatively to your index and a faulty one outright wrecked it (with complete ruin if you do not redo/retract), that would probably skew the metric optimisation to more carefully thought out, substantial research
- tdaltonc 10y ago
  I'm not sure that scientists need to be super objective for science to work. The history of science is so littered with grudges, feuds, and grit in spite of evidence that it's hard to see the ground of objective rationality that many assert is behind it. And pandemonium models [1] of computation show that the right rules of cummunication can produce astonishing order from chatic self interest. If scientst were forced to pre publish their methods and share all data -- science would probably work better. This is true even if this change had no effect on their behavior, or even if they made every effort to game the system. This imrpovement to science doesn't ask a committee to define or measure anything ineffable, and it doesn't expect individuals to change thier behavior. It just changes the rule of interaction in a way that better favors the systems epistomological progress. [1] https://en.wikipedia.org/wiki/Pandemonium_architecture https://en.wikipedia.org/wiki/Pandemonium_architecture
  - TeMPOraL 10y ago
    Objective scientists are a good thing, but are not - and should never be - considered an essencial component of the system. Any system that assumes honesty and fair play in its participants is doomed to failure. What science was aiming for is getting the true results from the aggregate - thus peer reviews, replication, etc. Of course the more noise you have, the less efficient the system is, so it's good to incentivize people to do honest, objective work - thus pre-publishing / pre-registering, sharing data and algorithms, etc. are all good and important goals. But so should be changing the metric affecting the aggregate - like making sure scientists are actually incentivized to replicate previous work.
- nonbel 10y ago
  This was figured out long ago. Before people should believe you know what you are talking about, you need to make precise and accurate predictions about the future (comet should reappear on x date) or perform engineering feats (pull a ship out of the water using only your own manpower). People just don't like this real solution because it means many (really, most) of the wild speculations they hold as dear dogma (and have been calling science) will need to be reassessed using the correct metric. There will undoubtedly be a lot of egg on many faces.
- bzbarsky 10y ago
  > how do you have any objectivity in allocation of funds etc. without some metric? Our need for "objectivity" here is to a large extent driven by the centralization of the funding allocation that http://mcadams.posc.mu.edu/ike.htm http://mcadams.posc.mu.edu/ike.htm warned about and has since come to pass. If funding were more decentralized, lack of objectivity in any particular funding source would be _much_ less of a problem.
  - tdaltonc 10y ago
    Specifically: The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present – and is gravely to be regarded.
    - bzbarsky 10y ago
      Indeed. Since then it has gone from "prospect" to "it's been this way for decades"... There's lots more in there that usually doesn't get quoted (unlike the bit about the military-industrial complex) and is spot-on prescient.
infinity0 10y ago
It wasn't obvious to people living thousands of years ago that science would be the way to move civilisation forward, but here we are. And just because it "doesn't follow" that these methods would improve science when applied to itself, doesn't mean that the latter idea is false either. We follow these principles out of observing history. But part of it is an act of faith, I'll give you that. What sort of "evidence" are you referring to, that it would be harmful?

wojcech 10y ago

So how do you ensure that sciencists stay careful and try to be objective, how do you have any objectivity in allocation of funds etc. without some metric? I hate the publish or perish system, but that is due to wrong metrics. If (purely pulling this out of my as, probably a horrible idea) a flimsy study counted negatively to your index and a faulty one outright wrecked it (with complete ruin if you do not redo/retract), that would probably skew the metric optimisation to more carefully thought out, substantial research

tdaltonc 10y ago
I'm not sure that scientists need to be super objective for science to work. The history of science is so littered with grudges, feuds, and grit in spite of evidence that it's hard to see the ground of objective rationality that many assert is behind it. And pandemonium models [1] of computation show that the right rules of cummunication can produce astonishing order from chatic self interest. If scientst were forced to pre publish their methods and share all data -- science would probably work better. This is true even if this change had no effect on their behavior, or even if they made every effort to game the system. This imrpovement to science doesn't ask a committee to define or measure anything ineffable, and it doesn't expect individuals to change thier behavior. It just changes the rule of interaction in a way that better favors the systems epistomological progress. [1] https://en.wikipedia.org/wiki/Pandemonium_architecture https://en.wikipedia.org/wiki/Pandemonium_architecture
- TeMPOraL 10y ago
  Objective scientists are a good thing, but are not - and should never be - considered an essencial component of the system. Any system that assumes honesty and fair play in its participants is doomed to failure. What science was aiming for is getting the true results from the aggregate - thus peer reviews, replication, etc. Of course the more noise you have, the less efficient the system is, so it's good to incentivize people to do honest, objective work - thus pre-publishing / pre-registering, sharing data and algorithms, etc. are all good and important goals. But so should be changing the metric affecting the aggregate - like making sure scientists are actually incentivized to replicate previous work.
nonbel 10y ago
This was figured out long ago. Before people should believe you know what you are talking about, you need to make precise and accurate predictions about the future (comet should reappear on x date) or perform engineering feats (pull a ship out of the water using only your own manpower). People just don't like this real solution because it means many (really, most) of the wild speculations they hold as dear dogma (and have been calling science) will need to be reassessed using the correct metric. There will undoubtedly be a lot of egg on many faces.
bzbarsky 10y ago
> how do you have any objectivity in allocation of funds etc. without some metric? Our need for "objectivity" here is to a large extent driven by the centralization of the funding allocation that http://mcadams.posc.mu.edu/ike.htm http://mcadams.posc.mu.edu/ike.htm warned about and has since come to pass. If funding were more decentralized, lack of objectivity in any particular funding source would be _much_ less of a problem.
- tdaltonc 10y ago
  Specifically: The prospect of domination of the nation's scholars by Federal employment, project allocations, and the power of money is ever present – and is gravely to be regarded.
  - bzbarsky 10y ago
    Indeed. Since then it has gone from "prospect" to "it's been this way for decades"... There's lots more in there that usually doesn't get quoted (unlike the bit about the military-industrial complex) and is spot-on prescient.

infinity0 10y ago

It wasn't obvious to people living thousands of years ago that science would be the way to move civilisation forward, but here we are. And just because it "doesn't follow" that these methods would improve science when applied to itself, doesn't mean that the latter idea is false either. We follow these principles out of observing history. But part of it is an act of faith, I'll give you that. What sort of "evidence" are you referring to, that it would be harmful?