6 ms·

The challenge will be to fix this whilst keeping the precise, mechanical and quantitative spirit of science alive. The reason why scientists have a natural "di

The challenge will be to fix this whilst keeping the precise, mechanical and quantitative spirit of science alive. The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. Yet the scientific culture of today is falling into that trap itself. The article criticises metric-driven incentives, but it is not metrics (the general concept) that are at fault. It is the choice of metric, and the meta-analysis of this that is lacking. These choices are themselves often backed up by non-scientific vague arguments, of the similar sort that scientists often criticise other fields as depending upon. We must certainly not conclude from these studies that quantitative analysis is itself what is at fault. I know the article doesn't explicitly say this, but it hints at it - using suggestive phrasing like "disdained sociological studies" and referring to all "metric incentives" as a single group - and it is a point I have seen made by many non-scientists. That is, using these flaws as a straw man to attack the very qualities of what has made science so successful and useful. To improve the situation, we must reject these straw-man arguments against science, and develop better methods that are more quantitive, over a broader spectrum of what is being analysed, and that are more self-critical.

YeGoblynQueenne 10y ago
I don't think the problem of the article is with quantitative methods per se. Rather, I read it as a gripe about the specific metrics employed, frex, the h-index. A problem is that any such metric is only ever going to be a proxy for some assumed real value, and such a proxy is always going to be controversial, particularly if it's used to control access to funding. Which I think at the end of the day is the big issue here. All this soul-searching is just a big attempt to decide what we spend money on. Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure.
- infinity0 10y ago
  It could be read either way - it doesn't talk too much about differences between metrics, and groups them as near-equivalent. Anyway the exact intention of the author/article is not so important. My point is to say that one way that you could (and I'm sure some will) interpret it, is not good. Certainly we need to look at our social reward/incentive systems from an adversarial point of view, in science, economics, politics and elsewhere. Game theory is useful for that, and it can help us develop systems that are less-easily susceptible to the flaws that the article mentions.
- hx87 10y ago
  > Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure. And who goes into finance, software engineering, or some other tangentially related field where the skillset is very useful.
Xcelerate 10y ago
> It is the choice of metric, and the meta-analysis of this that is lacking. They need something like a "Journal of Meta-Research", which would research the best ways to perform research. You'd think something like that would get a lot of funding...
richmarr 10y ago
> Yet the scientific culture of today is falling into that trap itself. You're probably being too kind here. The moment of actual falling was decades ago, or more.
maverick_iceman 10y ago
Any metric that is used to judge scientific performance will be subject to targeted manipulation and eventually lose its effectiveness.
- nabla9 10y ago
  Some metrics are more resistant to manipulation than others. Mechanism design (reverse game theory) is subfield of game theory deals with these issues.
  - danieltillett 10y ago
    Do you have any good resources on this topic. Edit. Prof. Google to the rescue [1]. 1. https://en.m.wikipedia.org/wiki/Mechanism_design https://en.m.wikipedia.org/wiki/Mechanism_design
    - nabla9 10y ago
      https://news.ycombinator.com/item?id=12981167 https://news.ycombinator.com/item?id=12981167
tdaltonc 10y ago
> whilst keeping the precise, mechanical and quantitative spirit of science alive It's not obvious to me that measuring scientists or research quality has anything to do with the "spirit of science." Just because science involves very carefully measuring the objects of study, it doesn't follow that science is best served by trying to meta-measure that process. And there is evidence that it can be harmful.
- wojcech 10y ago
  So how do you ensure that sciencists stay careful and try to be objective, how do you have any objectivity in allocation of funds etc. without some metric? I hate the publish or perish system, but that is due to wrong metrics. If (purely pulling this out of my as, probably a horrible idea) a flimsy study counted negatively to your index and a faulty one outright wrecked it (with complete ruin if you do not redo/retract), that would probably skew the metric optimisation to more carefully thought out, substantial research
  - tdaltonc 10y ago
    I'm not sure that scientists need to be super objective for science to work. The history of science is so littered with grudges, feuds, and grit in spite of evidence that it's hard to see the ground of objective rationality that many assert is behind it. And pandemonium models [1] of computation show that the right rules of cummunication can produce astonishing order from chatic self interest. If scientst were forced to pre publish their methods and share all data -- science would probably work better. This is true even if this change had no effect on their behavior, or even if they made every effort to game the system. This imrpovement to science doesn't ask a committee to define or measure anything ineffable, and it doesn't expect individuals to change thier behavior. It just changes the rule of interaction in a way that better favors the systems epistomological progress. [1] https://en.wikipedia.org/wiki/Pandemonium_architecture https://en.wikipedia.org/wiki/Pandemonium_architecture
    - TeMPOraL 10y ago
      Objective scientists are a good thing, but are not - and should never be - considered an essencial component of the system. Any system that assumes honesty and fair play in its participants is doomed to failure. What science was aiming for is getting the true results from the aggregate - thus peer reviews, replication, etc. Of course the more noise you have, the less efficient the system is, so it's good to incentivize people to do honest, objective work - thus pre-publishing / pre-registering, sharing data and algorithms, etc. are all good and important goals. But so should be changing the metric affecting the aggregate - like making sure scientists are actually incentivized to replicate previous work.
jimmaswell 10y ago
>The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. I've run into that myself, as described here: https://news.ycombinator.com/item?id=12271097 https://news.ycombinator.com/item?id=12271097
hammock 10y ago
And yet, the greatest scientists in history did NOT have the stereotypical Scientist's Disdain for sociology or softer topics. For example Einstein said things as "As far as [the laws of mathematics] are certain, they do not refer to reality" and "All religions, arts and sciences are branches of the same tree." There are plenty of brilliant scientists who are held back by their excessive rationality, or whatever you want to call it. Another great book on this topic as it relates to econometrics (as opposed to physics) is "The Romantic Economist" by Nicolson.
- bzbarsky 10y ago
  There's a difference between "disdain for sociology" and "disdain for the way sociology is drawing conclusions from the experiments it performs". Sociology is very important to study. The tools we have for it are not great, in various ways. For example, one can study people and societies by reading Balzac's writing, but the number of people who can produce that sort of thing is fairly limited. We can try to do controlled experiments, but the way we do it in practice is not great. We really really need better tools here...
  - TeMPOraL 10y ago
    Indeed, psychology and sociology are much more difficult fields of study than hard sciences. Physical laws, however tricky they are, don't seem to change at all, and tend to be the same everywhere you look. Psychology is about studying a behaviour of an advanced computing system that's about as smart as the researchers themselves. Sociology is about studying how those advanced computing systems interact with each other at scale. It's insanely difficult, and that's why it's so hard to even come up with an experimental setup that makes some sort of sense. Not to mention ethical issues (many experiments would be so much simpler if you could disregard the well-being of the test subjects). So yeah, personally, I have utmost respect for the complexities involved in sociology - while at the same time I absolutely hate all the bullshit that's being done because doing actual research feels too hard.
- posterboy 10y ago
  Might be just another Eselei of Einstein to say that, init? Sure he can go and say that, and the former is a widely held believe, but he got nothing on pythagoreans, does he? If numbers were surreal, what are we talking about then? If they are just in our imagination, what isn't? Where did he say that? Also, a tree has more than branches. Religion is like a rotten root or some other evil imagery, if you will. Edit: maybe he was a nihilist, I'd tolerate nil as the only non quantitative number. Sorry for the rant, but that's what you get for anecdotal evidence.
yk 10y ago
I think quantitative analysis is at fault. The thing is, mathematics is a language and I can as easily produce a model that encapsulates the lies we tell ourselves in Newspapers, as I can produce a model based on Hitler's Mein Kampf. Actually I thought yesterday about how to misconstruct a human detector for photos, one possibility would be to have face detection and then measure the average pixel color of the face. That would get me probably quite nice detection rates if I test on a dataset that has a racial markup as the average CS lecture. The thing here is, that I push the false negatives purposely to people of color. It is not hard then to invent a story about how 'mathematics proves' that black people are more similar to apes than white people. We currently just don't have anything to ground models of human interaction in,^1 and what is worse most people treat models just as previous generations treated prophecies. They don't understand math and believe in it because they don't understand it. ^1 I am actually not saying that models are worthless, I am saying that most complex models only show what the author wants them to show and they can be as easily manipulated as an essay.
- infinity0 10y ago
  The flaws you're pointing out, are flaws with the non-quantitative arguments used to justify choosing specific flawed quantitative models. This does not mean quantitative analysis as a whole, is flawed. Someone goes and builds a bridge. The bridge collapses because they applied mathematics in a way to maximise profit, with a safety margin just beyond what they can get away with by existing regulations. 5 million other bridge engineers also do the same, so that eventually bridge engineers get a bad rep, and people think bridges are awful awful things. This doesn't mean mathematics or quantitative analysis is at fault - and in fact these tools can be used to examine the incentives and other social dynamics that led to these situations, to be able to fix them more effectively in the future.
  - yk 10y ago
    That is what I meant with 'grounding.' I say that mathematics can be a useful tool, however contrast this with the situation in physics, we have comprehensive theories and we have the situation were we can be very certain that all future theories contain something that can be identified with atoms and even more, if one uses a mathematical theorem then we can be confident that there is something in nature that corresponds to the results. We don't have anything similar in all other intellectual endeavors, but people treat physical and non-physical models similar.
bzbarsky 10y ago
> but it is not metrics (the general concept) that are at fault I disagree, because of https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law The problem is that we want "quality science" (whatever that means!), but we don't know how to quantify that, or indeed really define it. So a quantitative metric will necessarily be measuring some sort of proxy or set of proxies for "quality", and then you will get people optimizing those proxies, not "quality". To the extent that those proxies miss something important, it will be underinvested in. Unfortunately, I don't have a better proposal, perhaps short of taking the warnings in http://mcadams.posc.mu.edu/ike.htm http://mcadams.posc.mu.edu/ike.htm to heart (the ones that are NOT about the military-industrial complex). Doing that might change the general funding climate sufficiently that the need for deciding "quality" like we do now may simply become less critical.
unabst 10y ago
Basically, science is incapable of introspection, at least at this moment in time. But a good scientist will make good science out of any social subject. Academic scientific activity is still mostly a social activity because it is done through the communities and the social interaction between scientists and institutions. Specific problems may be solved by specific brains or groups of brains, but the output and the collaboration is all social. The rewards and incentives are all social systems. And at the end of the day, any time we write a blog or post a comment, it's all social. This is social. But the disdain for social sciences by non-social scientists is discrimination only warranted to the extent they can avoid social and professional interactions with them, and feel immune from any criticism for their bigotry. The moment a scientist wishes to study the social aspects of science itself, they have no choice but to accept social science as a science. But this is okay because reputation has no consequence in problem solving, because reputation is a social device, and not a solvent. The track record of a field, a department of an institution, the history of publications, or the publications of an individual social scientist are all largely irrelevant when faced with your own research targets. The only thing relevant is the available data and the premises chosen for any model. There is no need to criticize the scientific integrity of the work of others if you can do better. Do it yourself if you have to.

YeGoblynQueenne 10y ago

I don't think the problem of the article is with quantitative methods per se. Rather, I read it as a gripe about the specific metrics employed, frex, the h-index. A problem is that any such metric is only ever going to be a proxy for some assumed real value, and such a proxy is always going to be controversial, particularly if it's used to control access to funding. Which I think at the end of the day is the big issue here. All this soul-searching is just a big attempt to decide what we spend money on. Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure.

infinity0 10y ago
It could be read either way - it doesn't talk too much about differences between metrics, and groups them as near-equivalent. Anyway the exact intention of the author/article is not so important. My point is to say that one way that you could (and I'm sure some will) interpret it, is not good. Certainly we need to look at our social reward/incentive systems from an adversarial point of view, in science, economics, politics and elsewhere. Game theory is useful for that, and it can help us develop systems that are less-easily susceptible to the flaws that the article mentions.
hx87 10y ago
> Who gets to be a scientist, and who gets to become a person with a lot of useless knowledge and an incurable sense of failure. And who goes into finance, software engineering, or some other tangentially related field where the skillset is very useful.

Xcelerate 10y ago

> It is the choice of metric, and the meta-analysis of this that is lacking. They need something like a "Journal of Meta-Research", which would research the best ways to perform research. You'd think something like that would get a lot of funding...

richmarr 10y ago

> Yet the scientific culture of today is falling into that trap itself. You're probably being too kind here. The moment of actual falling was decades ago, or more.

maverick_iceman 10y ago

Any metric that is used to judge scientific performance will be subject to targeted manipulation and eventually lose its effectiveness.

nabla9 10y ago
Some metrics are more resistant to manipulation than others. Mechanism design (reverse game theory) is subfield of game theory deals with these issues.
- danieltillett 10y ago
  Do you have any good resources on this topic. Edit. Prof. Google to the rescue [1]. 1. https://en.m.wikipedia.org/wiki/Mechanism_design https://en.m.wikipedia.org/wiki/Mechanism_design
  - nabla9 10y ago
    https://news.ycombinator.com/item?id=12981167 https://news.ycombinator.com/item?id=12981167

tdaltonc 10y ago

> whilst keeping the precise, mechanical and quantitative spirit of science alive It's not obvious to me that measuring scientists or research quality has anything to do with the "spirit of science." Just because science involves very carefully measuring the objects of study, it doesn't follow that science is best served by trying to meta-measure that process. And there is evidence that it can be harmful.

wojcech 10y ago
So how do you ensure that sciencists stay careful and try to be objective, how do you have any objectivity in allocation of funds etc. without some metric? I hate the publish or perish system, but that is due to wrong metrics. If (purely pulling this out of my as, probably a horrible idea) a flimsy study counted negatively to your index and a faulty one outright wrecked it (with complete ruin if you do not redo/retract), that would probably skew the metric optimisation to more carefully thought out, substantial research
- tdaltonc 10y ago
  I'm not sure that scientists need to be super objective for science to work. The history of science is so littered with grudges, feuds, and grit in spite of evidence that it's hard to see the ground of objective rationality that many assert is behind it. And pandemonium models [1] of computation show that the right rules of cummunication can produce astonishing order from chatic self interest. If scientst were forced to pre publish their methods and share all data -- science would probably work better. This is true even if this change had no effect on their behavior, or even if they made every effort to game the system. This imrpovement to science doesn't ask a committee to define or measure anything ineffable, and it doesn't expect individuals to change thier behavior. It just changes the rule of interaction in a way that better favors the systems epistomological progress. [1] https://en.wikipedia.org/wiki/Pandemonium_architecture https://en.wikipedia.org/wiki/Pandemonium_architecture
  - TeMPOraL 10y ago
    Objective scientists are a good thing, but are not - and should never be - considered an essencial component of the system. Any system that assumes honesty and fair play in its participants is doomed to failure. What science was aiming for is getting the true results from the aggregate - thus peer reviews, replication, etc. Of course the more noise you have, the less efficient the system is, so it's good to incentivize people to do honest, objective work - thus pre-publishing / pre-registering, sharing data and algorithms, etc. are all good and important goals. But so should be changing the metric affecting the aggregate - like making sure scientists are actually incentivized to replicate previous work.

jimmaswell 10y ago

>The reason why scientists have a natural "disdain" for sociological studies, is because they don't have these qualities. Many of the arguments aren't convincing from a critical angle, and only convincing to people who have a pre-existing bias towards certain conclusions. I've run into that myself, as described here: https://news.ycombinator.com/item?id=12271097 https://news.ycombinator.com/item?id=12271097

hammock 10y ago

And yet, the greatest scientists in history did NOT have the stereotypical Scientist's Disdain for sociology or softer topics. For example Einstein said things as "As far as [the laws of mathematics] are certain, they do not refer to reality" and "All religions, arts and sciences are branches of the same tree." There are plenty of brilliant scientists who are held back by their excessive rationality, or whatever you want to call it. Another great book on this topic as it relates to econometrics (as opposed to physics) is "The Romantic Economist" by Nicolson.

bzbarsky 10y ago
There's a difference between "disdain for sociology" and "disdain for the way sociology is drawing conclusions from the experiments it performs". Sociology is very important to study. The tools we have for it are not great, in various ways. For example, one can study people and societies by reading Balzac's writing, but the number of people who can produce that sort of thing is fairly limited. We can try to do controlled experiments, but the way we do it in practice is not great. We really really need better tools here...
- TeMPOraL 10y ago
  Indeed, psychology and sociology are much more difficult fields of study than hard sciences. Physical laws, however tricky they are, don't seem to change at all, and tend to be the same everywhere you look. Psychology is about studying a behaviour of an advanced computing system that's about as smart as the researchers themselves. Sociology is about studying how those advanced computing systems interact with each other at scale. It's insanely difficult, and that's why it's so hard to even come up with an experimental setup that makes some sort of sense. Not to mention ethical issues (many experiments would be so much simpler if you could disregard the well-being of the test subjects). So yeah, personally, I have utmost respect for the complexities involved in sociology - while at the same time I absolutely hate all the bullshit that's being done because doing actual research feels too hard.
posterboy 10y ago
Might be just another Eselei of Einstein to say that, init? Sure he can go and say that, and the former is a widely held believe, but he got nothing on pythagoreans, does he? If numbers were surreal, what are we talking about then? If they are just in our imagination, what isn't? Where did he say that? Also, a tree has more than branches. Religion is like a rotten root or some other evil imagery, if you will. Edit: maybe he was a nihilist, I'd tolerate nil as the only non quantitative number. Sorry for the rant, but that's what you get for anecdotal evidence.

yk 10y ago

I think quantitative analysis is at fault. The thing is, mathematics is a language and I can as easily produce a model that encapsulates the lies we tell ourselves in Newspapers, as I can produce a model based on Hitler's Mein Kampf. Actually I thought yesterday about how to misconstruct a human detector for photos, one possibility would be to have face detection and then measure the average pixel color of the face. That would get me probably quite nice detection rates if I test on a dataset that has a racial markup as the average CS lecture. The thing here is, that I push the false negatives purposely to people of color. It is not hard then to invent a story about how 'mathematics proves' that black people are more similar to apes than white people. We currently just don't have anything to ground models of human interaction in,^1 and what is worse most people treat models just as previous generations treated prophecies. They don't understand math and believe in it because they don't understand it. ^1 I am actually not saying that models are worthless, I am saying that most complex models only show what the author wants them to show and they can be as easily manipulated as an essay.

infinity0 10y ago
The flaws you're pointing out, are flaws with the non-quantitative arguments used to justify choosing specific flawed quantitative models. This does not mean quantitative analysis as a whole, is flawed. Someone goes and builds a bridge. The bridge collapses because they applied mathematics in a way to maximise profit, with a safety margin just beyond what they can get away with by existing regulations. 5 million other bridge engineers also do the same, so that eventually bridge engineers get a bad rep, and people think bridges are awful awful things. This doesn't mean mathematics or quantitative analysis is at fault - and in fact these tools can be used to examine the incentives and other social dynamics that led to these situations, to be able to fix them more effectively in the future.
- yk 10y ago
  That is what I meant with 'grounding.' I say that mathematics can be a useful tool, however contrast this with the situation in physics, we have comprehensive theories and we have the situation were we can be very certain that all future theories contain something that can be identified with atoms and even more, if one uses a mathematical theorem then we can be confident that there is something in nature that corresponds to the results. We don't have anything similar in all other intellectual endeavors, but people treat physical and non-physical models similar.

bzbarsky 10y ago

> but it is not metrics (the general concept) that are at fault I disagree, because of https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law The problem is that we want "quality science" (whatever that means!), but we don't know how to quantify that, or indeed really define it. So a quantitative metric will necessarily be measuring some sort of proxy or set of proxies for "quality", and then you will get people optimizing those proxies, not "quality". To the extent that those proxies miss something important, it will be underinvested in. Unfortunately, I don't have a better proposal, perhaps short of taking the warnings in http://mcadams.posc.mu.edu/ike.htm http://mcadams.posc.mu.edu/ike.htm to heart (the ones that are NOT about the military-industrial complex). Doing that might change the general funding climate sufficiently that the need for deciding "quality" like we do now may simply become less critical.

unabst 10y ago

Basically, science is incapable of introspection, at least at this moment in time. But a good scientist will make good science out of any social subject. Academic scientific activity is still mostly a social activity because it is done through the communities and the social interaction between scientists and institutions. Specific problems may be solved by specific brains or groups of brains, but the output and the collaboration is all social. The rewards and incentives are all social systems. And at the end of the day, any time we write a blog or post a comment, it's all social. This is social. But the disdain for social sciences by non-social scientists is discrimination only warranted to the extent they can avoid social and professional interactions with them, and feel immune from any criticism for their bigotry. The moment a scientist wishes to study the social aspects of science itself, they have no choice but to accept social science as a science. But this is okay because reputation has no consequence in problem solving, because reputation is a social device, and not a solvent. The track record of a field, a department of an institution, the history of publications, or the publications of an individual social scientist are all largely irrelevant when faced with your own research targets. The only thing relevant is the available data and the premises chosen for any model. There is no need to criticize the scientific integrity of the work of others if you can do better. Do it yourself if you have to.