3 ms·
If the benefits can't be measured, why are we wasting money on them? I can also give you oodles of unmeasureable and unquantifiable benefits. Just send me a big
by yummyfajitas 13y ago
If the benefits can't be measured, why are we wasting money on them? I can also give you oodles of unmeasureable and unquantifiable benefits. Just send me a big check and then don't try to box me in with oversight or accountability.
(There are of course some measurable benefits from the current system. For example: http://www.opensecrets.org/industries/indus.php?ind=W04 http://www.opensecrets.org/industries/indus.php?ind=W04 )
- rmk2 13y agoExcellent, alongside the Humanities, then, we can also close down physics, because, clearly, if benefits can't be measured, why waste money? Many of the "impact" arguments assume that the only victims will be the humanities, a sacrifice lots of people are very willing to make. (The original article talks, albeit not quite as strongly, of a hemorrhaging that is slowly happening in terms of funding, jobs and perspective.) However, there is some other, less obvious, collateral damage: physics, mathematics (except for applied maths), economics all share similar problems. And here are some choice quotes about the UK's REF and its "Impact" pilot study, with "Impact" being an integral part of the future score (20%, which is going to go up to 25% for the next one) that decides about research spending from Research Councils over the next couple years, due in 2014: > "Physicists had some particular concerns about the challenge of assessing impact in the manner foreseen by HEFCE. 1. The difficulty in attributing UoA-specific ‘ownership’ of impact that arises from the work of a large collaboration, either experimental (in our case, a ten-institute international collaboration working on experiments at CERN) or theoretical (an eight-institute collaboration using High Performance Computing facilities). 2. The blurry distinction between the ‘use of’ new high performance computers and the ‘stimulation of’ the development of these machines. At what point does close collaboration with a company such as IBM in the development of teraflop or petaflop computers become ‘impact’? 4. Commercial confidentiality became an issue. Even in this pilot exercise, and making use of the ‘Confidential’ tag on the submissions, we still had to remove some of the strongest material concerning cooperation with industrial companies because of confidentiality agreements. In REF itself, this would be a huge problem. It should be noted that this also contributed to the Institute of Physics giving up on a recent exercise to try to quantify the impact of Physics. It is impossible to quantify the impact of public understanding of science initiatives such as TV and radio appearances, popular lectures, books, etc at the scale of individuals or groups. Across the community, such work clearly has an important impact, but this cannot be sensibly quantified." (p. 29) > "Several respondents noted that research impact was most readily observed and measured where the distance between the two realms, research and impact, was relatively short and the pathways rather direct: for example, applied scientists working in collaboration with a private business seeking expressly to translate the generic insights into commercial advantage. The socio-economic impact of more fundamental research might go unnoticed and unremarked most of the time, reflecting the diffuse, cumulative and rather unpredictable nature of intellectual advances. Indeed, one might imagine that the occasion where a fundamental breakthrough produces a relatively immediate and direct social improvement is rare, even anomalous. In practical terms, this means the impact case studies track back to particular types of people and types of work. It might also mean that some of the most consequential socio-economic contributions will be excluded by virtue of their rather indirect link with cutting-edge research. There was a suggestion that institutions had taken the low hanging fruit, and that in some sense the portfolio of case studies was not a good representation of the breadth of research undertaken. This sentiment appears to be rooted in a general sense that impact is most obvious in some narrow areas, within subjects rather than between them, and largely unknowable, at least in practical terms, in most instances. There were also suggestions made that a proportion of the more significant, evident impacts was linked to people and activities outside the very best academic research, albeit dependent upon that work, and which would be overlooked as a result. There is always going to be an over-reliance on a small number of sub-groups, even individuals, whose research happens to lend itself to non-academic impact. The impact submission is not necessarily representative of the UoA as a whole." (p. 30) > "The biggest challenge was the need to acquire evidence to reveal the reach and significance of a given impact. There were many practical issues that stood in the way, and the response of the majority, for the pilot exercise at least, was to firstly focus on the more obvious cases and, secondly, to use whatever narratives, references and statistics that came to hand readily." (p. 31) Source: REF Research Impact Pilot Exercise Lessons-Learned Project Feedback on Pilot Submissions, November 2010 (http://www.ref.ac.uk/pubs/refimpactpilotlessons-learnedfeedbackonpilotsubmissions/ http://www.ref.ac.uk/pubs/refimpactpilotlessons-learnedfeedb...)
- yummyfajitas 13y agoI'm quite happy to cut unproductive parts of physics, math and economics. Cutting the entirety of these fields seems counterproductive, since the benefits of the fields as a whole are quite clear. Semiconductor physics alone has certainly paid for all of physics and operations research alone has certainly paid for all of math. But if we can identify useless segments of these fields, I favor cutting them. I see no good reason any particular endeavor should be exempt from a cost/benefit analysis.
- waqf 13y agoRight, the whole problem is that most of the subfields look probably-useless until someone goes all the way down the path and discovers a use. So the benefit side of the cost-benefit looks like huge benefit times small probability, and that's a calculation that's very sensitive to error. To continue your example: one of my ancestors worked on investigating the semiconducting properties of germanium in the '30s, but he couldn't think of anything it might be useful for. Oddly enough, he's not mentioned in the history books.
- yummyfajitas 13y agoAs I said, "if we can identify useless segments of these fields". I didn't say we could actually do that. I'm not a strong proponent of cutting scientific funding. Science contains a lot of stories about random seemingly useless research coming in handy in 10-30 years. Medieval literature has far fewer stories, however. (Incidentally, in case you think we don't currently attempt to predict the future uses of assorted subfields, you've clearly never looked at the NSF budget.)
- _delirium 13y agoI would put the burden of proof on the 'radicals' in this case: if you believe that the traditional scientific research system can be cut in half, and/or replaced by startups and other endeavors, and still produce the same or more advances in science, I'd like to see evidence supporting that hypothesis. Put in Bayesian terms, I put a reasonably strong prior on the current system successfully generating science, so radical changes to it, like significantly gutting universities, need evidence to justify them. If we had metrics that fully explained where scientific advances come from and how they feed back into engineering and other areas, in at least the medium-term, we could use those as a guide for changes. But the metrics so far are about as good as lines-of-code metrics: not completely noise, but relatively low explanatory value. Metrics are only good if they can be validated: if you can come up with metrics for scientific productivity and also show that your metrics explain a large portion of scientific advances and their later use, then they might be worth taking seriously. So far, though, the metrics are relatively poor. But that does not prove the opposite: that scientific advances don't happen, and don't get later commercialized. The fact that one doesn't fully understand why scientific advances happen, and have not successfully found a way to attach lines-of-code-style metrics to measuring them, is not actual evidence that the current system for scientific research doesn't work, or that any particular proposed change will be neutral to positive.
- yummyfajitas 13y ago...if you believe that the traditional scientific research system can be cut in half... Where did you get the idea that I believe this? I'm simply arguing that no endeavor should be exempt from cost/benefit analysis.
- _delirium 13y agoIf there's a validated metric on the 'benefit' side, sure. What I'm skeptical of is making up some unrigorous numbers to do quantitative cost-benefit analysis when a system isn't well understood enough to be quantified with any degree of confidence in the numbers, at least at present. Assuming anything we can't measure well yet is ~0 is also not a very rigorous approach. One cautious approach could be to try to come up with lower and upper bounds on benefits, e.g. that with high confidence, we can conclude that academic physics research, or academic geography research, or whatever, has had at least $N but not more than $M impact on the U.S. economy, taking into account direct and indirect effects over a period of, say, 30 years. I suspect the space between those bounds would be large, but we would at least be able to say for sure (for certain values of "sure") that quantifiable benefit lies within a certain range. I'd be interested in that research, if there is any, but these kinds of estimates are notoriously difficult to establish, given that history isn't an experimental science.