4 ms·

I think computer models just mean you do replication the other way around. Of course anyone who runs the same program gets the same results, so ordinary replic

by bjaress 14y ago

I think computer models just mean you do replication the other way around. Of course anyone who runs the same program gets the same results, so ordinary replication is pointless--instead you answer the same question with a totally different program and see if your answer is close. And then, of course, you wait and see if the predictions of the models come true. You can't reset the world and re-test it, but you can re-run the models and ask for more prediction in the future and wait some more. Climate scientists do both of those things all the time because they're in one of the most heavily scrutinized fields. Isolating variables just means you compare two setups with everything the same except one. That's actually one of the things the models are for. You can't re-run the world without humans, but you can re-run the model with humans turned off. Then someone else can do the same with their totally different model. And if both of your answers match reality with humans on and each other with humans off, well maybe the difference between humans on and humans off is the impact of humans. Or maybe not. But adding more different models helps. In short: climate science generates testable hypotheses, does replication, and isolates variables. It's possible they're wrong (and publishing source code is a good idea) but they don't have a methodology problem. And they're probably right. Also, Michael Crichton basically wrote Hollywood scripts in novel form. He's not a good source on anything.

ArbitraryLimits 14y ago
> Of course anyone who runs the same program gets the same results, so ordinary replication is pointless That would be true, if anyone actually distributed their actual code. Pick a journal article at random in any field that describes results from a computational model, and 99% of the paper will describe the results and not the model. The paper will never contain the complete code (which is fair enough, since it would be too long); 1 paper out of 100 will have excerpts of the code, and another 10/100 will have a URL that claims to have the code. If you actually follow that link, you'll find that 2-3 times out of 10 the code won't actually compile or run, and 9 times out of 10, the figures in the paper were generated by tweeking some parameters not defined in the paper whose particular values the author never recorded, and so even the author couldn't reproduce what he actually published, even if he wanted to. Some people are trying to make institutional change here, see http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pdf http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pd... But really it's a pretty sad state of affairs.
nkurz 14y ago
> He's not a good source on anything. Perhaps you mean this in the sense that we should trust in science and not in authority? Because as hack writers go, his academic qualifications are better than most. He's an intelligent guy, and it seems fair to give at least some weight to his opinion. Equally, it seems unfair to presume a priori that he's not a good source of information. At the least, you'd should argue "He's looney about X, which we all agree is false, therefore we should not trust him on Y". Simply saying "He's looney about X" doesn't add much information and merely pits your authority against his. I agree with your summary of computer models, and your basic judgement of climate scientists, but fear that many of the influential climate science papers don't adhere to this standard. Frequently the technique is to tweak the parameters of a number of models until each creates "realistic" results, generate a small number (1-3) of simulations with each model, and then create an unweighted average of this ensemble so that the high and low estimates cancel. The meaning of this is much harder to interpret than the case where all the models predict a similar outcome with identical inputs. http://www.ipcc.ch/ipccreports/tar/wg1/347.htm#tab91 http://www.ipcc.ch/ipccreports/tar/wg1/347.htm#tab91 ---- CRICHTON, (John) Michael. American. Born in Chicago, Illinois, October 23, 1942. Died in Los Angeles, November 4, 2008. Educated at Harvard University, Cambridge, Massachusetts, A.B. (summa cum laude) 1964 (Phi Beta Kappa). Henry Russell Shaw Travelling Fellow, 1964-65. Visiting Lecturer in Anthropology at Cambridge University, England, 1965. Graduated Harvard Medical School, M.D. 1969; post-doctoral fellow at the Salk Institute for Biological Sciences, La Jolla, California 1969-1970. Visiting Writer, Massachusetts Institute of Technology, 1988. http://www.michaelcrichton.net/aboutmichaelcrichton-biography.html http://www.michaelcrichton.net/aboutmichaelcrichton-biograph...
- Luc 14y ago
  Crichton's scientific standards can't have been THAT high, considering how he got hoodwincked into believing Jack Houck's whole spoon bending spiel. I've attended one of Houck's parties and the whole thing was quite remarkable, not for the spoon bending but for the gullability of 90% of the people there.
  - glanch 14y ago
    Wait, can someone explain how/why the spoon bending thing works? What's the trick?

by bjaress 14y ago

ArbitraryLimits 14y ago

> Of course anyone who runs the same program gets the same results, so ordinary replication is pointless That would be true, if anyone actually distributed their actual code. Pick a journal article at random in any field that describes results from a computational model, and 99% of the paper will describe the results and not the model. The paper will never contain the complete code (which is fair enough, since it would be too long); 1 paper out of 100 will have excerpts of the code, and another 10/100 will have a URL that claims to have the code. If you actually follow that link, you'll find that 2-3 times out of 10 the code won't actually compile or run, and 9 times out of 10, the figures in the paper were generated by tweeking some parameters not defined in the paper whose particular values the author never recorded, and so even the author couldn't reproduce what he actually published, even if he wanted to. Some people are trying to make institutional change here, see http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pdf http://www-stat.stanford.edu/~wavelab/Wavelab_850/wavelab.pd... But really it's a pretty sad state of affairs.

nkurz 14y ago

> He's not a good source on anything. Perhaps you mean this in the sense that we should trust in science and not in authority? Because as hack writers go, his academic qualifications are better than most. He's an intelligent guy, and it seems fair to give at least some weight to his opinion. Equally, it seems unfair to presume a priori that he's not a good source of information. At the least, you'd should argue "He's looney about X, which we all agree is false, therefore we should not trust him on Y". Simply saying "He's looney about X" doesn't add much information and merely pits your authority against his. I agree with your summary of computer models, and your basic judgement of climate scientists, but fear that many of the influential climate science papers don't adhere to this standard. Frequently the technique is to tweak the parameters of a number of models until each creates "realistic" results, generate a small number (1-3) of simulations with each model, and then create an unweighted average of this ensemble so that the high and low estimates cancel. The meaning of this is much harder to interpret than the case where all the models predict a similar outcome with identical inputs. http://www.ipcc.ch/ipccreports/tar/wg1/347.htm#tab91 http://www.ipcc.ch/ipccreports/tar/wg1/347.htm#tab91 ---- CRICHTON, (John) Michael. American. Born in Chicago, Illinois, October 23, 1942. Died in Los Angeles, November 4, 2008. Educated at Harvard University, Cambridge, Massachusetts, A.B. (summa cum laude) 1964 (Phi Beta Kappa). Henry Russell Shaw Travelling Fellow, 1964-65. Visiting Lecturer in Anthropology at Cambridge University, England, 1965. Graduated Harvard Medical School, M.D. 1969; post-doctoral fellow at the Salk Institute for Biological Sciences, La Jolla, California 1969-1970. Visiting Writer, Massachusetts Institute of Technology, 1988. http://www.michaelcrichton.net/aboutmichaelcrichton-biography.html http://www.michaelcrichton.net/aboutmichaelcrichton-biograph...

Luc 14y ago
Crichton's scientific standards can't have been THAT high, considering how he got hoodwincked into believing Jack Houck's whole spoon bending spiel. I've attended one of Houck's parties and the whole thing was quite remarkable, not for the spoon bending but for the gullability of 90% of the people there.
- glanch 14y ago
  Wait, can someone explain how/why the spoon bending thing works? What's the trick?