3 ms·

Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the

Your concerns are certainly legitimate, and I'm sure there are some situations where they apply. However, in many physical sciences (I would venture to say the vast majority), simulations are not chaotic. They are based on stochastic models that have converging behavior. So, in general, it really is fair for most people to rely on the stability of the simulations. As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used. Let me give an example from my field (high energy physics). The pre-eminent event generator of choice is called PYTHIA. It incorporates as much known physics as possible, and is continually updated. There are groups that regularly convene to compare the distributions it produces to those seen in real experiments and adjust the parameters to improve the results. The code is open source, so you can readily make improvements and submit them to the maintainers for inclusion in official releases. Now the primary uses of these simulations are to either tune your analysis to separate signal from background or calculate corrections for different kinds of detector acceptance issues. No reviewer would ever accept a publication that used simulation if the paper did not include clear evidence that the simulation was valid, usually in the form of a data/simulation comparison. Regarding reproducabilility, I think you're a little off the mark. The idea is not that you should simply repeat my analysis to get the same answer (this important verification step should be, and usually is, present in all scientific groups). Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer. Edit: grammar

jerf 17y ago
"As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.... my field (high energy physics)" Bad example, inasmuch as it is too good. Particle physics has petabytes of data (exabytes yet?) to test against and is very connected to the real world. With that check you can't stray very far. "Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer." I would say a precondition of that step is first that I can reproduce your work. If I can't even reproduce your computation it's not even worth trying to reproduce the eventual results, as your eventual results are too questionable to begin with, as far as I am concerned. If I can't reproduce the computation you might as well have just pulled them from your bum.
- Retric 17y ago
  Weather simulations have even more data; they are less accurate for other reasons. Also, running someone else’s exact simulation again is useless. You need to start from scratch (or some vary well accepted baseline libraries) for it to be useful.
  - jerf 17y ago
    "Weather simulations have even more data; they are less accurate for other reasons." I doubt anyone can beat particle physics for sheer information quantity. I seriously doubt that we have petabytes of real weather information to feed our simulations. I can find some references online to petabyte stores for weather simulation results online, but that's ultimately just cache, not information (in the information theoretic sense). This is part of what I mean when I talk about the information theory, and how you can't get more information in the information-theoretic sense than the sum total of the simulation and the original data. Weather simulations may chew through terabytes or petabytes of RAM in the simulation phase, but they are not fed that much data. If the people involved mistake it for real information, then this is also part of what I mean when I say that once you get into using computers in a big way my training does indeed start giving me standing to complain again by even the most rigid "stay out of my science" standards. Secondly, your assumption that running the simulation again is useless in a world where you can casually assume that you have all the data and the simulation and can have the contempt bred of familiarity for the whole process. In the real world, if you can't replicate the results, how can you criticize the model? I think there's still some "the computer said it, it must be right" underlying your answer; you can't assume the computer model is worth anything, it must be proved and debated and peer reviewed, which is not possible if you can't even get it to run and get the same results. The model is still subject to scientific inquiry, it can't be given a free pass. Independent replication of results is also desirable, but you need both. In a world where nobody can replicate the results and it's hard to verify the simulation against the real world, it's too damned easy to end up with the Feynman electron mass situation where the selection effects from the researchers dominate the putative results of the simulations. The researcher summaries of the results of some runs of some models you can't see or execute and some data you can't get at happen to align... what does it mean? Frankly, who knows?

jerf 17y ago

"As for your other points, the issue of simulation accuracy is already taken very seriously in the areas where I've seen it used.... my field (high energy physics)" Bad example, inasmuch as it is too good. Particle physics has petabytes of data (exabytes yet?) to test against and is very connected to the real world. With that check you can't stray very far. "Ideally, you should collect your own data, make your own simulations, and do your own analysis. Then we should see if we got the same answer." I would say a precondition of that step is first that I can reproduce your work. If I can't even reproduce your computation it's not even worth trying to reproduce the eventual results, as your eventual results are too questionable to begin with, as far as I am concerned. If I can't reproduce the computation you might as well have just pulled them from your bum.

Retric 17y ago
Weather simulations have even more data; they are less accurate for other reasons. Also, running someone else’s exact simulation again is useless. You need to start from scratch (or some vary well accepted baseline libraries) for it to be useful.
- jerf 17y ago
  "Weather simulations have even more data; they are less accurate for other reasons." I doubt anyone can beat particle physics for sheer information quantity. I seriously doubt that we have petabytes of real weather information to feed our simulations. I can find some references online to petabyte stores for weather simulation results online, but that's ultimately just cache, not information (in the information theoretic sense). This is part of what I mean when I talk about the information theory, and how you can't get more information in the information-theoretic sense than the sum total of the simulation and the original data. Weather simulations may chew through terabytes or petabytes of RAM in the simulation phase, but they are not fed that much data. If the people involved mistake it for real information, then this is also part of what I mean when I say that once you get into using computers in a big way my training does indeed start giving me standing to complain again by even the most rigid "stay out of my science" standards. Secondly, your assumption that running the simulation again is useless in a world where you can casually assume that you have all the data and the simulation and can have the contempt bred of familiarity for the whole process. In the real world, if you can't replicate the results, how can you criticize the model? I think there's still some "the computer said it, it must be right" underlying your answer; you can't assume the computer model is worth anything, it must be proved and debated and peer reviewed, which is not possible if you can't even get it to run and get the same results. The model is still subject to scientific inquiry, it can't be given a free pass. Independent replication of results is also desirable, but you need both. In a world where nobody can replicate the results and it's hard to verify the simulation against the real world, it's too damned easy to end up with the Feynman electron mass situation where the selection effects from the researchers dominate the putative results of the simulations. The researcher summaries of the results of some runs of some models you can't see or execute and some data you can't get at happen to align... what does it mean? Frankly, who knows?