3 ms·
They fear what would happen to their reputations if it were obvious how the sausage was being made. (Not unique to climate change: my academic AI research prod
by patio11 16y ago
They fear what would happen to their reputations if it were obvious how the sausage was being made. (Not unique to climate change: my academic AI research produced decent results on some data sets with code of truly abominable quality. Then again, I wasn't asking anybody to bet the economy on my results.)
Plus, if people actually provided code, someone might actually get it into their heads to run it. That can't happen. No, literally, it can't happen. The level of professionalism with regards to source control, documentation, distribution, etc in most academic labs is insufficient to allow the code to be executed outside of the environment that it actually executes on. If you put a tarball up somewhere, somebody who tries to run it is going to get a compile error because they're missing library foo or running on an architecture which doesn't match the byte lengths hardcoded into the assembly file, and then they're going to email you, and that is going to suck your time doing "customer support" when you should be doing what academics actually get paid to do: write grant proposals.
This, by the way, means that peer review by necessity consists of checking that you cited the right people, scratched the right backs, and wrote your paper in the style currently in fashion in your discipline, because reproducing calculations or data sets is virtually impossible.
- philk 16y agoI think pretty much everything looks more impressive before you see how it actually works.
- patio11 16y agoSeriously. Relatedly, it is seriously impressive that systems this comprehensively screwed up seem to still converge on producing acceptable work much of the time. Big companies manage to get us all fed and fly us around the world. That scares the bejeesus out of me. I have put my lives in the hands of someone who was selected by an HR department (assisted by, even worse, a union)
- DevX101 16y agoConvergence of scientific results could be validation of the theory. But it could also be because people are anxious to publish contradictory results. From Richard Feynman's1974 CalTech Commencement: "We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease. But this long history of learning how to not fool ourselves--of having utter scientific integrity--is, I'm sorry to say, something that we haven't specifically included in any particular course that I know of. We just hope you've caught on by osmosis"
- randallsquared 16y agoanxious to publish Should be "anxious not to publish", or "anxious about publishing", I think.
- ZeroGravitas 16y agoI get your point, but those results were still diverging from the original wrong answer and converging on the right one, just not di/converging as fast as would be ideal.
- arethuza 16y agoI don't know about that - I've seen a lot of industrial/manufacturing processes over the years that were damned impressive and had lots of complexities that I had no idea existed.
- cromulent 16y agoThe only exception that I can think of is GPS (not the receivers, the system). Most people simply know that it involves satellites somehow. That system delights me in its cleverness, effectiveness, and scalability.
- abstractbill 16y agoNuclear magnetic resonance, as used to make MRI scans, is pretty damn impressive if you ask me.
- kristiandupont 16y agoThe first time I looked at the code for a large 3D game engine, I felt this. It was almost like someone explaining a magic trick for you, you go "oh, so it all just worked because I was looking from this direction and that detail is something most people don't notice". Very disappointing, and it actually ruined it a bit for me.
- pama 16y agoI am optimistic that more codes will become available soon. This subject is dear to the heart of many scientists and is often discussed -- see this recent HN item: http://news.ycombinator.com/item?id=1789134 http://news.ycombinator.com/item?id=1789134
- roel_v 16y ago"Then again, I wasn't asking anybody to bet the economy on my results." To be fair, no climate scientist is asking anyone to bet the economy on their results, it's the various interest groups that pressure scientists into giving an answer that is less nuanced as they would have liked and then blame them when things go wrong. (at least for the 10 or so I know personally, but I'm pretty sure no others do, either). Actually this is true for all scientists I know in field that have political relevance.
- patio11 16y agoPublicly available statements and evidence of past behavior has led me to reject my default charitable supposition that climate scientists and interests groups are disjoint sets.
- ANH 16y agoI've attended climate science conferences and worked for years writing code for and with climate scientists. Down in the trenches, pretty much the only thing I saw was scientists interested in accurate results. I don't, however, claim to have worked with nearly everyone in the field. But I do on occasion feel compelled to defend my former coworkers (I work for myself now).
- ericb 16y agoCan you link some examples?
- roel_v 16y agoWell I would nuance that, in that there are many interest groups why try to sway public opinion by posing as scientists. Like ANH I write code for and have worked with climate scientists, actual ones who have no real political or financial motivation and just really try to model the world around us as accurately as possible. So I guess it boils down to the definition of 'scientist'. I don't call every quack who get a book on Lulu or marketer who is paid to write scientific-sounding 'position papers' a scientist.
- nollidge 16y ago
- liuliu 16y agoI don't buy the point that once source code is published, you have to do all the "customer support" stuff. You don't have any obligation to do customer support so that you should be able to decide when you'd like to reply the request or whether reply it at all. Source code is a very precise way to exhibit the work you have done. It is so precise that even particular machine can understand it. I am really happy to see that recently in NIPS more and more people published their MATLAB code along with their papers.
- St-Clock 16y agoOh, you must do "customer support"! Otherwise, they will use your code and report 1) that your code does not work or that 2) their code is better because they did not use your code well. And for code that is not easy to install (e.g., requires many libraries, frameworks, dev tools, etc.), you need to spend time on documentation and even then, you often need to answer questions like "how can i create an eclipse plug-in to use your code or lxml failed to install"...
- liuliu 16y agoMostly you only concern about the criticism from within scientific community. Since your result is published, generally, people with background will spend time to figure out how to use it by themselves. And their criticism is only relevant when it gets published. At that point, you are surely obligated to response such criticism. Most of the time, it is really not relevant that some random people send you email and have such "eclipse plug-in" problem.
- RBerenguel 16y agoPeople with background usually don't have the time to really discover how it works to make it run. They'll just ask.
- St-Clock 16y ago"some random people send you email and have such "eclipse plug-in" problem" -> They are not random people, they are Ph.D. students working on their thesis and you can be assured that they will find a way to publish their comparison one way or the other. Some Ph.D. students will go at great length to figure out other's code before asking a question, but some will suck your energy big time and still do a poor job at representing your work. Now compare this to those who do not release their code: they can only be evaluated on their published results and they can spend their time on their next study/research work. My point is that there is not a lot of incentive to release code: it requires a lot of your time, it is not always clear that granting agencies/employers consider this as being more positive than publications, and it exposes your work to all sorts of unfair replication/comparison. Now, if research work without released code were never cited, that would be a terrific incentive to release your code :-) P.S. I speak from experience and I release the code of my most important research work.
- yock 16y agoIf your process isn't sufficiently documented so that your professional peers could reproduce your results with your data, then your process is unreliable. We berate academics all the time for improper or incomplete documentation of manual experiments. I can't see any reason why source code should be held to a lesser standard.
- lutorm 16y agoWell, really it's the algorithm that needs to be published so others can reproduce the results. In principle, publishing code is similar to publishing the sequence to put together your experimental apparatus, which isn't done either.