6 ms·

As someone with personal experience inside academic psychology and academic computer science, it's my impression that psychology is far stronger as a field, hav

by tedks 13y ago

As someone with personal experience inside academic psychology and academic computer science, it's my impression that psychology is far stronger as a field, having a much more rigorous community dedicated to scientific truth, and exhaustive standards of review and experimental design. But, I'm sure that your general opinion is well-informed as well, or you wouldn't make such strong claims about a very broad field. What's your background in psychology, and what makes you say the things you do about it?

deleted 13y ago
[deleted]
colechristensen 13y ago
>But, I'm sure that your general opinion is well-informed as well, or you wouldn't make such strong claims about a very broad field. I don't believe you are being genuine here at all. This passive-aggressive, sarcastic response is not appreciated. If you don't like what I wrote, respond directly and honestly.
- jdn 13y ago
  Well dodged
adrianbye 13y ago
Perhaps your experience inside psychology is limited? Nobel prize winner Daniel Kahneman[1] would disagree with you, along with social psychologist Jonathan Haidt[2]. [1] http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... [2] http://www.nytimes.com/2011/02/08/science/08tier.html http://www.nytimes.com/2011/02/08/science/08tier.html
- btilly 13y ago
  This point is critical. A study can have a good methodology. But if nobody tries to replicate it, you have no chance of finding out that you are missing something critical. And with something as complex as people, "missing something critical" is something that we need to figure out. And this isn't exactly a new problem for psychology. Feynman somewhat famously questioned a lot of classic research with http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html. I say somewhat famously because people in areas like physics have all read that essay. But in my experience psychologists are a little less inclined to read the criticism. Even at the best of times, replication of results in a complex system is hard. See, for instance, http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... for evidence that most published medical research is wrong. But with psychology it is even worse. We lump people together by symptom, not by root cause. It is like lumping together people with migraines, head injuries, and caffeine headaches together. Then you find out that coffee beats a placebo for helping headaches, and give them all coffee! You think I'm exaggerating? Read http://www.nimh.nih.gov/about/director/2013/transforming-diagnosis.shtml http://www.nimh.nih.gov/about/director/2013/transforming-dia... for verification that the NIH considers it a high priority research item to come up with classifications based on root causes, so that we have a chance of even being able to do useful research! Yes, psychology has a lot of well-meaning people, and they stumble across a lot of interesting stuff. But it should be read with a critical eye, because the field as a whole does not perform reliably enough to compel our trust.
  - thaumasiotes 13y ago
    > But with psychology it is even worse. We lump people together by symptom, not by root cause. It is like lumping together people with migraines, head injuries, and caffeine headaches together. Then you find out that coffee beats a placebo for helping headaches, and give them all coffee! This happens (in medicine, not just psychology), and there's definitely room for hypothetical improvement. But in many cases, it's hard to blame people, because there's no way to do better without doing an unknown but gargantuan amount of research. For example, lupus is (or was when I had the discussion with my mother) defined as exhibiting X of a list of Y symptoms, where Y > X. This leads to situations like the following: 1. You, the patient, present with X-1 symptoms of lupus. By definition, you don't have lupus, and treatment for lupus is not warranted. 2. Time passes. 3. You, the patient, come back to your doctor with one additional symptom of lupus. Medical doctrine now states that you have lupus (not so interesting) and that, more interestingly, those X-1 earlier symptoms are symptoms of lupus (which you didn't have when you developed those symptoms), not of some other yet-undiagnosed problem. You can now be treated for lupus. Obviously, that satisfies no one and is crying out for a more "reality-based" definition. The problem is that no one has a candidate for the actual physical causes of lupus. The best anyone's ever been able to do is recognize that the same set of treatments are broadly effective against a constellation of symptoms, which may seem to be related to each other or not, but which co-occur to some degree in people who respond to these treatments. Of course it's a high priority to learn what the root causes of symptoms are, but that doesn't mean it's easy or even, in the general case, possible. You have to start somewhere.
- tedks 13y ago
  Oh no, I'm well aware of those -- but academic computer science is even worse! It's very new to have artifact evaluation at all, meaning almost any published result that represents an evaluation of a system (as opposed to something purely theoretical) is probably bunk. Almost no academic systems are released publicly for evaluation, so reproduction isn't just something that should be done, but isn't -- it's entirely impossible. This is basically the same as publishing a psych paper with a Methods section including a line that says "and then magic occurs". It's like an unsafe cast in a programming language -- you totally escape the guarantees provided by the wider structure of the academic system. Academia in general has serious problems. There's a fundamental problem of incentive, and until that's solved, nothing's really going to change. But psychology is definitely unfairly maligned. (It doesn't help that plenty of people doing doing what's almost psuedoscience are still harbored in a lot of psych departments, but their methods are still more rigorous than HCI research in most cases.)
qq66 13y ago
Psychology isn't really comparable to computer science since computer science is the study of a manmade system, making it more like mathematics. Computer science is not a traditional "science" studying the mechanisms of an unknown phenomenon, so concepts like "experimental design" aren't easily mappable.
- tedks 13y ago
  Theoretical computer science is much like mathematics. But virtually no other area in computer science is, even slightly. If you look at computer science papers in machine learning, systems, human-computer interaction, networking, robotics, or architecture, you'll almost always see an experiment of some sort. These are typically benchmarks, but there are plenty of other examples of empiricism in computer science. But the experiments in computer science are horribly designed! They lack almost any control, and use statistical methods that would make psychologists wince. Hypothesis testing is rare to the point of nonexistance. (Psychology sometimes adjusts too far in the other direction, using complex analyses where a simpler one would do.) Further, experiments are nearly never replicable, at all, because the systems evaluated are never made public, or the experimental environment is never recorded. For a great treatment of this trend in systems, see http://www-plan.cs.colorado.edu/diwan/asplos09.pdf http://www-plan.cs.colorado.edu/diwan/asplos09.pdf. Pointing out a systematic artifact in that many experimental designs would be career-making in psychology, or any other science! But you can look at modern papers and see exactly the same mistake being made.

by tedks 13y ago

deleted 13y ago

[deleted]

colechristensen 13y ago

>But, I'm sure that your general opinion is well-informed as well, or you wouldn't make such strong claims about a very broad field. I don't believe you are being genuine here at all. This passive-aggressive, sarcastic response is not appreciated. If you don't like what I wrote, respond directly and honestly.

jdn 13y ago
Well dodged

adrianbye 13y ago

Perhaps your experience inside psychology is limited? Nobel prize winner Daniel Kahneman[1] would disagree with you, along with social psychologist Jonathan Haidt[2]. [1] http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... [2] http://www.nytimes.com/2011/02/08/science/08tier.html http://www.nytimes.com/2011/02/08/science/08tier.html

btilly 13y ago
This point is critical. A study can have a good methodology. But if nobody tries to replicate it, you have no chance of finding out that you are missing something critical. And with something as complex as people, "missing something critical" is something that we need to figure out. And this isn't exactly a new problem for psychology. Feynman somewhat famously questioned a lot of classic research with http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html. I say somewhat famously because people in areas like physics have all read that essay. But in my experience psychologists are a little less inclined to read the criticism. Even at the best of times, replication of results in a complex system is hard. See, for instance, http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... for evidence that most published medical research is wrong. But with psychology it is even worse. We lump people together by symptom, not by root cause. It is like lumping together people with migraines, head injuries, and caffeine headaches together. Then you find out that coffee beats a placebo for helping headaches, and give them all coffee! You think I'm exaggerating? Read http://www.nimh.nih.gov/about/director/2013/transforming-diagnosis.shtml http://www.nimh.nih.gov/about/director/2013/transforming-dia... for verification that the NIH considers it a high priority research item to come up with classifications based on root causes, so that we have a chance of even being able to do useful research! Yes, psychology has a lot of well-meaning people, and they stumble across a lot of interesting stuff. But it should be read with a critical eye, because the field as a whole does not perform reliably enough to compel our trust.
- thaumasiotes 13y ago
  > But with psychology it is even worse. We lump people together by symptom, not by root cause. It is like lumping together people with migraines, head injuries, and caffeine headaches together. Then you find out that coffee beats a placebo for helping headaches, and give them all coffee! This happens (in medicine, not just psychology), and there's definitely room for hypothetical improvement. But in many cases, it's hard to blame people, because there's no way to do better without doing an unknown but gargantuan amount of research. For example, lupus is (or was when I had the discussion with my mother) defined as exhibiting X of a list of Y symptoms, where Y > X. This leads to situations like the following: 1. You, the patient, present with X-1 symptoms of lupus. By definition, you don't have lupus, and treatment for lupus is not warranted. 2. Time passes. 3. You, the patient, come back to your doctor with one additional symptom of lupus. Medical doctrine now states that you have lupus (not so interesting) and that, more interestingly, those X-1 earlier symptoms are symptoms of lupus (which you didn't have when you developed those symptoms), not of some other yet-undiagnosed problem. You can now be treated for lupus. Obviously, that satisfies no one and is crying out for a more "reality-based" definition. The problem is that no one has a candidate for the actual physical causes of lupus. The best anyone's ever been able to do is recognize that the same set of treatments are broadly effective against a constellation of symptoms, which may seem to be related to each other or not, but which co-occur to some degree in people who respond to these treatments. Of course it's a high priority to learn what the root causes of symptoms are, but that doesn't mean it's easy or even, in the general case, possible. You have to start somewhere.
tedks 13y ago
Oh no, I'm well aware of those -- but academic computer science is even worse! It's very new to have artifact evaluation at all, meaning almost any published result that represents an evaluation of a system (as opposed to something purely theoretical) is probably bunk. Almost no academic systems are released publicly for evaluation, so reproduction isn't just something that should be done, but isn't -- it's entirely impossible. This is basically the same as publishing a psych paper with a Methods section including a line that says "and then magic occurs". It's like an unsafe cast in a programming language -- you totally escape the guarantees provided by the wider structure of the academic system. Academia in general has serious problems. There's a fundamental problem of incentive, and until that's solved, nothing's really going to change. But psychology is definitely unfairly maligned. (It doesn't help that plenty of people doing doing what's almost psuedoscience are still harbored in a lot of psych departments, but their methods are still more rigorous than HCI research in most cases.)

qq66 13y ago

Psychology isn't really comparable to computer science since computer science is the study of a manmade system, making it more like mathematics. Computer science is not a traditional "science" studying the mechanisms of an unknown phenomenon, so concepts like "experimental design" aren't easily mappable.

tedks 13y ago
Theoretical computer science is much like mathematics. But virtually no other area in computer science is, even slightly. If you look at computer science papers in machine learning, systems, human-computer interaction, networking, robotics, or architecture, you'll almost always see an experiment of some sort. These are typically benchmarks, but there are plenty of other examples of empiricism in computer science. But the experiments in computer science are horribly designed! They lack almost any control, and use statistical methods that would make psychologists wince. Hypothesis testing is rare to the point of nonexistance. (Psychology sometimes adjusts too far in the other direction, using complex analyses where a simpler one would do.) Further, experiments are nearly never replicable, at all, because the systems evaluated are never made public, or the experimental environment is never recorded. For a great treatment of this trend in systems, see http://www-plan.cs.colorado.edu/diwan/asplos09.pdf http://www-plan.cs.colorado.edu/diwan/asplos09.pdf. Pointing out a systematic artifact in that many experimental designs would be career-making in psychology, or any other science! But you can look at modern papers and see exactly the same mistake being made.