17 ms·

Are Experts Real?

187 points by tlb 6y ago

pa7x1 6y ago
Experts are real, unfortunately knowledge in many fields is built with very poor standards of quality. The experts in those fields are vastly knowledgeable in falsehoods. While the general public questions if we should be funding the next big project in physics, the real question we should be asking is if we should be funding almost anything else as long as they are not willing to apply the same levels of rigor.
- ndiscussion 6y ago
  That's right, unfortunately most of science has been debunked ie replication crisis. Sadly, the populace doesn't see it that way.
  - orzig 6y ago
    Whoa - [citation needed] on "Most of science has been debunked" We've learned to be much more skeptical, and the previous standards were scandalous, but there's a difference between "too much of" and "most of"
    - ndiscussion 6y ago
      Here's a start, almost every single field is implicated: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Economics, social sciences, even medicine.
      - Der_Einzige 6y ago
        And don't forget machine learning! If your paper in ML doesn't provide code, I assume that it's not reproducible and the authors made a conscious decision to not release the code...
      - QuesnayJr 6y ago
        Social psychology has a replication crisis, because they had a culture of never questioning a published result. You can see that the vast majority of that article is devoted to problems within psychology. (Medicine also seems to have problems.) In economics 1/3 of 18 studies failed to replicate? A study that small, what are the odds that study itself would replicate? Note that this claim is restricted to experimental studies, which are usually testing human decision-making on economic questions. This is the field of economics closest to social psychology. It seems plausible to me that the model of "get a bunch of people in a room and ask them some questions" is not a reliable methodology. Economics as a whole may have a problem with replication, but we can't tell from the information on that page. There are whole fields that aren't even mentioned -- no physics, no chemistry.
      - chalst 6y ago
        That is not most of science. There are more biochemists than all those fields combined. The fields affected have three properties: they deal with complex entities, they are interested in phenomena that is generally investigated using statistical methods and they deal with noisy data. For those fields, we need new methodologies, but other sciences do not have a comparable problem.
    - nradov 6y ago
      Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
      - berekuk 6y ago
        On the other hand, here's a hilarious response how Ioannidis is wrong in his claim from one of the most well-known replication crisis blogs: https://replicationindex.com/2020/12/24/ioannidis-is-wrong-most-of-the-time/ https://replicationindex.com/2020/12/24/ioannidis-is-wrong-m...
        nradov 6y ago
        Why do you suppose they chose to publish this hit piece as a blog post instead of going through the peer review process with a real journal article? Or at least a formal letter to PLoS Medicine?
        berekuk 6y ago
        I'm not sure. Maybe they did both and the real article is somethere in peer review pipeline. (Do you imply that no posts on replicationindex should be trusted since they're not properly peer reviewed?) One possible reason is that blog posts are more widely shared on social media, and I agree that this is kind of a hit piece, mostly as a reaction to Ioannidis' contrarian stance on COVID-19, so publicity was probably an objective here.
breck 6y ago
Prediction: in 10 years no "expert" will not be putting their work in Git. Git provides a zero cost (in time and money) way to share your full work with audibility; where fixing mistakes continuously is expected and not something to be embarrassed about; where people can instantly clone to reproduce and test better ideas upon your work; where collaborating with proper attribution is done automatically, et cetera. Look in any field—epidemiology, computer science, physics, chemistry, medicine, et cetera. If the leading experts aren't using git yet, look for the ones that are. They will be the leading experts of tomorrow.
- deleted 6y ago
  [deleted]
- jdsully 6y ago
  I like that prediction, but git is actually a bit of a step back from older more traditional source control systems. Rewriting history is far more common with git to the point where teams have to make explicit policies on when that is allowed. Even if git becomes common in science outside of code, I don't think it would overcome the social pressure to appear infallible.
  - breck 6y ago
    > Rewriting history is far more common with git How do you mean? It's impossible. You git push something, and then you leave an immutable trail that someone somewhere may have cloned so even if you were to force push or delete you can't erase it. Assuming it was of any importance and someone was watching (but if not of any importance, then no big deal to force push/delete).
    - 0xffff2 6y ago
      Saying something is immutable because someone may have saved an old copy before you mutated your own copy sounds very fragile to my ears.
      - 6y ago
dragontamer 6y ago
Yes, experts are real. Unfortunately, experts are only expert in a few subjects, and experts (well... all humans even) are bad at knowing when they've left their field of expertise. Linus Torvalds for example is clearly an expert programmer. And then suddenly he talks about AVX512 and I wonder if he knows anything about low-level programming or optimization. (https://www.realworldtech.com/forum/?threadid=193189&curpostid=193190 https://www.realworldtech.com/forum/?threadid=193189&curpost...). Case in point: with Rocket lake (on 14nm nonetheless), the power-limits associated with AVX512 have already been largely lifted. It didn't even take a few months before Linus's words became obsolete. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... 100MHz difference single-core, 0MHz difference multicore. Virtually no penalty for using AVX512 on new systems. Linus was ultimately trapped in the Skylake-X way of thinking, and not looking forward to the next chips.
- zug_zug 6y ago
  Did you read the article? The article is specifically talking about cases where experts are incorrect within their field of expertise and then cited as a source of truth in published papers. The question of the article isn't really a boolean, it clearly shows some experts are real and some aren't and asks the more important question -- how do we discern when the institutionalized "expert" in an area is really an expert?
  - dragontamer 6y ago
    > Did you read the article? The article is specifically talking about cases where experts are incorrect within their field of expertise and then cited as a source of truth in published papers. Yes. But I still think that Linus is a good character to discuss within the scope of the article. Linus is clearly considered an expert, and many people follow his bombastic words (retweeting, or using them to form the basis of their argument around here). "Because Linus said so" is a common way to end various arguments. Linus's opinion is truly "expert", people copy it without further analysis because... well... Linus kinda deserves it. He is an expert after all. It doesn't mean Linus is always right. -------- The "feedback loop" of the open source world is pretty quick. If you're a bad programmer, people figure it out when they start working with your code. But the "feedback loop" of "chip ISA opinions" is slow. There's not really any way to prove, or disprove, whether or not AVX512 is a good or bad idea. Linus's expertise in programming doesn't necessarily translate to chip ISA design. And as such, Linus can get details (expected power-characteristics of future chips) dramatically wrong.
WC3w6pXxgGd 6y ago
Great article. I have had many debates with people within the last year about the concept of "experts," and have frequently found it almost impossible to break through the myth that 1. Experts exist, and you should believe everything they say. 2. Expertise is fungible, so an "expert" in one small sliver of healthcare is also somehow an overall expert in healthcare. An example of 2 was a conversation I had recently with someone in the healthcare industry who, in this single conversation, said - Cuba has better healthcare than the US - All doctors would be fine with the State forcing them to provide their services because doctors are not profit-driven and only care about The Greater Good.
tboyd47 6y ago
Great article. > Do you have it in you to ignore 800 epidemiologists, and is it actually good idea? Yes, I do, because I was tracking the wave of non-stop bullshit coming from scientists closely throughout all of 2020, and got completely fed up. But I'm not the norm (yet), and no, it's not actually a good solution. We're in the beginning of the story of the boy who cried wolf. Eventually the whole town ended up like me: fully willing to ignore and oppose the mainstream information channels. When a real wolf came, the boy told the truth once out of 1,000 times, and the townspeople all ignored him and got their sheep eaten up. We need to find a way to crowd-source our trust mechanism if "trust the experts" starts to fail. My suggestion is that any "expert" who volunteers an opinion that later gets falsified should be canceled permanently from the public sphere, with no forgiveness. They can go on publishing in scientific journals and selling books, but their opinion should not be accepted by the general public anymore. The other solution is to keep ignoring the boy and start watching your own sheep.
- cltby 6y ago
  Many argue that the experts ultimately got it right (and are thus redeemed). I strongly disagree. Even when the public health types corrected course, their behavior left a bad taste in my mouth. There was no acknowledgement of having been wrong, no introspection on how their errors came to be, no discussion how they could be avoided in the future. Just aggressive enforcement of the new party line couched in appeals to authority. The whole enterprise reeks of blind dogmatism.
  - tboyd47 6y ago
    > Many argue that the experts ultimately got it right (and are thus redeemed). Perhaps their opinion on that one matter they corrected was redeemed, but their general lack of understanding about the limits of their own knowledge was exposed, which is more damning for someone called an "expert."
    - TheTrotters 6y ago
      And in time-sensitive situations getting something "ultimately right" is akin to getting it wrong.
DanielBMarkham 6y ago
This essay points out a pervasive problem that doesn't get much attention (aside from fear-mongering) Because it's a topic so easily prone to hyperbole, it helps if we can reduce it down to a few premises. - Science is science, but the practice of science is a social system. People get tenures, jobs, promoted, invitation to speak, etc. by how well they do in the judgment of their peers - Political systems exist for political reasons, ie, if you don't have a reproducible, falsifiable test in a feedback loop somewhere, your career and goals are all about image. There's no alternative, either for you or your field of study (That doesn't mean other sciences aren't important or aren't really science.) - If somebody appears on TV or in the media playing the role of an expert, it is, by definition, because they wanted to appear on TV or in the media. The goals of being popular and asked about important questions and the goals of whatever your field are ... two different things. The reporter doesn't care who figured out X. They care who can bring excitement and emotion to talking about X. - All this leads directly to the problem of the average science news consumer: any communication you receive about science or research is the result of a lot of conflicting interests. "Being good science" is way down the list, sadly. I imagine the percentage of average readers who understand p-hacking is below 5% - None of this is a secret. The average news consumer knows it, even if they're not able to articulate it - Add in things like the reproducibility crisis, and we've got a huge problem as a species. We've got one flashlight to light our way in the darkness and it turns out there's a loose connection in there somewhere
- cbozeman 6y ago
  This is a fantastic comment, and spot on I think. I concur with everything you've said.
- ska 6y ago
  > ...by definition, because they wanted to appear on TV or in the media I suspect this misses some nuance; there is a meaningful difference between people who appear on the media because they want exposure, and people who do it because they want their field of study to be well represented. For example there are people to accept interviews when they genuinely dislike the process, because they felt it was a duty that came along with their chair, etc. I doubt these are ever the most dynamic interviews, but they are common. I agree this doesn't really extend to the level of the same old bobble heads pulled up on prime time news TV, but don't thing you can paint all of the media exposure with this brush.
deleted 6y ago
[deleted]
zug_zug 6y ago
> "This year, we’ve come to better appreciate the fallibility and shortcomings of numerous well-established institutions (“masks don’t work”)… while simultaneously entrenching more heavily mechanisms that assume their correctness (“removing COVID misinformation”)." I suspect in a cultural moment of mass-skepticism of even valid systems (e.g. masks/election/vaccines/wikipedia) it's very hard to touch on the fallibility of those systems without putting people on the defensive. It shouldn't be in an ideal world -- making these things more auditable and rigorous only makes them less prone to questioning in the long term, but I suppose many feel faith in the system is more important at this moment than incremental improvement. [However this really doesn't answer the n=59 conundrum, which I think requires a different lens]
Bakary 6y ago
I remember this concept being mentioned elsewhere as "friendly" and "hostile" learning environments
- JadeNB 6y ago
  Epstein's excellent book Range uses the terms, though I think they don't originate there.
rusticpenn 6y ago
Regardless of the quality of knowledge, people with better social and networking skills will become better known than those without.
- TheTrotters 6y ago
  Among the general public? Definitely. Social and networking skills also matter within your professional community but they are not a substitute for publishing in top journals/conferences.
  - rusticpenn 6y ago
    Perhaps your experience is different. Having the right name as the first or last author helps a lot in peer review and publication. Even in “double blind” reviews, it’s easy to find the authors usually.
    - angrais 6y ago
      Sort of true, but not quite. You may infer that a paper is from a certain lab or that the first author may be someone specific as it builds on their work... But last author? Not as easy as supervisors/lab leads can change.
      - rusticpenn 6y ago
        Its not the supervisors who become famous. Its the lead Professor of the lab. Usually in engineering papers/journals the first/last author is the lead professor of the lab, who does nothing more than get the funding for the project.
bgorman 6y ago
One of my theories based on life experience is that many fields do not attract the brightest people to begin with, and consequently many of the ideas from these fields are not pressure-tested enough. In todays academia, it is basically impossible to fail getting a PhD in many fields provided you are ok being miserable for years at a time. At the same time academia is filled with a ton of bs like publicly funded research being released behind paywalls, professors receiving grants for projects that are already completed, students being denied graduation util their project can be submitted to the highest impact journal possible etc, having to do a long postdoc to get a tenure track position etc... For smart people that value financial freedom, academia is a terrible option at this point in time. The current coronavirus fiasco has laid bare the lack of critical thinking in the life-sciences fields.
- nksv 6y ago
  "One of my theories based on life experience is that many fields do not attract the brightest people to begin with, and consequently many of the ideas from these fields are not pressure-tested enough." very well articulated, i have thought about this many times, basically a lot of physical sciences face this
moomin 6y ago
I think this argument conflates two things, one of which is true and slightly terrifying. The terrifying thing is how few scientists understand the statistical underpinnings of their studies. The conflation happens because he thinks this is stratified by field. In practice, there are plenty of sociologists with a decent grasp on this, and a truly astonishing number of biologists who don't. Don't believe me? Read the studies of biology papers and how many have serious problems in their statistical treatment. So we've got a serious problem with how science is being done. Probably doesn't affect whether or not expertise exists, or which fields it exists in. Unless we're going to start saying biology isn't real.
- jldugger 6y ago
  > Unless we're going to start saying biology isn't real. It probably helps to recognize that statisticians caught one of the first modern biology researchers fudging his numbers: https://en.wikipedia.org/wiki/Gregor_Mendel#Mendelian_paradox https://en.wikipedia.org/wiki/Gregor_Mendel#Mendelian_parado...
  - lmm 6y ago
    Except they didn't; they alleged that the numbers had been fudged on rather scant evidence.
    - jldugger 6y ago
      As I understand it, Fisher had two complaints. First, that the trifactorial data was just wildly too accurate (using a chi-squared approach). This seems to be resolved in the scientific community as an off-by-one error: if instead of 10 seeds, Gregor had used 11 on occasion, that's enough to resolve the issue. The other issue is the chi-squared analysis of Mendel's experiments in aggregate, which when pooled also too neatly fit the pattern he wanted to illustrate. And as best I know, nobody has adaquately disputed that analysis.
      - lmm 6y ago
        The only actual analysis I saw came up with a p-value of about 0.07, which is just a complete nothingburger. There's no explanation for that because there's nothing to explain.
klmadfejno 6y ago
IMO Experts are more wise to the problems of their domain than intelligent. Intelligence relates to the execution of mental tasks and the recall of specific facts. Wisdom relates to the planning of the approach to solve a given task. It is whether or not the nature of the problem lends itself to expertise that defines if experts are particularly relevant for a given domain.
- 13415 6y ago
  I'd say that experts are mostly knowledgeable, experienced, and broadly respected within their scientific domain. I don't see how intelligence or wisdom are important factors - other than for the fact that most accomplished scientists are probably rather intelligent.
- deleted 6y ago
  [deleted]
dmitryminkovsky 6y ago
The Masks Fiasco of 2020 was indeed a noteworthy example of experts-gone-wild, but my favorite is still the Monty Hall problem [0], because unlike epidemiology where proof stems from evidence-based experimentation, the Monty Hall fiasco saw a wide range of experts in mathematics/statistics—a field where truth does not require experimentation but only logic-based proof—loudly and publicly denigrate a person with no expertise in their field, only to be wrong. But I think the an even more interesting question that this post does not cover is the role experts play in our culture. Neil Postman discusses experts at length in Technopoly: The Surrender of Culture to Technology (1992), wherein he characterizes them as basically the shaman-priests who control the Overton window [1] in a world where morality and social acceptability no longer stem from things like religion: > ...expertise is a second important technical means by which Technopoly strives furiously to control information. There have, of course, always been experts, even in tool-using cultures. The pyramids, Roman roads, the Strasbourg Cathedral, could hardly have been built without experts. But the expert in Technopoly has two characteristics that distinguish him or her from experts of the past. First, Technopoly’s experts tend to be ignorant about any matter not directly related to their specialized area. The average psychotherapist, for example, barely has even superficial knowledge of literature, philosophy, social history, art, religion, and biology, and is not expected to have such knowledge. Second, like bureaucracy itself (with which an expert may or may not be connected), Technopoly’s experts claim dominion not only over technical matters but also over social, psychological, and moral affairs. In the United States, we have experts in how to raise children, how to educate them, how to be lovable, how to make love, how to influence people, how to make friends. There is no aspect of human relations that has not been technicalized and therefore relegated to the control of experts. ... > The role of the expert is to concentrate on one field of knowledge, sift through all that is available, eliminate that which has no bearing on a problem, and use what is left to assist in solving a problem. This process works fairly well in situations where only a technical solution is required and there is no conflict with human purposes—for example, in space rocketry or the construction of a sewer system. It works less well in situations where technical requirements may conflict with human purposes, as in medicine or architecture. And it is disastrous when applied to situations that cannot be solved by technical means and where efficiency is usually irrelevant, such as in education, law, family life, and problems of personal maladjustment. I assume I do not need to convince the reader that there are no experts—there can be no experts—in child-rearing and lovemaking and friend-making. All of this is a figment of the Technopolist’s imagination, made plausible by the use of technical machinery, without which the expert would be totally disarmed and exposed as an intruder and an ignoramus. > Technical machinery is essential to both the bureaucrat and the expert, and may be regarded as a third mechanism of information control. I do not have in mind such “hard” technologies as the computer—which must, in any case, be treated separately, since it embodies all that Technopoly stands for. I have in mind “softer” technologies such as IQ tests, SATs, standardized forms, taxonomies, and opinion polls. Some of these I discuss in detail in chapter eight, “Invisible Technologies,” but I mention them here because their role in reducing the types and quantity of information admitted to a system often goes unnoticed, and therefore their role in redefining traditional concepts also goes unnoticed. There is, for example, no test that can measure a person’s intelligence. Intelligence is a general term used to denote one’s capacity to solve real-life problems in a variety of novel contexts. It is acknowledged by everyone except experts that each person varies greatly in such capacities, from consistently effective to consistently ineffective, depending on the kinds of problems requiring solution. If, however, we are made to believe that a test can reveal precisely the quantity of intelligence a person has, then, for all institutional purposes, a score on a test becomes his or her intelligence. The test transforms an abstract and multifaceted meaning into a technical and exact term that leaves out everything of importance. One might even say that an intelligence test is a tale told by an expert, signifying nothing. Nonetheless, the expert relies on our believing in the reality of technical machinery, which means we will reify the answers generated by the machinery. We come to believe that our score is our intelligence, or our capacity for creativity or love or pain. We come to believe that the results of opinion polls are what people believe, as if our beliefs can be encapsulated in such sentences as “I approve” and “I disapprove." [0] https://en.wikipedia.org/wiki/Monty_Hall_problem#Vos_Savant_and_the_media_furor https://en.wikipedia.org/wiki/Monty_Hall_problem#Vos_Savant_... [1] https://en.wikipedia.org/wiki/Overton_window https://en.wikipedia.org/wiki/Overton_window
xyzelement 6y ago
I want to share this tid-bit. When I studied portfolio management, I had to learn not just how to talk to people about their risk tolerances, but how to ascertain whether they knew their own tolerances. The concept was called "calibration" - or - how well is someone able to know that they are wrong/don't know something. Basically, "high calibration" is the opposite of Dunning-Kruger in my mind. The text cited a study that measured calibration by profession. The highest calibrated profession was meteorologists. It makes sense if you think about it: I prognosticate rain tomorrow, tomorrow comes and it's sunny - I am forced to confront the fact that my prognosis was wrong. So not only is there a way to know who's better and who's worse, but also constant reminder that "I can be wrong despite my best work" The lowest calibration profession was doctors. Which also makes sense. Imagine I am a doctor and I miss an early sign of a disease. The patient dies 5 years later - I may not even know, and even if I do, very unlikely am I confronted with the specific point in time 5 years prior where I made the wrong call. So the thing that matters with expertise is - how often is the person getting feedback. In the article they cite chess as having "real experts" - of course. Because a chess player gets feedback on how good they are every time they win or lose a match. On the flip side, how often is someone in a very soft field getting feedback? What's the way for an "expert" in say Gender Studies, to wake up in the morning and be confronted with "the thing I did yesterday didn't work?" The other thing about experts is they are optimizing for a single thing. Eg ask your compliance person at work if you can do something, the answer is "NO" because they are optimizing for not being sued, but in reality that's not the only concern. Or in the pandemic, ask the economist if we should lock down - they'll say no because they understand the financial consequences. Ask a medical person and they'll say yes because they are focused solely on the number of sick people. In reality the world is complex and there are tradeoffs. Related, experts are often concerned with high-impact issues which may be different from individual issues. EG: in the USA, by default, kids get the Hep B vaccine on their first day of life. On country level this makes sense because there are a lot of mothers out there who have Hep B and don't know it. But when my wife was giving birth, we knew to opt out despite the recommendation because we understood that the experts are optimizing for a thing that doesn't relate to us.
- jldugger 6y ago
  Any chance you can recall enough of that paper to find a citation? I'd be interested to read such a paper.
lordnacho 6y ago
I think stats is the culprit. Or lack of it. If there's one area of study that ties together observations and models, it's stats. But stats suffers from the same problem that programming does: people think it's a tool that they just need to know a little bit of, so they don't defer to the proper authorities in the area. Of course this is just anecdotes, but I do have a friend who did stats as a phd and this is the attitude that seems to be common: stats (and code) are minor tools that won't affect the outcome of our experiment.
- cutemonster 6y ago
  Some ppl who studied social sciences might even be uninterested in statistics and look at it as off topic. After all, most people aren't good at mathematics (i think) and someone who, say, chose to study psychology might have preferred that there wasn't any math stats courses in the program at all. (I know one such person in real life)
nonpolitical 6y ago
This reminds me of my layperson puzzlement: Can we apply and verify the results from CERN experiments beyond experimental settings? If we can't, how can we know the results from CERN experiments aren't just artifacts of the experimental apparatuses?
- losvedir 6y ago
  That's a good question. You're right that a random grad student in a lab can't replicate, e.g., the discovery of the Higgs boson! However, at least for the Higgs (and I believe this is standard for most such large experiments), there were actually two different groups using two different apparatuses, both at CERN. There was the CMS[0] detector and ATLAS[1] detector, used and analyzed by two different groups of physicists, resulting in two papers[2,3], published on the same day. [0] https://en.wikipedia.org/wiki/Compact_Muon_Solenoid https://en.wikipedia.org/wiki/Compact_Muon_Solenoid [1] https://en.wikipedia.org/wiki/ATLAS_experiment https://en.wikipedia.org/wiki/ATLAS_experiment [2] https://arxiv.org/abs/1207.7235 https://arxiv.org/abs/1207.7235 [3] https://arxiv.org/abs/1207.7214 https://arxiv.org/abs/1207.7214
  - nonpolitical 6y ago
    Thanks! Pity that I guess I won't be able to understand the papers linked in the foreseeable future.
deleted 6y ago
[deleted]
yboris 6y ago
Surprised that the classic "Conditions for intuitive expertise: A failure to disagree." paper by Daniel Kahneman and Gary Klein is not referenced https://psycnet.apa.org/record/2009-13007-001 https://psycnet.apa.org/record/2009-13007-001 My takeaway: without short feedback loops it is hard to build expertise. If it takes 2 years to learn the individual you paroled reoffended, you can't learn (so there are virtually no expertise in parole officers or university admissions). In these scenarios, simple linear models outperform "experts" by a wide margin. See: "In Praise of Epistemic Irresponsibility: How Lazy and Ignorant Can You Be?" by Michael Bishop https://philpapers.org/rec/BISIPO https://philpapers.org/rec/BISIPO
d23 6y ago
A lot of the points are interesting, but I couldn't help but get hung up on the link to the covid analyist who supposedly beat the experts, at https://www.maby.app/covid/ https://www.maby.app/covid/ A couple of immediate red flags: the predictions stop in early May, and there appears to be no weighting of how "important" a given prediction is. Perhaps the former is just because a contest ended at that time. For the latter, of course, importance is subjective, but ranking these two as equivalent is silly: * How many cases per day will Washington state average for the week ending June 7? (Tom: good prediction, Experts: bad) * How many US COVID deaths will occur in 2020? (Tom: bad prediction, Experts: good)
eesmith 6y ago
Teller attributes Bohr as saying: "An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field." I prefer something closer to "An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them". 'Cause even experts make mistakes in their own field. This piece seems to think an expert cannot make a mistake in a given field of expertise, which is "utter nonsense, of course." I don't like this example: "If you look at the papers citing Viechtbauer et al., you will find dozens of them simply using N=59, regardless of the problem they're studying" Paging through the linked-to Google Scholar search finds 90 results. Of the first 12 with readable text, only two seem correctly described that way (and max 3), suggesting under two dozen examples. A spot check of the remainder suggests that pattern holds. I'll show my work: #1 says "A sample size of 30 patients per group was chosen as recommended for pilot studies to achieve an appropriate level of statistical power [30]." They selected 60, and "1 patient was lost to follow-up" giving 59. So this wasn't N=59 but N=60. - https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD&utm_medium=cpc&utm_campaign=JMIR_TrendMD_0 https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD... . 30 was chosen because "It corresponds to the detection of a potential difference of 21% between groups for a power of 80% and an alpha significance level of 5%." which is not the default example from Viechtbauer. #2 does not use N=59. The number 59 occurs because: "The study included 16 patients with diabetes (59 ± 8.8 years; 8 men and 8 women)" - https://www.sciencedirect.com/science/article/pii/S0958259218300919?casa_token=7GVLsoy6nDcAAAAA:glWnYg27NQjB6ZjYV6NxbJQNAlI0-o694dpZyO4nme3xFDAfWFrh8dv1i0wCheNX2CnJlYxddA https://www.sciencedirect.com/science/article/pii/S095825921... #3 uses N=59, though I don't know if it was used incorrectly because I can only read the abstract and citation list - https://journals.sagepub.com/doi/pdf/10.1177/0306624X19858487?casa_token=9iQatZFhsc8AAAAA:2fgaomaQLIxdvPp_P1NKoD25-kPwghHXbl3LmxT-SBbY4mIjtJ0obJ6m7qAoOCCk60pEqUttB1FQ https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848... #4 specifically does NOT use Viechtbaueret, writing: "Viechtbaueret al., (2020) recommends sample size between 10-40 as adequate for feasibility study. Linacre, (2015) suggests a sample of thirty (30) respondents as appropriate for general purposes. This study however uses 60 respondents becauseadequate sample and item produce a good item, person separation and reliability in Rasch measurement (Fisher, Elbaum, & Coulter, 2010)." - https://journals.sagepub.com/doi/pdf/10.1177/0306624X19858487?casa_token=9iQatZFhsc8AAAAA:2fgaomaQLIxdvPp_P1NKoD25-kPwghHXbl3LmxT-SBbY4mIjtJ0obJ6m7qAoOCCk60pEqUttB1FQ https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848... #5 uses N=43 ("With an alpha value of 0.05 and with an 80% power we need a sample size of at least 43 patients to detect a difference in validity and assess reliability") - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ . This is not the default example from Viechtbauer. #6 uses N=59, using calculations directly from Viechtbaueret. It appears to be used correctly. #7 uses N=53, commenting "There is no gold standard for sample size calculation in pilot studies. Sample sizes as small as 10 (Hertzog, 2008) and as large as 59 (Viechtbauer et al., 2015) have been recommended." - https://journals.lww.com/nursingresearchonline/FullText/2018/09000/Enhancing_Men_s_Awareness_of_Testicular_Disorders.2.aspx https://journals.lww.com/nursingresearchonline/FullText/2018... #8 uses N=30 for each region, writing "Tamaño de la muestra: en cada región, se llevó a cabo un estudio piloto con aproximadamente 30 pacientes" - http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1134-928X2020000400001 http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1... . The 59 occurs in, for example, "La reducción también fue estadísticamente significativa (59%) al tener en cuenta el coste del tiempo de enfermería". #9 points to the N=59 recommendation (correctly, for their experiment) but actually uses N=71 - https://www.mdpi.com/1422-0067/20/8/2024/pdf https://www.mdpi.com/1422-0067/20/8/2024/pdf #10 appears to have the discussed N=59 problem as it doesn't mention the 0.05 effect - https://mhealth.jmir.org/2016/3/e109/ https://mhealth.jmir.org/2016/3/e109/ . #11 says "Pilot studies should include at least 12 patients [30,31], and cluster-randomized studies need more patients to correct for the clustering [32]. To have sufficient power for this pilot study, we intended to include 15 patients per nurse." where [31] is Viechtbauer. The value 59 is because "data of 149 (98%) patients were available of whom 66 (44%) were in the intervention group; ‡ = data of 135 (89%) were available of whom 59 (44%) were in the intervention group". - https://www.sciencedirect.com/science/article/pii/S0261561419331656?casa_token=G9Wp2q11YQYAAAAA:t8UiqCu8LB1egMYID6J-BAm7FpBnkXAN2-nbzVXdPSdt2mRAB3CXHzsFrmjhXw1t7Rmjq_GdWQ https://www.sciencedirect.com/science/article/pii/S026156141... #12 says "In line with recommended sample sizes of pilot feasibility trials, 83–85 40 patients is deemed sufficient to explore feasibility, acceptability and potentially efficacy of the intervention, assuming retention rates of 80%, the true population consent rate will be with a margin of error of 13% (95% binomial exact CI)". The only "59" comes from citations. - https://www.sciencedirect.com/science/article/pii/S0261561419331656?casa_token=G9Wp2q11YQYAAAAA:t8UiqCu8LB1egMYID6J-BAm7FpBnkXAN2-nbzVXdPSdt2mRAB3CXHzsFrmjhXw1t7Rmjq_GdWQ https://www.sciencedirect.com/science/article/pii/S026156141... I'm not sure how we're to have a take-home lesson when the false negatives so overwhelm the claimed evidence. I'm also hung up on: "Capitalism and evolution work so well because of their merciless feedback mechanisms." because of obvious questions like "does capitalism work well for slaves?"
hikerclimber 6y ago
i don't think so... you can gain expertise in anything.
altarius 6y ago
One of my problems with this line of argument is that it provides a sort relativism of opinion, every opinion is just as valid as any other. And then you're at post-truth and post-fact sadly... If Carlsen is a fake, so is every chess player he has ever played, the whole system? Epidemiologists lied about masks (or they really didn't know better 3-5 months into the pandemic, also not good), discount everything else said? I've heard these arguments often lately :/ Experts not being "real" implies that there's no absolute expertise whatsoever and implicitly legitimizes dismissing any argument whatsoever on a single failure and allows people to believe that any opinion (including their own) is just as valid. I think this is in part to blame for our "post-fact/post-truth" phenomenon. Many people decide experts or government were wrong/have lied before and can no longer be trusted. And when an outsider comes along with contrarian but convenient ideas, why not believe them, nobody else is to be trusted after all and every opinion is valid. Any expert opinion can be wrong but the prior is that an expert opinion is much more likely to be correct than your own, or a politician's, or a blogger's. NB: I want to note that the author's point is much more nuanced and (more importantly!) narrow, but it's not evident from the title or a superficial reading.
- mcguire 6y ago
  "Epidemiologists lied about masks (or they really didn't know better 3-5 months into the pandemic, also not good)..." Technically, if my timeline is right[1], about 2 months into the pandemic, for his example. 3-4 months generally. Plus they had a prior belief that masks wouldn't be useful. [1] The world outside China learned about the thing at the end of December; his example is the first week of March.
- nickthemagicman 6y ago
  'Experts' often disagree. Which makes you wonder if the whole system is actually invalid. Experts are merely one data point, at the end of the day you have to look at the entirety of the data yourself to make a decision.
mcguire 6y ago
On March 4, 2020, Michelle Odden tweeted about 800 epidemiologists not wearing masks. Because I don't commute anymore, I'm way behind in my podcasts. Three podcasts I listen to, and am behind in, are the Naked Scientists (a British popular science broadcast), and the Nature and Science podcasts (which cover weekly scientific news). The interesting thing is that, as I listen to them now, I'm going through a live view of the past with present hindsight. It's fun. One interesting thing is that aerosol, airborne transmission of many viruses was (and is?) controversial, or at least an open research question. (Recognition of aerosol transmission of infectious agents: a commentary [Jan 2019] https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-3707-y https://bmcinfectdis.biomedcentral.com/articles/10.1186/s128...) Up until at least mid-April, the default belief of the medical community seems to have been that Corona viruses were not, or at least not particularly, airborne. Instead it would be spread by physical contact (contaminated surfaces, etc.) or large droplets produced by coughing or sneezing. As a result, the default use of masks was not particularly recommended.[1] (And that's before you consider the scarcity of PPE for medical users at the time.) After about mid-April, the facts on the ground seem to change, and things like aerosol transmission and asymptomatic spreaders suddenly become a much larger concern. (And a lot of people on my podcasts got sheepish with the egg on their faces.) As a result, it's not particularly surprising that an epidemiologist would not consider wearing a mask to be particularly useful in early March. [1] One thing the article does not touch on is the tendency of actual experts to get feelings akin to religious hysteria based on internal fights, with the resulting dramatic statements spilling over into outside conversations. (I had this one paper ripped to shreds by a old-time network guy because I and my advisor had the temerity to consider keepalives/heartbeats.) I suspect there may be something behind the default assumption up until April that masks were bad.
mncharity 6y ago
People seriously misunderstand expertise. There's the perennial news stories like "Harvard MBA's don't know what causes Earth's seasons!?! LOL!". And the "He's a Scientist, so he must know...". And the "my local TV news meteorologist proved climate change isn't real". And in TFA, "But sometimes [experts] say something so obviously incorrect, [...] that the only possible explanation is that they understand very little about their field. [disillusioned gasp!]". So, you're a computer engineer? Wow, you must be sooooooo smart. An Expert! And did you read Huckleberry Finn in school like me? Yes, thirty years ago? Great, what do you think of the raft trip as an allegory for mumble mumble? What, you've no idea?!? - LOL. But I'm glad you're an Engineer - tell me what kind of solar panels should I buy, and about driving trains. And, Ha ha, you didn't even notice the computer cable was loose - you obviously understand very little about computers. If the last time someone touched something was in middle school, don't be surprised to find a middle schooler's understanding. If you ask a protein chemist a quantum chemistry question, don't be surprised to get a decades-old graduate student's answer. Don't be surprised if a respected fruit fly biologist, believes earthy-crunchy nuttery about the human heath benefits of eating nuts. Don't be surprised by scientists gossiping at conferences "Yeah, don't believe work from [first-tier PI] on topic X - he sees what he wants to see". Don't be surprised when a first-tier stone cold empirical physicist, says "I don't need to test it - I trust my gut to tell me how well my teaching is working". Expertise can become ramshackle startlingly fast as you move away from someone's own research focus. Or sometimes from an advanced teaching focus. A focus with fear of being embarrassed by getting things wrong in front of double-checking picky peers. Getting things right is hard, and takes effort - which has to be incentivized, and monitored. And it's not just "what is expertise" which is taught badly. If I told you "Yay, I found a good Professor of Computer Science, to write a Computer textbook, including language design, microservices, chip taping, and UX - I'm sure he will do a correct and insightful job of it", what would you think? So, what if you heard "Yay, I found a good Professor of Chemistry, to write a Chemistry textbook, including bonding, nuclei, materials, and nanostructures - I'm sure he will do a correct and insightful job of it"? Or how about "Welcome to <big textbook publisher>, new graduates, new science textbook writers. Don't worry that you only have a liberal arts degree, and no background in science at all, because we have a Scientist on call for questions!". Sigh. People seriously misunderstand expertise.
ncmncm 6y ago
The example of magnetohydrodynamicists vs astrophysicists doesn't seem entirely idle.
keithnz 6y ago
one of the problems of experts is that at the edge of their field of expertise, they aren't experts of the edge. It's just they have developed expertise in the building blocks to allow them to live at the edge. Quite often we are interested in things happening at the edge, and the "experts' can't really provide information at high levels of certainty.
blueflame7 6y ago
Yes. Dr Fauci has won 12 times in a row at the national infectious diseases championship. He is THE expert
deleted 6y ago
[deleted]
CyberRabbi 6y ago
Experts do exist but I think many alleged experts often give into the temptation to use the expertise in their field to represent expertise in a field in which they cannot claim the same level of expertise. This is my opinion because I’ve experienced the temptation to do just that many times and I’ve witnessed instances where that has happened as well. I don’t think this behavior is limited to people who have expertise in any field but it might be most damaging when experts do commit that logical fallacy.
makz 6y ago
Well, I’m an expert and as far as I know I’m real, so my answer is yes.
karlhughes 6y ago
My main problem with experts is they share the same cognitive biases as the rest of us: https://www.karllhughes.com/posts/experts https://www.karllhughes.com/posts/experts It's great to have academic qualifications, but you still have such a limited sphere of experience and knowledge - even more so the higher you go in academia.
obviouslynotme 6y ago
Scientists are the new priest class. They deliver edicts and answer the public's questions about the universe. The problem is that the answer to 99% of the public's questions is "I don't know" or "It's very complicated." We don't like that. We want the right answer, right now, delivered in one simple sentence. We find the one goober who hasn't walked inside of a lab in decades who wants to feel important to give us what we want. Worse is that people envision scientists as unbiased saints. That's just not real. Real science is about one thing and one thing only. Money, aka grants. Grants get you labs and underlings which bring more money and maybe someday tenure. And once you have tenure, you can finally publish what you really think about what interests you instead of chasing funding fads. Except that part never happens. That means you study the subjects that give funding. You give papers that increase the chance of more funding for more papers. Your accuracy? Doesn't matter. Funding matters. No one is going to check your work anyhow beyond simple peer review. Someone tries to recreate your work or recalculate your numbers? Tell them you lost the data. It's a bit embarrassing but no one is going to call you on this stuff anyhow. There is only one solution to the replication crisis. Funded third-party replication for all experiments. Money is the root of this problem. It has to be the solution. How many scientists are going to publish "mistakes" if they know that four different teams at four other universities are going to automatically attempt replication?
- DaiPlusPlus 6y ago
  > Real science is about one thing and one thing only. Money, aka grants no.
  - obviouslynotme 6y ago
    If you think that real research isn't dependent on funding, then I really don't know what to tell you.
- bonoboTP 6y ago
  Many people only interface with "science" through school classes and popular science magazines / documentaries / IFL science / antitheist memes, arguing against hoaxes, flat Earth, creationism etc. People dislike your answer because they have no idea how the sausage is made, when actually your description is quite realistic.
_cs2017_ 6y ago
We accept that any complex software, even well-designed and well-tested, can malfunction -- often spectacularly. What, then, can we expect from humans? Consider that virtually no effort went into the design; and the test framework is horrific, relying primarily on trial and error. The natural outcome is that any human "expert" is no more reliable than your average startup codebase. In fact, it's a wonder that human societies manage to organize themselves into anything resembling civilization.
analog31 6y ago
I think experts exist, but there's a problem with translating expert opinion in evolving fields into actionable advice that can be communicated to the masses through the news, and that can withstand aggressive, organized skeptical attack. Okay, that's a bit of an exaggeration, just because it rolls a whole bunch of different things into one blanket statement, and sounds like a rant, which it kind of is. The world is fucked. What's the point of an organized skeptical attack on experts? I think I can attribute to Imre Lakatos the thought that if you can completely discredit the experts, and make it impossible to decide on facts, then you create a power vacuum that can only be filled by those who are capable of acting in the absence of facts, such as thugs and tyrants. You have effectively censored the experts with greater efficiency than any government censorship program could ever dream of. I believe this is the precise reason for "the war on the truth."
logicchains 6y ago
From an economic perspective there's a relatively simple solution: requiring experts to wager a certain amount of their own money on a measurable outcome of their prediction when making public policy recommendations. If somebody's not confident enough in their own prediction to put their own money on it, then that prediction shouldn't be used as the basis for policies that can affect millions of peoples' lives.
- ineedasername 6y ago
  Science is not about being right every time. Stating a hypothesis publicly as you're experimenting to determine it's validity shouldn't require a wager.
  - caddemon 6y ago
    To be fair, betting isn't about being right every time either. To really make bank on a site like DraftKings you need to be good at the meta strategy, not just good at predicting sports outcomes. Obviously it's not a good idea to require a bet to make a public statement, but I also wouldn't mind seeing scientists be held more accountable for important decisions. How often are study sections evaluated after the fact, for example?
    - ineedasername 6y ago
      If we're talking public policy decisions, I don't think scientists should be punished when acting on the best available, but incomplete data, especially when two equally reasonable experts within the field can disagree on its interpretation.
ineedasername 6y ago
"Are experts real?" Yes. They are also sometimes wrong. Also sometimes a person fakes expertise. Also sometimes an outsider to a field makes a useful discovery or insight the experts missed. None of the above preclude the existence of actual experts.
peter_l_downs 6y ago
To be fair, the CalPERS board the hired Meng is absolutely corrupt _and_ inept on top of it. At least Meng is now gone. If you want to read about some even more absurd missteps that he made while he was CIO, see Naked Capitalism. The archives have the full stories but the link below has the highlights: self-dealing, lying, misrepresenting his credentials as an investor, and oh yeah, cancelling a massive hedge against market downturns right before coronavirus hit because he didn't understand how hedges work https://www.nakedcapitalism.com/2020/08/calpers-chief-investment-officer-ben-meng-resigns-following-our-exposing-his-false-felonious-financial-disclosure-filings-and-private-equity-conflicts-of-interest.html https://www.nakedcapitalism.com/2020/08/calpers-chief-invest... Honestly it was the least convincing part of the blogpost's overall argument -- in any large-enough pool of people of course there will be a few charlatans and frauds.
hawksprite 6y ago
Shout out to Hexo! I don't see it on here often and it's a great static blog generation tool.
im3w1l 6y ago
The intelligence of a neural network is in the edges. And I think partially, the same goes for social networks. An armchair expert can know better and be more extensively read than a credentialed expert. But as people will not trust (=edge) him as much, he will still be less useful to the group.
throwitaway1235 6y ago
I'm not grasping why if one professional chess player is a fraud, all else are? Would the assumption be that the chess fraud was receiving move instructions from someone else? Again, not sure how that would philosophically impugn all other chess masters.
- barry-cotter 6y ago
  If the top chess master is a fraud everyone else who respected him and his expertise is also either a fraud or was deluded by a fraud. If you can’t tell who’s a fraud in your field when they have the kind of scrutiny being no. 1 invites there’s no there there.

187 points by tlb 6y ago

Experts are real, unfortunately knowledge in many fields is built with very poor standards of quality. The experts in those fields are vastly knowledgeable in falsehoods. While the general public questions if we should be funding the next big project in physics, the real question we should be asking is if we should be funding almost anything else as long as they are not willing to apply the same levels of rigor.

ndiscussion 6y ago
That's right, unfortunately most of science has been debunked ie replication crisis. Sadly, the populace doesn't see it that way.
- orzig 6y ago
  Whoa - [citation needed] on "Most of science has been debunked" We've learned to be much more skeptical, and the previous standards were scandalous, but there's a difference between "too much of" and "most of"
  - ndiscussion 6y ago
    Here's a start, almost every single field is implicated: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis Economics, social sciences, even medicine.
    - Der_Einzige 6y ago
      And don't forget machine learning! If your paper in ML doesn't provide code, I assume that it's not reproducible and the authors made a conscious decision to not release the code...
    - QuesnayJr 6y ago
      Social psychology has a replication crisis, because they had a culture of never questioning a published result. You can see that the vast majority of that article is devoted to problems within psychology. (Medicine also seems to have problems.) In economics 1/3 of 18 studies failed to replicate? A study that small, what are the odds that study itself would replicate? Note that this claim is restricted to experimental studies, which are usually testing human decision-making on economic questions. This is the field of economics closest to social psychology. It seems plausible to me that the model of "get a bunch of people in a room and ask them some questions" is not a reliable methodology. Economics as a whole may have a problem with replication, but we can't tell from the information on that page. There are whole fields that aren't even mentioned -- no physics, no chemistry.
    - chalst 6y ago
      That is not most of science. There are more biochemists than all those fields combined. The fields affected have three properties: they deal with complex entities, they are interested in phenomena that is generally investigated using statistical methods and they deal with noisy data. For those fields, we need new methodologies, but other sciences do not have a comparable problem.
  - nradov 6y ago
    Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124 https://doi.org/10.1371/journal.pmed.0020124
    - berekuk 6y ago
      On the other hand, here's a hilarious response how Ioannidis is wrong in his claim from one of the most well-known replication crisis blogs: https://replicationindex.com/2020/12/24/ioannidis-is-wrong-most-of-the-time/ https://replicationindex.com/2020/12/24/ioannidis-is-wrong-m...
      - nradov 6y ago
        Why do you suppose they chose to publish this hit piece as a blog post instead of going through the peer review process with a real journal article? Or at least a formal letter to PLoS Medicine?
        berekuk 6y ago
        I'm not sure. Maybe they did both and the real article is somethere in peer review pipeline. (Do you imply that no posts on replicationindex should be trusted since they're not properly peer reviewed?) One possible reason is that blog posts are more widely shared on social media, and I agree that this is kind of a hit piece, mostly as a reaction to Ioannidis' contrarian stance on COVID-19, so publicity was probably an objective here.

breck 6y ago

Prediction: in 10 years no "expert" will not be putting their work in Git. Git provides a zero cost (in time and money) way to share your full work with audibility; where fixing mistakes continuously is expected and not something to be embarrassed about; where people can instantly clone to reproduce and test better ideas upon your work; where collaborating with proper attribution is done automatically, et cetera. Look in any field—epidemiology, computer science, physics, chemistry, medicine, et cetera. If the leading experts aren't using git yet, look for the ones that are. They will be the leading experts of tomorrow.

deleted 6y ago
[deleted]
jdsully 6y ago
I like that prediction, but git is actually a bit of a step back from older more traditional source control systems. Rewriting history is far more common with git to the point where teams have to make explicit policies on when that is allowed. Even if git becomes common in science outside of code, I don't think it would overcome the social pressure to appear infallible.
- breck 6y ago
  > Rewriting history is far more common with git How do you mean? It's impossible. You git push something, and then you leave an immutable trail that someone somewhere may have cloned so even if you were to force push or delete you can't erase it. Assuming it was of any importance and someone was watching (but if not of any importance, then no big deal to force push/delete).
  - 0xffff2 6y ago
    Saying something is immutable because someone may have saved an old copy before you mutated your own copy sounds very fragile to my ears.
    - 6y ago

dragontamer 6y ago

Yes, experts are real. Unfortunately, experts are only expert in a few subjects, and experts (well... all humans even) are bad at knowing when they've left their field of expertise. Linus Torvalds for example is clearly an expert programmer. And then suddenly he talks about AVX512 and I wonder if he knows anything about low-level programming or optimization. (https://www.realworldtech.com/forum/?threadid=193189&curpostid=193190 https://www.realworldtech.com/forum/?threadid=193189&curpost...). Case in point: with Rocket lake (on 14nm nonetheless), the power-limits associated with AVX512 have already been largely lifted. It didn't even take a few months before Linus's words became obsolete. https://travisdowns.github.io/blog/2020/08/19/icl-avx512-freq.html https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre... 100MHz difference single-core, 0MHz difference multicore. Virtually no penalty for using AVX512 on new systems. Linus was ultimately trapped in the Skylake-X way of thinking, and not looking forward to the next chips.

zug_zug 6y ago
Did you read the article? The article is specifically talking about cases where experts are incorrect within their field of expertise and then cited as a source of truth in published papers. The question of the article isn't really a boolean, it clearly shows some experts are real and some aren't and asks the more important question -- how do we discern when the institutionalized "expert" in an area is really an expert?
- dragontamer 6y ago
  > Did you read the article? The article is specifically talking about cases where experts are incorrect within their field of expertise and then cited as a source of truth in published papers. Yes. But I still think that Linus is a good character to discuss within the scope of the article. Linus is clearly considered an expert, and many people follow his bombastic words (retweeting, or using them to form the basis of their argument around here). "Because Linus said so" is a common way to end various arguments. Linus's opinion is truly "expert", people copy it without further analysis because... well... Linus kinda deserves it. He is an expert after all. It doesn't mean Linus is always right. -------- The "feedback loop" of the open source world is pretty quick. If you're a bad programmer, people figure it out when they start working with your code. But the "feedback loop" of "chip ISA opinions" is slow. There's not really any way to prove, or disprove, whether or not AVX512 is a good or bad idea. Linus's expertise in programming doesn't necessarily translate to chip ISA design. And as such, Linus can get details (expected power-characteristics of future chips) dramatically wrong.

WC3w6pXxgGd 6y ago

Great article. I have had many debates with people within the last year about the concept of "experts," and have frequently found it almost impossible to break through the myth that 1. Experts exist, and you should believe everything they say. 2. Expertise is fungible, so an "expert" in one small sliver of healthcare is also somehow an overall expert in healthcare. An example of 2 was a conversation I had recently with someone in the healthcare industry who, in this single conversation, said - Cuba has better healthcare than the US - All doctors would be fine with the State forcing them to provide their services because doctors are not profit-driven and only care about The Greater Good.

tboyd47 6y ago

Great article. > Do you have it in you to ignore 800 epidemiologists, and is it actually good idea? Yes, I do, because I was tracking the wave of non-stop bullshit coming from scientists closely throughout all of 2020, and got completely fed up. But I'm not the norm (yet), and no, it's not actually a good solution. We're in the beginning of the story of the boy who cried wolf. Eventually the whole town ended up like me: fully willing to ignore and oppose the mainstream information channels. When a real wolf came, the boy told the truth once out of 1,000 times, and the townspeople all ignored him and got their sheep eaten up. We need to find a way to crowd-source our trust mechanism if "trust the experts" starts to fail. My suggestion is that any "expert" who volunteers an opinion that later gets falsified should be canceled permanently from the public sphere, with no forgiveness. They can go on publishing in scientific journals and selling books, but their opinion should not be accepted by the general public anymore. The other solution is to keep ignoring the boy and start watching your own sheep.

cltby 6y ago
Many argue that the experts ultimately got it right (and are thus redeemed). I strongly disagree. Even when the public health types corrected course, their behavior left a bad taste in my mouth. There was no acknowledgement of having been wrong, no introspection on how their errors came to be, no discussion how they could be avoided in the future. Just aggressive enforcement of the new party line couched in appeals to authority. The whole enterprise reeks of blind dogmatism.
- tboyd47 6y ago
  > Many argue that the experts ultimately got it right (and are thus redeemed). Perhaps their opinion on that one matter they corrected was redeemed, but their general lack of understanding about the limits of their own knowledge was exposed, which is more damning for someone called an "expert."
  - TheTrotters 6y ago
    And in time-sensitive situations getting something "ultimately right" is akin to getting it wrong.

DanielBMarkham 6y ago

This essay points out a pervasive problem that doesn't get much attention (aside from fear-mongering) Because it's a topic so easily prone to hyperbole, it helps if we can reduce it down to a few premises. - Science is science, but the practice of science is a social system. People get tenures, jobs, promoted, invitation to speak, etc. by how well they do in the judgment of their peers - Political systems exist for political reasons, ie, if you don't have a reproducible, falsifiable test in a feedback loop somewhere, your career and goals are all about image. There's no alternative, either for you or your field of study (That doesn't mean other sciences aren't important or aren't really science.) - If somebody appears on TV or in the media playing the role of an expert, it is, by definition, because they wanted to appear on TV or in the media. The goals of being popular and asked about important questions and the goals of whatever your field are ... two different things. The reporter doesn't care who figured out X. They care who can bring excitement and emotion to talking about X. - All this leads directly to the problem of the average science news consumer: any communication you receive about science or research is the result of a lot of conflicting interests. "Being good science" is way down the list, sadly. I imagine the percentage of average readers who understand p-hacking is below 5% - None of this is a secret. The average news consumer knows it, even if they're not able to articulate it - Add in things like the reproducibility crisis, and we've got a huge problem as a species. We've got one flashlight to light our way in the darkness and it turns out there's a loose connection in there somewhere

cbozeman 6y ago
This is a fantastic comment, and spot on I think. I concur with everything you've said.
ska 6y ago
> ...by definition, because they wanted to appear on TV or in the media I suspect this misses some nuance; there is a meaningful difference between people who appear on the media because they want exposure, and people who do it because they want their field of study to be well represented. For example there are people to accept interviews when they genuinely dislike the process, because they felt it was a duty that came along with their chair, etc. I doubt these are ever the most dynamic interviews, but they are common. I agree this doesn't really extend to the level of the same old bobble heads pulled up on prime time news TV, but don't thing you can paint all of the media exposure with this brush.

deleted 6y ago

[deleted]

zug_zug 6y ago

> "This year, we’ve come to better appreciate the fallibility and shortcomings of numerous well-established institutions (“masks don’t work”)… while simultaneously entrenching more heavily mechanisms that assume their correctness (“removing COVID misinformation”)." I suspect in a cultural moment of mass-skepticism of even valid systems (e.g. masks/election/vaccines/wikipedia) it's very hard to touch on the fallibility of those systems without putting people on the defensive. It shouldn't be in an ideal world -- making these things more auditable and rigorous only makes them less prone to questioning in the long term, but I suppose many feel faith in the system is more important at this moment than incremental improvement. [However this really doesn't answer the n=59 conundrum, which I think requires a different lens]

Bakary 6y ago

I remember this concept being mentioned elsewhere as "friendly" and "hostile" learning environments

JadeNB 6y ago
Epstein's excellent book Range uses the terms, though I think they don't originate there.

rusticpenn 6y ago

Regardless of the quality of knowledge, people with better social and networking skills will become better known than those without.

TheTrotters 6y ago
Among the general public? Definitely. Social and networking skills also matter within your professional community but they are not a substitute for publishing in top journals/conferences.
- rusticpenn 6y ago
  Perhaps your experience is different. Having the right name as the first or last author helps a lot in peer review and publication. Even in “double blind” reviews, it’s easy to find the authors usually.
  - angrais 6y ago
    Sort of true, but not quite. You may infer that a paper is from a certain lab or that the first author may be someone specific as it builds on their work... But last author? Not as easy as supervisors/lab leads can change.
    - rusticpenn 6y ago
      Its not the supervisors who become famous. Its the lead Professor of the lab. Usually in engineering papers/journals the first/last author is the lead professor of the lab, who does nothing more than get the funding for the project.

bgorman 6y ago

One of my theories based on life experience is that many fields do not attract the brightest people to begin with, and consequently many of the ideas from these fields are not pressure-tested enough. In todays academia, it is basically impossible to fail getting a PhD in many fields provided you are ok being miserable for years at a time. At the same time academia is filled with a ton of bs like publicly funded research being released behind paywalls, professors receiving grants for projects that are already completed, students being denied graduation util their project can be submitted to the highest impact journal possible etc, having to do a long postdoc to get a tenure track position etc... For smart people that value financial freedom, academia is a terrible option at this point in time. The current coronavirus fiasco has laid bare the lack of critical thinking in the life-sciences fields.

nksv 6y ago
"One of my theories based on life experience is that many fields do not attract the brightest people to begin with, and consequently many of the ideas from these fields are not pressure-tested enough." very well articulated, i have thought about this many times, basically a lot of physical sciences face this

moomin 6y ago

I think this argument conflates two things, one of which is true and slightly terrifying. The terrifying thing is how few scientists understand the statistical underpinnings of their studies. The conflation happens because he thinks this is stratified by field. In practice, there are plenty of sociologists with a decent grasp on this, and a truly astonishing number of biologists who don't. Don't believe me? Read the studies of biology papers and how many have serious problems in their statistical treatment. So we've got a serious problem with how science is being done. Probably doesn't affect whether or not expertise exists, or which fields it exists in. Unless we're going to start saying biology isn't real.

jldugger 6y ago
> Unless we're going to start saying biology isn't real. It probably helps to recognize that statisticians caught one of the first modern biology researchers fudging his numbers: https://en.wikipedia.org/wiki/Gregor_Mendel#Mendelian_paradox https://en.wikipedia.org/wiki/Gregor_Mendel#Mendelian_parado...
- lmm 6y ago
  Except they didn't; they alleged that the numbers had been fudged on rather scant evidence.
  - jldugger 6y ago
    As I understand it, Fisher had two complaints. First, that the trifactorial data was just wildly too accurate (using a chi-squared approach). This seems to be resolved in the scientific community as an off-by-one error: if instead of 10 seeds, Gregor had used 11 on occasion, that's enough to resolve the issue. The other issue is the chi-squared analysis of Mendel's experiments in aggregate, which when pooled also too neatly fit the pattern he wanted to illustrate. And as best I know, nobody has adaquately disputed that analysis.
    - lmm 6y ago
      The only actual analysis I saw came up with a p-value of about 0.07, which is just a complete nothingburger. There's no explanation for that because there's nothing to explain.

klmadfejno 6y ago

IMO Experts are more wise to the problems of their domain than intelligent. Intelligence relates to the execution of mental tasks and the recall of specific facts. Wisdom relates to the planning of the approach to solve a given task. It is whether or not the nature of the problem lends itself to expertise that defines if experts are particularly relevant for a given domain.

13415 6y ago
I'd say that experts are mostly knowledgeable, experienced, and broadly respected within their scientific domain. I don't see how intelligence or wisdom are important factors - other than for the fact that most accomplished scientists are probably rather intelligent.
deleted 6y ago
[deleted]

dmitryminkovsky 6y ago

The Masks Fiasco of 2020 was indeed a noteworthy example of experts-gone-wild, but my favorite is still the Monty Hall problem [0], because unlike epidemiology where proof stems from evidence-based experimentation, the Monty Hall fiasco saw a wide range of experts in mathematics/statistics—a field where truth does not require experimentation but only logic-based proof—loudly and publicly denigrate a person with no expertise in their field, only to be wrong. But I think the an even more interesting question that this post does not cover is the role experts play in our culture. Neil Postman discusses experts at length in Technopoly: The Surrender of Culture to Technology (1992), wherein he characterizes them as basically the shaman-priests who control the Overton window [1] in a world where morality and social acceptability no longer stem from things like religion: > ...expertise is a second important technical means by which Technopoly strives furiously to control information. There have, of course, always been experts, even in tool-using cultures. The pyramids, Roman roads, the Strasbourg Cathedral, could hardly have been built without experts. But the expert in Technopoly has two characteristics that distinguish him or her from experts of the past. First, Technopoly’s experts tend to be ignorant about any matter not directly related to their specialized area. The average psychotherapist, for example, barely has even superficial knowledge of literature, philosophy, social history, art, religion, and biology, and is not expected to have such knowledge. Second, like bureaucracy itself (with which an expert may or may not be connected), Technopoly’s experts claim dominion not only over technical matters but also over social, psychological, and moral affairs. In the United States, we have experts in how to raise children, how to educate them, how to be lovable, how to make love, how to influence people, how to make friends. There is no aspect of human relations that has not been technicalized and therefore relegated to the control of experts. ... > The role of the expert is to concentrate on one field of knowledge, sift through all that is available, eliminate that which has no bearing on a problem, and use what is left to assist in solving a problem. This process works fairly well in situations where only a technical solution is required and there is no conflict with human purposes—for example, in space rocketry or the construction of a sewer system. It works less well in situations where technical requirements may conflict with human purposes, as in medicine or architecture. And it is disastrous when applied to situations that cannot be solved by technical means and where efficiency is usually irrelevant, such as in education, law, family life, and problems of personal maladjustment. I assume I do not need to convince the reader that there are no experts—there can be no experts—in child-rearing and lovemaking and friend-making. All of this is a figment of the Technopolist’s imagination, made plausible by the use of technical machinery, without which the expert would be totally disarmed and exposed as an intruder and an ignoramus. > Technical machinery is essential to both the bureaucrat and the expert, and may be regarded as a third mechanism of information control. I do not have in mind such “hard” technologies as the computer—which must, in any case, be treated separately, since it embodies all that Technopoly stands for. I have in mind “softer” technologies such as IQ tests, SATs, standardized forms, taxonomies, and opinion polls. Some of these I discuss in detail in chapter eight, “Invisible Technologies,” but I mention them here because their role in reducing the types and quantity of information admitted to a system often goes unnoticed, and therefore their role in redefining traditional concepts also goes unnoticed. There is, for example, no test that can measure a person’s intelligence. Intelligence is a general term used to denote one’s capacity to solve real-life problems in a variety of novel contexts. It is acknowledged by everyone except experts that each person varies greatly in such capacities, from consistently effective to consistently ineffective, depending on the kinds of problems requiring solution. If, however, we are made to believe that a test can reveal precisely the quantity of intelligence a person has, then, for all institutional purposes, a score on a test becomes his or her intelligence. The test transforms an abstract and multifaceted meaning into a technical and exact term that leaves out everything of importance. One might even say that an intelligence test is a tale told by an expert, signifying nothing. Nonetheless, the expert relies on our believing in the reality of technical machinery, which means we will reify the answers generated by the machinery. We come to believe that our score is our intelligence, or our capacity for creativity or love or pain. We come to believe that the results of opinion polls are what people believe, as if our beliefs can be encapsulated in such sentences as “I approve” and “I disapprove." [0] https://en.wikipedia.org/wiki/Monty_Hall_problem#Vos_Savant_and_the_media_furor https://en.wikipedia.org/wiki/Monty_Hall_problem#Vos_Savant_... [1] https://en.wikipedia.org/wiki/Overton_window https://en.wikipedia.org/wiki/Overton_window

xyzelement 6y ago

I want to share this tid-bit. When I studied portfolio management, I had to learn not just how to talk to people about their risk tolerances, but how to ascertain whether they knew their own tolerances. The concept was called "calibration" - or - how well is someone able to know that they are wrong/don't know something. Basically, "high calibration" is the opposite of Dunning-Kruger in my mind. The text cited a study that measured calibration by profession. The highest calibrated profession was meteorologists. It makes sense if you think about it: I prognosticate rain tomorrow, tomorrow comes and it's sunny - I am forced to confront the fact that my prognosis was wrong. So not only is there a way to know who's better and who's worse, but also constant reminder that "I can be wrong despite my best work" The lowest calibration profession was doctors. Which also makes sense. Imagine I am a doctor and I miss an early sign of a disease. The patient dies 5 years later - I may not even know, and even if I do, very unlikely am I confronted with the specific point in time 5 years prior where I made the wrong call. So the thing that matters with expertise is - how often is the person getting feedback. In the article they cite chess as having "real experts" - of course. Because a chess player gets feedback on how good they are every time they win or lose a match. On the flip side, how often is someone in a very soft field getting feedback? What's the way for an "expert" in say Gender Studies, to wake up in the morning and be confronted with "the thing I did yesterday didn't work?" The other thing about experts is they are optimizing for a single thing. Eg ask your compliance person at work if you can do something, the answer is "NO" because they are optimizing for not being sued, but in reality that's not the only concern. Or in the pandemic, ask the economist if we should lock down - they'll say no because they understand the financial consequences. Ask a medical person and they'll say yes because they are focused solely on the number of sick people. In reality the world is complex and there are tradeoffs. Related, experts are often concerned with high-impact issues which may be different from individual issues. EG: in the USA, by default, kids get the Hep B vaccine on their first day of life. On country level this makes sense because there are a lot of mothers out there who have Hep B and don't know it. But when my wife was giving birth, we knew to opt out despite the recommendation because we understood that the experts are optimizing for a thing that doesn't relate to us.

jldugger 6y ago
Any chance you can recall enough of that paper to find a citation? I'd be interested to read such a paper.

lordnacho 6y ago

I think stats is the culprit. Or lack of it. If there's one area of study that ties together observations and models, it's stats. But stats suffers from the same problem that programming does: people think it's a tool that they just need to know a little bit of, so they don't defer to the proper authorities in the area. Of course this is just anecdotes, but I do have a friend who did stats as a phd and this is the attitude that seems to be common: stats (and code) are minor tools that won't affect the outcome of our experiment.

cutemonster 6y ago
Some ppl who studied social sciences might even be uninterested in statistics and look at it as off topic. After all, most people aren't good at mathematics (i think) and someone who, say, chose to study psychology might have preferred that there wasn't any math stats courses in the program at all. (I know one such person in real life)

nonpolitical 6y ago

This reminds me of my layperson puzzlement: Can we apply and verify the results from CERN experiments beyond experimental settings? If we can't, how can we know the results from CERN experiments aren't just artifacts of the experimental apparatuses?

losvedir 6y ago
That's a good question. You're right that a random grad student in a lab can't replicate, e.g., the discovery of the Higgs boson! However, at least for the Higgs (and I believe this is standard for most such large experiments), there were actually two different groups using two different apparatuses, both at CERN. There was the CMS[0] detector and ATLAS[1] detector, used and analyzed by two different groups of physicists, resulting in two papers[2,3], published on the same day. [0] https://en.wikipedia.org/wiki/Compact_Muon_Solenoid https://en.wikipedia.org/wiki/Compact_Muon_Solenoid [1] https://en.wikipedia.org/wiki/ATLAS_experiment https://en.wikipedia.org/wiki/ATLAS_experiment [2] https://arxiv.org/abs/1207.7235 https://arxiv.org/abs/1207.7235 [3] https://arxiv.org/abs/1207.7214 https://arxiv.org/abs/1207.7214
- nonpolitical 6y ago
  Thanks! Pity that I guess I won't be able to understand the papers linked in the foreseeable future.

deleted 6y ago

[deleted]

yboris 6y ago

Surprised that the classic "Conditions for intuitive expertise: A failure to disagree." paper by Daniel Kahneman and Gary Klein is not referenced https://psycnet.apa.org/record/2009-13007-001 https://psycnet.apa.org/record/2009-13007-001 My takeaway: without short feedback loops it is hard to build expertise. If it takes 2 years to learn the individual you paroled reoffended, you can't learn (so there are virtually no expertise in parole officers or university admissions). In these scenarios, simple linear models outperform "experts" by a wide margin. See: "In Praise of Epistemic Irresponsibility: How Lazy and Ignorant Can You Be?" by Michael Bishop https://philpapers.org/rec/BISIPO https://philpapers.org/rec/BISIPO

d23 6y ago

A lot of the points are interesting, but I couldn't help but get hung up on the link to the covid analyist who supposedly beat the experts, at https://www.maby.app/covid/ https://www.maby.app/covid/ A couple of immediate red flags: the predictions stop in early May, and there appears to be no weighting of how "important" a given prediction is. Perhaps the former is just because a contest ended at that time. For the latter, of course, importance is subjective, but ranking these two as equivalent is silly: * How many cases per day will Washington state average for the week ending June 7? (Tom: good prediction, Experts: bad) * How many US COVID deaths will occur in 2020? (Tom: bad prediction, Experts: good)

eesmith 6y ago

Teller attributes Bohr as saying: "An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field." I prefer something closer to "An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them". 'Cause even experts make mistakes in their own field. This piece seems to think an expert cannot make a mistake in a given field of expertise, which is "utter nonsense, of course." I don't like this example: "If you look at the papers citing Viechtbauer et al., you will find dozens of them simply using N=59, regardless of the problem they're studying" Paging through the linked-to Google Scholar search finds 90 results. Of the first 12 with readable text, only two seem correctly described that way (and max 3), suggesting under two dozen examples. A spot check of the remainder suggests that pattern holds. I'll show my work: #1 says "A sample size of 30 patients per group was chosen as recommended for pilot studies to achieve an appropriate level of statistical power [30]." They selected 60, and "1 patient was lost to follow-up" giving 59. So this wasn't N=59 but N=60. - https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD&utm_medium=cpc&utm_campaign=JMIR_TrendMD_0 https://medinform.jmir.org/2019/4/e14044/?utm_source=TrendMD... . 30 was chosen because "It corresponds to the detection of a potential difference of 21% between groups for a power of 80% and an alpha significance level of 5%." which is not the default example from Viechtbauer. #2 does not use N=59. The number 59 occurs because: "The study included 16 patients with diabetes (59 ± 8.8 years; 8 men and 8 women)" - https://www.sciencedirect.com/science/article/pii/S0958259218300919?casa_token=7GVLsoy6nDcAAAAA:glWnYg27NQjB6ZjYV6NxbJQNAlI0-o694dpZyO4nme3xFDAfWFrh8dv1i0wCheNX2CnJlYxddA https://www.sciencedirect.com/science/article/pii/S095825921... #3 uses N=59, though I don't know if it was used incorrectly because I can only read the abstract and citation list - https://journals.sagepub.com/doi/pdf/10.1177/0306624X19858487?casa_token=9iQatZFhsc8AAAAA:2fgaomaQLIxdvPp_P1NKoD25-kPwghHXbl3LmxT-SBbY4mIjtJ0obJ6m7qAoOCCk60pEqUttB1FQ https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848... #4 specifically does NOT use Viechtbaueret, writing: "Viechtbaueret al., (2020) recommends sample size between 10-40 as adequate for feasibility study. Linacre, (2015) suggests a sample of thirty (30) respondents as appropriate for general purposes. This study however uses 60 respondents becauseadequate sample and item produce a good item, person separation and reliability in Rasch measurement (Fisher, Elbaum, & Coulter, 2010)." - https://journals.sagepub.com/doi/pdf/10.1177/0306624X19858487?casa_token=9iQatZFhsc8AAAAA:2fgaomaQLIxdvPp_P1NKoD25-kPwghHXbl3LmxT-SBbY4mIjtJ0obJ6m7qAoOCCk60pEqUttB1FQ https://journals.sagepub.com/doi/pdf/10.1177/0306624X1985848... #5 uses N=43 ("With an alpha value of 0.05 and with an 80% power we need a sample size of at least 43 patients to detect a difference in validity and assess reliability") - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6886622/ . This is not the default example from Viechtbauer. #6 uses N=59, using calculations directly from Viechtbaueret. It appears to be used correctly. #7 uses N=53, commenting "There is no gold standard for sample size calculation in pilot studies. Sample sizes as small as 10 (Hertzog, 2008) and as large as 59 (Viechtbauer et al., 2015) have been recommended." - https://journals.lww.com/nursingresearchonline/FullText/2018/09000/Enhancing_Men_s_Awareness_of_Testicular_Disorders.2.aspx https://journals.lww.com/nursingresearchonline/FullText/2018... #8 uses N=30 for each region, writing "Tamaño de la muestra: en cada región, se llevó a cabo un estudio piloto con aproximadamente 30 pacientes" - http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1134-928X2020000400001 http://scielo.isciii.es/scielo.php?script=sci_arttext&pid=S1... . The 59 occurs in, for example, "La reducción también fue estadísticamente significativa (59%) al tener en cuenta el coste del tiempo de enfermería". #9 points to the N=59 recommendation (correctly, for their experiment) but actually uses N=71 - https://www.mdpi.com/1422-0067/20/8/2024/pdf https://www.mdpi.com/1422-0067/20/8/2024/pdf #10 appears to have the discussed N=59 problem as it doesn't mention the 0.05 effect - https://mhealth.jmir.org/2016/3/e109/ https://mhealth.jmir.org/2016/3/e109/ . #11 says "Pilot studies should include at least 12 patients [30,31], and cluster-randomized studies need more patients to correct for the clustering [32]. To have sufficient power for this pilot study, we intended to include 15 patients per nurse." where [31] is Viechtbauer. The value 59 is because "data of 149 (98%) patients were available of whom 66 (44%) were in the intervention group; ‡ = data of 135 (89%) were available of whom 59 (44%) were in the intervention group". - https://www.sciencedirect.com/science/article/pii/S0261561419331656?casa_token=G9Wp2q11YQYAAAAA:t8UiqCu8LB1egMYID6J-BAm7FpBnkXAN2-nbzVXdPSdt2mRAB3CXHzsFrmjhXw1t7Rmjq_GdWQ https://www.sciencedirect.com/science/article/pii/S026156141... #12 says "In line with recommended sample sizes of pilot feasibility trials, 83–85 40 patients is deemed sufficient to explore feasibility, acceptability and potentially efficacy of the intervention, assuming retention rates of 80%, the true population consent rate will be with a margin of error of 13% (95% binomial exact CI)". The only "59" comes from citations. - https://www.sciencedirect.com/science/article/pii/S0261561419331656?casa_token=G9Wp2q11YQYAAAAA:t8UiqCu8LB1egMYID6J-BAm7FpBnkXAN2-nbzVXdPSdt2mRAB3CXHzsFrmjhXw1t7Rmjq_GdWQ https://www.sciencedirect.com/science/article/pii/S026156141... I'm not sure how we're to have a take-home lesson when the false negatives so overwhelm the claimed evidence. I'm also hung up on: "Capitalism and evolution work so well because of their merciless feedback mechanisms." because of obvious questions like "does capitalism work well for slaves?"

hikerclimber 6y ago

i don't think so... you can gain expertise in anything.

altarius 6y ago

One of my problems with this line of argument is that it provides a sort relativism of opinion, every opinion is just as valid as any other. And then you're at post-truth and post-fact sadly... If Carlsen is a fake, so is every chess player he has ever played, the whole system? Epidemiologists lied about masks (or they really didn't know better 3-5 months into the pandemic, also not good), discount everything else said? I've heard these arguments often lately :/ Experts not being "real" implies that there's no absolute expertise whatsoever and implicitly legitimizes dismissing any argument whatsoever on a single failure and allows people to believe that any opinion (including their own) is just as valid. I think this is in part to blame for our "post-fact/post-truth" phenomenon. Many people decide experts or government were wrong/have lied before and can no longer be trusted. And when an outsider comes along with contrarian but convenient ideas, why not believe them, nobody else is to be trusted after all and every opinion is valid. Any expert opinion can be wrong but the prior is that an expert opinion is much more likely to be correct than your own, or a politician's, or a blogger's. NB: I want to note that the author's point is much more nuanced and (more importantly!) narrow, but it's not evident from the title or a superficial reading.

mcguire 6y ago
"Epidemiologists lied about masks (or they really didn't know better 3-5 months into the pandemic, also not good)..." Technically, if my timeline is right[1], about 2 months into the pandemic, for his example. 3-4 months generally. Plus they had a prior belief that masks wouldn't be useful. [1] The world outside China learned about the thing at the end of December; his example is the first week of March.
nickthemagicman 6y ago
'Experts' often disagree. Which makes you wonder if the whole system is actually invalid. Experts are merely one data point, at the end of the day you have to look at the entirety of the data yourself to make a decision.

mcguire 6y ago

On March 4, 2020, Michelle Odden tweeted about 800 epidemiologists not wearing masks. Because I don't commute anymore, I'm way behind in my podcasts. Three podcasts I listen to, and am behind in, are the Naked Scientists (a British popular science broadcast), and the Nature and Science podcasts (which cover weekly scientific news). The interesting thing is that, as I listen to them now, I'm going through a live view of the past with present hindsight. It's fun. One interesting thing is that aerosol, airborne transmission of many viruses was (and is?) controversial, or at least an open research question. (Recognition of aerosol transmission of infectious agents: a commentary [Jan 2019] https://bmcinfectdis.biomedcentral.com/articles/10.1186/s12879-019-3707-y https://bmcinfectdis.biomedcentral.com/articles/10.1186/s128...) Up until at least mid-April, the default belief of the medical community seems to have been that Corona viruses were not, or at least not particularly, airborne. Instead it would be spread by physical contact (contaminated surfaces, etc.) or large droplets produced by coughing or sneezing. As a result, the default use of masks was not particularly recommended.[1] (And that's before you consider the scarcity of PPE for medical users at the time.) After about mid-April, the facts on the ground seem to change, and things like aerosol transmission and asymptomatic spreaders suddenly become a much larger concern. (And a lot of people on my podcasts got sheepish with the egg on their faces.) As a result, it's not particularly surprising that an epidemiologist would not consider wearing a mask to be particularly useful in early March. [1] One thing the article does not touch on is the tendency of actual experts to get feelings akin to religious hysteria based on internal fights, with the resulting dramatic statements spilling over into outside conversations. (I had this one paper ripped to shreds by a old-time network guy because I and my advisor had the temerity to consider keepalives/heartbeats.) I suspect there may be something behind the default assumption up until April that masks were bad.

mncharity 6y ago

People seriously misunderstand expertise. There's the perennial news stories like "Harvard MBA's don't know what causes Earth's seasons!?! LOL!". And the "He's a Scientist, so he must know...". And the "my local TV news meteorologist proved climate change isn't real". And in TFA, "But sometimes [experts] say something so obviously incorrect, [...] that the only possible explanation is that they understand very little about their field. [disillusioned gasp!]". So, you're a computer engineer? Wow, you must be sooooooo smart. An Expert! And did you read Huckleberry Finn in school like me? Yes, thirty years ago? Great, what do you think of the raft trip as an allegory for mumble mumble? What, you've no idea?!? - LOL. But I'm glad you're an Engineer - tell me what kind of solar panels should I buy, and about driving trains. And, Ha ha, you didn't even notice the computer cable was loose - you obviously understand very little about computers. If the last time someone touched something was in middle school, don't be surprised to find a middle schooler's understanding. If you ask a protein chemist a quantum chemistry question, don't be surprised to get a decades-old graduate student's answer. Don't be surprised if a respected fruit fly biologist, believes earthy-crunchy nuttery about the human heath benefits of eating nuts. Don't be surprised by scientists gossiping at conferences "Yeah, don't believe work from [first-tier PI] on topic X - he sees what he wants to see". Don't be surprised when a first-tier stone cold empirical physicist, says "I don't need to test it - I trust my gut to tell me how well my teaching is working". Expertise can become ramshackle startlingly fast as you move away from someone's own research focus. Or sometimes from an advanced teaching focus. A focus with fear of being embarrassed by getting things wrong in front of double-checking picky peers. Getting things right is hard, and takes effort - which has to be incentivized, and monitored. And it's not just "what is expertise" which is taught badly. If I told you "Yay, I found a good Professor of Computer Science, to write a Computer textbook, including language design, microservices, chip taping, and UX - I'm sure he will do a correct and insightful job of it", what would you think? So, what if you heard "Yay, I found a good Professor of Chemistry, to write a Chemistry textbook, including bonding, nuclei, materials, and nanostructures - I'm sure he will do a correct and insightful job of it"? Or how about "Welcome to <big textbook publisher>, new graduates, new science textbook writers. Don't worry that you only have a liberal arts degree, and no background in science at all, because we have a Scientist on call for questions!". Sigh. People seriously misunderstand expertise.

ncmncm 6y ago

The example of magnetohydrodynamicists vs astrophysicists doesn't seem entirely idle.

keithnz 6y ago

one of the problems of experts is that at the edge of their field of expertise, they aren't experts of the edge. It's just they have developed expertise in the building blocks to allow them to live at the edge. Quite often we are interested in things happening at the edge, and the "experts' can't really provide information at high levels of certainty.

blueflame7 6y ago

Yes. Dr Fauci has won 12 times in a row at the national infectious diseases championship. He is THE expert

deleted 6y ago

[deleted]

CyberRabbi 6y ago

Experts do exist but I think many alleged experts often give into the temptation to use the expertise in their field to represent expertise in a field in which they cannot claim the same level of expertise. This is my opinion because I’ve experienced the temptation to do just that many times and I’ve witnessed instances where that has happened as well. I don’t think this behavior is limited to people who have expertise in any field but it might be most damaging when experts do commit that logical fallacy.

makz 6y ago

Well, I’m an expert and as far as I know I’m real, so my answer is yes.

karlhughes 6y ago

My main problem with experts is they share the same cognitive biases as the rest of us: https://www.karllhughes.com/posts/experts https://www.karllhughes.com/posts/experts It's great to have academic qualifications, but you still have such a limited sphere of experience and knowledge - even more so the higher you go in academia.

obviouslynotme 6y ago

Scientists are the new priest class. They deliver edicts and answer the public's questions about the universe. The problem is that the answer to 99% of the public's questions is "I don't know" or "It's very complicated." We don't like that. We want the right answer, right now, delivered in one simple sentence. We find the one goober who hasn't walked inside of a lab in decades who wants to feel important to give us what we want. Worse is that people envision scientists as unbiased saints. That's just not real. Real science is about one thing and one thing only. Money, aka grants. Grants get you labs and underlings which bring more money and maybe someday tenure. And once you have tenure, you can finally publish what you really think about what interests you instead of chasing funding fads. Except that part never happens. That means you study the subjects that give funding. You give papers that increase the chance of more funding for more papers. Your accuracy? Doesn't matter. Funding matters. No one is going to check your work anyhow beyond simple peer review. Someone tries to recreate your work or recalculate your numbers? Tell them you lost the data. It's a bit embarrassing but no one is going to call you on this stuff anyhow. There is only one solution to the replication crisis. Funded third-party replication for all experiments. Money is the root of this problem. It has to be the solution. How many scientists are going to publish "mistakes" if they know that four different teams at four other universities are going to automatically attempt replication?

DaiPlusPlus 6y ago
> Real science is about one thing and one thing only. Money, aka grants no.
- obviouslynotme 6y ago
  If you think that real research isn't dependent on funding, then I really don't know what to tell you.
bonoboTP 6y ago
Many people only interface with "science" through school classes and popular science magazines / documentaries / IFL science / antitheist memes, arguing against hoaxes, flat Earth, creationism etc. People dislike your answer because they have no idea how the sausage is made, when actually your description is quite realistic.

_cs2017_ 6y ago

We accept that any complex software, even well-designed and well-tested, can malfunction -- often spectacularly. What, then, can we expect from humans? Consider that virtually no effort went into the design; and the test framework is horrific, relying primarily on trial and error. The natural outcome is that any human "expert" is no more reliable than your average startup codebase. In fact, it's a wonder that human societies manage to organize themselves into anything resembling civilization.

analog31 6y ago

I think experts exist, but there's a problem with translating expert opinion in evolving fields into actionable advice that can be communicated to the masses through the news, and that can withstand aggressive, organized skeptical attack. Okay, that's a bit of an exaggeration, just because it rolls a whole bunch of different things into one blanket statement, and sounds like a rant, which it kind of is. The world is fucked. What's the point of an organized skeptical attack on experts? I think I can attribute to Imre Lakatos the thought that if you can completely discredit the experts, and make it impossible to decide on facts, then you create a power vacuum that can only be filled by those who are capable of acting in the absence of facts, such as thugs and tyrants. You have effectively censored the experts with greater efficiency than any government censorship program could ever dream of. I believe this is the precise reason for "the war on the truth."

logicchains 6y ago

From an economic perspective there's a relatively simple solution: requiring experts to wager a certain amount of their own money on a measurable outcome of their prediction when making public policy recommendations. If somebody's not confident enough in their own prediction to put their own money on it, then that prediction shouldn't be used as the basis for policies that can affect millions of peoples' lives.

ineedasername 6y ago
Science is not about being right every time. Stating a hypothesis publicly as you're experimenting to determine it's validity shouldn't require a wager.
- caddemon 6y ago
  To be fair, betting isn't about being right every time either. To really make bank on a site like DraftKings you need to be good at the meta strategy, not just good at predicting sports outcomes. Obviously it's not a good idea to require a bet to make a public statement, but I also wouldn't mind seeing scientists be held more accountable for important decisions. How often are study sections evaluated after the fact, for example?
  - ineedasername 6y ago
    If we're talking public policy decisions, I don't think scientists should be punished when acting on the best available, but incomplete data, especially when two equally reasonable experts within the field can disagree on its interpretation.

ineedasername 6y ago

"Are experts real?" Yes. They are also sometimes wrong. Also sometimes a person fakes expertise. Also sometimes an outsider to a field makes a useful discovery or insight the experts missed. None of the above preclude the existence of actual experts.

peter_l_downs 6y ago

To be fair, the CalPERS board the hired Meng is absolutely corrupt _and_ inept on top of it. At least Meng is now gone. If you want to read about some even more absurd missteps that he made while he was CIO, see Naked Capitalism. The archives have the full stories but the link below has the highlights: self-dealing, lying, misrepresenting his credentials as an investor, and oh yeah, cancelling a massive hedge against market downturns right before coronavirus hit because he didn't understand how hedges work https://www.nakedcapitalism.com/2020/08/calpers-chief-investment-officer-ben-meng-resigns-following-our-exposing-his-false-felonious-financial-disclosure-filings-and-private-equity-conflicts-of-interest.html https://www.nakedcapitalism.com/2020/08/calpers-chief-invest... Honestly it was the least convincing part of the blogpost's overall argument -- in any large-enough pool of people of course there will be a few charlatans and frauds.

hawksprite 6y ago

Shout out to Hexo! I don't see it on here often and it's a great static blog generation tool.

im3w1l 6y ago

The intelligence of a neural network is in the edges. And I think partially, the same goes for social networks. An armchair expert can know better and be more extensively read than a credentialed expert. But as people will not trust (=edge) him as much, he will still be less useful to the group.

throwitaway1235 6y ago

I'm not grasping why if one professional chess player is a fraud, all else are? Would the assumption be that the chess fraud was receiving move instructions from someone else? Again, not sure how that would philosophically impugn all other chess masters.

barry-cotter 6y ago
If the top chess master is a fraud everyone else who respected him and his expertise is also either a fraud or was deluded by a fraud. If you can’t tell who’s a fraud in your field when they have the kind of scrutiny being no. 1 invites there’s no there there.