5 ms·
Smart Folk Often Full of Crap, Study Finds
- andy99 6y agoThe concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important to know how well you can trust the prediction of a ML model (ironically to that you can do something like send low confidence predictions for manual review, which apparently might not be that calibrated either)... Deep learning models in particular are not good at accurately estimating confidence, so there is a whole body of work on getting better confidence estimates, e.g. [0]. It's nice to see that people make the same kind of mistakes. [0]Guo et al., https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599
- mxcrossb 6y agoGiven that people had a bimodal distribution of confidence, I would assume that means a lot of the perceived overconfidence come from people answering a question wrong that they thought was easy. I wonder if the test takers would say “that was a trick question”.
- hn_throwaway_99 6y agoYeah, I had the thought that the math might work out in such a way that this is the expected result. That is, if you say you are "50%" confident, it really means you have no confidence - simply guessing would get you 50%, so you can't go less than that. But on the other end, for people who say they're 100% or 95% confident, it's pretty much all downside since you can't go over 100%.
- ummonk 6y agoYeah I'm going through https://www.congap.com/ https://www.congap.com/ and I see quite a few questions where the correct answer is often the one that goes against intuition. Thus, low information responders might make a reasoned guess and assign somewhat greater than 50% confidence when in reality they have a less than 50% chance of getting it right. Incidentally, I pretty quickly learned to guess the most unusual answer for questions I have no direct information about.
- pmiller2 6y ago> Incidentally, I pretty quickly learned to guess the most unusual answer for questions I have no direct information about. I suspect that's a strategy many people who are both educated and intelligent would pick up on.
- ncmncm 6y agoI am not completely sure I believe that Black Magin Woman was first performed by Feetwood Mac [sic]. And I confess I have no idea what a stop side is. But apparently I was correct to guess that it does not have six sides. And I remain dubious that the Mona Lisa was painted by someone other than Leonardo.
- microtherion 6y agoI did not know that about Black Magic Woman, but (give or take an "l") it's true: https://secondhandsongs.com/work/12143/versions#nav-entity https://secondhandsongs.com/work/12143/versions#nav-entity
- ncmncm 6y agoTaking an "l" and a "c", it's very definitely false.
- pmiller2 6y agoGoogle will easily convince you of the first two. The last one is speculation.
- ncmncm 6y agoGoogle doesn't tell me word one about "Black Magin Woman" and "Feetwood Mac"; or about "stop sides". If it tells you, I feel 100% confident that you are hallucinating. The quiz reliably informed me that I am consistently underconfident.
- pmiller2 6y agoWell, then, you could have communicated more clearly without being deliberately obtuse. Did you also communicate to the author that there were many typos in the survey? If not, you probably just deliberately skewed the survey for your own amusement.
- 6y ago
- rahimnathwani 6y agoThe book 'How to measure anything' talks about how to improve your own calibration. From what I recall (it's been a while since I read it), the author recommended testing yourself to estimate a bunch of questions with numerical answers. For example: what's the height of the empire state building? You write down a range for each answer, trying to make the range narrow enough that you're just about 80% confident that the actual answer is within the range. By doing this repeatedly, and periodically reviewing your cumulative correct rate, you can calibrate appropriately (e.g. widening or narrowing your ranges for future questions).
- smabie 6y agoJob applicants for junior trading (market making) roles are often given similar questions. An example question might be: "Make me a market on the height of the empire state building." By this the interviewer means: what price would you buy at and what price would you sell at? So for example I might respond 1000/2000. which means I'm buying at 1000ft and selling at 2000ft. Presumably, I believe the true height is somewhere between these two numbers (ideally 1500ft), at a certain confidence level. The interviewer might ask you to quote a spread given a confidence interval. The interviewer might then lift your bid or ask and then ask you to make a new market. There's a couple factors at play here: 1. The interviewer has more information than you (presumably), so whether he bought from or sold to you is valuable information. You might want to raise your quotes if he buys in order to capitalize on the new information. 2. You want to decrease your risk. So for example if he just lifted your bid, you might want to lower your quotes: offer a lower price to buy and a lower price to sell. A good market maker capture the difference between the bid-ask spread: the price for which you buy at and the price that you sell at (like any other middle man). By lowering your quotes, you are increasing the probability that you will flatten out your risk (you are not trying to speculate on the height of the empire state building). These two things are in contention, and the interviewer wants to see how you think about it/deal with it. Also the interviewer wants to ascertain whether your confidence levels are correct over multiple questions. ie if the confidence level is 80%, then you should be wrong (true value lies outside your range) about 20% of time. It is not good to be either under or over confident. Accurately dealing with uncertainty is what they are looking for. Anyways I thought this was a very similar process that you were referring to, except you are dealing with money. Which always helps to clarify things!
- WalterBright 6y agoSmart is how well your brain works. Education is the things your brain knows. Wisdom is the combination. Smart does not imply having knowledge.
- teh_infallible 6y agoYou can be smart, intelligent, and educated without being wise. In fact, it’s the norm.
- pmiller2 6y agoI agree. I would say intelligence is what you have naturally, education is what you acquire in school, and wisdom is what you acquire through experience. These are not interchangeable.
- jb775 6y agoI'd say smart is what you have naturally, education is what you acquire in school, intelligence is a snapshot of what you actively remember from education plus what you've learned via recent experience, and wisdom is a scratchpad where your brain makes a note whenever it comes across something that's relevant to all of the above.
- wyqydsyq 6y agoI prefer to think of it as wisdom is what you know intrinsically, education is what you have learned, intelligence is your capacity for learning
- philipswood 6y agoNice summary, but I dislike the fact that education has made the default understanding of these categories to be about retaining information/knowledge rather that your ability to compute/reason/intuit. Yes, familiarity with a field should improve skill, but I like to think that one can be smart, intelligent or wise in a situation that does not involve previous information about it. Someone exposed to a brand new game or puzzle can be smart, intelligent or wise in solving it separately (to some degree) from having prior information.
- exophoric 6y agoA study on confidence without confidence intervals?
- indigodaddy 6y agoI did a double take at the title, thought it had to be an Onion article..
- strait 6y agoArea Man Finds Study Finding Him Full of Crap, Full of Crap
- philipswood 6y agoYeah, its a bit disingenuous: it's highlighting a single conclusion noted in passing that's not part of the main thrust of the post.
- voidhorse 6y agoI think anyone who has inhabited planet earth for more than 25-30 years or so starts to realize that most of people are either: 1. “Geniuses” of speculation and every topic under the sun (overconfident braggarts that make sweeping claims that are difficult to criticize because they are so general; such dreamers talk big but really have little expertise) 2. People that actually are “deep-knowledge” experts, but only in one particular, very specialized domain and can otherwise barely tie their shoelaces And that the 1. Kind of people make visionary, ridiculous claims that are then tempered and shored up by the 2. Kind of people behind the scenes that actually do the work so as not to cause horrific catastrophes and that true genius is rare and hardly ever recognizable as such within the genius’s own lifetime. If you have no knowledge but boundless confidence and a good sense for what kind of statements you can make without actually resulting in direct (i.e. traceable to your statements) physical or financial injury to other people you can get pretty far in life. I’m aware of the irony that the claims in this post put me in the 1. category.
- Kye 6y agoThere's a subset of 1 who are good at making wild guesses while leaving open the possibility they're wrong so 2 has an opportunity to share what they know.
- tapland 6y agoI don't have actual expert knowledge in anything but a bit of knowledge of a broad range of things. So there's certainly a spectrum of "1" where you have all levels of actual knowledge. It reads like someone is just grumpy dividing it up like that.
- xwolfi 6y agoThere are also a lot of people who know nothing and struggle to understand anything new and don't either brag about it or have any deeper field on understand. They can also usually tie their shoelace.
- voidhorse 6y ago
- backtoyoujim 6y agohow does this affect my micro-dosing schedule ?
- itronitron 6y agoI have doubts about this study. It could be that participants are thinking about the questions differently than the author anticipates. For example, one could argue that a stop sign has ten sides and not eight because it is a physical object with a front and back. And if it has circular bolt holes for mounting to a post then you could also say it has many more sides than ten.
- atian 6y agoThis is fair to be part of the study although could be varied a bit more. The error that participants make in assumptions is surprise that the market does price.
- valand 6y agoThe study asserts assumption to the participants that the stop sign is 2D object "sides" usually is a term describing 2d objects, 3d objects are described by faces, edges, and vertices
- adrian_mrd 6y agoMaybe a quick litmus test to identify such people is those who refer to themselves as ‘thought leaders’ on LinkedIn /s
- fallingfrog 6y agoThe math behind this is wrong. It’s taking a naive average of your confidence scores so that if your average confidence level is 95% then you must think you got 9.5 of 10 correct. No, if I answered 95% confidence on all 10 then my estimate of the probability I got at least one wrong is 1 - .95^10 = 40%. Whereas if I answered 100% confidence on 9 and 50% confidence on one, then i am predicting 50% probability that I got 9, and 50% probability I got all 10. I assume the paper is riddled with similar mathematical errors. You can’t just average probabilities together like that.
- deleted 6y ago[deleted]