10 ms·
Resident physicians' exam scores tied to patient survival
- Bostonian 2y ago'Board exam performance was powerfully linked to patient risk of dying or hospital readmission. For example, there was an 8 percent reduction in the odds of dying within seven days of hospitalization in patients of physicians who scored in the top 25 percent on the exam, compared to the patients of physicians who scored in the bottom 25 percent on the exam, which was still a passing grade.'
- goodluckchuck 2y agoControlled hospital quality? I figure the best credentialed doctors go to the best hospitals, where patients receive a lot of other care aside from the MD.
- sebmellen 2y agoThis is the most important question in the thread.
- pcthrowaway 2y agoDid they also control across types of medicine? If the higher-scoring doctors go into types of care which are more competitive, could those practices have lower patient mortality within 7 days? For example, maybe burn unit care is high-mortality and low-barrier, compared to sleep medicine which is low-mortality and high-barrier (I don't know how accurate this is, just providing some hypotheticals for clarity)
- helsinkiandrew 2y agoFrom the article: >The researchers compared outcomes for patients within the same hospitals who were cared for by doctors with different exam scores. This allowed the researchers to eliminate, or at least minimize, the effect of differences in patient populations, hospital resources, and other variations that might influence the odds of patient death or readmission, independent of a doctor’s performance.
- ETH_start 2y ago[flagged]
- colechristensen 2y agoI’m all for diversity but that admissions gap is just racism. You can’t have separate entrances for your establishment based on what folks look like, the group you prefer getting better service doesn’t make it equality.
- sagarm 2y agoI know right? Black applicants with high MCAT scores were rejected in favor of white applicants with low MCAT scores! Just unbelievable.
- NotYourLawyer 2y agoUnbelievable because it’s the opposite of what the link shows?
- sagarm 2y ago96% acceptance rate for black candidates with high MCAT scores, but a nonzero acceptance rate for white candidates with low scores. Maybe there are other factors, and they're correlated with the buckets being used here?
- NotYourLawyer 2y agoIt’s not the case that every single black applicant gets admitted before a single white/Asian applicant does. The point is that it’s much, much easier for a black applicant to get admitted. A black applicant with GPA and MCAT scores in the lowest bucket still has a 56% chance of admission. That’s on par with an Asian applicant who has GPA and MCAT scores in the highest bucket.
- yyyyz 2y ago
- crooked-v 2y agoWith the absolute absurdity the residency process, and the focus entirely on new doctors just after that residency, I have to wonder how much of this just corresponds to whoever's lucky enough to be the kind of high-powered mutant who can survive multiple years of 80- to 100-hour week schedules designed by a man who was high on cocaine and morphine 24/7 (seriously, look it up, it's true). There are going to be a lot of people who need an extended sabattical to recover from that before they'll be effective at anything at all, which makes any kind of baseline of test scores really suspect to me.
- derbOac 2y agoYeah I wondered how much of this is accounted for by some general resiliency thing or circumstances during residency or something along those lines.
- SpicyLemonZest 2y agoDoes the difference matter in this context, though? Medicine isn't like other professions where it's no big deal to have some fraction of the workforce be bad at their jobs. I'm not so status-quo-biased that I'd support 100 hour residencies, but I'm skeptical of reform proposals that focus on doctors' working conditions rather than patient outcomes. If some filtering process leads to better patient outcomes, I think we should retain it, even if it's quite stressful for the doctors who have to go through it.
- the_d3f4ult 2y agoFair point. There's some data showing patient outcomes are worse when managed by overworked residents-in-training, but I think you're referring to outcomes post-residency. i.e. Physicians should squeeze as much training as possible into the allotted years. This is reasonable, especially for surgical specialties where procedural reps are a commodity for trainees. I'd be more open to this line of reasoning if physician's salaries had kept pace with inflation over the last 30 years and if if we hadn't tacitly accepted a much, much lower standard of training in the form of DNPs, CRNAs and PAs who are now practicing independently in a lot of regions. You can't demand that people make extraordinary sacrifices without extraordinary compensation. For contrast, most European countries have a much longer post-residency training process that is more humane. Caveat being that students enter medical school directly from high school and don't have student loans. It's also worth pointing out that in the US a LOT of those 100 hours are not spent in direct patient care. They're spent doing chores ('scut') that are not directly tied to patient care. Think: Calling insurance companies for prior authorization for your supervisor or filling out FMLA paperwork for one of your supervisors' patients. As a resident you don't have the ability to say "no" to these tasks.
- PaulKeeble 2y agoMedicine is now absurdly complex, it's far more than a person can possibly learn especially if trying to be up to date with modern research. The more you can memorise correctly and pattern match the better. Many patients are failed in the current system, most not fatally but their lives are damaged and it's not uncommon for more complex diseases to have 90% of sufferers never getting a diagnosis until they die from the disease. Something has to change drastically in how medicine is organised because it's not working in its current iteration as the difficulty goes up and up.
- smgit 2y agoIts the environment that compounds the complexity. Go down the list of Largest companies by revenue in the US and 8 in top 20 are related to "health" - are they running hospitals? are they pharma companies? No. They run pharmacy benefits management, health insurance and drug distribution. The estimate is 4-5 Trillion flows throw these firms. Which is larger than the GDP of India. So this gigantic structure has emerged that doesn't really make too much profit btw (very similar to Amazon Platform Economics) but is layer upon layer upon layer of cash flow passing through middlemen. Drastic change requires new ideas about what do we do about all these middlemen who shape the environment on top of which everything exists.
- ethbr1 2y agoThe biggest problem with the US health system? Complexity. It's impossible to fix overly complex systems. Simplify, simplify, simplify, and then the fixes become trivial. In the US case, that means banning most of the middle-layers.
- zdragnar 2y agoAlas, independent middle layers have long been the US solution to avoiding monopolies. This is the whole reason car manufacturers can't sell directly to consumers, and micro breweries can't sell to consumers except for on-site purchases. Breweries in particular have to sell to distributors, who sell to stores. Banning the middle layers here (absent other changes) just means that the companies that replace their spots in the top 20 will be vertically integrated conglomerates that manufacturer, distribute, prescribe and provide insurance (i.e. payment plans) for pharmaceutical drugs.
- jmcgough 2y agoEdit: Well, that's embarrassing. I hadn't realized that the link is to a new 2024 study on IM board scores and patient outcomes. My post is in regards to a 2023 study on USMLE scores and patient outcomes that was pretty widely discussed. Healthcare worker here. Sheriffofsodium did a great video poking holes at this study: https://youtu.be/JKS9Y-nCnKs?si=VPsUNSoepltbg4Hu https://youtu.be/JKS9Y-nCnKs?si=VPsUNSoepltbg4Hu It's 45 minutes so I don't expect people to watch it, but he makes several important points, including: - This study was performed by USMLE insiders, the only ones with access to this private data. USMLE does not share this data publicly so it's impossible to verify. - As the USMLE makes millions of dollars from these exams, they have a clear conflict of interest. - The differences in patient outcome are AT BEST of marginal clinical significance, which the authors of the study even state in the paper. There is better scientific evidence that female surgeons have better patient outcomes on average: https://pubmed.ncbi.nlm.nih.gov/37647075/ https://pubmed.ncbi.nlm.nih.gov/37647075/
- zameerb1 2y agothe OP is referring to a different study about Board not USMLE
- NotYourLawyer 2y ago[flagged]
- jmcgough 2y agoNo one is hiring based on test scores though? The bar to even get into med school is so insanely high that most people able to get in and become doctors were already upper-middle or high SES. The only point in the entire process where "DEI" matters is feeder programs for underprivileged students, the type of people who can't afford to pay for MCAT tutors etc.
- SpicyLemonZest 2y agoResident physician hiring is strongly based on test scores, specifically the USMLE Step 1. It's true that scores in the board exams the OP discusses aren't super relevant to hiring, though.
- dillydogg 2y agoIt is step 2 now, given that step 1 is now pass/fail. But yes step 2 is the single most important factor in residency match
- evulhotdog 2y agoIt very much depends on the specialty, too.
- SkyPuncher 2y agoI married my wife shortly before she started med school. Scores are basically the entire name of the game. Sure you’re not hired into your attending job based on scores, but med school and residency are largely based on scores.
- NotYourLawyer 2y agoObviously I don’t mean “hire” in the narrow sense. We shouldn’t admit people to medical school based on DEI any more than we should hire them after medical school based on DEI.
- eigenblake 2y agoDoctors aren't machines, they're humans. I have not yet read the full paper, only the article, but I already see something really big and important to look out for. When I read the full thing, the question I'll be asking is "what's the likelihood that the self-esteem of doctors was directly intervened on by the exam taking process itself." How do you control for the loss in confidence that learning of your test performance gives you? How are we certain that learning your score on the board exam doesn't make you more conservative (or riskier) with how you treat patients as a psychological effect?
- the_real_cher 2y agoloss of confidence? lol what?
- AStonesThrow 2y agoYeah but the patient is just a biological machine. This machine can easily be divided into organs and apportioned among specialists. The machine is easily understood by a corpus of research and laboratory experimentation. . Many inputs can be placed in the machine by physicians, and the outputs are known. The biological machines can easily be isolated from environment, or monitored with high technology, and assigned numbers in databases to be processed in data centers. Value is extracted from the biological machines mostly from government and 3rd party sources, so there is no real need to rely on machines having a means or will of their own. There is no compelling reason to treat humans any different from automobiles for the purposes of medicine and medical treatment. In fact humans are less genetically diverse than motor vehicles, and A new model year will always produce a bumper crop of lemons to work on.
- rscho 2y agoThe common misconception of someone with a hard science education. > Many inputs can be placed in the machine by physicians, and the outputs are known. The biological machines can easily be isolated from environment, or monitored with high technology, and assigned numbers in databases to be processed in data centers. We aren't even close to that level of understanding.
- iancmceachern 2y agoI worked for the "Father of Robotic Surgury" and once in a company-wide meting he said "The general public would be pretty happy with the average surgeons results, but they would be horrified by the below average surgeons results". Their goal was to bridge that gap with robotics.
- AStonesThrow 2y agoLike bridging it in this sense? https://en.wikipedia.org/wiki/Brooklyn_Bridge#Culture https://en.wikipedia.org/wiki/Brooklyn_Bridge#Culture
- Calavar 2y agoI'm not a surgeon myself, but when I was in medical school the program director of our local general surgery residency told me that in terms of hand skills 90% of surgeons are more or less average, 10% are masters, and 10% are horrific. (So basically a bell curve with very thin tails.) He also said the correlation between test scores and surgical hand skills was pretty week.
- fsckboy 2y ago>correlation between test scores and surgical hand skills was pretty week even if hand skills is orthogonal to test scores, test scores could still be highly correlated with outcomes
- deleted 2y ago[deleted]
- MichaelZuo 2y agoThat adds up to 110%…
- kinggrowler 2y agoHe's a surgeon, not a mathematician.
- dartharva 2y agoIs this surprising? High exam scores are an indication of discipline and good prioritisation - factors that evidently reflect on the physician's professional performance.
- marcosdumay 2y agoIt's evidence that those exams are doing something right. Whether it's surprising or not, it's up to you. But it's something that should be measured once in a while.
- zdragnar 2y agoWe (in the US) have spent years deprioritizing standardized testing for college admissions on the (public) justification that they don't reflect potential for success. Likewise, there's been an element of testing = racial discrimination in all sorts of fields, such as: https://fairtest.org/article/legal-attack-biased-firefighter-tests/ https://fairtest.org/article/legal-attack-biased-firefighter... And: https://www.theguardian.com/world/2009/jun/29/connecticut-firefighters-supreme-court https://www.theguardian.com/world/2009/jun/29/connecticut-fi... The fact that this study alleges a direct link between exam scores and performance is itself bucking the zeitgeist.
- NoImmatureAdHom 2y agoIt isn't exactly news that doctors with better test scores are better doctors, but this is additional evidence. The article doesn't touch on race, but very deliberately. To anyone on the inside, the silence is deafening. In the U.S. med schools been matriculating many unqualified "underrepresented minority" (black, hispanic, native American, Hawaiian) medical students for a long time. This is unfair to patients and doctors, especially competent brown doctors, because it is now the case that you get a very strong signal about how how good a doctor is simply by the color of his or her skin. Which is messed up. AAMC has the data (https://www.aamc.org/data-reports/students-residents/data/facts-applicants-and-matriculants https://www.aamc.org/data-reports/students-residents/data/fa... , table A-18). This is after the 2023 Supreme Court decision, so the spreads are a little wider in e.g. 2022 data. MCAT scores range from a minimum of 472 to a max of 528, which is stupid and a deliberate tactic to make the differences between groups seem small. Subtracting 472 from each average score, 2024 average MCAT scores look like this for matriculants: 41.9: Asian 40.2: White 36.9: Hawaiian 34.4: Black 33.9: Hispanic 31.3: American Indian These are very large differences which you can absolutely expect to show up in doctor performance. Everyone has to pass the same boards during / after med school, but that's just going to cut out some of the worst. Among those who pass, the unqualified minority students who were admitted to med school because of their skin color will still be concentrated at the bottom of the distribution. Do you know what they call the guy who finished last in his med school class? "Doctor".
- dqv 2y ago> The article doesn't touch on race, but very deliberately. To anyone on the inside, the silence is deafening. ??? The NPI registry doesn't indicate the race of registered providers, only their sex. Really bizarre to call a limitation of the available data "deliberate".
- NoImmatureAdHom 2y agoIt's possible to put together multiple data sources. There are certain things everyone reading this will already know. It's like reporting "educational attainment" rather than g or IQ in studies...everyone knows what it implies, you just can't say it. Anyway: 1) Board scores are strongly linked to patient outcomes (this paper) 2) We already know test scores vary strongly with observable characteristics like race 3) It's a very safe bet that board scores vary with race in the same way that MCAT scores vary with race Therefore, 4) We can have a very good idea of how good a doctor is based on observable characteristics like race Which is a thing the article immediately, obviously, and loudly implies but of course couldn't say for fear of censorship, losing jobs, etc.
- doctorpangloss 2y agoA chatbot can also score very highly on these tests. What do you think the survival rate of ChatGPT’s patients will be?
- rowanG077 2y agoProbably pretty high in diagnosis at least?
- rscho 2y agoMost patients are unable to communicate their symptoms accurately enough. Which is why you need to see them in person, talk with them, and examine them. Not saying a robot couldn't perform, but certainly not a simple chatbot. Despite what some papers say.
- rowanG077 2y agoI wonder how much this is simply. Smart people do better on tests. Smart people make better doctors.
- rscho 2y agoI don't think smart people make better docs. I'm USMLE 90+ percentile, and not particularly clever. It is however, important to be clever enough to understand what you read. Good docs are humble, meticulous and knowledgeable. Stellar docs are excellent communicators.
- rowanG077 2y agoThe study at least proves better test taking strongly predicts outcomes, test scores are correlated with intelligence as countless studies prove. It may be the case that some non-clever people get high test scores. That doesn't dismiss the general conclusion.
- rscho 2y agoNo, no contradiction. I said: high USMLE score != smart. GP said: good doctor = smart. Study says: good doc = high USMLE score. As I also said: good doc = understand what you read.
- rscho 2y agoThe article says 'board exam' which is quite different from USMLE. So, it's established: I can't read, and I'm not especially clever. It all checks out ! :-)
- deleted 2y ago[deleted]
- gadders 2y agoReminds me of the old joke: What do you call the person that came bottom of their class in Med School? Doctor.
- bufferoverflow 2y ago[flagged]
- thijson 2y ago[flagged]
- ForTheKidz 2y ago[flagged]
- apognwsi 2y agoi did not read the study. an obvious confounding factor is that doctors with better board scores are hired into better hospitals with better patient populations, and thus better outcomes.
- deleted 2y ago[deleted]
- dh5 2y agoThis was controlled for as stated in the article: > The researchers compared outcomes for patients within the same hospitals who were cared for by doctors with different exam scores. This allowed the researchers to eliminate, or at least minimize, the effect of differences in patient populations, hospital resources, and other variations that might influence the odds of patient death or readmission, independent of a doctor’s performance.
- HelloMcFly 2y agoThis is also pretty much the easiest thing the factor out through mixed effects modeling (among other methods if required). But your statement that higher scoring physicians go to places with healthier patient populations is not correct across all disciplines. Often it can be the opposite: the best physicians go to the major hospitals (usually but not always university affiliated) located in major population centers that draw in the sickest/worst/rarest cases from the surrounding geography.
- deleted 2y ago[deleted]
- palijer 2y agoThe people who ran the study also thought of this and controlled for it.
- fn-mote 2y ago> Gray, Lipner, McDonald, and Vandergrift reported that they are employees of ABIM [American Board of Internal Medicine]. Landon reported receiving consulting fees from ABIM for ongoing work during the conduct of the study. A study that shows the board test is effective, sponsored by the board? I might be reassured by more detailed statistics about the analysis. Even top 25% vs bottom 25% - how much actual variation in score are we talking about? What is the probability that someone scoring in the top 25% is actually in the top/bottom 25%? We imagine a big gap but that’s not necessarily true. Consider exam scores of 85 90 90 95…
- gunian 2y ago[flagged]
- uslic001 2y agoDEI policies in surgical residencies are leading to below average surgical skills. Surgical residencies used to be a pyramid based system with only the best being chosen based on merit. That changed over 20 years ago and we are now paying the price.