13 ms·

As a physician, I often think about how we lack truely objective assessment of patient outcomes (either in the context of evaluating physician competence or, pr

As a physician, I often think about how we lack truely objective assessment of patient outcomes (either in the context of evaluating physician competence or, probably more importantly, assessing and improving upon clinical practises). I would really appreciate insight on how this could be achieved. There are several issues which are particularly vexing: - The distinct lack of verifiable, objective markers of physician competence. - Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. - Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent. - This is made worse by the hierarchical nature of clinical medicine and deference to seniority and title. - Medicine is parochial so clinical practises for the same disorder vary tremendously. You might be treated a dozen different ways for the same disorder and presentation depending on the facility and especially on the specialty that ends up treating you. - Outcomes are not necessarily determined by clinician ability. There are several other factors at play: the pre- and post-care (such as work-up by ancillary staff or ICU care after a surgery), the cohesiveness of the facility and its efficiencies (or lack thereof), availability and preferences for resources such as medical devices, drugs and hospital equipment which may be largely out of the hands of the physician.

lowglow 9y ago
No lie this sounds like a great fit for a reputation based system of openly evaluated experiments on an immutable, open, decentralized ledger.
- neuro_imager 9y ago
  By "reputation based", do you mean patient report outcomes or something else? Could you expand on "openly evaluated experiments", maybe with an example?
  - lowglow 9y ago
    Give participants (patients, doctors) access to subscribe to health data or symptoms (input) and create a consensus of possible treatment plans, iterate through those plans ordered by some metric (reputation, time, cost, availability). When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct. You could even use this as human-in-the-loop machine learning model training for open models. If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/ https://synapse.ai/ Happy to reach out and chat more.
    - neuro_imager 9y ago
      > Give participants (patients, doctors) access to subscribe to health data or symptoms (input) and create a consensus of possible treatment plans, iterate through those plans ordered by some metric (reputation, time, cost, availability). I'm still not sure I understand what you mean. Could you give an example? > When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct. It's not that simple. Most patients are not either 'cured' or 'not cured'. There are multiple possible outcomes at each stage of diagnosis and management. > You could even use this as human-in-the-loop machine learning model training for open models. Doubtful. Current machine learning algorithms are painfully inept when given clinical data (outside of very limited use cases). I'd love to be proven wrong though. > If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/ https://synapse.ai/ Happy to reach out and chat more. Sure, email?
      - lowglow 9y ago
        dan [at] synapse [dot] ai
    - nradov 9y ago
      This is ridiculous and completely unrealistic. Most conditions are never really cured and diagnoses are seldom "proven correct". We only have varying levels of improvement and confidence. Patients generally lack the skills to confirm anything themselves, especially because most of them don't understand causality. Did my condition improve because of my physician's treatment, or in spite of it?
epmaybe 9y ago
I have this same conversation with my non-healthcare friends almost yearly. You touched on everything I saw, but I'm not sure I understand what you mean by: > Reported outcomes are often erroneous or frankly fraudulent Could you elaborate?
- neuro_imager 9y ago
  Outcomes (including complication rates) are typically recorded based on history, clinical examination and lab and imaging results. - History and physical exam are extremely subjective. - Medical notes and records can be massaged to sound better than reality (The patient didn't have a stroke after their operation. They had "minor lower limb weakness which we anticipate will improve over the next few weeks") or attributed to all sorts of things (patient's cognitive function hasn't worsened, they have some "fatigue" following surgery). - People can be very selective about which studies are ordered. - Junior and ancillary staff are reluctant to report poor outcomes about established senior people. I'll give a real-life example: Professor X is a "world-renowned expert" on neurosurgery at an academic mecca. He claims a "1-2%" complication rate on aneurysm surgeries in the clinic and at any conferences. His residents know he performs outdated surgeries with spectacularly bad outcomes but no one would be willing to talk about this openly unless they want to torpedo their career prospects.
  - epmaybe 9y ago
    Probably going to sound naive, as I'm not at this stage of my training yet, but if not saying anything led to patients dying or suffering, I don't think it's worth whatever career prospects were potentially attainable. It's ethically horrendous to consider that would be my life until I too held a superiority complex.
    - Spooky23 9y ago
      Never underestimate the power of a mortgage to make smart people bend over backwards to not notice something. Someone close to me, a very experienced nurse, reported some misconduct on the part of a senior physician. This was both required by law and an ethical duty. She essentially got blackballed from that hospital as a result.
sandworm101 9y ago
Don't focus on the complicated ones (the old people) where things like 5-year survival rates are so muddled up in overlapping conditions and inevitable decline. I once presented at an ER at 4am as a 23yo with every symptom of appendicitis. They shaved me, but then the ultrasound tech arrived for work early. It turned out to be something else and I didn't need surgery. They admitted that 20% of diagnosed appendicitis turn out to be something else once they cut in, many still needing surgery but some no. Ten years later I related this story to another surgeon and he was downright angry at 20%. His hospital had that stat down to 10% (in men) and wanted it down to 5%. Those stats, basic conditions with standard treatments but perhaps a complicated diagnosis, are the better starting point. I'd bet good money that the hospital at 5% for appendicitis diagnosis is also doing better at the more complex stuff too.
- neuro_imager 9y ago
  In fairness, misdiagnosis of appendicitis is largely down to utilisation of technology. Most modern institutions utilise imaging (ultrasound/CT) and this results in a high specificity for appendicitis (especially CT, which doesn't suffer from the user variability that ultrasound does). Old school surgeons will insist that they can reliably make the diagnosis without imaging (and that's probably where you get a 20% false positive from). Some institutions will push this notion too - either to reduce imaging expenditure or increase surgical turnover (sad but true).
  - brohee 9y ago
    When is knowingly not following best practices actually malpractice? I wonder if ambulance chasers don't have an actually useful role to play in that fight...
    - neuro_imager 9y ago
      The first part of your comment is a great question. Its extremely hard to definitively define best practises. You'd be surprised how many practitioners still argue for what look like clearly outdated ways of doing things. There's also the argument that in some people's hands an outdated way of doing things is safer than a newer (better?) way of doing things in their hands. eg. A surgeon who has been practising for 30 years but only recently learned to use a laparoscope. I doubt ambulance chasers do anything other than muddy the waters and enrich themselves at the healthcare systems expense.
MarkMc 9y ago
> I often think about how we lack truely objective assessment of patient outcomes We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. > - Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. > - Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent. I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.
- neuro_imager 9y ago
  > We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. > This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. Agree. > I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not. That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).
RachelF 9y ago
I agree. As an engineer or programmer, you tend to get immediate and pretty conclusive feedback about whether the system you are building works. You can iterate and fix your mistakes. My experience with the world of medicine is that it is way less scientific and evidence based. This is because it can be more complex, but it does lead to doctors being able to ignore the fact that their treatments don't work. Bad doctors can bury their mistakes and blame other factors, whereas bad engineers quickly get found out.
- taurath 9y ago
  I’m not so sure about that. How many programmers that are senior build up a system only to have it nearly fall to pieces after they leave?
  - lmm 9y ago
    Software maintainability is hard to measure and test, and programmers who are bad at writing maintainable software can hide in the noise. But we do at least have some level of objective accountability in terms of "can the program do the thing?" Doctors largely don't even have that much.
- amrrs 9y ago
  I think that holds true for Politicians also especially since there is no standard framework to see whether what they have done/built had worked.
BurningFrog 9y ago
I doubt the lack of objective assessment an unfortunate accident. More likely things are this way because the relevant people and institutions don't want their performance measured, and they have the power to keep things this way.
- Gatsky 9y ago
  Not sure what your relevant experience is, but in my experience large healthcare institutions are constantly measuring their performance. The problem is comparing it in a useful way across different institutions.
  - Fomite 9y ago
    Exactly this. My whole job is based on the torrent of data large healthcare institutions collect. The problem is not that people aren't writing things down - it's what to make of it.
  - BurningFrog 9y ago
    I have zero experience from the industry, but I still want to defend my point: "Comparing it in a useful way across different institutions" seems like a BIG deal! Institutions revealed to be inferior would have to improve close down. Those proven to do great work would be richly rewarded, but at all levels there'd be a new level of scrutiny. This would be of huge benefit for patients, save tons of lives etc, but also put a lot of strain on doctors and hospitals, and I can see how people with power in those fields don't want this to happen and thus don't make it happen. It's also entirely possible I babble too much about things I know nothing about here. If so, I'd appreciate learning about it!
indubitable 9y ago
I'm curious about your thoughts on a hypothetical. Imagine we have a person who, from childhood, is given daily observation, care, assessment and treatment by the truly 'best' doctors available -- assuming we were able to develop a means to measure that. But aside from that he would live his life in a relatively average upper class way in terms of diet/nutrition, activity, and so on. What would you expect to change? How would you expect his life expectancy to vary against the general population? What if we take general population to mean the general population of Japan? This is relevant as I think the views here help determine where the focus in healthcare ought be.
- neuro_imager 9y ago
  Sure, optimising environmental factors would undoubtably be helpful and is a laudable goal.
Dwolb 9y ago
I did some market research back in the day and there was a lot negative opinions on even attempting to measure performance based on patient outcomes because “each case is unique”. To me there were two things at play: possibly the sample size of number of patients doctors see with the same problem might be too small and there was genuinely a lot of fear that the doctor’s skill could be boiled down to protocols and statistics. I think the biggest hurdle toward improving patient outcomes will be alleviating the fear and then after that figuring out how to give better context to the data.
- tinokid 9y ago
  Their fear (for their own professional standing and income stream) is justified, but the objections to measuring performance are special pleading. 1) "Each patient is unique" just like "each student is unique"...but we still give standardized tests (to the grumbling of below-average teachers and administrators) 2) If the sample size is too small because the disease is rare, there may be a genuine measurement problem. If the doctor or hospital's sample size is too small but the illness is common, then they should be referring those patients to providers/facilities with the requisite experience levels.
  - zaarn 9y ago
    If a common disease has not sufficient experience in a hospital I'd argue they should pull in experts from elsewhere and try to gain the experience necessary, otherwise you just end up exporting the problem.
  - Fordrus 9y ago
    There is substantial argument that our focus on standardized testing is literally ruining the advantages traditionally associated with a Western Education. But then, perhaps I would be a "below-average" teacher, and my opinion might therefore similarly be discarded as worthless. We'll never know for certain, because I don't make a habit of boarding sinking ships - especially when I'm slated to receive blame for their sinking after boarding them. (And I did at one point very much want to enter the field of secondary school education.) (Slight modification: I won't board a sinking ship and take blame for it's sinking after the fact without some substantial advantage, such as excellent remuneration, being offered as well - teaching offers no such advantage, except perhaps self-actualization, and I can neither eat nor sleep in that.)
packeted 9y ago
This is exactly the problem we have been working on at the startup I founded, Outcomes.com. As a physician and training surgeon before moving out to the Bay Area, I was amazed that we had very poor visibility and almost no data on how our patients did after treatments beside very crude measures, like whether the patient had major complications or died. Our focus has been on capturing patient-reported outcomes - that is the outcome of care as experienced by the patient, measured using objective and validated surveys that are often specific to the condition or treatment. There is now a movement among payers and Medicare to incorporate these kind of patient-centered outcome measures in to reimbursement, although change is admittedly slow. I'd love to talk to people wanting to make an impact in this field, my email address is francis at outcomes.com
- nradov 9y ago
  Perhaps I'm missing something, but how could patient-reported outcomes possibly be objective? Patients are notoriously bad at assessing their own conditions and establishing causality for changes. We know that patients tend to give positive survey results if the physician was "nice" and if he wrote a prescription, regardless of actual quality of care.
  - packeted 9y ago
    Most patient-reported outcome measures are designed to provide an objective assessment of a patient's health status, for example in urology many of the surveys ask specific questions about urinary symptoms, eg. how many times did you have to get up in the night to urinate, or in orthopedics whether you had difficulty performing specific tasks related to your joint. Some aspects are always going to be subjective (eg. impact on quality of life or pain) and IMHO that's OK and we should absolutely attempt to measure them, not least because that information could help inform the treatment itself. Also, by measuring the changes in response over time for a patient, you can attempt to control for individual biases. I agree that patient satisfaction surveys (in the UK, categorized as patient reported experience measures) can be very prone to bias and while important, are not necessarily correlated with outcomes.
deleted 9y ago
[deleted]
throwawa 9y ago
> Medicine is parochial so clinical practises for the same disorder vary tremendously Oh my, I've been suffering for back/neck pain for years. I will randomly get flare-up (usually when it gets cold or when I do sport) and things will get worse and worse. I've been going to different PT/phisio/osteopath for the last 10 years and nothing never changed. My last doctor told me an X-ray wouldn't do much and sent me to another PT... Now I'm considering visiting a chiropractor eventhough reddit has told me not to trust them... But I mostly just endure the pain every day.
- Joakal 9y ago
  I don't know what you're going through, but for a friend with sciatica (or what the consensus was!), they went through several doctors, health foods, drugs and chiro. The Chiro helped a little but the pain largely remained. They tried fitness (mostly running) and it cleared most of the pain after two weeks of exercise. I understand the pain, keep trying and try whatever's best for you.
- neuro_imager 9y ago
  There's no easy answer for back/neck pain. I will tell you that yoga/pilates/other non-ballistic, moderate resistance exercise that targets stabiliser muscles helps many people. I would avoid surgery at all costs unless you have cord compression, a focal lesion or a fracture. I'd be very wary of chiropractors - I've seen several patients with vertebral artery dissections (with incapacitating brain stem strokes!) because of neck manipulation. Of course, I have a selection bias but even a small risk for this seams like too much.
- dennis_jeeves 9y ago
  Unsolicited suggestion: try https://www.regenexx.com/ https://www.regenexx.com/ I have following Chris Centeno, M.D. for some years. As far as I know he tries all/some approaches including stem cells, chiropractors, exercises etc. If you do end you being treated by them, let me know the outcome - good or bad.
ekianjo 9y ago
> Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. Every individual is unique and yet we are able to segment folks using multiple variables and draw predictions for a bunch of things across businesses - it's just like saying "every shopper online is unique", which is true, yet we still find ways to influence them of target them in somewhat meaningful ways. You should be able to do the same with patients, by clustering them in larger groups that make sense (for cancer many trials refer to specific mutations nowadays to define whether a patient is expected to get efficacy from a specific drug). The only question in the end is whether or not you have enough sample size for each doctor. Maybe not. Then it would make sense to at least draw comparisons between groups of specialists across multiple hospitals, at least. Even if metrics are imperfect to begin with, there is no excuse not to start at least, and improve over time.
ska 9y ago
Do you feel the various evidence based medicine initiatives can not addressing these issues effectively?
snarf21 9y ago
One of the hard truths you have here is outcomes != ability. Treatment compliance is almost always a much bigger factor. Compliance can also be greatly affected by the patient experience and relationship with the care provider. I think the thing we really need is a better patient experience model that maximizes trust and information transfer and helps ensure compliance. (Disclaimer: I led a team building one of the top patient experience platforms for several years)
- DontGiveTwoFlux 9y ago
  That is probably the biggest takeaway for me from the article. The insufferable Dr. Warwick gets his patients to comply. That is one of his cutting edge skills.
  - bshanks 9y ago
    Another possibility is that Warwick's manner might be acting as a filter for patients who are already determined to go the extra mile for their health, and causing any other patients to leave. Perhaps any doctor would have better stats if they got to have the same pool of patients as those who remained with Warwick, without having other less motivated patients to drag them down. We can't know if this is the case until we collect the data on patients who left Warwick's care.
FLUX-YOU 9y ago
>The distinct lack of verifiable, objective markers of physician competence. This is the same for software, and is equally annoying (especially as a self-taught Dev), but physicians at least have the assessment of people who have had to endure intellectually difficult course work. You also have several assessments of direct physician ability (boards) and possibly other things. As for software, I can basically follow tutorials, watch some videos, maybe do a few small projects, and then be able to convince someone to actually give me a job. That's my only measurement -- whether someone will pay for my skills. You guys are much closer to objectively verifying someone's ability, so hopefully that doesn't seem as vexing for you.
k2enemy 9y ago
There is another danger related to your second point. If you can't completely control for the risk of a patient, then you create an incentive for doctors to turn away high risk patients. This effect has been observed in real life: https://www.kellogg.northwestern.edu/faculty/satterthwaite/Research/2003-0520%20Dranove%20et%20al%20re%20Report%20Cards%20(JPE).pdf https://www.kellogg.northwestern.edu/faculty/satterthwaite/R...
tinokid 9y ago
Randomize patients to providers with the same certification level in delineated service areas. It will require formalizing the informal process you describe (whereby "cases with the highest levels of difficulty are often treated by the most experienced people"), most likely by creating additional certification levels.

lowglow 9y ago

No lie this sounds like a great fit for a reputation based system of openly evaluated experiments on an immutable, open, decentralized ledger.

neuro_imager 9y ago
By "reputation based", do you mean patient report outcomes or something else? Could you expand on "openly evaluated experiments", maybe with an example?
- lowglow 9y ago
  Give participants (patients, doctors) access to subscribe to health data or symptoms (input) and create a consensus of possible treatment plans, iterate through those plans ordered by some metric (reputation, time, cost, availability). When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct. You could even use this as human-in-the-loop machine learning model training for open models. If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/ https://synapse.ai/ Happy to reach out and chat more.
  - neuro_imager 9y ago
    > Give participants (patients, doctors) access to subscribe to health data or symptoms (input) and create a consensus of possible treatment plans, iterate through those plans ordered by some metric (reputation, time, cost, availability). I'm still not sure I understand what you mean. Could you give an example? > When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct. It's not that simple. Most patients are not either 'cured' or 'not cured'. There are multiple possible outcomes at each stage of diagnosis and management. > You could even use this as human-in-the-loop machine learning model training for open models. Doubtful. Current machine learning algorithms are painfully inept when given clinical data (outside of very limited use cases). I'd love to be proven wrong though. > If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/ https://synapse.ai/ Happy to reach out and chat more. Sure, email?
    - lowglow 9y ago
      dan [at] synapse [dot] ai
  - nradov 9y ago
    This is ridiculous and completely unrealistic. Most conditions are never really cured and diagnoses are seldom "proven correct". We only have varying levels of improvement and confidence. Patients generally lack the skills to confirm anything themselves, especially because most of them don't understand causality. Did my condition improve because of my physician's treatment, or in spite of it?

epmaybe 9y ago

I have this same conversation with my non-healthcare friends almost yearly. You touched on everything I saw, but I'm not sure I understand what you mean by: > Reported outcomes are often erroneous or frankly fraudulent Could you elaborate?

neuro_imager 9y ago
Outcomes (including complication rates) are typically recorded based on history, clinical examination and lab and imaging results. - History and physical exam are extremely subjective. - Medical notes and records can be massaged to sound better than reality (The patient didn't have a stroke after their operation. They had "minor lower limb weakness which we anticipate will improve over the next few weeks") or attributed to all sorts of things (patient's cognitive function hasn't worsened, they have some "fatigue" following surgery). - People can be very selective about which studies are ordered. - Junior and ancillary staff are reluctant to report poor outcomes about established senior people. I'll give a real-life example: Professor X is a "world-renowned expert" on neurosurgery at an academic mecca. He claims a "1-2%" complication rate on aneurysm surgeries in the clinic and at any conferences. His residents know he performs outdated surgeries with spectacularly bad outcomes but no one would be willing to talk about this openly unless they want to torpedo their career prospects.
- epmaybe 9y ago
  Probably going to sound naive, as I'm not at this stage of my training yet, but if not saying anything led to patients dying or suffering, I don't think it's worth whatever career prospects were potentially attainable. It's ethically horrendous to consider that would be my life until I too held a superiority complex.
  - Spooky23 9y ago
    Never underestimate the power of a mortgage to make smart people bend over backwards to not notice something. Someone close to me, a very experienced nurse, reported some misconduct on the part of a senior physician. This was both required by law and an ethical duty. She essentially got blackballed from that hospital as a result.

sandworm101 9y ago

Don't focus on the complicated ones (the old people) where things like 5-year survival rates are so muddled up in overlapping conditions and inevitable decline. I once presented at an ER at 4am as a 23yo with every symptom of appendicitis. They shaved me, but then the ultrasound tech arrived for work early. It turned out to be something else and I didn't need surgery. They admitted that 20% of diagnosed appendicitis turn out to be something else once they cut in, many still needing surgery but some no. Ten years later I related this story to another surgeon and he was downright angry at 20%. His hospital had that stat down to 10% (in men) and wanted it down to 5%. Those stats, basic conditions with standard treatments but perhaps a complicated diagnosis, are the better starting point. I'd bet good money that the hospital at 5% for appendicitis diagnosis is also doing better at the more complex stuff too.

neuro_imager 9y ago
In fairness, misdiagnosis of appendicitis is largely down to utilisation of technology. Most modern institutions utilise imaging (ultrasound/CT) and this results in a high specificity for appendicitis (especially CT, which doesn't suffer from the user variability that ultrasound does). Old school surgeons will insist that they can reliably make the diagnosis without imaging (and that's probably where you get a 20% false positive from). Some institutions will push this notion too - either to reduce imaging expenditure or increase surgical turnover (sad but true).
- brohee 9y ago
  When is knowingly not following best practices actually malpractice? I wonder if ambulance chasers don't have an actually useful role to play in that fight...
  - neuro_imager 9y ago
    The first part of your comment is a great question. Its extremely hard to definitively define best practises. You'd be surprised how many practitioners still argue for what look like clearly outdated ways of doing things. There's also the argument that in some people's hands an outdated way of doing things is safer than a newer (better?) way of doing things in their hands. eg. A surgeon who has been practising for 30 years but only recently learned to use a laparoscope. I doubt ambulance chasers do anything other than muddy the waters and enrich themselves at the healthcare systems expense.

MarkMc 9y ago

> I often think about how we lack truely objective assessment of patient outcomes We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. > - Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. > - Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent. I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.

neuro_imager 9y ago
> We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon. I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence. > This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate. Agree. > I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not. That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).

RachelF 9y ago

I agree. As an engineer or programmer, you tend to get immediate and pretty conclusive feedback about whether the system you are building works. You can iterate and fix your mistakes. My experience with the world of medicine is that it is way less scientific and evidence based. This is because it can be more complex, but it does lead to doctors being able to ignore the fact that their treatments don't work. Bad doctors can bury their mistakes and blame other factors, whereas bad engineers quickly get found out.

taurath 9y ago
I’m not so sure about that. How many programmers that are senior build up a system only to have it nearly fall to pieces after they leave?
- lmm 9y ago
  Software maintainability is hard to measure and test, and programmers who are bad at writing maintainable software can hide in the noise. But we do at least have some level of objective accountability in terms of "can the program do the thing?" Doctors largely don't even have that much.
amrrs 9y ago
I think that holds true for Politicians also especially since there is no standard framework to see whether what they have done/built had worked.

BurningFrog 9y ago

I doubt the lack of objective assessment an unfortunate accident. More likely things are this way because the relevant people and institutions don't want their performance measured, and they have the power to keep things this way.

Gatsky 9y ago
Not sure what your relevant experience is, but in my experience large healthcare institutions are constantly measuring their performance. The problem is comparing it in a useful way across different institutions.
- Fomite 9y ago
  Exactly this. My whole job is based on the torrent of data large healthcare institutions collect. The problem is not that people aren't writing things down - it's what to make of it.
- BurningFrog 9y ago
  I have zero experience from the industry, but I still want to defend my point: "Comparing it in a useful way across different institutions" seems like a BIG deal! Institutions revealed to be inferior would have to improve close down. Those proven to do great work would be richly rewarded, but at all levels there'd be a new level of scrutiny. This would be of huge benefit for patients, save tons of lives etc, but also put a lot of strain on doctors and hospitals, and I can see how people with power in those fields don't want this to happen and thus don't make it happen. It's also entirely possible I babble too much about things I know nothing about here. If so, I'd appreciate learning about it!

indubitable 9y ago

I'm curious about your thoughts on a hypothetical. Imagine we have a person who, from childhood, is given daily observation, care, assessment and treatment by the truly 'best' doctors available -- assuming we were able to develop a means to measure that. But aside from that he would live his life in a relatively average upper class way in terms of diet/nutrition, activity, and so on. What would you expect to change? How would you expect his life expectancy to vary against the general population? What if we take general population to mean the general population of Japan? This is relevant as I think the views here help determine where the focus in healthcare ought be.

neuro_imager 9y ago
Sure, optimising environmental factors would undoubtably be helpful and is a laudable goal.

Dwolb 9y ago

I did some market research back in the day and there was a lot negative opinions on even attempting to measure performance based on patient outcomes because “each case is unique”. To me there were two things at play: possibly the sample size of number of patients doctors see with the same problem might be too small and there was genuinely a lot of fear that the doctor’s skill could be boiled down to protocols and statistics. I think the biggest hurdle toward improving patient outcomes will be alleviating the fear and then after that figuring out how to give better context to the data.

tinokid 9y ago
Their fear (for their own professional standing and income stream) is justified, but the objections to measuring performance are special pleading. 1) "Each patient is unique" just like "each student is unique"...but we still give standardized tests (to the grumbling of below-average teachers and administrators) 2) If the sample size is too small because the disease is rare, there may be a genuine measurement problem. If the doctor or hospital's sample size is too small but the illness is common, then they should be referring those patients to providers/facilities with the requisite experience levels.
- zaarn 9y ago
  If a common disease has not sufficient experience in a hospital I'd argue they should pull in experts from elsewhere and try to gain the experience necessary, otherwise you just end up exporting the problem.
- Fordrus 9y ago
  There is substantial argument that our focus on standardized testing is literally ruining the advantages traditionally associated with a Western Education. But then, perhaps I would be a "below-average" teacher, and my opinion might therefore similarly be discarded as worthless. We'll never know for certain, because I don't make a habit of boarding sinking ships - especially when I'm slated to receive blame for their sinking after boarding them. (And I did at one point very much want to enter the field of secondary school education.) (Slight modification: I won't board a sinking ship and take blame for it's sinking after the fact without some substantial advantage, such as excellent remuneration, being offered as well - teaching offers no such advantage, except perhaps self-actualization, and I can neither eat nor sleep in that.)

packeted 9y ago

This is exactly the problem we have been working on at the startup I founded, Outcomes.com. As a physician and training surgeon before moving out to the Bay Area, I was amazed that we had very poor visibility and almost no data on how our patients did after treatments beside very crude measures, like whether the patient had major complications or died. Our focus has been on capturing patient-reported outcomes - that is the outcome of care as experienced by the patient, measured using objective and validated surveys that are often specific to the condition or treatment. There is now a movement among payers and Medicare to incorporate these kind of patient-centered outcome measures in to reimbursement, although change is admittedly slow. I'd love to talk to people wanting to make an impact in this field, my email address is francis at outcomes.com

nradov 9y ago
Perhaps I'm missing something, but how could patient-reported outcomes possibly be objective? Patients are notoriously bad at assessing their own conditions and establishing causality for changes. We know that patients tend to give positive survey results if the physician was "nice" and if he wrote a prescription, regardless of actual quality of care.
- packeted 9y ago
  Most patient-reported outcome measures are designed to provide an objective assessment of a patient's health status, for example in urology many of the surveys ask specific questions about urinary symptoms, eg. how many times did you have to get up in the night to urinate, or in orthopedics whether you had difficulty performing specific tasks related to your joint. Some aspects are always going to be subjective (eg. impact on quality of life or pain) and IMHO that's OK and we should absolutely attempt to measure them, not least because that information could help inform the treatment itself. Also, by measuring the changes in response over time for a patient, you can attempt to control for individual biases. I agree that patient satisfaction surveys (in the UK, categorized as patient reported experience measures) can be very prone to bias and while important, are not necessarily correlated with outcomes.

deleted 9y ago

[deleted]

throwawa 9y ago

> Medicine is parochial so clinical practises for the same disorder vary tremendously Oh my, I've been suffering for back/neck pain for years. I will randomly get flare-up (usually when it gets cold or when I do sport) and things will get worse and worse. I've been going to different PT/phisio/osteopath for the last 10 years and nothing never changed. My last doctor told me an X-ray wouldn't do much and sent me to another PT... Now I'm considering visiting a chiropractor eventhough reddit has told me not to trust them... But I mostly just endure the pain every day.

Joakal 9y ago
I don't know what you're going through, but for a friend with sciatica (or what the consensus was!), they went through several doctors, health foods, drugs and chiro. The Chiro helped a little but the pain largely remained. They tried fitness (mostly running) and it cleared most of the pain after two weeks of exercise. I understand the pain, keep trying and try whatever's best for you.
neuro_imager 9y ago
There's no easy answer for back/neck pain. I will tell you that yoga/pilates/other non-ballistic, moderate resistance exercise that targets stabiliser muscles helps many people. I would avoid surgery at all costs unless you have cord compression, a focal lesion or a fracture. I'd be very wary of chiropractors - I've seen several patients with vertebral artery dissections (with incapacitating brain stem strokes!) because of neck manipulation. Of course, I have a selection bias but even a small risk for this seams like too much.
dennis_jeeves 9y ago
Unsolicited suggestion: try https://www.regenexx.com/ https://www.regenexx.com/ I have following Chris Centeno, M.D. for some years. As far as I know he tries all/some approaches including stem cells, chiropractors, exercises etc. If you do end you being treated by them, let me know the outcome - good or bad.

ekianjo 9y ago

> Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians. Every individual is unique and yet we are able to segment folks using multiple variables and draw predictions for a bunch of things across businesses - it's just like saying "every shopper online is unique", which is true, yet we still find ways to influence them of target them in somewhat meaningful ways. You should be able to do the same with patients, by clustering them in larger groups that make sense (for cancer many trials refer to specific mutations nowadays to define whether a patient is expected to get efficacy from a specific drug). The only question in the end is whether or not you have enough sample size for each doctor. Maybe not. Then it would make sense to at least draw comparisons between groups of specialists across multiple hospitals, at least. Even if metrics are imperfect to begin with, there is no excuse not to start at least, and improve over time.

ska 9y ago

Do you feel the various evidence based medicine initiatives can not addressing these issues effectively?

snarf21 9y ago

One of the hard truths you have here is outcomes != ability. Treatment compliance is almost always a much bigger factor. Compliance can also be greatly affected by the patient experience and relationship with the care provider. I think the thing we really need is a better patient experience model that maximizes trust and information transfer and helps ensure compliance. (Disclaimer: I led a team building one of the top patient experience platforms for several years)

DontGiveTwoFlux 9y ago
That is probably the biggest takeaway for me from the article. The insufferable Dr. Warwick gets his patients to comply. That is one of his cutting edge skills.
- bshanks 9y ago
  Another possibility is that Warwick's manner might be acting as a filter for patients who are already determined to go the extra mile for their health, and causing any other patients to leave. Perhaps any doctor would have better stats if they got to have the same pool of patients as those who remained with Warwick, without having other less motivated patients to drag them down. We can't know if this is the case until we collect the data on patients who left Warwick's care.

FLUX-YOU 9y ago

>The distinct lack of verifiable, objective markers of physician competence. This is the same for software, and is equally annoying (especially as a self-taught Dev), but physicians at least have the assessment of people who have had to endure intellectually difficult course work. You also have several assessments of direct physician ability (boards) and possibly other things. As for software, I can basically follow tutorials, watch some videos, maybe do a few small projects, and then be able to convince someone to actually give me a job. That's my only measurement -- whether someone will pay for my skills. You guys are much closer to objectively verifying someone's ability, so hopefully that doesn't seem as vexing for you.

k2enemy 9y ago

There is another danger related to your second point. If you can't completely control for the risk of a patient, then you create an incentive for doctors to turn away high risk patients. This effect has been observed in real life: https://www.kellogg.northwestern.edu/faculty/satterthwaite/Research/2003-0520%20Dranove%20et%20al%20re%20Report%20Cards%20(JPE).pdf https://www.kellogg.northwestern.edu/faculty/satterthwaite/R...

tinokid 9y ago

Randomize patients to providers with the same certification level in delineated service areas. It will require formalizing the informal process you describe (whereby "cases with the highest levels of difficulty are often treated by the most experienced people"), most likely by creating additional certification levels.