19 ms·

IBM's Watson will be used to make decisions about cancer care in 14 hospitals

139 points by Brakenshire 12y ago

gkoberger 12y ago
I wonder if the Jeopardy appearances helped Watson (by getting the name out) or hindered it (by making people think it's just a parlor trick). Either way, this is seriously fascinating: I'm excited to see what other applications it has.
- duaneb 12y ago
  People aren't stupid, and if it was a parlour trick it was an exceptionally well-executed on. People, even those who don't have a grasp on the computational complexity of language and reasoning, seem legitimately impressed by the performance.
  - sgt101 12y ago
    Yes, and it was a real research collaboration - a brilliant insight - to see that the Jeopardy language was constrained and tractable to analysis that would allow a mapping to a solver. I have not been involved in these efforts or the community closely enough to say, but I think that things like controlled english http://en.wikipedia.org/wiki/Attempto_Controlled_English http://en.wikipedia.org/wiki/Attempto_Controlled_English and http://web.eecs.umich.edu/~soar/sitemaker/docs/pubs/Interactive_task_learning_for_simple_games.pdf http://web.eecs.umich.edu/~soar/sitemaker/docs/pubs/Interact... may have been inspired by the success of Watson in Jeopardy. On the other hand I was running a question answering question when Watson started, and the PI said to me "oh yeah, but that's cheating..." And on the third hand we stopped and closed that project pretty well six months later. So - there's a lot to think about!
sjg007 12y ago
This is more of a research PR piece than anything.
- OscarCunningham 12y ago
  For certain tasks, AI has long been more accurate than doctors. But it isn't used because doctors and patients don't trust it. So perhaps PR is exactly the missing piece of the puzzle.
  - dcherman 12y ago
    I don't understand this mindset of not using technology due to a lack of trust. I can easily understand not blindly following an AI's findings, however they should probably be item #1 for the attending doctor to investigate. The worst case scenario is that the doctor disagrees with the findings and continues investigation. The best case scenario is that you hit the nail on the head much more quickly than if it was an entirely manual process.
    - smrtinsert 12y ago
      One answer is very simple: $$$. Study 12 years of your life to be a specialist only to be ultimately replaced by some installed machine at a Walmart Pharmacy.
    - duaneb 12y ago
      > I don't understand this mindset of not using technology due to a lack of trust. The more complex a tool is, the more likely it is to have flaws. Is it any surprise that the medical industry is slow to trust tools where the feared negative outweigh the few clear positives? I would rather trust a robot to operate on me rather than diagnose me. The human will be able to adapt and communicate to me while doing so.
      - wamsachel 12y ago
        You know that when a doctor is done talking with you, he/she leaves the room and looks up a diagnosis on a computer right?
        duaneb 12y ago
kifler 12y ago
Would be interesting to see what kind of liability this could open them up to. It had thought that Toronto was a US city, for example.
- Hobotron1 12y ago
  I imagine the same as CT/X-Ray manufacturers. It's going to be used as a tool, not a replacement for an oncologist.
- peter303 12y ago
  They went through a round of liability when expert systems were medical introduced in the 1980s. I dont know if they ever past the toy stage, Expert system-like rule-checking is imbedded in pharmacy programs and the such. but not advertised as the central engine of the code,but an auxilary.
- duaneb 12y ago
  At the moment, it's just diagnosing. Hopefully a) watson will provide the reasoning, enabling debugging and obvious errors to be spotted, and b) the doctors will continue to filter watson's answers for some time, shielding it from liability.
- duckmysick 12y ago
  I imagine there are some renowned physicians in Europe and Asia who also don't know that Toronto is a Canadian city. Or other arbitrary pieces of trivia for that matter. It doesn't stop them from being successful at their job. Conversely, knowing which city has airports named after a WWII lieutenant and a WWII battle doesn't protect from malpractice.
WalterBright 12y ago
More accurately it will be used as a diagnostic advisor.
madaxe_again 12y ago
This is like using a sledgehammer to put up pictures. You simply do not need the kind of parallel compute grunt something like watson can provide to do correlation analysis as they describe. You could cobble this together in {dirty secret scripting language of choice} and run it on your laptop. This is a PR piece - not the article, the activity - to bring forth the idea of computers making decisions about healthcare, based on metrics - not humans, based on metrics and compassion. The cynic in me sees this as setting a disquieting precedent in the direction of healthcare being distributed according not only to patients prospects and treatment needs, but other factors which a financially liable party, say an insurer, would be interested in - such as the earning potential of the patient. I'm pretty sure there was a Star Trek episode about this.
- zobzu 12y ago
  It's hard to judge and yes it's of course a PR piece. That said there's an interesting part to it. Where it applies, I'd welcome computer based decision or control (for ex. surgical operations where precision is a must) over humans. I'd trust it with my life more than the humans directly, ultimately. We're just not as reliable as the tools we create.
- maxander 12y ago
  The biology is a bit more complicated than that, though. Sure, if we had a big database of all the correlation factors in some easily computer-readable format, standardized testing procedures that lined up with these factors, and an army of experts keeping all of this up-to-date, it would be easy to write a script to make it all work. But we don't, and (in my off-the-cuff estimation) such a project could wind up being of Human Genome Project-esque scale. Watson is giving us a way of doing this when the available data is in a much less accommodating format (namely, scientific literature.) Not, perhaps, the biggest achievement ever, but pretty nifty and useful.
  - dunk010 12y ago
    Actually there are armies of experts keeping this all up to date, and some companies make a lot of money selling this as a service. The name of the role is "curator", and they basically map research to genetic variants, keeping a giant library up-to-date.
zkhalique 12y ago
About this prediction: "in the future, every decision mankind makes, every decision, is going to be informed by a cognitive system like Watson and, as a result, our lives in this world are going to be better for it." well if by "better" you mean "more machine-like and regulated like animals in a zoo". This is already somewhat true of consumer credit reports, and now this: http://www.ibtimes.com/china-use-big-data-rate-citizens-new-social-credit-system-1898711 http://www.ibtimes.com/china-use-big-data-rate-citizens-new-...
- Dewie3 12y ago
  Just train/program the AI to be more empathetic. :)
dba7dba 12y ago
A doctor (internist in primary care) friend told me once how he was glad he wasn't a heart surgeon (of some specialty that I don't recall). He said that surgeon in that specialty simply does one type of surgery all the time, as ordered by some other doctors. As far as he could tell, that surgeon did not have to do much mental work, but was rather like a technician. I wonder when work of doctor (diagnosing a disease) will be handled by a computer. I'd say in 100 years?
- visarga 12y ago
  We can do diagnosis now. All that is needed is a medical nurse to take symptoms and run them through Watson to get a diagnosis. The idea is that people in poor/remote regions could do this whole process themselves, with the help of a mobile phone and some medical sensors. There are already regions where it would be more beneficial to run with the current medial AI rather than leave things as they are.
nhswatson2 12y ago
Wouldn't work here in the UK. Watson: "I see you have cancer x - there are three effective drugs for that, but NICE has decided not to pay for them" Patient: "Fucking NHS bean counting cunts" Watson: "Indeed. Now, I have calculated it will cost the NHS a few pennies more between now and your death, what with the doctor, hospital, and terminal care, not to mention the endless drugs and things we will spend to slowly let you die." Patient: "I see, it's like the NHS doesn't have a scoody doo what they are actually doing, how things are spent." Watson: "Indeed".
grownseed 12y ago
The (very rough) process with next-gen sequencing is as follows: 1. one or more samples are taken from the patient (tumor/normal, peripheral blood, bone marrow, etc.) 2. the samples are sequenced, this can take different forms, involve different processes, but basically, specific sections of the genome are extracted, amplified, then turned into digital data 3. the data is aligned to a reference genome 4. the aligned data is scanned for variants (mutations), often in specific genes, and even specific sites of certain genes. This again involves different methods often used in combination 5. the variants are partially filtered out automatically (poor quality, bad reads, spurious, synonymous, etc.) 6. the remaining variants are then reviewed manually, relying on academic papers and historical data 7. what remains is reviewed again, taking the patient's history into account, and potential targeted treatments are inferred 8. the patient and his/her doctor make the final decision So from what I understand, even in an optimal scenario, Watson would be taking care of steps 6 and 7, which represent maybe 15% of the overall time. I believe the article presented here is rather misleading in this regard, as it seems to suggest that Watson would be taking care of the actual sequencing too. Additionally, I imagine that in most cases some humans would still need to review Watson's data. What the article doesn't mention either is access to the actual patient data, which in itself is a political and operational problem, not so much a technical one. Finally, I imagine Watson would infer the individual variant diagnoses from papers, which again, is far easier said than done (though I do believe that would be the point where Watson shines most). Parsing papers reliably for usable data is really, really hard, in no small part because the academic system is antiquated, as are paper formats. Even if you manage that, you still have to figure out whether the papers in question, or their authors, are actually reliable sources. And even then, you will likely end up with conflicting interpretations which will need to be sorted out. So while a nice fluff piece for IBM, I have serious doubts about the actually meaningful benefits Watson supposedly brings to the table. To be clear, I do believe that AI/ML can and do really help, and contrarily to what the article might want you to believe, people are already working with/on it. However making Watson look like it will solve all the current problems with cancer care is disingenuous at best.
- dekhn 12y ago
  I agree with your guess that Watson covers 6/7. From the clinical geneticists I've talked to, variants of unknown significance (where a person differences from the reference, the impact of the difference) are investigated manually through literature searches- often just basic text matches on the variant name. Regarding paper parsing: there are companies that employ hundreds of PhDs that read papers all day long and enter the data into a database based on ontology (Ingenuity). IBM would license that database and use it as a prior.
nl 12y ago
Everyone here has got the wrong idea about what Watson is doing here. It maybe doing sequence analysis, but I've never seen anything about that before. The real strength is the journal article reading mentioned in passing. What it does is read studies on new treatment options as they are released, and then when a person presents symptoms it will do a diagnosis and suggest treatment options along with supporting evidence. The natural language understanding needed to do the journal understanding is the real unique offering here. There are videos on YouTube showing the system. I'm on mobile so I'm not looking them up now.
- nl 12y ago
  Have a look at https://youtu.be/UFF9bI6e29U?t=2539 https://youtu.be/UFF9bI6e29U?t=2539 (should link to the 42 minute mark), where they go through slides of the Watson Healthcare UI, and it shows how the Watson reasoning system does diagnosis and suggests treatments.
- grownseed 12y ago
  Everyone here has got the wrong idea about what Watson is doing here. That would be because the article is very misleading, and without some background in the field, can you blame people for thinking that? Take for instance this paragraph: However the process is very time-consuming - a single patient's genome represents more than 100 gigabytes of data - and this needs to be combined with other medical records, journal studies and information about clinical trials. What would take a clinician weeks to analyse can be completed by Watson in only a few minutes. The first section seems to imply that the sequencing data and the analysis data are the same thing, which they aren't. As for the second section, it is simply not true. Clinicians only intervene at certain points in the process, and their work for a single patient certainly does not take "weeks", however the whole process does. But Watson would not cover the whole process, despite the impression being given. It is just shoddy reporting, taking shortcuts in the wrong places. What saddens me is that the general public will take the article at face value and expect their caretakers to live up to this fiction. Also, please see my other comment for more details, in particular about the job Watson is actually supposed to be taking care of: https://news.ycombinator.com/item?id=9501912 https://news.ycombinator.com/item?id=9501912
socialist_coder 12y ago
I have a daughter with Nephroblastomatosis, a super rare kind of kidney cancer that often lasts for years before just naturally going away (no one knows why), and dealing with the treatment options is agonizing. For example, she's been on 3 different kinds of chemotherapy for 12+ months. 2 of the chemotherapy drugs don't seem to do anything. The 3rd kind seemed to result in an immediate reduction of the nephroblastomatosis, but unfortunately this chemo drug is very harmful to your body and there is a limit to how much of the drug you can receive. Now, she is not on chemo anymore. Every 1.5 months we go in for a checkup, alternating between an MRI and sonogram. 2 times they have found "spots" on her kidneys. But, what to do about them? Additional chemo? Surgery? Wait and see? This is what I want a computer to tell me. What are the risks of surgery? What are the risks of losing more of your kidney? What are the risks of additional chemo? What are the risks of doing nothing? I would rather a computer had this information and gave its recommendation, rather than a group of doctors and us, the parents. We would be using the same logic in our decision, so it seems like the computer could do it more accurately and with zero bias. I welcome Watson into our health care system. Anything that takes some of the agony out of very very difficult treatment decisions will be a good thing.

139 points by Brakenshire 12y ago

I wonder if the Jeopardy appearances helped Watson (by getting the name out) or hindered it (by making people think it's just a parlor trick). Either way, this is seriously fascinating: I'm excited to see what other applications it has.

duaneb 12y ago
People aren't stupid, and if it was a parlour trick it was an exceptionally well-executed on. People, even those who don't have a grasp on the computational complexity of language and reasoning, seem legitimately impressed by the performance.
- sgt101 12y ago
  Yes, and it was a real research collaboration - a brilliant insight - to see that the Jeopardy language was constrained and tractable to analysis that would allow a mapping to a solver. I have not been involved in these efforts or the community closely enough to say, but I think that things like controlled english http://en.wikipedia.org/wiki/Attempto_Controlled_English http://en.wikipedia.org/wiki/Attempto_Controlled_English and http://web.eecs.umich.edu/~soar/sitemaker/docs/pubs/Interactive_task_learning_for_simple_games.pdf http://web.eecs.umich.edu/~soar/sitemaker/docs/pubs/Interact... may have been inspired by the success of Watson in Jeopardy. On the other hand I was running a question answering question when Watson started, and the PI said to me "oh yeah, but that's cheating..." And on the third hand we stopped and closed that project pretty well six months later. So - there's a lot to think about!

sjg007 12y ago

This is more of a research PR piece than anything.

OscarCunningham 12y ago
For certain tasks, AI has long been more accurate than doctors. But it isn't used because doctors and patients don't trust it. So perhaps PR is exactly the missing piece of the puzzle.
- dcherman 12y ago
  I don't understand this mindset of not using technology due to a lack of trust. I can easily understand not blindly following an AI's findings, however they should probably be item #1 for the attending doctor to investigate. The worst case scenario is that the doctor disagrees with the findings and continues investigation. The best case scenario is that you hit the nail on the head much more quickly than if it was an entirely manual process.
  - smrtinsert 12y ago
    One answer is very simple: $$$. Study 12 years of your life to be a specialist only to be ultimately replaced by some installed machine at a Walmart Pharmacy.
  - duaneb 12y ago
    > I don't understand this mindset of not using technology due to a lack of trust. The more complex a tool is, the more likely it is to have flaws. Is it any surprise that the medical industry is slow to trust tools where the feared negative outweigh the few clear positives? I would rather trust a robot to operate on me rather than diagnose me. The human will be able to adapt and communicate to me while doing so.
    - wamsachel 12y ago
      You know that when a doctor is done talking with you, he/she leaves the room and looks up a diagnosis on a computer right?
      - duaneb 12y ago

kifler 12y ago

Would be interesting to see what kind of liability this could open them up to. It had thought that Toronto was a US city, for example.

Hobotron1 12y ago
I imagine the same as CT/X-Ray manufacturers. It's going to be used as a tool, not a replacement for an oncologist.
peter303 12y ago
They went through a round of liability when expert systems were medical introduced in the 1980s. I dont know if they ever past the toy stage, Expert system-like rule-checking is imbedded in pharmacy programs and the such. but not advertised as the central engine of the code,but an auxilary.
duaneb 12y ago
At the moment, it's just diagnosing. Hopefully a) watson will provide the reasoning, enabling debugging and obvious errors to be spotted, and b) the doctors will continue to filter watson's answers for some time, shielding it from liability.
duckmysick 12y ago
I imagine there are some renowned physicians in Europe and Asia who also don't know that Toronto is a Canadian city. Or other arbitrary pieces of trivia for that matter. It doesn't stop them from being successful at their job. Conversely, knowing which city has airports named after a WWII lieutenant and a WWII battle doesn't protect from malpractice.

WalterBright 12y ago

More accurately it will be used as a diagnostic advisor.

madaxe_again 12y ago

This is like using a sledgehammer to put up pictures. You simply do not need the kind of parallel compute grunt something like watson can provide to do correlation analysis as they describe. You could cobble this together in {dirty secret scripting language of choice} and run it on your laptop. This is a PR piece - not the article, the activity - to bring forth the idea of computers making decisions about healthcare, based on metrics - not humans, based on metrics and compassion. The cynic in me sees this as setting a disquieting precedent in the direction of healthcare being distributed according not only to patients prospects and treatment needs, but other factors which a financially liable party, say an insurer, would be interested in - such as the earning potential of the patient. I'm pretty sure there was a Star Trek episode about this.

zobzu 12y ago
It's hard to judge and yes it's of course a PR piece. That said there's an interesting part to it. Where it applies, I'd welcome computer based decision or control (for ex. surgical operations where precision is a must) over humans. I'd trust it with my life more than the humans directly, ultimately. We're just not as reliable as the tools we create.
maxander 12y ago
The biology is a bit more complicated than that, though. Sure, if we had a big database of all the correlation factors in some easily computer-readable format, standardized testing procedures that lined up with these factors, and an army of experts keeping all of this up-to-date, it would be easy to write a script to make it all work. But we don't, and (in my off-the-cuff estimation) such a project could wind up being of Human Genome Project-esque scale. Watson is giving us a way of doing this when the available data is in a much less accommodating format (namely, scientific literature.) Not, perhaps, the biggest achievement ever, but pretty nifty and useful.
- dunk010 12y ago
  Actually there are armies of experts keeping this all up to date, and some companies make a lot of money selling this as a service. The name of the role is "curator", and they basically map research to genetic variants, keeping a giant library up-to-date.

zkhalique 12y ago

About this prediction: "in the future, every decision mankind makes, every decision, is going to be informed by a cognitive system like Watson and, as a result, our lives in this world are going to be better for it." well if by "better" you mean "more machine-like and regulated like animals in a zoo". This is already somewhat true of consumer credit reports, and now this: http://www.ibtimes.com/china-use-big-data-rate-citizens-new-social-credit-system-1898711 http://www.ibtimes.com/china-use-big-data-rate-citizens-new-...

Dewie3 12y ago
Just train/program the AI to be more empathetic. :)

dba7dba 12y ago

A doctor (internist in primary care) friend told me once how he was glad he wasn't a heart surgeon (of some specialty that I don't recall). He said that surgeon in that specialty simply does one type of surgery all the time, as ordered by some other doctors. As far as he could tell, that surgeon did not have to do much mental work, but was rather like a technician. I wonder when work of doctor (diagnosing a disease) will be handled by a computer. I'd say in 100 years?

visarga 12y ago
We can do diagnosis now. All that is needed is a medical nurse to take symptoms and run them through Watson to get a diagnosis. The idea is that people in poor/remote regions could do this whole process themselves, with the help of a mobile phone and some medical sensors. There are already regions where it would be more beneficial to run with the current medial AI rather than leave things as they are.

nhswatson2 12y ago

Wouldn't work here in the UK. Watson: "I see you have cancer x - there are three effective drugs for that, but NICE has decided not to pay for them" Patient: "Fucking NHS bean counting cunts" Watson: "Indeed. Now, I have calculated it will cost the NHS a few pennies more between now and your death, what with the doctor, hospital, and terminal care, not to mention the endless drugs and things we will spend to slowly let you die." Patient: "I see, it's like the NHS doesn't have a scoody doo what they are actually doing, how things are spent." Watson: "Indeed".

grownseed 12y ago

The (very rough) process with next-gen sequencing is as follows: 1. one or more samples are taken from the patient (tumor/normal, peripheral blood, bone marrow, etc.) 2. the samples are sequenced, this can take different forms, involve different processes, but basically, specific sections of the genome are extracted, amplified, then turned into digital data 3. the data is aligned to a reference genome 4. the aligned data is scanned for variants (mutations), often in specific genes, and even specific sites of certain genes. This again involves different methods often used in combination 5. the variants are partially filtered out automatically (poor quality, bad reads, spurious, synonymous, etc.) 6. the remaining variants are then reviewed manually, relying on academic papers and historical data 7. what remains is reviewed again, taking the patient's history into account, and potential targeted treatments are inferred 8. the patient and his/her doctor make the final decision So from what I understand, even in an optimal scenario, Watson would be taking care of steps 6 and 7, which represent maybe 15% of the overall time. I believe the article presented here is rather misleading in this regard, as it seems to suggest that Watson would be taking care of the actual sequencing too. Additionally, I imagine that in most cases some humans would still need to review Watson's data. What the article doesn't mention either is access to the actual patient data, which in itself is a political and operational problem, not so much a technical one. Finally, I imagine Watson would infer the individual variant diagnoses from papers, which again, is far easier said than done (though I do believe that would be the point where Watson shines most). Parsing papers reliably for usable data is really, really hard, in no small part because the academic system is antiquated, as are paper formats. Even if you manage that, you still have to figure out whether the papers in question, or their authors, are actually reliable sources. And even then, you will likely end up with conflicting interpretations which will need to be sorted out. So while a nice fluff piece for IBM, I have serious doubts about the actually meaningful benefits Watson supposedly brings to the table. To be clear, I do believe that AI/ML can and do really help, and contrarily to what the article might want you to believe, people are already working with/on it. However making Watson look like it will solve all the current problems with cancer care is disingenuous at best.

dekhn 12y ago
I agree with your guess that Watson covers 6/7. From the clinical geneticists I've talked to, variants of unknown significance (where a person differences from the reference, the impact of the difference) are investigated manually through literature searches- often just basic text matches on the variant name. Regarding paper parsing: there are companies that employ hundreds of PhDs that read papers all day long and enter the data into a database based on ontology (Ingenuity). IBM would license that database and use it as a prior.

nl 12y ago

Everyone here has got the wrong idea about what Watson is doing here. It maybe doing sequence analysis, but I've never seen anything about that before. The real strength is the journal article reading mentioned in passing. What it does is read studies on new treatment options as they are released, and then when a person presents symptoms it will do a diagnosis and suggest treatment options along with supporting evidence. The natural language understanding needed to do the journal understanding is the real unique offering here. There are videos on YouTube showing the system. I'm on mobile so I'm not looking them up now.

nl 12y ago
Have a look at https://youtu.be/UFF9bI6e29U?t=2539 https://youtu.be/UFF9bI6e29U?t=2539 (should link to the 42 minute mark), where they go through slides of the Watson Healthcare UI, and it shows how the Watson reasoning system does diagnosis and suggests treatments.
grownseed 12y ago
Everyone here has got the wrong idea about what Watson is doing here. That would be because the article is very misleading, and without some background in the field, can you blame people for thinking that? Take for instance this paragraph: However the process is very time-consuming - a single patient's genome represents more than 100 gigabytes of data - and this needs to be combined with other medical records, journal studies and information about clinical trials. What would take a clinician weeks to analyse can be completed by Watson in only a few minutes. The first section seems to imply that the sequencing data and the analysis data are the same thing, which they aren't. As for the second section, it is simply not true. Clinicians only intervene at certain points in the process, and their work for a single patient certainly does not take "weeks", however the whole process does. But Watson would not cover the whole process, despite the impression being given. It is just shoddy reporting, taking shortcuts in the wrong places. What saddens me is that the general public will take the article at face value and expect their caretakers to live up to this fiction. Also, please see my other comment for more details, in particular about the job Watson is actually supposed to be taking care of: https://news.ycombinator.com/item?id=9501912 https://news.ycombinator.com/item?id=9501912

socialist_coder 12y ago

I have a daughter with Nephroblastomatosis, a super rare kind of kidney cancer that often lasts for years before just naturally going away (no one knows why), and dealing with the treatment options is agonizing. For example, she's been on 3 different kinds of chemotherapy for 12+ months. 2 of the chemotherapy drugs don't seem to do anything. The 3rd kind seemed to result in an immediate reduction of the nephroblastomatosis, but unfortunately this chemo drug is very harmful to your body and there is a limit to how much of the drug you can receive. Now, she is not on chemo anymore. Every 1.5 months we go in for a checkup, alternating between an MRI and sonogram. 2 times they have found "spots" on her kidneys. But, what to do about them? Additional chemo? Surgery? Wait and see? This is what I want a computer to tell me. What are the risks of surgery? What are the risks of losing more of your kidney? What are the risks of additional chemo? What are the risks of doing nothing? I would rather a computer had this information and gave its recommendation, rather than a group of doctors and us, the parents. We would be using the same logic in our decision, so it seems like the computer could do it more accurately and with zero bias. I welcome Watson into our health care system. Anything that takes some of the agony out of very very difficult treatment decisions will be a good thing.