18 ms·
IBM Watson Overpromised and Underdelivered on AI Health Care
- bitcharmer 8y agoDon't want to sound too mean but this is not the first time IBM overpromised and underdelivered. Anyone surprised by this?
- dredmorbius 8y agoTech marketing tends to flaut the Scotty Principle to the extreme.
- PaulHoule 8y agoTo be fair, IBM has succeeded at big jobs too. For instance, they developed software for the Apollo mission and the Space Shuttle. The IBM/360. The IBM PC. AS/400.
- oldgradstudent 8y agoI don't think there's anyone who disagrees that IBM was a great company once. Do you have a more recent example? something that happened several years after Louis Gerstner first assumed leadership.
- Barrin92 8y agoThey're still solid in the supercomputer space, no? BlueGene, and Summit and Sierra more recently are IBM projects. Admittedly though given their pretty large size I can't name much else.
- trimbo 8y ago> Admittedly though given their pretty large size I can't name much else. Exactly. Even if 20,000 people worked on those supercomputers, mainframes and Watson, what do the other 350,000 employees work on? Consulting, and it's been that way since Gerstner.
- downrightmike 8y agoThey are really good at cashing checks and charging more on top of that: http://www.investigativepost.org/2018/02/19/taxpayers-fund-ibms-dysfunctional-call-center/ http://www.investigativepost.org/2018/02/19/taxpayers-fund-i...
- secfirstmd 8y agoYes IBM also developed for the Nazis during the Holocaust...
- colejohnson66 8y agoAnd? The people that did that are long gone
- maxxxxx 8y agoThe IBM before around 1990 and today’s IBM are totally different companies. The only thing that’s the same is the name. Same could be said for HP.
- peteradio 8y agoExcuse me but the logo is also the same. ;)
- return0 8y agoYep they are e.g. overhyping their neurmorphic platform too I hope that's not the case for their quantum computer
- craftinator 8y agoBut can their quantum computer factor the number "15"?
- jacquesm 8y agoAnything labelled 'Watson' should be seen as marketing, not product or consulting.
- mark_l_watson 8y agoI was going to agree with you, but then my last name is Watson :-) Seriously, I used IBM Watson on a consulting project a little over two years ago and I was disappointed. To be fair I should take another look. IBM has bought some good companies whose work is exposed in BlueBiz and other web services and the permanent free tier levels they give away rival the free tier levels from GCP. I think AutoML and auto data science web hosted offerings will be a huge market with right now Google, Amazon, and Microsoft leading the way. If I worked at IBM in a position of power, I would work hard to create a great developer experience for simplifying the use of machine learning, NLP, etc.
- jacquesm 8y agoIBM bit off more than they could chew with the AI/NLP project and their brand suffered as a consequence. Some problems are simply hard in the general case, though there are some spin-outs and spin-offs of that project for more limited domains that are successful.
- Retric 8y agoComparing the amount their brand suffered vs profits they may have come out ahead. AI got some real products from this hype cycle, but expectations where high enough that a crash was inevitable.
- mindslight 8y agoIsn't arbitraging off developed brands just a common contemporary playbook?
- xiphias2 8y agoThe big difference with Google and AutoML is that Google was already doing lots of machine learning in the past, just not deep learning. With deep learning it took the transition seriously internally as well: there's a huge push to get all models to use Tensorflow - even if the transition itself is painful - and this helps the infrastructure providers as well to gather feedback. With IBM it all started as a very successful research project, but instead of working more on the Watson system, it decided to rename dumber projects to be under the Watson umbrella as well thereby giving the impression that it's the same system that won in Jeopardy.
- imglorp 8y agoNot sure about Watson, but there was just this week a story about using pattern matching to improve breast cancer screening imaging. It seems the tech is decent but the problem is, it's unclear whom to sue if needed, so there's an obstacle to using it.
- kochikame 8y agoIf they'd marketed it this way from the beginning it would have been fine. AI is great at that kind of pattern-matching. The problem is that Watson was sold as a kind of "AI Doctor", which is pretty much ludicrous
- himaraya 8y ago> IBM > Overpromised and Underdelivered Sounds kind of catchy, to be honest.
- mlthoughts2018 8y agoIf you could work in a clever bit about ageism and poor severance packages, you’d have a strong contender for IBM corporate mission statement.
- taylodl 8y agoI figure we’re going to be reading articles similar to this about blockchain in a couple of years.
- TheReveller 8y agohttps://www.theregister.co.uk/2018/11/30/blockchain_study_finds_0_per_cent_success_rate/ https://www.theregister.co.uk/2018/11/30/blockchain_study_fi...
- seibelj 8y agoThe fact Bitcoin is worth $90 billion and has survived a decade of endless criticism, means blockchain is successful beyond anyone's wildest dreams a decade ago. I would say Bitcoin and Ethereum have been way more successful than Watson has.
- freehunter 8y agoPeople kept investing with Bernie Madoff for decades too, because his investments kept giving returns. People will keep shoveling money into Bitcoin as long as its value keeps swinging wildly. That's hardly a measure of success.
- seibelj 8y agoHow much will Bitcoin have to be worth, and for how long, to make it successful in your eyes? Or is the entire idea of a digital asset with intrinsic value impossible to you?
- SiempreViernes 8y agoSlot machines pull in a lot of money too, I guess you would label them a success?
- seibelj 8y agoUSDC (https://www.circle.com/en/usdc https://www.circle.com/en/usdc) is a dollar-backed cryptocurrency operating on Ethereum. Over $260 million has been converted into USDC, and it is used all over the globe to move small or very large amounts of money securely within seconds (https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 https://etherscan.io/token/0xa0b86991c6218b36c1d19d4a2e9eb0c...). This is a novel application of blockchain technology that is beyond a slot machine.
- enthd 8y agoI think quite a few companies(and maybe even a good majority of them) will end up overpromising and underdelivering when it comes to AI products.
- deleted 8y ago[deleted]
- qwerty456127 8y agoWhat actually is so hard about AI in health care? Why not just take a set of diagnostic indicators for inputs, map to conditions/treatments as outputs and train a neural net?
- blueadept111 8y agoEither the AI can generalize or it can't. If it can't, you don't need neural nets at all. If it can, you need to worry about whether it's generalizing correctly, which is definitely "so hard".
- deleted 8y ago[deleted]
- scottlocklin 8y agoSparse data. Neural nets (for example) do horribly at this sort of thing; they're data hungry. It's super difficult to beat something like linear regression in this sort of thing (ideally combined with domain knowledge -something neural approaches mostly fail at), and even linear regression gives awful results.
- 1e-9 8y agoThere are huge barriers to getting a healthcare AI product to market in the United States. First off, data is generally extremely difficult and expensive to gather due to the need to protect patient info according to the Health Insurance Portability and Accountability Act (HIPAA). To get representative data, you need to gather from many sites and the data needs to be anonymized and "truthed" by a medical expert who typically must examine multiple medical reports and case history for each patient. Each data collection site requires approval for data collection by an Institutional Review Board, which may take months or even years to get. Assuming you are able to get a reasonable amount of data for development and testing, you need to solve the actual AI problem (probably very difficult due to the extremely heterogeneous nature of disease you are dealing with) and then you need regulatory approval. Depending on the FDA classification of your product, this can be another many-month to many-year process of data collection and clinical studies.
- Ensorceled 8y ago
- randartie 8y ago>”But Watson won’t change its conclusions based on just four patients. To solve this problem, the Sloan Kettering experts created “synthetic cases” that Watson could learn from, essentially make-believe patients with certain demographic profiles and cancer characteristics.” Is this standard practice in machine learning? This sounds more like regular programming to get exactly the outcome you want.
- scottlocklin 8y agoIt's standard practice for idiots. And apparently "Sloan Kettering experts." There are semi-supervised techniques to do stuff like this in a more systematic/automated way, but you still don't get anything for free: the outcome depends on the priors used to do the semi-supervised voodoo. In a generous moment I might assume this is what they meant, but it's still dumb.
- 1e-9 8y agoTo me, the main mistake was the series of commercials giving the strong impression that IBM already had this incredible Artificial General Intelligence that was indistinguishable from a highly intelligent human and was solving a myriad of difficult practical problems better than any expert. I suspect that most who were well-versed in AI felt the ads were disingenuous from the start. I know I did. I think the marketing campaign would have better served IBM had it laid out their commitment to achieve these things without sounding like they were already there.
- Waterluvian 8y agoIn my opinion you don't even need to be an expert. - if you're a generally competent, learned person - and the new product feels like a giant leap from anything that exists already - it's likely not all it claims to be.
- scrooched_moose 8y agoI've always had a 20% threshold If anyone is promising a 20% or greater improvement over the current leaders, I immediately file it away as a scam until I see extraordinary proof. Other than some very immature industries, even 20% leaps don't happen without earth shattering once-in-a-generation breakthroughs. Some call me a pessimist, and occasionally I'm unfairly dismissive of new technologies. But for every one I get wrong, I get 99 others correct.
- randomsearch 8y agoWhen Google first arrived, I’d say the improvement was way more than 20%. Did you experience that?
- Rexxar 8y agoSearch was not a mature industry at all.
- 8y ago
- tosser1234 8y agoI worked on this project and there were a lot of issues. Two of the biggest were: * Whatever the quality of the technology (which I personally never saw as that compelling) was wrapped up in terribly written research code, making it practically impossible to setup and use. * The Jeopardy demo was made possible by the existence of a marked-up source of general knowledge (Wikipedia), a ready-made bank of questions and answers from past shows (j-archive.org), and the fact that practically anyone had the ability to curate more Q&A pairs. This is almost totally different than the medical use case where the knowledge is wrapped up in proprietary textbooks and papers and the only people able to curate training data are medical professionals.
- telchar 8y agoThat's not really true. The UMLS has a large graph of marked up medical domain knowledge that can be used. It's not as specific as one might want for developing an AI autodoc, but it's quite a bit better than what is available in most fields. It's actually quite similar to what one can derive from Wikipedia.
- cknoxrun 8y agoUnfortunately UMLS and Wikipedia are not good enough for drug information. Structuring that data requires a fine balance of tooling (with some NLP) and a good team of biocurators (doctors, pharmacists, pharmacologists and domain experts).
- nickpsecurity 8y agoQuora has a nice set of answers about why machine learning isn't used so much in medical: https://www.quora.com/Why-isn%E2%80%99t-machine-learning-more-widely-used-for-medical-diagnoses https://www.quora.com/Why-isn%E2%80%99t-machine-learning-mor... Skip to Jae Won Joe's answer to see a case study with a patient showing the interactive process. Then, especially look at ending questions about missing or deceptive data. Seems like their good diagnoses comes from a combination of domain data and expertly reading people in front of them. Machines suck at the part I emphasized. The data sets might not reflect a lot of stuff like that, too. Who knows.
- maxander 8y agoA bunch of the projects described, and the (technical) difficulties encountered, make me wonder if GPT-2-style systems would have better odds. Is anyone looking into applying that to medical/scientific text NLP problems? I mean, GPT-2 still often produces nonsense more similar to dream imagery than useful reasoning, but I gather most medical residency students are half-asleep most of the time anyway, so... :)
- return0 8y ago> still often produces nonsense more similar to dream imagery than useful reasoning, Is there any example of gpt-2 in the wild that is not nonsense?
- 0898 8y agoIBM sold Watson as a black box with a magic genie inside. You could point it at your data and it would tell you the answer before you'd even thought of a question. No wonder we're disappointed.
- Randypea 8y agoI was recently diagnosed with neuroendocrine cancer, which my PCP had been misdiagnosed for 10 years as IBS. This is more the norm than the exception for people with this type of cancer, Steve Jobs included. It's a perfect example of where AI can likely diagnosis what my PCP could not. AI tools need to find cancer problems to solve which are more suited to their capabilities. e.g. does anybody know of a company working on "AI for cancer screening"? This is desperately needed and would have helped me.
- jononor 8y agoI'm am sorry to hear that. I don't know about companies, but cancer screening with machine learning is a very active topic in academia at least. Other topics include outcome prediction, and analysis of treatment alternatives.
- Randypea 8y agoThanks. Actually, my cancer is very weak. On a scale of 1 to 10, it's aggressiveness is 1-2. But one of the liver tumors is very and will be removed soon. I am starting a side project to work on cancer screen if anyone can point me to anyone willing to partner on this, I would appreciate the help.
- 1e-9 8y agoUnfortunately, there are still many cancers detected far too late for effective treatment. It sounds like you were indeed fortunate to have a less aggressive form. AI for cancer screening generally falls under the category of "Computer-Aided Detection" or CAD. The commercial and academic CAD efforts tend to be organized by the primary anatomical site of cancer and the detection method (e.g. X-Ray, CT-Scan, PET, ultrasound, blood test) . Was your primary the pancreas or intestine? Are you wanting to contribute to an imaging detection method or something else? I might be able to help you identify someone working in the area depending on your goals.
- Randypea 8y agoMy cancer was finally found in my intestine. This is another opportunity. My primary oncologist and local surgeon were telling me the primary tumor, which has metastasized to a very large liver tumor, could not be found. I did my own research and found they had ordered the wrong type of imaging scan. Only after I pushed to have the correct scan (Gallium 68 PET/CT scan) was the primary tumor found. This was a "lack of information" for my local oncologist. Computer-aided diagnosis would have helped him. An additional new symptom (flushing) appeared and my PCP recognized a specific cheap blood test was needed that led to the cancer being found. I am happy to contribute to an imaging study. But I want to work on cancer screening. What kind of automation/screening would be needed to prevent 10 years of misdiagnosis by my PCP? ... not only for this type of cancer but for all of the top 15-20 types of cancer. People are not being screened. How can we make screening affordable? And how can we raise awareness of possible misdiagnosis and or affordable screening?
- sumoboy 8y agoThe best reference about Watson last year was on reddit, "Watson is a brand, not a "thing". It covers any technologies that IBM sells in the cognitive space. They are further broken up into the business areas." All hype and marketing built on open source tools, all while AWS, Google, and Azure build out bigger cloud offerings. As OP mentioned, they used to be a great company.
- altmind 8y agoI've heard some day: "Watson is a brand name for IBM consulancy services". These people formulate what watson is for your specific business.
- SCAQTony 8y agoWatson should rebrand then. You can't have a colossal failure such as the one reported by Stat News: “multiple examples of unsafe and incorrect treatment recommendations.” — [Memorial Sloan Kettering Cancer Center] and expect to move on from it as if they were growing pains.
- ethbro 8y agoI call this the Netflix ML effect. If your answer is mission critical, probabilistic ML isn't advanced, explainable, or reliable enough for your problem. If your answer is of the great-to-be-right, meh-to-be-wrong sort (e.g. ranking movie recommendations), then you can and should go nuts with ML. And if someone really wants to do an ML project on the former, do everything you can to transform it into the latter.
- djakjxnanjak 8y agoIf you can tune the specifics-sensitivity curve, you should be able to handle both cases. You have to be willing to refuse to provide an answer when you have low confidence.
- 8y ago
- Cactus2018 8y ago[from wikipedia] Thomson Reuters sold Thomson Healthcare to Veritas Capital for US$1.25 billion On June 6, 2012. The new company, Truven Health Analytics, became an independent organization solely focused on healthcare. Truven is a portmanteau of the words "trusted" and "proven". IBM Corporation acquired Truven on February 18, 2016, and merged with IBM's Watson Health unit.
- rocgf 8y agoAt one point, there was an API available via the IBM Cloud that you could use in order to ask Watson questions. I've rarely been that underwhelmed. After using that, I realized that Watson was mostly a gimmick.
- zeropnc 8y agoIt’s called Bluemix - it’s hilarously underwhelming
- barbecue_sauce 8y agoThe greatest thing about BlueMix was its promotional budget. Saw The Force Awakens on IBM's dime, and got about $50 worth of movie theater gift cards. All I had to do was sit through a 20 minute presentation about BlueMix case study integrations.
- chaoticmass 8y agoI saw Force Awakens via IBM as well, and I got a cool light saber toy! https://twitter.com/chaoticmass/status/677708888083439616 https://twitter.com/chaoticmass/status/677708888083439616 I did look at the BlueMix platform, and it seemed like a big mix of various APIs I could tap into. Some of them looked neat, but I never had a reason to really use any of them.
- barbecue_sauce 8y agoApparently they decided BlueMix was a stupid name and just renamed it IBM Cloud.
- thatoneuser 8y agoWhat were the limits?
- dre85 8y agoI always find it kind of silly when AI is just thrown into a field with the notion that they'll just deal with the messy, subjective and unstructured data (like hand written medical notes for example) as is. For me it makes much more sense to try to clean up and structure the data from the start instead. Maybe come up with some data acquisition compromise that is both UX friendly and give rise to more structure and consistency.
- jackfoxy 8y agoMy company does machine learning checked by physical models. Our single biggest problem (and management is finally waking up to it) is curating the incoming data. And this is in a mature industry (oil & gas).
- ethbro 8y agoMy biggest surprise has been how little everyone is aware of their data quality. The only explanation I've been able to come up with is that when it's all human processed, Joe 2nd-link-in-the-chain just deals with all the inconsistencies as best he can to get his job done, and never reports issues up.
- mr_toad 8y agoWithout those data inconsistencies to fix up every week, Joe would probably be out of a job.
- ethbro 8y agoBut Joe typically hates dealing with the inconsistencies, and he can tell you exactly how they could be fixed. It generally seems like (a) the suggestions for fixes are impractical to implement (overly detrimental effect on counterparty), (b) Joe isn't empowered organizationally to suggest fixes that will be implemented, or (c) Joe doesn't have access to the IT tools to implement fixes himself.
- 8y ago
- tus87 8y agoAI over-promising and under-delivering? Never.
- moocowtruck 8y agoWELCOME TO IBM
- densone 8y agoIBM MAKE great commercials. Those commercials are just as gimmicky as most of the products.
- newen 8y agoMain problem is Watson its like a million different things all with the same name. So no one actually knows what they are selling when they talk to you about Watson. Literally I had one IBM guy ask to check out their Watson and it took 5 minutes of them showing me the demo before I figured out what it was (cloud based Nvidia Digits alternative). If that particular Watson had an actual name, I probably would've heard about it before and I'd actually be able to recommend it to someone.
- fybe 8y agoDuring my time in IBM I was walking by the "main" Watson server everyday into work. I was always impressed at the package, and what ever they said it could do. Finally got my hands on it and was incredibly disappointed. Lacking features, archaic interface, and not delivering on what it was promised. Talked to some guys working for the Watson team and they told me that half of the AI stuff is just them doing it manually under the guise of "integration and configuration". I did have fun "talking" to Watson whenever I was bored at work, trying to make him swear
- colejohnson66 8y agoSo Watson is just IBM’s Mechanical Turk?