5 ms·
> Jefferies pulls from an audit of a partnership between IBM Watson and MD Anderson as a case study for IBM’s broader problems scaling Watson. MD Anderson cut i
by xienze 9y ago
> Jefferies pulls from an audit of a partnership between IBM Watson and MD Anderson as a case study for IBM’s broader problems scaling Watson. MD Anderson cut its ties with IBM after wasting $60 million on a Watson project that was ultimately deemed, “not ready for human investigational or clinical use.”
Well, can't say I'm surprised. I used to work on that project a few years ago, basically the idea was that Watson would look at a patient's medical record, figure out what medications they're on, what symptoms they had, etc. and cross-reference all that with the medical knowledge it had ingested from vast amounts of medical literature. In theory, Watson could figure out what medications the patient should or should not be using, a proper course of treatment, etc.
There were two major problems:
First, it turns out your medical record is mostly written in narrative form, i.e., "John Smith is a 45 year old male...", "Patient is taking X mg of Y twice daily", "Patient was administered X ml of Y on 3/1/2016", etc. In other words, there's basically no structured data, so just figuring out the patient's stats, vitals, medications, and treatment dosages was an adventure in NLP. All that stuff was written in sentence form, and of course how things were written depended on who wrote it in the first place. It was really, really hard to make sure Watson actually had correct information about the patient in the first place.
Second, all that medical literature that was being ingested? Regular old, don't-know-anything-about-medicine programmers were the ones writing the rules the manipulating the data extracted via NLP. Well guess what, if you're not a domain expert you're bound to get things wrong.
Put those two things together and we would frequently get recommendations that were wildly incorrect, but that's to be expected when you get garbage input being fed into algorithms written by people who aren't domain experts.
- Deinumite 9y agoThis basically sums up my brief stint in healthcare. Every hospital / doctor wants to talk about the wealth of medical information they are sitting on... as you said most notes are just transcribed ad lib, or input into rigid EMR systems, there is no middle ground. Also the language spoken by healthcare software is HL7 which is another nightmare in itself.
- throwaway7645 9y agoPlease tell me of this HL7.
- Dlotan 9y agoIts the protocol in which health applications talk to each other. It is extremely complicated to say the least.
- filereaper 9y agoSo if I understand correctly, there was dirty/malformed data which was difficult to interpret, and when sent to a ML algorithm not tuned by a domain expert led to bad results. This applies to all ML work, why is Watson exempt from it? Conversely if this the canonical failure case, why's everyone so harsh on Watson?
- xienze 9y ago> This applies to all ML work, why is Watson exempt from it? I'm not following, Watson was most assuredly not exempt from it. > Conversely if this the canonical failure case, why's everyone so harsh on Watson? Because they were attempting to use Watson to help make potentially life-and-death decisions?
- filereaper 9y ago>> This applies to all ML work, why is Watson exempt from it? >I'm not following, Watson was most assuredly not exempt from it. Apologies, I meant it as not having clean data with badly tuned models should be obvious root cause of failure to the analysts who are familiar with this industry. As in it may take more work over time to improve the situation, and that this problem occurs commonly in other areas of AI/ML as well. >> Conversely if this the canonical failure case, why's everyone so harsh on Watson? >Because they were attempting to use Watson to help make potentially life-and-death decisions? Touche. I have no arguments here.
- deleted 9y ago[deleted]
- PKop 9y agoPerhaps because they charged $60 million for it?
- semi-extrinsic 9y agoWhat I sincerely don't get about IT is why customers sign contracts where they fork over the mulah even when nothing of value is delivered to them. Why the actual fuck? I mean, if Delta ordered a super-advanced new fancy 787 as a tech demo, and Boeing came back with "sorry, our experts were unable to make anything work, so we're going to take all your money and not give you anything back", Delta would rightly tell Boeing to fuck right off and demand that they refund the money.
- RobLach 9y agoAs I understand it IBM now pulls in a team of domain experts to help tune their algs and even markets this as an advantage over their competitors. That makes the MD Anderson case more interesting as it mostly means that MD Anderson spent $60 million to help IBM figure out the importance of that. It begins to seem obvious how IBM stays so impervious despite being so sluggish. If your marketing and sales teams can subsidize the engineering and design side so heavily by wooing the executives that actually sign the contracts, you can spin it as just "where the technology was at that time". Selling on hype gets you in the door and establishes you as player, and eventually the tech will catch up. Being 1-2 years behind blurs when you're signing 5 year enterprise-scale deals that took a year to organize and negotiate. And if your products really are a dead end, you've got enough of a war chest to just aquihire an entire branch of your company, like IBM did buying Softlayer to just become IBM Cloud Services.
- nopinsight 9y agoThe acquihire fallback plan might work with other areas, but the valuation of a top AI team/startup is now in the high tens or even hundreds of millions. Almost all single contracts would not be sufficient to pay for that. Moreover, the acquirer must be willing to pay salaries competitive with Google, Facebook, and the likes to keep the team. From what I heard, IBM is not willing to do that. If anyone has information to the contrary, please feel free to correct me.