8 ms·

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a d

by bitL 8y ago

I was recently "playing" with some radiology data. I had no chance to identify diagnoses myself with untrained eyes, something that probably takes years for a decent radiologist to master. Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training. In 4 out of 12 categories this classifier beat best human performing radiologists. Now the very same model with no other change than adjusting number of categories could be used in any image classification task, likely with state-of-art results. I was surprised when I applied it to another, completely unrelated dataset and got >92% accuracy right away. If you think this is a symptom of AI winter, then you are probably wasting time on outdated/dysfunctional models or models that aren't suited for what you want to accomplish. Looking e.g. at Google Duplex (better voice synchronization than Vocaloid I use for making music), this pushed state-of-art to unbelievable levels in hard-to-address domains. I believe the whole SW industry will be living next 10 years from gradual addition of these concepts into production. If you think Deep (Reinforcement) Learning is going to solve AGI, you are out of luck. If you however think it's useless and won't bring us anywhere, you are guaranteed to be wrong. Frankly, if you are daily working with Deep Learning, you are probably not seeing the big picture (i.e. how horrible methods used in real-life are and how you can easily get very economical 5% benefit of just plugging in Deep Learning somewhere in the pipeline; this might seem little but managers would kill for 5% of extra profit).

imant 8y ago
Would you mind sharing which other, unrelated dataset you have used the model on?
- bitL 8y ago
  I can't unfortunately, proprietary stuff being plugged into existing business right now.
- Faaak 8y ago
  You can take a look at: https://github.com/sfikas/medical-imaging-datasets https://github.com/sfikas/medical-imaging-datasets or http://www.radrounds.com/profiles/blogs/list-of-open-access-medical-imaging-datasets http://www.radrounds.com/profiles/blogs/list-of-open-access-...
deleted 8y ago
[deleted]
flamedoge 8y ago
Next winter will probably going to be going over that 92% across all domains.
- bitL 8y ago
  Possibly, but will it be called AI winter, if e.g. average human has 88% accuracy and best human 97%?
jeffreyrogers 8y ago
I don't think the claim is that AI isn't useful. It's that it's oversold. In any case, I don't think you can tell much about how well your classifier is working for something like cancer diagnoses unless you know how many false negatives you have (and how that compares to how many false negatives a radiologist makes).
- bitL 8y ago
  There are two sides to this: - how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful? - there are metrics for capturing true/false positives/negatives one can focus on during learning optimization From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.39, which sounds really bad. Your system manages to push average to 0.44, which might be worse than the best radiologist out there, but better than an average radiologist [1]. Is this really being oversold? (I am not addressing possible problems with overly optimistic datasets etc. which are real concerns) [1] https://stanfordmlgroup.github.io/projects/chexnet/ https://stanfordmlgroup.github.io/projects/chexnet/
  - salawat 8y ago
    Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of problem. AI's as we call them today are better referred to as expert systems. AI carries too much baggage to be thrown around Willy nilly. An expert system may beat out a human at interpreting large unintuitive datasets, but they aren't generally testable, and like it or not, it will remain a tough sell in any situation where lives are on the line. I'm not saying it isn't worth researching, but AI will continue to fight an uphill battle in terms of public acceptance outside of research or analytics spaces, and overselling or being anything but straightforward about what is going on under the hood will NOT help.
khawkins 8y ago
AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is being exposed to stories warning about the singularity by well respected people like Stephen Hawking and Elon Musk (http://time.com/3614349/artificial-intelligence-singularity-stephen-hawking-elon-musk/ http://time.com/3614349/artificial-intelligence-singularity-...). Autonomous vehicles are on the roads and Boston Dynamics is showing very real robot demonstrations. Deep learning is breaking records in what we thought was possible with machine learning. All of this progress has excited an irrational exuberance in the general public. But people don't have a good concept of what these technologies can't do, mainly because researchers, business people, and journalists don't want to tell them--they want the money and attention. But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.
- varelse 8y ago
  I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field. That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi... My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.
ujal 8y ago
I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times. State of the art in numbers: Image Classification - ~$55, 9hrs (ImageNet) Object Detection - ~$40, 6hrs (COCO) Machine Translation - ~$40, 6hrs (WMT '14 EN-DE) Question Answering - ~$5, 0.8hrs (SQuAD) Speech recognition - ~$90, 13hrs (LibriSpeech) Language Modeling - ~$490, 74hrs (LM1B) "If you think Deep (Reinforcement) Learning is going to solve AGI, you are out of luck" -- I don't know. Duplex equipped with a way to minimize his own uncertainties sounds quite scary.
- varelse 8y ago
  Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witchery-was-faked-213add710da7 https://medium.com/@Michael_Spencer/google-duplex-demo-witch... Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoice-chat-bot-phone-call-demo https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic... Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google does remain the master of its own hype though.
  - sah2ed 8y ago
    > Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ? IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers. 0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854-41da-b6e2-5e55b619283e.html https://www.axios.com/google-ai-demo-questions-9a57afad-9854...
    - varelse 8y ago
      You're right, it was Steve Ballmer who nearly vanquished Microsoft at a time when Google was the company to work for in tech and kept doing amazing things. At least IMO. Unfortunately, by the time of my brief stint at Google, the place was a professional dead-end where most of the hirees got smoke blown up their patooties at orientation about how amazing they were to be accepted into Google, only to be blind allocated into me-too MVPs of stuff they'd read about on TechCrunch. All IMO of course. That said, I met the early Google Brain team there and I apparently made a sufficiently negative first impression for one of their leaders to hold a grudge against me 6 years later, explaining at last who it was that had blacklisted me there. So at least that mystery is solved. PS It was pretty obvious these were voice actors in a studio conversing with the AI. That is impressive, but speaking as a former DJ myself, when one has any degree of voice training, one pronounces words without much accent and without slurring them together. Google will likely never admit anything here: they don't have to. But I will give Alphabet a point for Waymo being the most professionally-responsible self-driving car effort so far. Compare and contrast with Tesla and Uber.
timr 8y ago
"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training." OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice. You're basically making the author's case: deep learning clearly outperforms on certain classes of problems, and easily "generalizes" to modest performance on lots of others. But leaping from that to "radiology robots are almost here!" is folly.
- bitL 8y ago
  Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than humans and other ones easier detectable by humans would slip through. There is also higher chance that next state-of-art model would push it significantly over 83% or best human radiologist at some point in the future, so it might not be very economical to train humans to become even better (i.e. dedicate your life to focus on radiology diagnostics only).
  - timr 8y ago
    "Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless. Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates relative to cost), comparison to average statistics is academic. But I don't really need to tell you this -- your comment has so many caveats that you're clearly already aware of the limits of the method.
    - bitL 8y ago
      What would be the good metrics then? Of course metrics are just indicators that can be interpreted incorrectly. Still, we have to measure something tangible. What would you propose? I am aware of limitations and would gladly use something better... Some people mention Matthews correlation coefficients, Youden's J statistic, Cohen's kappa etc. but I haven't seen them in any Deep Learning paper so far and I bet they have large blindspots as well.
eanzenberg 8y ago
ROC AUC is fairly useless when you have disparate costs in the errors. Try precision-recall.
- bitL 8y ago
  I mentioned F1 in some later comment.
ASalazarMX 8y ago
> Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training Of course! Using DenseNet-BC-100-12 to increase ROC AUC, it was so obvious!
- deleted 8y ago
  [deleted]
mathattack 8y ago
How much of this can we pin on IBM's overhype of Watson?
machinelearning 8y ago
Yea this sounds extremely unlikely unless the other dataset has a fairly easy decision boundary. The kind of cross-domain transfer learning you seem to think deep neural networks have is nothing I've observed before in my formal studies of neural network

by bitL 8y ago

imant 8y ago

Would you mind sharing which other, unrelated dataset you have used the model on?

bitL 8y ago
I can't unfortunately, proprietary stuff being plugged into existing business right now.
Faaak 8y ago
You can take a look at: https://github.com/sfikas/medical-imaging-datasets https://github.com/sfikas/medical-imaging-datasets or http://www.radrounds.com/profiles/blogs/list-of-open-access-medical-imaging-datasets http://www.radrounds.com/profiles/blogs/list-of-open-access-...

deleted 8y ago

[deleted]

flamedoge 8y ago

Next winter will probably going to be going over that 92% across all domains.

bitL 8y ago
Possibly, but will it be called AI winter, if e.g. average human has 88% accuracy and best human 97%?

jeffreyrogers 8y ago

I don't think the claim is that AI isn't useful. It's that it's oversold. In any case, I don't think you can tell much about how well your classifier is working for something like cancer diagnoses unless you know how many false negatives you have (and how that compares to how many false negatives a radiologist makes).

bitL 8y ago
There are two sides to this: - how good humans are in detecting cancer (hint: not very good) and if having an automated system even as a "second opinion" next to an expert might not be useful? - there are metrics for capturing true/false positives/negatives one can focus on during learning optimization From studies you might have noticed that expert radiologists have e.g. F1-score at 0.45 and on average they score 0.39, which sounds really bad. Your system manages to push average to 0.44, which might be worse than the best radiologist out there, but better than an average radiologist [1]. Is this really being oversold? (I am not addressing possible problems with overly optimistic datasets etc. which are real concerns) [1] https://stanfordmlgroup.github.io/projects/chexnet/ https://stanfordmlgroup.github.io/projects/chexnet/
- salawat 8y ago
  Alright. What is the cost of a false positive in that case? The problem AI runs into is that with too much faith in the machine, people STOP thinking and believe the machine. Where you might get a .44 detection rate on radiology data alone, that radiologist with a .39 or a doctor can consult alternate streams of information. The AI may still be helpful in reinforcing a decision to continue scrutinizing a set of problem. AI's as we call them today are better referred to as expert systems. AI carries too much baggage to be thrown around Willy nilly. An expert system may beat out a human at interpreting large unintuitive datasets, but they aren't generally testable, and like it or not, it will remain a tough sell in any situation where lives are on the line. I'm not saying it isn't worth researching, but AI will continue to fight an uphill battle in terms of public acceptance outside of research or analytics spaces, and overselling or being anything but straightforward about what is going on under the hood will NOT help.

khawkins 8y ago

AI winters are a result of a massive disparity between the expectations of the general public and the reality of where the technology currently sits. Just like an asset bubble, the value of the industry as a whole pops as people collectively realize that AI, while not being worthless, is worth significantly less than they thought. Understand that in pop-sci circles over the past several years the general public is being exposed to stories warning about the singularity by well respected people like Stephen Hawking and Elon Musk (http://time.com/3614349/artificial-intelligence-singularity-stephen-hawking-elon-musk/ http://time.com/3614349/artificial-intelligence-singularity-...). Autonomous vehicles are on the roads and Boston Dynamics is showing very real robot demonstrations. Deep learning is breaking records in what we thought was possible with machine learning. All of this progress has excited an irrational exuberance in the general public. But people don't have a good concept of what these technologies can't do, mainly because researchers, business people, and journalists don't want to tell them--they want the money and attention. But eventually the general public wises up to the unfulfillment of expectations, and drives their attention elsewhere. Here we have the AI winter.

varelse 8y ago
I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment. Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field. That said, if you put your ~10,000 hours into this, and keep up with the field, it's pretty impressive what high-dimensional classification and regression can do. Judea Pearl concurs: https://www.theatlantic.com/technology/archive/2018/05/machine-learning-is-stuck-on-asking-why/560675/ https://www.theatlantic.com/technology/archive/2018/05/machi... My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.

ujal 8y ago

I don't get the sentiment of the article either. I can't speak for researchers but software engineers are living through very exciting times. State of the art in numbers: Image Classification - ~$55, 9hrs (ImageNet) Object Detection - ~$40, 6hrs (COCO) Machine Translation - ~$40, 6hrs (WMT '14 EN-DE) Question Answering - ~$5, 0.8hrs (SQuAD) Speech recognition - ~$90, 13hrs (LibriSpeech) Language Modeling - ~$490, 74hrs (LM1B) "If you think Deep (Reinforcement) Learning is going to solve AGI, you are out of luck" -- I don't know. Duplex equipped with a way to minimize his own uncertainties sounds quite scary.

varelse 8y ago
Duplex was impressive but cheap street magic: https://medium.com/@Michael_Spencer/google-duplex-demo-witchery-was-faked-213add710da7 https://medium.com/@Michael_Spencer/google-duplex-demo-witch... Microsoft OTOH quietly shipped the equivalent in China last month: https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoice-chat-bot-phone-call-demo https://www.theverge.com/2018/5/22/17379508/microsoft-xiaoic... Google has lost a lot of steam lately IMO. Facebook is releasing better tools and Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google does remain the master of its own hype though.
- sah2ed 8y ago
  > Microsoft, the company they nearly vanquished a decade ago, is releasing better products. Google nearly vanquished Microsoft a decade ago? Where can I read more about this bit of history :) ? IMO, Axios [0] seem to do a better job of criticizing Google's Duplex AI claims, as they repeatedly reached out to their contacts at Google for answers. 0: https://www.axios.com/google-ai-demo-questions-9a57afad-9854-41da-b6e2-5e55b619283e.html https://www.axios.com/google-ai-demo-questions-9a57afad-9854...
  - varelse 8y ago
    You're right, it was Steve Ballmer who nearly vanquished Microsoft at a time when Google was the company to work for in tech and kept doing amazing things. At least IMO. Unfortunately, by the time of my brief stint at Google, the place was a professional dead-end where most of the hirees got smoke blown up their patooties at orientation about how amazing they were to be accepted into Google, only to be blind allocated into me-too MVPs of stuff they'd read about on TechCrunch. All IMO of course. That said, I met the early Google Brain team there and I apparently made a sufficiently negative first impression for one of their leaders to hold a grudge against me 6 years later, explaining at last who it was that had blacklisted me there. So at least that mystery is solved. PS It was pretty obvious these were voice actors in a studio conversing with the AI. That is impressive, but speaking as a former DJ myself, when one has any degree of voice training, one pronounces words without much accent and without slurring them together. Google will likely never admit anything here: they don't have to. But I will give Alphabet a point for Waymo being the most professionally-responsible self-driving car effort so far. Compare and contrast with Tesla and Uber.

timr 8y ago

"Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training." OK, but 83% ROC/AUC is nothing to be bragging about. ROC/AUC routinely overstates the performance of a classifier anyway, and even so, ~80% values aren't that great in any domain. I wouldn't trust my life to that level of performance, unless I had no other choice. You're basically making the author's case: deep learning clearly outperforms on certain classes of problems, and easily "generalizes" to modest performance on lots of others. But leaping from that to "radiology robots are almost here!" is folly.

bitL 8y ago
Yeah, but the point here was that radiologists on average fared even worse. 83% is not impressive, but better than what we have right now in real-world with real people, as sad as it is. Obviously, best radiologists would outperform it right now, but average ones, likely stressed under heavy workload might not be able to beat it. And of course, this classifier probably works on certain visual structures better than humans and other ones easier detectable by humans would slip through. There is also higher chance that next state-of-art model would push it significantly over 83% or best human radiologist at some point in the future, so it might not be very economical to train humans to become even better (i.e. dedicate your life to focus on radiology diagnostics only).
- timr 8y ago
  "Yeah, but the point here was that radiologists on average fared even worse." Except they don't. See the table in the original post. Also, comparing the "average" radiologist by F1 scores from a single experiment (as you've done in other comments here) is meaningless. Unless my doctor is exactly average (and isn't incorporating additional information, or smart enough to be optimizing for false positive/negative rates relative to cost), comparison to average statistics is academic. But I don't really need to tell you this -- your comment has so many caveats that you're clearly already aware of the limits of the method.
  - bitL 8y ago
    What would be the good metrics then? Of course metrics are just indicators that can be interpreted incorrectly. Still, we have to measure something tangible. What would you propose? I am aware of limitations and would gladly use something better... Some people mention Matthews correlation coefficients, Youden's J statistic, Cohen's kappa etc. but I haven't seen them in any Deep Learning paper so far and I bet they have large blindspots as well.

eanzenberg 8y ago

ROC AUC is fairly useless when you have disparate costs in the errors. Try precision-recall.

bitL 8y ago
I mentioned F1 in some later comment.

ASalazarMX 8y ago

> Just by using DenseNet-BC-100-12 I ended up with 83% ROC AUC after a few hours of training Of course! Using DenseNet-BC-100-12 to increase ROC AUC, it was so obvious!

deleted 8y ago
[deleted]

mathattack 8y ago

How much of this can we pin on IBM's overhype of Watson?

machinelearning 8y ago

Yea this sounds extremely unlikely unless the other dataset has a fairly easy decision boundary. The kind of cross-domain transfer learning you seem to think deep neural networks have is nothing I've observed before in my formal studies of neural network