20 ms·

There are many parallels with seismic interpretation here. Many companies/etc keep promising to "revolutionize" interpretation and remove the need for the "ted

by jofer 5y ago

There are many parallels with seismic interpretation here. Many companies/etc keep promising to "revolutionize" interpretation and remove the need for the "tedious" work of a geologist/geophysicist. This is very appealing to management for a wide variety of reasons, so it gets a lot of funding. What folks miss is that an interpreter _isn't_ just drawing lines / picking reflectors. That's less than 1% of the time spent, if you're doing it right. Instead, the interpreter's role is to incorporate all of the information from _outside_ the image. E.g. "sure we see this here, but it can't be X because we see Y in this other area", or "there must be a fault in this unimaged area because we see a fold 10 km away". By definition, you're in a data poor environment. The features you're interested in are almost never what's clearly imaged -- instead, you're predicting what's in that unimaged, "mushy" area over there through fundamental laws of physics like conservation of mass and understanding of the larger regional context. Those are deeply difficult to incorporate in machine learning in practice. Put a different way, the role is not to come up with a reasonable realization from an image or detect feature X in an image. It's to outline the entire space of physically valid solutions and, most importantly, reject the non-physically valid solutions.

ludamad 5y ago
I wonder how often these projects truly need someone with on the ground experience guiding it, as the textbook tasks as you say are easy for even the humans
boleary-gl 5y ago
This is a really good point and example. I spent 10 years in mammography software, and I saw first hand how many outside factors can impact a physician's decision to biopsy or not a given artifact on an image. Things like family history, patient's history, cycle timing, age, weight, other risk factors all play a role in a smart radiologist making the right decision for that patient. And the pattern recognition on top of that is really hard - it's not just about the pattern you see at a particular spot in an image, it's the whole image in the context of what that looks like. Could ML get better over time with this? Sure...but they've been using CAD in mammography for decades and it still hasn't replaced radiologists at all. Could a model be made to include those over variables? Sure...but again the complexity of that kind of decision making is something that requires a lot more "intelligence" than any AI or ML system exhibits today and in my mind in the foreseeable future. Just collecting that data in a structured, consistent way is more challenging than people realize.
- mumblemumble 5y ago
  I see the same thing in natural language processing. A lot of important details come from outside the four corners of the document. Ironically, I often find myself in the unenviable position of being the machine learning person who's trying to convince people that machine learning is probably not a good fit for their problem. Or, worse, taking some more fuzzy-seeming position like, "Yes, we could solve this problem with machine learning, but it would actually cost more money than paying humans to do it by hand." Part of why I hate hate hate the term "AI" is because you simply can't call something artificial intelligence and then expect them to understand that it's not actually useful for doing anything that requires any kind of intelligence.
  - mcguire 5y ago
    There's an old AI joke that all actual problems reduce to the artificial general intelligence problem---everything else, by definition, doesn't require intelligence.
    - mumblemumble 5y ago
      There's some truth to that. But I'd also argue that there's a tendency to try to bill every single kind of spinoff technology that the artificial intelligence community has produced as artificial intelligence. Which a bit like characterizing mixing up a glass of Tang as a kind of space exploration.
  - 2sk21 5y ago
    You are absolutely correct. In fact, most NLP software ignores the formatting of documents which conveys a lot of information as well. For example, section headings must be treated differently from the text that makes up the body of a section. Its very hard to even determine section headings and then its hard to take advantage of them since the big transformer models simply accept a stream of unspecialized tokens.
    - deleted 5y ago
      [deleted]
shiftpgdn 5y ago
Surely you've seen the improvement over the last 5-6 years from machine learning in all the interpretation toolsets. The last place I worked we internally had a seismic inversion tool that blew all the commercial suites out the water. I'm currently contracting for an AI/ML service company currently that has a synthetic welllog tool that is can apparently beat the pants off actual well logging tools for a fraction of the cost (though I'm not a geologist or petrophysicist so I can't personally verify this.) I think the problem is more the media and advertisers likes to paint the picture of a magical AI tool which will instantly solve all your problems and do all the work instead of a fulcrum to make doing the actual work significantly easier.
- Workaccount2 5y ago
  I am wondering (knowing nothing about this) if there is an issue with the approach to acquire data that it putting AI in a difficult position. This is akin to trying to train and AI to walk in the footsteps of a geophysicist, rather than making new footsteps for the AI. I guess I would extend this to radiology too since it seems to be the same issue. Let me give an example: People often mention that truck drivers are safe from automation because lots of last mile work is awkward and non-standard, requiring humans to navigate the bizarre atypical situations the truck encounter. Training an AI to handle all this is far harder than getting it to drive on a highway. What is often left out though is the idea that the infrastructure can/will change to accommodate the short comings of AI. This could look like warehouses having a "conductor" on staff who commandeers trucks for the tricky last bit of getting on the dock. Or perhaps preset radar and laser path guidance for the tight spots. I'd imagine most large volume shippers would build entire new warehouses just to accommodate automated trucks. A long time ago people noted that horses offered much more versatility than cars since roads were rocky and muddy. How do you make a car than can traverse the terrain a horse does? You don't, you pave all the roads.
- woeirua 5y ago
  Automatic interpretation has been a thing for decades and the promise of replacing a geoscientist completely is always just over the horizon. Even with DL. The new tools are better yes, but honestly I wouldn’t invest in this space. Conventional interpretation is dead in the US. All the geos got laid off. I’m going to call bullshit. No artificially generated well log is going to ever be better than a physically measured log.
smaddox 5y ago
Interesting perspective. What's your take on tools that use AI/ML to accelerate applying an interpretation over a full volume? For example: https://youtu.be/mLgKtmLY3cs https://youtu.be/mLgKtmLY3cs
- jofer 5y ago
  Bluntly, they're useless except for a few niche cases. Anything they're capable of picking up _isn't_ what you're actually concerned about as an interpeter. Sure they're good at picking reflectors in the shallow portion of the volume. No one cares about picking reflectors. That's not what you're doing as interpreter. A good example is the faults in that video. Sure, it did a great job at picking the tiny-but-well-imaged and mostly irrelevant faults. Those are the sort of things you'd almost always ignore because they don't matter it detail for most applications. The faults you care about are the ones that those methods more-or-less always fail to recognize. The significant faults are almost never imaged directly. Instead, they're inferred from deformed stratigraphy. It's definitely possible to automatically predict them using basic principles of structural geology, but it's exactly the type of thing that these sort of image-focused "automated interpretation" methods completely miss. Simply put: These methods miss the point. They produce something that looks good, but isn't relevant to the problems that you're trying to solve. No one cares about the well-imaged portion that these methods do a good job with. They automate the part that took 0 time to begin with.
  - deeviant 5y ago
    You seem extremely biased against AI in general, to the point where I very much doubt anybody would benefit from hearing your opinions on it.
    - jofer 5y ago
      I work in machine learning these days. I'm not biased against it -- it's literally my profession. I'm biased against a specific category of applications that are being heavily pushed by people who don't actually understand the problem they're purporting to solve. Put another way, the automated tools produce verifiably non-physical results nearly 100% of the time. The video there is a great example -- none of those faults could actually exist. They're close, but are all require violations of conservation of mass when compared to the horizons also picked by the model. Until "automated interpretation" tools start incorporating basic validation and physical constraints, they're just drawing lines. An interpretation is a _4D_ model. You _have_ to show how it developed through time -- it's part of the definition of "interpretation" and what distinguishes it from picking reflectors. I have strong opinions because I've spent decades working in this field on both sides. I've been an exploration geologist _and_ I've developed automated interpretation tools. I've also worked outside of the oil industry in the broader tech industry. I happen to think that structural geology is rather relevant to this problem. The law of conservation of mass still applies. You don't get to ignore it. All of these tools completely ignore it and product results that are physically impossible.
woeirua 5y ago
I have but one upvote to give, but as someone who worked as an interpreter and then moved onto the software side this is the problem that 99% of people don’t get. You can train a DL model to pick every horizon, but you can’t train to pick the horizon of interest. Same with faults. Let’s not even get started with poorly imaged areas.
- tachyonbeam 5y ago
  IMO a part of the problem here is that you have a misunderstanding on the part of deep learning people. They look at radiology, and they say "these people are just interpreting these pictures, we can train a deep learning model to do that better". Maybe there's a bit of arrogance too, this idea that deep learning can surpass human performance in every field with enough data. That may be the case, but not if you fundamentally misunderstood the problem that needs to be solved, and the data you need to solve radiology, for instance, isn't all in the image. Somewhat related: another area where DL seems to fail is anything that requires causal reasoning. The progress in robotics, for instance, hasn't been all that great. People will use DL for perception, but so far, using deep reinforcement learning for control only makes sense for really simple problems such as balancing your robot. When it comes to actually controlling what the robot is going to do next at a high level, people still write rules as programming code. In terms of radiology and causal reasoning, you could imagine that if you added extra information that allows the model to deduce "this can't be a cancerous tumor because we've performed this other test", you would want your software to make that diagnosis reliably. You can't have it misdiagnose when the tumor is on the right side of the ribcage 30% of the time because there wasn't enough training data where that other test was performed. Strange failure modes like that are unacceptable.
  - triska 5y ago
    Expanding on this, particularly regarding causal reasoning and rules, what I find especially puzzling is the desire to apply deep learning even in cases where the rules are explicitly known already, and the actual challenge would have been to reliably automate the application of the known, explicitly available rules. Such cases include for example the application of tax law: Yes, it is complex and maybe cannot be automated entirely. However, even today, computer programs handle a large percentage of the arising cases automatically in many governments, and these programs often already have automated mechanisms to delegate a certain percentage of (randomly chosen, maybe weighted according to certain criteria) cases to humans for manual assessment and quality checks, also a case of rule-based reasoning. Even fraud detection can likely be better automated by encoding and applying the rules that auditors already use to detect suspicious cases. The issue today is that all these rules are hard-coded, and the programs need to be rewritten and redeployed every time the laws change.
riedel 5y ago
I guess some animals are also good at seismic interpretation. For radiology we first need to beat pigeons: https://www.mentalfloss.com/article/71455/pigeons-good-radiologists-spotting-breast-cancer-scientists-say https://www.mentalfloss.com/article/71455/pigeons-good-radio... (there was a HN post I think on this) Actually mammography screening is done to my knowledge with out any background which could bias the decision. But here humans are fast anyways and even pidgins don't promise a relevant price cut. When complicated decisions need to be made . E.g on treatment we will have other problems with ai...
deleted 5y ago
[deleted]
mark_l_watson 5y ago
We used AI to analyze seismic data in the DARPA nuclear test monitoring system in the 1980s. I don’t think that it was considered to have anything but a fully automated system. That said, we had a large budget, and great teams of geophysicists and computer scientists, and 38 data collection stations around the world. In my experience, throwing money and resources at difficult problems usually gets those problems solved.
- jofer 5y ago
  Very different sort of seismic data, FWIW. You're referring to seismology and deciding whether something is a blast or a standard double-couple earthquake. That's fairly straightfoward, as it's mostly a matter of getting enough data from different angles. Lots of data processing and ambiguity, but in the end, you're inverting for a relatively simple mathematical model (the focal mechanism): https://en.wikipedia.org/wiki/Focal_mechanism https://en.wikipedia.org/wiki/Focal_mechanism I'm referring to reflection seismic, where you're fundamentally interpreting an image after all of the processing to make the image (i.e. basically making a mathematic lens) has already been done.
magicalhippo 5y ago
And maybe look for things that are not expected... My dad went to take a shoulder x-ray in preparation for a small bit of surgery. In the corner of the image the radiologist noticed something that didn't look right. He took more pictures, this time of the lungs, and quickly escalated the case. My dad had fought cancer, and it turned out the cancer had spread to his lungs. He had gone to regular checks every six months for several years at that point, but the original cancer was in a different part of his body. For a year prior he'd been short of breath, and they'd given him asthma medication... until he went to get that shoulder x-ray.
- chefkoch 5y ago
  As a cancer patient that feels like negligance.
  - magicalhippo 5y ago
    I agree. Essentially the same scenario has happened twice in my close circle since my dad. Sadly it seems treatment here is very much focused on the organ, not the patient. Hence why I tell people I come across who's diagnosed for the first time: learn where your cancer might spread to, and be very vigilant of changes/pain in those areas.
mikesabbagh 5y ago
>Many companies/etc keep promising to "revolutionize" It is all about the money baby If u fall and go to the ER, u get an xray to rule out a fracture. Many times the radiologist will read it after you leave the ER, yet he gets paid. If u think ML cant read a trauma xray, and offer a faster service, you are wrong!! The problem is who gets paid, and who is paying the malpractice insurance Check out in China, they have MRI machines with ML built in. U get the results before u get dressed!!
- catblast01 5y ago
  Who do you think I’d rather go after for malpractice? Someone that went to school for many years dedicated to medicine or the idiot stiffs behind a machine that can’t even spell the word “you”. That is in large part also what it is really about. Having said that I do ML research on cross-sectional neuroimaging, and basically everything you said is nonsense.
duxup 5y ago
I recall stories of IBM Watson's failures were focused around how they sold it as just dumping data into the machine and wonders coming out. Meanwhile actual implementation customers weren't ready / were frustrated with how much prepping the data was required, how time consuming it was, and in a lot of ways how each situation was a sort of research project of its own. It seems like any successful AI system will require the team working with the data to be experts in the actual data, or in this case experts in radiology ... and take a long time to really find out good outcomes / processes, if there are any to be found. Add the fact that the medical industry is super iterative / science takes a long time to really figure out ... that's a big job.
- visarga 5y ago
  There's no free ride, ML is data centric, you got to get close and personal with the data and its quality. That means 90% of our time is spent on data prep and evaluations. Getting to know the weak points of your dataset takes a lot of effort and custom tool building. Speaking from experience.
TuringNYC 5y ago
>> seismic interpretation here Strong disagree here. Lets put aside the math and focus on money. I dont know much about seismic interpretation, but I know a lot about Radiology+CV/ML. I was CTO+CoFounder for three years full time of a venture-backed Radiology+CV/ML startup. From what I can see, there is a huge conflict of interest w/r/t Radiology (and presumably any medical field) in the US. Radiologists make a lot of money -- and given their jobs are not tied to high CoL regions (as coders jobs are), they make even more on a CoL-adjust basis. Automating these jobs is the equivalent of killing the golden goose. Further, Radiologists standards of practice are driven partly by their board (The American Board of Radiology) and the supply of labor is also controlled by them (The American Board of Radiology) by way of limited residency spots to train new radiologists. So Radiologists (or any medical specialist) can essentially control the supply of labor, and control the standards of best practice, essentially allowing continued high salaries by way of artificial scarcity. WHY ON EARTH WOULD THEY WANT THEIR WORK AUTOMATED AWAY? My experience during my startup was lots of radiologists mildly interested in CV/ML/AI, interested in lots of discussions, interested in paid advisory roles, interested in paid CMO figurehead-positions, but mostly dragging their feet and hindering real progress, presumably because of the threat it posed. Every action item was hindered by a variety of players in the ecosystem. In fact, most of our R&D and testing was done overseas in a more friendly single payer system. I dont see how the US's fee-for-service model for Radiology is ever compatible with real progress to drive down costs or drive up volume/value. Not surprisingly, we made a decision to mostly move on. You can see Enlitic (a competitor) didnt do well either despite the star-studded executive team. Another competitor (to be unnamed) appears to have shifted from models to just licensing data. Same for IBM/Merge. Going back to seismic interpretation -- this cant be compared to Radiology from a follow-the-money perspective because seismic interpretation isnt effectively a cartel. Happy to speak offline if anyone is curious about specific experiences. DM me.
- grogenaut 5y ago
  Would you say this is similar to challenges in other fields such as law?
  - TuringNYC 5y ago
markus_zhang 5y ago
So it seems that instead of an image recognition algo we need to feed years of univ education into the AI.
audit 5y ago
I think you are onto something. The feedback from radiologists I get, about companies like path.ai and similar -- is that they are 'evolutionary' dead-ends (meaning that they need to exist to show that something should not be done that way). They lack innovativness not just in technology but also in the overall process. That is, they are missing innovation around overall context in which pathologists or radiologists work. Process includes steps (and steps of steps), information sources, information feedback loops, etc. Certainly, there is also a view, that the overall imaging process needs to evolve more (sort of like we need smart highways for safe self-driving cars)
derf_ 5y ago
I don't know anything about seismology, and I am going to put aside the money and focus on the math. > The features you're interested in are almost never what's clearly imaged -- instead, you're predicting what's in that unimaged, "mushy" area over there through fundamental laws of physics like conservation of mass and understanding of the larger regional context. Those are deeply difficult to incorporate in machine learning in practice. I was part of a university research lab over 15 years ago that was doing exactly this [1], with just regular old statistics (no AI/ML required). By modeling the variability of the stuff that you could see easily, you could produce priors strong enough to eek out the little bit of signal from the mush (which is basically what the actual radiologists do, which we know because they told us). It isn't a turn-key, black box solution like deep learning pretends to be. It takes a long time, it is highly dependent on getting good data sets, and years of labor goes into a basically bespoke solution for a single radiology problem, but the results agree with a human as closely as humans agree with each other. You also get the added bonus of understanding the relationships you are modeling when you are done. From university lab to clinically proven diagnosis tool is of course a longer road, and I have not been involved in these projects for a long time, but my point is that the math problem on its own is tractable. [1] http://midag.cs.unc.edu/ http://midag.cs.unc.edu/
timeu 5y ago
Funny, when I was young my father who was a geophyisicst (now retired) took me to seismological surveys, where we hit with a hammer on a plate to generate shock waves and measure velocity (mostly for building tunnels through montains). After a hard day of physical work, my father was drawing these lines at specific points of the recorded dataset (sorry I lack the proper terms and vocabulary because it was like 20 years ago) and there are some patterns you can use to identifiy those spots but most of it was based on his intuition and years of experience. Sometimes he let me do the interpretation and all I had was the those patterns, but many times he had to correct it because of what you describe (experience, terrain, environment, etc)

by jofer 5y ago

ludamad 5y ago

I wonder how often these projects truly need someone with on the ground experience guiding it, as the textbook tasks as you say are easy for even the humans

boleary-gl 5y ago

This is a really good point and example. I spent 10 years in mammography software, and I saw first hand how many outside factors can impact a physician's decision to biopsy or not a given artifact on an image. Things like family history, patient's history, cycle timing, age, weight, other risk factors all play a role in a smart radiologist making the right decision for that patient. And the pattern recognition on top of that is really hard - it's not just about the pattern you see at a particular spot in an image, it's the whole image in the context of what that looks like. Could ML get better over time with this? Sure...but they've been using CAD in mammography for decades and it still hasn't replaced radiologists at all. Could a model be made to include those over variables? Sure...but again the complexity of that kind of decision making is something that requires a lot more "intelligence" than any AI or ML system exhibits today and in my mind in the foreseeable future. Just collecting that data in a structured, consistent way is more challenging than people realize.

mumblemumble 5y ago
I see the same thing in natural language processing. A lot of important details come from outside the four corners of the document. Ironically, I often find myself in the unenviable position of being the machine learning person who's trying to convince people that machine learning is probably not a good fit for their problem. Or, worse, taking some more fuzzy-seeming position like, "Yes, we could solve this problem with machine learning, but it would actually cost more money than paying humans to do it by hand." Part of why I hate hate hate the term "AI" is because you simply can't call something artificial intelligence and then expect them to understand that it's not actually useful for doing anything that requires any kind of intelligence.
- mcguire 5y ago
  There's an old AI joke that all actual problems reduce to the artificial general intelligence problem---everything else, by definition, doesn't require intelligence.
  - mumblemumble 5y ago
    There's some truth to that. But I'd also argue that there's a tendency to try to bill every single kind of spinoff technology that the artificial intelligence community has produced as artificial intelligence. Which a bit like characterizing mixing up a glass of Tang as a kind of space exploration.
- 2sk21 5y ago
  You are absolutely correct. In fact, most NLP software ignores the formatting of documents which conveys a lot of information as well. For example, section headings must be treated differently from the text that makes up the body of a section. Its very hard to even determine section headings and then its hard to take advantage of them since the big transformer models simply accept a stream of unspecialized tokens.
  - deleted 5y ago
    [deleted]

shiftpgdn 5y ago

Surely you've seen the improvement over the last 5-6 years from machine learning in all the interpretation toolsets. The last place I worked we internally had a seismic inversion tool that blew all the commercial suites out the water. I'm currently contracting for an AI/ML service company currently that has a synthetic welllog tool that is can apparently beat the pants off actual well logging tools for a fraction of the cost (though I'm not a geologist or petrophysicist so I can't personally verify this.) I think the problem is more the media and advertisers likes to paint the picture of a magical AI tool which will instantly solve all your problems and do all the work instead of a fulcrum to make doing the actual work significantly easier.

Workaccount2 5y ago
I am wondering (knowing nothing about this) if there is an issue with the approach to acquire data that it putting AI in a difficult position. This is akin to trying to train and AI to walk in the footsteps of a geophysicist, rather than making new footsteps for the AI. I guess I would extend this to radiology too since it seems to be the same issue. Let me give an example: People often mention that truck drivers are safe from automation because lots of last mile work is awkward and non-standard, requiring humans to navigate the bizarre atypical situations the truck encounter. Training an AI to handle all this is far harder than getting it to drive on a highway. What is often left out though is the idea that the infrastructure can/will change to accommodate the short comings of AI. This could look like warehouses having a "conductor" on staff who commandeers trucks for the tricky last bit of getting on the dock. Or perhaps preset radar and laser path guidance for the tight spots. I'd imagine most large volume shippers would build entire new warehouses just to accommodate automated trucks. A long time ago people noted that horses offered much more versatility than cars since roads were rocky and muddy. How do you make a car than can traverse the terrain a horse does? You don't, you pave all the roads.
woeirua 5y ago
Automatic interpretation has been a thing for decades and the promise of replacing a geoscientist completely is always just over the horizon. Even with DL. The new tools are better yes, but honestly I wouldn’t invest in this space. Conventional interpretation is dead in the US. All the geos got laid off. I’m going to call bullshit. No artificially generated well log is going to ever be better than a physically measured log.

smaddox 5y ago

Interesting perspective. What's your take on tools that use AI/ML to accelerate applying an interpretation over a full volume? For example: https://youtu.be/mLgKtmLY3cs https://youtu.be/mLgKtmLY3cs

jofer 5y ago
Bluntly, they're useless except for a few niche cases. Anything they're capable of picking up _isn't_ what you're actually concerned about as an interpeter. Sure they're good at picking reflectors in the shallow portion of the volume. No one cares about picking reflectors. That's not what you're doing as interpreter. A good example is the faults in that video. Sure, it did a great job at picking the tiny-but-well-imaged and mostly irrelevant faults. Those are the sort of things you'd almost always ignore because they don't matter it detail for most applications. The faults you care about are the ones that those methods more-or-less always fail to recognize. The significant faults are almost never imaged directly. Instead, they're inferred from deformed stratigraphy. It's definitely possible to automatically predict them using basic principles of structural geology, but it's exactly the type of thing that these sort of image-focused "automated interpretation" methods completely miss. Simply put: These methods miss the point. They produce something that looks good, but isn't relevant to the problems that you're trying to solve. No one cares about the well-imaged portion that these methods do a good job with. They automate the part that took 0 time to begin with.
- deeviant 5y ago
  You seem extremely biased against AI in general, to the point where I very much doubt anybody would benefit from hearing your opinions on it.
  - jofer 5y ago
    I work in machine learning these days. I'm not biased against it -- it's literally my profession. I'm biased against a specific category of applications that are being heavily pushed by people who don't actually understand the problem they're purporting to solve. Put another way, the automated tools produce verifiably non-physical results nearly 100% of the time. The video there is a great example -- none of those faults could actually exist. They're close, but are all require violations of conservation of mass when compared to the horizons also picked by the model. Until "automated interpretation" tools start incorporating basic validation and physical constraints, they're just drawing lines. An interpretation is a _4D_ model. You _have_ to show how it developed through time -- it's part of the definition of "interpretation" and what distinguishes it from picking reflectors. I have strong opinions because I've spent decades working in this field on both sides. I've been an exploration geologist _and_ I've developed automated interpretation tools. I've also worked outside of the oil industry in the broader tech industry. I happen to think that structural geology is rather relevant to this problem. The law of conservation of mass still applies. You don't get to ignore it. All of these tools completely ignore it and product results that are physically impossible.

woeirua 5y ago

I have but one upvote to give, but as someone who worked as an interpreter and then moved onto the software side this is the problem that 99% of people don’t get. You can train a DL model to pick every horizon, but you can’t train to pick the horizon of interest. Same with faults. Let’s not even get started with poorly imaged areas.

tachyonbeam 5y ago
IMO a part of the problem here is that you have a misunderstanding on the part of deep learning people. They look at radiology, and they say "these people are just interpreting these pictures, we can train a deep learning model to do that better". Maybe there's a bit of arrogance too, this idea that deep learning can surpass human performance in every field with enough data. That may be the case, but not if you fundamentally misunderstood the problem that needs to be solved, and the data you need to solve radiology, for instance, isn't all in the image. Somewhat related: another area where DL seems to fail is anything that requires causal reasoning. The progress in robotics, for instance, hasn't been all that great. People will use DL for perception, but so far, using deep reinforcement learning for control only makes sense for really simple problems such as balancing your robot. When it comes to actually controlling what the robot is going to do next at a high level, people still write rules as programming code. In terms of radiology and causal reasoning, you could imagine that if you added extra information that allows the model to deduce "this can't be a cancerous tumor because we've performed this other test", you would want your software to make that diagnosis reliably. You can't have it misdiagnose when the tumor is on the right side of the ribcage 30% of the time because there wasn't enough training data where that other test was performed. Strange failure modes like that are unacceptable.
- triska 5y ago
  Expanding on this, particularly regarding causal reasoning and rules, what I find especially puzzling is the desire to apply deep learning even in cases where the rules are explicitly known already, and the actual challenge would have been to reliably automate the application of the known, explicitly available rules. Such cases include for example the application of tax law: Yes, it is complex and maybe cannot be automated entirely. However, even today, computer programs handle a large percentage of the arising cases automatically in many governments, and these programs often already have automated mechanisms to delegate a certain percentage of (randomly chosen, maybe weighted according to certain criteria) cases to humans for manual assessment and quality checks, also a case of rule-based reasoning. Even fraud detection can likely be better automated by encoding and applying the rules that auditors already use to detect suspicious cases. The issue today is that all these rules are hard-coded, and the programs need to be rewritten and redeployed every time the laws change.

riedel 5y ago

I guess some animals are also good at seismic interpretation. For radiology we first need to beat pigeons: https://www.mentalfloss.com/article/71455/pigeons-good-radiologists-spotting-breast-cancer-scientists-say https://www.mentalfloss.com/article/71455/pigeons-good-radio... (there was a HN post I think on this) Actually mammography screening is done to my knowledge with out any background which could bias the decision. But here humans are fast anyways and even pidgins don't promise a relevant price cut. When complicated decisions need to be made . E.g on treatment we will have other problems with ai...

deleted 5y ago

[deleted]

mark_l_watson 5y ago

We used AI to analyze seismic data in the DARPA nuclear test monitoring system in the 1980s. I don’t think that it was considered to have anything but a fully automated system. That said, we had a large budget, and great teams of geophysicists and computer scientists, and 38 data collection stations around the world. In my experience, throwing money and resources at difficult problems usually gets those problems solved.

jofer 5y ago
Very different sort of seismic data, FWIW. You're referring to seismology and deciding whether something is a blast or a standard double-couple earthquake. That's fairly straightfoward, as it's mostly a matter of getting enough data from different angles. Lots of data processing and ambiguity, but in the end, you're inverting for a relatively simple mathematical model (the focal mechanism): https://en.wikipedia.org/wiki/Focal_mechanism https://en.wikipedia.org/wiki/Focal_mechanism I'm referring to reflection seismic, where you're fundamentally interpreting an image after all of the processing to make the image (i.e. basically making a mathematic lens) has already been done.

magicalhippo 5y ago

And maybe look for things that are not expected... My dad went to take a shoulder x-ray in preparation for a small bit of surgery. In the corner of the image the radiologist noticed something that didn't look right. He took more pictures, this time of the lungs, and quickly escalated the case. My dad had fought cancer, and it turned out the cancer had spread to his lungs. He had gone to regular checks every six months for several years at that point, but the original cancer was in a different part of his body. For a year prior he'd been short of breath, and they'd given him asthma medication... until he went to get that shoulder x-ray.

chefkoch 5y ago
As a cancer patient that feels like negligance.
- magicalhippo 5y ago
  I agree. Essentially the same scenario has happened twice in my close circle since my dad. Sadly it seems treatment here is very much focused on the organ, not the patient. Hence why I tell people I come across who's diagnosed for the first time: learn where your cancer might spread to, and be very vigilant of changes/pain in those areas.

mikesabbagh 5y ago

>Many companies/etc keep promising to "revolutionize" It is all about the money baby If u fall and go to the ER, u get an xray to rule out a fracture. Many times the radiologist will read it after you leave the ER, yet he gets paid. If u think ML cant read a trauma xray, and offer a faster service, you are wrong!! The problem is who gets paid, and who is paying the malpractice insurance Check out in China, they have MRI machines with ML built in. U get the results before u get dressed!!

catblast01 5y ago
Who do you think I’d rather go after for malpractice? Someone that went to school for many years dedicated to medicine or the idiot stiffs behind a machine that can’t even spell the word “you”. That is in large part also what it is really about. Having said that I do ML research on cross-sectional neuroimaging, and basically everything you said is nonsense.

duxup 5y ago

I recall stories of IBM Watson's failures were focused around how they sold it as just dumping data into the machine and wonders coming out. Meanwhile actual implementation customers weren't ready / were frustrated with how much prepping the data was required, how time consuming it was, and in a lot of ways how each situation was a sort of research project of its own. It seems like any successful AI system will require the team working with the data to be experts in the actual data, or in this case experts in radiology ... and take a long time to really find out good outcomes / processes, if there are any to be found. Add the fact that the medical industry is super iterative / science takes a long time to really figure out ... that's a big job.

visarga 5y ago
There's no free ride, ML is data centric, you got to get close and personal with the data and its quality. That means 90% of our time is spent on data prep and evaluations. Getting to know the weak points of your dataset takes a lot of effort and custom tool building. Speaking from experience.

TuringNYC 5y ago

>> seismic interpretation here Strong disagree here. Lets put aside the math and focus on money. I dont know much about seismic interpretation, but I know a lot about Radiology+CV/ML. I was CTO+CoFounder for three years full time of a venture-backed Radiology+CV/ML startup. From what I can see, there is a huge conflict of interest w/r/t Radiology (and presumably any medical field) in the US. Radiologists make a lot of money -- and given their jobs are not tied to high CoL regions (as coders jobs are), they make even more on a CoL-adjust basis. Automating these jobs is the equivalent of killing the golden goose. Further, Radiologists standards of practice are driven partly by their board (The American Board of Radiology) and the supply of labor is also controlled by them (The American Board of Radiology) by way of limited residency spots to train new radiologists. So Radiologists (or any medical specialist) can essentially control the supply of labor, and control the standards of best practice, essentially allowing continued high salaries by way of artificial scarcity. WHY ON EARTH WOULD THEY WANT THEIR WORK AUTOMATED AWAY? My experience during my startup was lots of radiologists mildly interested in CV/ML/AI, interested in lots of discussions, interested in paid advisory roles, interested in paid CMO figurehead-positions, but mostly dragging their feet and hindering real progress, presumably because of the threat it posed. Every action item was hindered by a variety of players in the ecosystem. In fact, most of our R&D and testing was done overseas in a more friendly single payer system. I dont see how the US's fee-for-service model for Radiology is ever compatible with real progress to drive down costs or drive up volume/value. Not surprisingly, we made a decision to mostly move on. You can see Enlitic (a competitor) didnt do well either despite the star-studded executive team. Another competitor (to be unnamed) appears to have shifted from models to just licensing data. Same for IBM/Merge. Going back to seismic interpretation -- this cant be compared to Radiology from a follow-the-money perspective because seismic interpretation isnt effectively a cartel. Happy to speak offline if anyone is curious about specific experiences. DM me.

grogenaut 5y ago
Would you say this is similar to challenges in other fields such as law?
- TuringNYC 5y ago

markus_zhang 5y ago

So it seems that instead of an image recognition algo we need to feed years of univ education into the AI.

audit 5y ago

I think you are onto something. The feedback from radiologists I get, about companies like path.ai and similar -- is that they are 'evolutionary' dead-ends (meaning that they need to exist to show that something should not be done that way). They lack innovativness not just in technology but also in the overall process. That is, they are missing innovation around overall context in which pathologists or radiologists work. Process includes steps (and steps of steps), information sources, information feedback loops, etc. Certainly, there is also a view, that the overall imaging process needs to evolve more (sort of like we need smart highways for safe self-driving cars)

derf_ 5y ago

I don't know anything about seismology, and I am going to put aside the money and focus on the math. > The features you're interested in are almost never what's clearly imaged -- instead, you're predicting what's in that unimaged, "mushy" area over there through fundamental laws of physics like conservation of mass and understanding of the larger regional context. Those are deeply difficult to incorporate in machine learning in practice. I was part of a university research lab over 15 years ago that was doing exactly this [1], with just regular old statistics (no AI/ML required). By modeling the variability of the stuff that you could see easily, you could produce priors strong enough to eek out the little bit of signal from the mush (which is basically what the actual radiologists do, which we know because they told us). It isn't a turn-key, black box solution like deep learning pretends to be. It takes a long time, it is highly dependent on getting good data sets, and years of labor goes into a basically bespoke solution for a single radiology problem, but the results agree with a human as closely as humans agree with each other. You also get the added bonus of understanding the relationships you are modeling when you are done. From university lab to clinically proven diagnosis tool is of course a longer road, and I have not been involved in these projects for a long time, but my point is that the math problem on its own is tractable. [1] http://midag.cs.unc.edu/ http://midag.cs.unc.edu/

timeu 5y ago

Funny, when I was young my father who was a geophyisicst (now retired) took me to seismological surveys, where we hit with a hammer on a plate to generate shock waves and measure velocity (mostly for building tunnels through montains). After a hard day of physical work, my father was drawing these lines at specific points of the recorded dataset (sorry I lack the proper terms and vocabulary because it was like 20 years ago) and there are some patterns you can use to identifiy those spots but most of it was based on his intuition and years of experience. Sometimes he let me do the interpretation and all I had was the those patterns, but many times he had to correct it because of what you describe (experience, terrain, environment, etc)