6 ms·

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearl

If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. I have already nearly stopped using Google search for anything, in favor of GPT-4. GPT-4 has helped me very quickly prototype things that I normally would have had to spend hours researching. GPT-4 has also created custom curriculum for me to help me learn various things for which I have struggled to find good books/tutorials online. There will always be many areas in which a solid human intellect and well-honed human judgment is still useful, but much of the less critical work will yield to LLMs. If we call the difference between GPT-3 and GPT4 1x, then I would expect to see a 2x-5x improvement in LLM capability within the next few years just based on how much great work has recently gone into shrinking really big models so they can run on smaller hardware. LLMs are not digital human minds, they are simply very good information synthesizers. Information synthesis happens to be what most white collar professionals get paid to do with their brains.

hyperliner 3y ago
[dead]
doctorpangloss 3y ago
> If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. While I don't know you personally and I'm not talking about you specifically, "just" feeling very excited about something doesn't Make This About You, the prognostications don't actually help you Be a Part of This. This is the same energy as being really into COVID-19 (what was it called, "corona-scrolling?"), the gambling-fueled crypto boom, the retail stock trading booms. It's kind of adjacent to the fallacy Feelings that are More Strongly Held Are More Valid. It's like you could use these things as a barometer for the health of a social media channel. Anyway, I'm sure there's a name for this part of the hype cycle - tapping into how people want to be a part of the number one biggest, hottest trend by, at the very least, talking about it all the time, trying to compete against each other by sucking the most air out of all rooms through ever-greater hyperbole. One thing's for sure: there's some guy who found some massive low hanging fruit in some neural network designs by carefully choosing brackets in chained matrix multiplications. That's one guy, and he definitely Made It About Him in a way that makes sense. How many people on Earth have enough knowledge to see things like that, maybe 1,000-10,000? It's so exclusive, in a sense. I guess my point is, you're pretty far off the mark. In the interest of curiosity: Standard practice in medicine is to examine the patient before giving an opinion, a big roadblock for even the most competent of multi-modal models. Also, most people are biased in that they are intimately familiar with their own lived health problems, so they fill in tons of blanks, discounting the value of inquiry during a consult, whereas doctors may see you for less than a few hours a year, LLMs much less so.
- lumost 3y ago
  I strongly doubt that the LLM club will remain exclusive for very long, or that the innovation in this space was restricted to 10k people. At a minimum, I’ve already begun using gpt-4 as a consultant for my fathers medical issues. I can reasonably get confirmation/dis confirmation of whether a medical experts opinion is BS or not. As anyone who has dealt with the medical system in the US can attest - the majority of opinions you get from experts are either BS, outdated, or folks talking their own expertise. Asking the question to GPT-4 can help you decide when to talk to a different doctor, or try something new entirely.
  - whatshisface 3y ago
    This adds a whole new dimension to AI safety, it's one thing to dumb it down so it can't do bad stuff, but when you do that you're making it give worse medical advice to people who are going to take that advice.
    - staunton 3y ago
      It's possible most publicly available models (or at least "models the general public uses") will eventually refuse to give medical advice
- refulgentis 3y ago
  Apologies, I may misunderstand: there's a lot of Novel All Caps Concepts Even Though Caps Is For Well Worn Cliches. I Think Maybe You Think They're For Accusations? Insults? Big Concepts? For Handwaving At Big Concepts You've Divined Other People Are Thinking? The last paragraph caught my attention though you were a bit abstract _and_ obtuse, due to the structure being "brief aside for the curious: LLMs don't have bodies" My job and existence is centered around enabling doctors via LLMs and it absolutely is astounding to and for them. Funnily enough, it's the simplest thing: citations and easy UI to reach them. They perceive it as quick literature review on demand, as far as I can tell, that means they get to skip about 5 minutes across 4-8 articles checking links for relevancy. I don't think OP was proposing that LLMs will replace doctors as you seem to think. I can't identify anything in their post suggesting that.
zmmmmm 3y ago
> If we call the difference between GPT-3 and GPT4 1x, then I would expect to see a 2x-5x improvement in LLM capability within the next few years it's interesting this is probably the assumption of many people leading to the current hype cycle. But in reality it's not how progress tends to happen which is a much more punctuated equilibrium type phenomenon. My guess is it's actually quite likely that most of the progress will be made on maturing things and underlying capability will plateau.
nradov 3y ago
That is not obvious, and at least in the case of physicians is unlikely to come to pass. They don't just take a written list of signs and symptoms as input, and produce a diagnosis as output. Often the evidence they use to make a diagnosis isn't even written down anywhere. And for most primary care cases, the diagnosis is only a minor part of the workflow. The hard part is in working with the patient to make a practical treatment plan, and then course correcting as needed based on results. This is as much art as science, and relies on tacit knowledge that no LLM can access. In order to automate most medical care we'll need multiple breakthroughs that go way beyond LLMs. Linear improvements in LLMs won't get us there. LLMs do have potential to improve other areas of clinical practice such as charting.
dogmatism 3y ago
Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR systems that don't interchange) and summarize or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" The NEJM CPC's are after the hard work has already been done to get the data to present the case
- arcticfox 3y ago
  > Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant, although I didn't. I told it to roleplay a potential follow-up: > "Patient with fatigue, weight loss, mild fever. CBC shows anemia, elevated WBC. CMP reveals elevated liver enzymes. TSH normal. CXR clear. Consider ESR, CRP, ANA, and abdominal ultrasound. Possible infection or autoimmune condition. Referral to a hematologist may be warranted."
  - dogmatism 3y ago
    I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part
    - Sai_ 3y ago
      You’re being dogmatic about your, what seems to me, objections to LLM skills. Making data ingestible is just laborious and unappetising to an engineer. So hand it over to a data entry operator who’ll do this happily for a wage. I’m not sure why the issue of making data available to an LLM is a failure of the LLM. Btw, OpenAI’s LLM can very well parse images, read handwriting, fix sloppy spellings, and extract objects from unstructured data.
intended 3y ago
I’m very glad you set the argument this way, since it very neatly sets the stage for the core disagreement to be argued. > Simply very good information synthesizers > Simple very good syntactic synthesizers I argue that theorists tend to adhere to the former, and practitioners the latter. Consider prototyping: For an expert, it costs nothing (or near nothing) to check the output. You even start better, because you know how to ask the right questions to the tool in the first place. LLMs dont get the semantics right, they get syntactic correlations right. When a subject is common place (literature), the correlations are good enough to get the reasoning right. There is an IMPRESSIVE amount that gets done with just this. However, expecting to get reasoning right is the bridge too far. Wrong expectations of tools, result in bad management projects and failures. You can test this out right now. IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. I did, I know why it can’t work as a result. The simple question to ask is “whats your error rate, and your hallucination rate?” LLMs are always ‘hallucinating’. I do hope, that the difference eventually becomes academic. That there is so much training material, that correlation is equal to reasoning. However, it will not be reasoning/semantic prediction. Finally - the real world work has properties that are emergent. You can read every spec sheet you like, but when you start assembling components together, they are going to do weird things.
- IanCal 3y ago
  > IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. What do you mean? I've had GPT-4 (3.5 I think too) talking to itself, setup as a planner and someone critiquing a plan, the end result is much better. > However, expecting to get reasoning right is the bridge too far. Hmm, how do you position othello-gpt in this? It builds a world model and makes moves based on that, so it's not just correlation of input (x,y,z typically followed by A so respond A).
  - intended 3y ago
    1) Set up multiple prompts to work together and make a web page. See how far that goes, before it’s nonsense talking to nonsense. 2) Production is proof. Get a working LLM based reasoning system that doesnt need its hand held. The process of failing is educational enough. For the record, I would love it if these things work.
RandomLensman 3y ago
The real mental work is when your client, patient, etc. struggles to articulate the problems, data is wrong or mislabeled, etc.

hyperliner 3y ago

[dead]

doctorpangloss 3y ago

> If it's not already obvious, LLMs are going to be doing most of the mental work currently performed by doctors, lawyers, accountants, etc. While I don't know you personally and I'm not talking about you specifically, "just" feeling very excited about something doesn't Make This About You, the prognostications don't actually help you Be a Part of This. This is the same energy as being really into COVID-19 (what was it called, "corona-scrolling?"), the gambling-fueled crypto boom, the retail stock trading booms. It's kind of adjacent to the fallacy Feelings that are More Strongly Held Are More Valid. It's like you could use these things as a barometer for the health of a social media channel. Anyway, I'm sure there's a name for this part of the hype cycle - tapping into how people want to be a part of the number one biggest, hottest trend by, at the very least, talking about it all the time, trying to compete against each other by sucking the most air out of all rooms through ever-greater hyperbole. One thing's for sure: there's some guy who found some massive low hanging fruit in some neural network designs by carefully choosing brackets in chained matrix multiplications. That's one guy, and he definitely Made It About Him in a way that makes sense. How many people on Earth have enough knowledge to see things like that, maybe 1,000-10,000? It's so exclusive, in a sense. I guess my point is, you're pretty far off the mark. In the interest of curiosity: Standard practice in medicine is to examine the patient before giving an opinion, a big roadblock for even the most competent of multi-modal models. Also, most people are biased in that they are intimately familiar with their own lived health problems, so they fill in tons of blanks, discounting the value of inquiry during a consult, whereas doctors may see you for less than a few hours a year, LLMs much less so.

lumost 3y ago
I strongly doubt that the LLM club will remain exclusive for very long, or that the innovation in this space was restricted to 10k people. At a minimum, I’ve already begun using gpt-4 as a consultant for my fathers medical issues. I can reasonably get confirmation/dis confirmation of whether a medical experts opinion is BS or not. As anyone who has dealt with the medical system in the US can attest - the majority of opinions you get from experts are either BS, outdated, or folks talking their own expertise. Asking the question to GPT-4 can help you decide when to talk to a different doctor, or try something new entirely.
- whatshisface 3y ago
  This adds a whole new dimension to AI safety, it's one thing to dumb it down so it can't do bad stuff, but when you do that you're making it give worse medical advice to people who are going to take that advice.
  - staunton 3y ago
    It's possible most publicly available models (or at least "models the general public uses") will eventually refuse to give medical advice
refulgentis 3y ago
Apologies, I may misunderstand: there's a lot of Novel All Caps Concepts Even Though Caps Is For Well Worn Cliches. I Think Maybe You Think They're For Accusations? Insults? Big Concepts? For Handwaving At Big Concepts You've Divined Other People Are Thinking? The last paragraph caught my attention though you were a bit abstract _and_ obtuse, due to the structure being "brief aside for the curious: LLMs don't have bodies" My job and existence is centered around enabling doctors via LLMs and it absolutely is astounding to and for them. Funnily enough, it's the simplest thing: citations and easy UI to reach them. They perceive it as quick literature review on demand, as far as I can tell, that means they get to skip about 5 minutes across 4-8 articles checking links for relevancy. I don't think OP was proposing that LLMs will replace doctors as you seem to think. I can't identify anything in their post suggesting that.

zmmmmm 3y ago

> If we call the difference between GPT-3 and GPT4 1x, then I would expect to see a 2x-5x improvement in LLM capability within the next few years it's interesting this is probably the assumption of many people leading to the current hype cycle. But in reality it's not how progress tends to happen which is a much more punctuated equilibrium type phenomenon. My guess is it's actually quite likely that most of the progress will be made on maturing things and underlying capability will plateau.

nradov 3y ago

That is not obvious, and at least in the case of physicians is unlikely to come to pass. They don't just take a written list of signs and symptoms as input, and produce a diagnosis as output. Often the evidence they use to make a diagnosis isn't even written down anywhere. And for most primary care cases, the diagnosis is only a minor part of the workflow. The hard part is in working with the patient to make a practical treatment plan, and then course correcting as needed based on results. This is as much art as science, and relies on tacit knowledge that no LLM can access. In order to automate most medical care we'll need multiple breakthroughs that go way beyond LLMs. Linear improvements in LLMs won't get us there. LLMs do have potential to improve other areas of clinical practice such as charting.

dogmatism 3y ago

Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit and for a follow up visit, read the prior visit's note and orders, search/track down all the test results (may be at more than one institution with multiple EHR systems that don't interchange) and summarize or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" The NEJM CPC's are after the hard work has already been done to get the data to present the case

arcticfox 3y ago
> Call me when an LLM can ingest a pts medical record from 3 different sources in multiple formats ranging from structured data to free form text stored (I shit you not) as TIFF files, extract the actual information, and present it for review prior to a visit LLMs were born to do this. > or even just be able to take verbal orders like it used to be: "get a CBC, CMP, TSH, and CXR" GPT-4 of course knew what that meant, although I didn't. I told it to roleplay a potential follow-up: > "Patient with fatigue, weight loss, mild fever. CBC shows anemia, elevated WBC. CMP reveals elevated liver enzymes. TSH normal. CXR clear. Consider ESR, CRP, ANA, and abdominal ultrasound. Possible infection or autoimmune condition. Referral to a hematologist may be warranted."
- dogmatism 3y ago
  I know an LLM can do this when given the data in an ingestible form it's the taking of data from disparate sources including literal paper faxes that's the hard part
  - Sai_ 3y ago
    You’re being dogmatic about your, what seems to me, objections to LLM skills. Making data ingestible is just laborious and unappetising to an engineer. So hand it over to a data entry operator who’ll do this happily for a wage. I’m not sure why the issue of making data available to an LLM is a failure of the LLM. Btw, OpenAI’s LLM can very well parse images, read handwriting, fix sloppy spellings, and extract objects from unstructured data.

intended 3y ago

I’m very glad you set the argument this way, since it very neatly sets the stage for the core disagreement to be argued. > Simply very good information synthesizers > Simple very good syntactic synthesizers I argue that theorists tend to adhere to the former, and practitioners the latter. Consider prototyping: For an expert, it costs nothing (or near nothing) to check the output. You even start better, because you know how to ask the right questions to the tool in the first place. LLMs dont get the semantics right, they get syntactic correlations right. When a subject is common place (literature), the correlations are good enough to get the reasoning right. There is an IMPRESSIVE amount that gets done with just this. However, expecting to get reasoning right is the bridge too far. Wrong expectations of tools, result in bad management projects and failures. You can test this out right now. IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. I did, I know why it can’t work as a result. The simple question to ask is “whats your error rate, and your hallucination rate?” LLMs are always ‘hallucinating’. I do hope, that the difference eventually becomes academic. That there is so much training material, that correlation is equal to reasoning. However, it will not be reasoning/semantic prediction. Finally - the real world work has properties that are emergent. You can read every spec sheet you like, but when you start assembling components together, they are going to do weird things.

IanCal 3y ago
> IF an LLM is able to do information synthesis, it is trivial to set up parallel prompts to work as teams. Try it out. What do you mean? I've had GPT-4 (3.5 I think too) talking to itself, setup as a planner and someone critiquing a plan, the end result is much better. > However, expecting to get reasoning right is the bridge too far. Hmm, how do you position othello-gpt in this? It builds a world model and makes moves based on that, so it's not just correlation of input (x,y,z typically followed by A so respond A).
- intended 3y ago
  1) Set up multiple prompts to work together and make a web page. See how far that goes, before it’s nonsense talking to nonsense. 2) Production is proof. Get a working LLM based reasoning system that doesnt need its hand held. The process of failing is educational enough. For the record, I would love it if these things work.

RandomLensman 3y ago

The real mental work is when your client, patient, etc. struggles to articulate the problems, data is wrong or mislabeled, etc.