4 ms·

Wrong. One anecdote is one data point. If one person witnesses a murder that is a data point. The court doesn't just dismiss the data point because they need a

Wrong. One anecdote is one data point. If one person witnesses a murder that is a data point. The court doesn't just dismiss the data point because they need a sample size of 1000 witnesses for any statement to be meaningful. This article here says something meaningful about AI and it also says something damning about doctors. Also there have been studies: https://www.sciencedirect.com/science/article/abs/pii/S019606442300642X?dgcid=author#fig1 https://www.sciencedirect.com/science/article/abs/pii/S01960...

rvz 3y ago
Wrong again. This is a case of reliability and just because there is one anecdote of the 'correct' diagnosis, it hardly means it is reliable overall. In fact you wouldn't use these AI systems and trust their judgement at face value without the involvement of human doctors because by default, you don't trust their outputs. This is on top of the fact that these AI systems are unexplainable black boxes which cannot transparently reason and explain their own decisions. It also appears that this 'research' study is quite frankly a surface level one with an extremely low sample size of 30 rather than the 1,000 sample size you suggested in your example. So I can only dismiss that study since it is not sufficient enough to result in conclusive evidence of reliability.
- corethree 3y ago
  You are dismissing the study not because you need 1000 samples. You are dismissing it because you're biased. 30 witnesses to a murder is highly reliable. Any judge or court who denies that is biased. 17 doctors who couldn't diagnose a simple issue for 10 years is a demonstration of total unreliability.
  - rvz 3y ago
    > You are dismissing the study not because you need 1000 samples. You are dismissing it because you're biased. Biased to what? You gave ONE study with a very small sample size of 30, which is a far cry of the 1,000 sample size you suggested? It is weak evidence and can be dismissed. That hardly means that the next study which another group attempts to reproduce is conclusive 'evidence' that it is more reliable than doctors. > 17 doctors who couldn't diagnose a simple issue for 10 years is a demonstration of total unreliability. So this ONE anecdote immediately means that ChatGPT is far more reliable than human doctors and can be used for medical advice? How many cases are there when ChatGPT gets its response incorrect out of 1,000 cases?
    - corethree 3y ago
      >Biased to what? You gave ONE study with a very small sample size of 30, which is a far cry of the 1,000 sample size you suggested? Go read what I said again because you completely misinterpreted. I said a 1000 sample size isn't needed by a judge. A judge can convict a murderer off of 1 witness. >That hardly means that the next study which another group attempts to reproduce is conclusive 'evidence' that it is more reliable than doctors. This is what gets me. Do some people really go through reality only deeming things that have been under the lens of a scientific study to be real? Is the sky blue? Are you alive? Do you need a scientific study to prove it to you? Are you so enamored with science that the only way you can speculate or debate a topic is if the scientific method has been applied rigorously? Bro, just use chatGPT a couple of times. You'll find it's reliability is roughly above 50% for queries and questions no other machine on the face of the earth can answer. It beats humans consistently at speed and in many instances where creativity is required. >So this ONE anecdote immediately means that ChatGPT is far more reliable than human doctors and can be used for medical advice? How many cases are there when ChatGPT gets its response incorrect out of 1,000 cases? No but a sample size of 30 and beating out 17 doctors spending a decade analyzing one child in mere seconds is reasonable evidence that chatGPT is the precursor to the thing that will eat the entire medical diagnosis industry alive. Right now the machine is better only in certain instances. The delta from zero to now signifies another delta in the impending future.
      - rvz 3y ago
        > Go read what I said again because you completely misinterpreted. I said a 1000 sample size isn't needed by a judge. A judge can convict a murderer off of 1 witness. That is not how this works. A sample size of 1 or 30 in ONE study in the medical industry is hardly convincing to physicians and other medical professions to conclusively prove that an experiment works reliably for others. Thus, you have been comparing apples to oranges all this time which research in the clinical practice does not apply to whatever you are trying to compare with in the legal practice. You need multiple, peer-reviewed and reproducible studies which verifies the claims set in the research paper and a larger sample size is favoured over an extremely small one. Not just 'past cases' or 'case studies'. That only shows that someone has done it within those disclosed limitations and in this case, it does NOT conclusively answer the reliability question. Henceforth, it scratches the surface and can be dismissed as insufficient evidence that does not prove reliability. > This is what gets me. Do some people really go through reality only deeming things that have been under the lens of a scientific study to be real? Is the sky blue? Are you alive? Do you need a scientific study to prove it to you? Are you so enamored with science that the only way you can speculate or debate a topic is if the scientific method has been applied rigorously? As I have already explained you have just confused yourself with a nonsensical comparison right from the start. Had you actually read the paper you pasted instead of the title, you would have realize the non-determinstic nature of LLMs would mean that it is a limitation that would require further studies and methodologies (including larger more sample sizes) conclusively prove that ChatGPT is far more reliable than human doctors - even enough to fully replace them. So framing my point to believe as if I am attempting to prove a tautology isn't what this is. LLMs are non-deterministic black-box models that requires rigorous evaluations and experiments and suggesting that one surface-level study with a limited sample size as proof that LLMs like ChatGPT are more reliable than doctors is beyond ludicrous, especially using that shallow research paper as an example. > No but a sample size of 30 and beating out 17 doctors spending a decade analyzing one child in mere seconds is reasonable evidence that chatGPT is the precursor to the thing that will eat the entire medical diagnosis industry alive. Again. One example isn't sufficient evidence to result into jumping to such wild conclusions of 'eating the entire medical diagnosis industry alive'. You have already admitted, it is not even a reliable or transparent enough to be used for medical advice especially with my question being unanswered. I'll be more specific: For every case it gives the correct output on a diagnosis, how many cases are there when ChatGPT gets its response incorrect out of 1,000 cases? ONE case study which scratches the surface with a product that is an opaque non-deterministic tool, is something that physicians and clinicians are extremely skeptical on as fully replacement which not only it cannot fully replace them, it will always require a human doctor to check its diagnosis regardless, especially when something goes wrong.

rvz 3y ago

Wrong again. This is a case of reliability and just because there is one anecdote of the 'correct' diagnosis, it hardly means it is reliable overall. In fact you wouldn't use these AI systems and trust their judgement at face value without the involvement of human doctors because by default, you don't trust their outputs. This is on top of the fact that these AI systems are unexplainable black boxes which cannot transparently reason and explain their own decisions. It also appears that this 'research' study is quite frankly a surface level one with an extremely low sample size of 30 rather than the 1,000 sample size you suggested in your example. So I can only dismiss that study since it is not sufficient enough to result in conclusive evidence of reliability.

corethree 3y ago
You are dismissing the study not because you need 1000 samples. You are dismissing it because you're biased. 30 witnesses to a murder is highly reliable. Any judge or court who denies that is biased. 17 doctors who couldn't diagnose a simple issue for 10 years is a demonstration of total unreliability.
- rvz 3y ago
  > You are dismissing the study not because you need 1000 samples. You are dismissing it because you're biased. Biased to what? You gave ONE study with a very small sample size of 30, which is a far cry of the 1,000 sample size you suggested? It is weak evidence and can be dismissed. That hardly means that the next study which another group attempts to reproduce is conclusive 'evidence' that it is more reliable than doctors. > 17 doctors who couldn't diagnose a simple issue for 10 years is a demonstration of total unreliability. So this ONE anecdote immediately means that ChatGPT is far more reliable than human doctors and can be used for medical advice? How many cases are there when ChatGPT gets its response incorrect out of 1,000 cases?
  - corethree 3y ago
    >Biased to what? You gave ONE study with a very small sample size of 30, which is a far cry of the 1,000 sample size you suggested? Go read what I said again because you completely misinterpreted. I said a 1000 sample size isn't needed by a judge. A judge can convict a murderer off of 1 witness. >That hardly means that the next study which another group attempts to reproduce is conclusive 'evidence' that it is more reliable than doctors. This is what gets me. Do some people really go through reality only deeming things that have been under the lens of a scientific study to be real? Is the sky blue? Are you alive? Do you need a scientific study to prove it to you? Are you so enamored with science that the only way you can speculate or debate a topic is if the scientific method has been applied rigorously? Bro, just use chatGPT a couple of times. You'll find it's reliability is roughly above 50% for queries and questions no other machine on the face of the earth can answer. It beats humans consistently at speed and in many instances where creativity is required. >So this ONE anecdote immediately means that ChatGPT is far more reliable than human doctors and can be used for medical advice? How many cases are there when ChatGPT gets its response incorrect out of 1,000 cases? No but a sample size of 30 and beating out 17 doctors spending a decade analyzing one child in mere seconds is reasonable evidence that chatGPT is the precursor to the thing that will eat the entire medical diagnosis industry alive. Right now the machine is better only in certain instances. The delta from zero to now signifies another delta in the impending future.
    - rvz 3y ago
      > Go read what I said again because you completely misinterpreted. I said a 1000 sample size isn't needed by a judge. A judge can convict a murderer off of 1 witness. That is not how this works. A sample size of 1 or 30 in ONE study in the medical industry is hardly convincing to physicians and other medical professions to conclusively prove that an experiment works reliably for others. Thus, you have been comparing apples to oranges all this time which research in the clinical practice does not apply to whatever you are trying to compare with in the legal practice. You need multiple, peer-reviewed and reproducible studies which verifies the claims set in the research paper and a larger sample size is favoured over an extremely small one. Not just 'past cases' or 'case studies'. That only shows that someone has done it within those disclosed limitations and in this case, it does NOT conclusively answer the reliability question. Henceforth, it scratches the surface and can be dismissed as insufficient evidence that does not prove reliability. > This is what gets me. Do some people really go through reality only deeming things that have been under the lens of a scientific study to be real? Is the sky blue? Are you alive? Do you need a scientific study to prove it to you? Are you so enamored with science that the only way you can speculate or debate a topic is if the scientific method has been applied rigorously? As I have already explained you have just confused yourself with a nonsensical comparison right from the start. Had you actually read the paper you pasted instead of the title, you would have realize the non-determinstic nature of LLMs would mean that it is a limitation that would require further studies and methodologies (including larger more sample sizes) conclusively prove that ChatGPT is far more reliable than human doctors - even enough to fully replace them. So framing my point to believe as if I am attempting to prove a tautology isn't what this is. LLMs are non-deterministic black-box models that requires rigorous evaluations and experiments and suggesting that one surface-level study with a limited sample size as proof that LLMs like ChatGPT are more reliable than doctors is beyond ludicrous, especially using that shallow research paper as an example. > No but a sample size of 30 and beating out 17 doctors spending a decade analyzing one child in mere seconds is reasonable evidence that chatGPT is the precursor to the thing that will eat the entire medical diagnosis industry alive. Again. One example isn't sufficient evidence to result into jumping to such wild conclusions of 'eating the entire medical diagnosis industry alive'. You have already admitted, it is not even a reliable or transparent enough to be used for medical advice especially with my question being unanswered. I'll be more specific: For every case it gives the correct output on a diagnosis, how many cases are there when ChatGPT gets its response incorrect out of 1,000 cases? ONE case study which scratches the surface with a product that is an opaque non-deterministic tool, is something that physicians and clinicians are extremely skeptical on as fully replacement which not only it cannot fully replace them, it will always require a human doctor to check its diagnosis regardless, especially when something goes wrong.