3 ms·

You give 3 data analysts the same question on a typical enterprise database. How many correct answers do you get? How many different answers? Based on anecdota

by zurfer 2y ago

You give 3 data analysts the same question on a typical enterprise database. How many correct answers do you get? How many different answers? Based on anecdotal experience (we should make a study) LLMs don't make more errors than human analysts IF given the same information.

latexr 2y ago
What a tired argument. Is this what the field is reduced to? Computers are supposed to be fast, deterministic, and correct. It is profoundly disappointing that we’re regressing to a system that is as mediocre as the average bullshitter and laud it as an achievement. The thing with humans is that you can build trust. I know exactly who to ask if I have a question about music, or medicine, or a myriad of other topics. I know those people will know the answers and be able to assess their level of confidence in them. If they don’t know, they can figure it out. If they are mistaken, they’ll come back and correct themselves without me having to do anything. Comparing LLMs to random humans is the wrong methodology. Of course, Upton Sinclair had a point so I don’t expect to convince someone who is monetarily invested in having this broken assumption succeed.
- 7thpower 2y ago
  Why are you pretending this is a binary outcome? There are plenty of use cases where this is a great starting place to be able to jump in and start to ask questions. I have worked on a variety of acquisitions where I didn’t need to have 100% certainty, I just needed a starting place to make sense of some bizarre home grown financials and something like this would’ve been great to be able to quickly probe and come back with thoughtful questions. Instead I spent my time either tediously figuring out a schema or waiting for someone in finance to come back with ad-hoc analysis that also had a bunch of errors.
  - latexr 2y ago
    > Why are you pretending this is a binary outcome? You seem confused. My argument is not at all concerned with outcomes and is not binary in the slightest. My point is clearly spelled out: > Comparing LLMs to random humans is the wrong methodology. I’m not saying LLMs are never useful or anything of the sort. What I am saying is that defending LLMs on the basis of “some random undefined human would do the same” is a poor argument which shows a deep misunderstanding of human collaboration.
- jacobr1 2y ago
  Is not the answer here the same? Build the trust in the AI systems. Benchmarks help and are a start. Consistent results over time are another measure. We'll learn over the next few years which models, or which companies developing model can be trusted for certain sets of tasks. If these exceed alternatives, and have consistency over a margin I can reason about, I'll make use of them, same as any other tool.
  - latexr 2y ago
    I can ask the exact same question of an LLM multiple times and get different answers with the same degree of confidence. Hard to trust that, and also hard to fix. Which wouldn’t be so problematic if people didn’t just turn off their brains when interacting with them. Either way, everything you’re suggesting are possibilities for the future, which may or may not pan out. The bad comparisons to humans are happening today.

by zurfer 2y ago

latexr 2y ago

What a tired argument. Is this what the field is reduced to? Computers are supposed to be fast, deterministic, and correct. It is profoundly disappointing that we’re regressing to a system that is as mediocre as the average bullshitter and laud it as an achievement. The thing with humans is that you can build trust. I know exactly who to ask if I have a question about music, or medicine, or a myriad of other topics. I know those people will know the answers and be able to assess their level of confidence in them. If they don’t know, they can figure it out. If they are mistaken, they’ll come back and correct themselves without me having to do anything. Comparing LLMs to random humans is the wrong methodology. Of course, Upton Sinclair had a point so I don’t expect to convince someone who is monetarily invested in having this broken assumption succeed.

7thpower 2y ago
Why are you pretending this is a binary outcome? There are plenty of use cases where this is a great starting place to be able to jump in and start to ask questions. I have worked on a variety of acquisitions where I didn’t need to have 100% certainty, I just needed a starting place to make sense of some bizarre home grown financials and something like this would’ve been great to be able to quickly probe and come back with thoughtful questions. Instead I spent my time either tediously figuring out a schema or waiting for someone in finance to come back with ad-hoc analysis that also had a bunch of errors.
- latexr 2y ago
  > Why are you pretending this is a binary outcome? You seem confused. My argument is not at all concerned with outcomes and is not binary in the slightest. My point is clearly spelled out: > Comparing LLMs to random humans is the wrong methodology. I’m not saying LLMs are never useful or anything of the sort. What I am saying is that defending LLMs on the basis of “some random undefined human would do the same” is a poor argument which shows a deep misunderstanding of human collaboration.
jacobr1 2y ago
Is not the answer here the same? Build the trust in the AI systems. Benchmarks help and are a start. Consistent results over time are another measure. We'll learn over the next few years which models, or which companies developing model can be trusted for certain sets of tasks. If these exceed alternatives, and have consistency over a margin I can reason about, I'll make use of them, same as any other tool.
- latexr 2y ago
  I can ask the exact same question of an LLM multiple times and get different answers with the same degree of confidence. Hard to trust that, and also hard to fix. Which wouldn’t be so problematic if people didn’t just turn off their brains when interacting with them. Either way, everything you’re suggesting are possibilities for the future, which may or may not pan out. The bad comparisons to humans are happening today.