6 ms·

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator

by brentm 1y ago

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great but also there are a lot of questions on if this technology will ultimately lead to the productivity gains that many think are guaranteed in next few years. There is a non zero chance it ends up actually hurting productivity because of all the time wasted trying to get it to produce magic results.

andrewmutz 1y ago
What we are seeing with our customers is that LLM errors are a very manageable problem. End users adapt pretty quickly to the idea that AI systems aren't perfect. In many cases AI products are doing tasks that used to be done by humans and these humans were making mistakes too, so the end user is used to the idea that the task will get accomplished with some non-zero error rate. You just need to build your products in a manner where the user has the ability to easily double check the results whenever they like. Then they can audit as they see fit, in order to get used to the accuracy level and to apply additional scrutiny to cases that are very important to their business.
- brentm 1y ago
  Yea I just think the true unlock in productivity will come from not requiring a human in the loop.
- insane_dreamer 1y ago
  > the user has the ability to easily double check the results whenever they like if the user is able to so easily verify that the results are accurate, that means that they are able to generate accurate results through other means, which means they don't need the LLM in the first place
  - tom_m 1y ago
    True, but people are so enamored by what they can do that they rarely seem to think about that. We will over spend on AI purely because we think it's cool.
  - Ukv 1y ago
    I don't think that's necessarily true - many tasks are difficult to solve but easy to verify. If I ask "place names that end with um", or "good ideas for a birthday party" I can pretty much verify the answer just by reading it. In other cases, clicking through to check that a linked source supports a claim is easier than researching to find and summarize the source in the first place would be.
    - insane_dreamer 1y ago
      > I can pretty much verify the answer just by reading it Only if you have domain knowledge. In both of your examples, you have to 1) know geography to determine whether "Técolum" and "Tolum" are indeed city names or just made up; and 2) know what might be acceptable ("good idea") or not at a birthday party. Yes, it'll probably save you some time, but it's not orders of magnitude. > In other cases, clicking through to check that a linked source supports a claim this supposes that the AI provides a link for every fact. Google search + Gemini does, but most LLM interfaces don't. secondly, if I have to click through every link and read through the source to determine whether details of a "summary" are correct or not, that really does not save me much time from conducting a search and looking through the linked sources myself Anecdote from a couple of weeks ago. My wife's professor sent her 5 citations and summaries related to a medical research project. She didn't say they were LLM generated, but it was obvious (to me, not my wife) they were, by the formatting alone. None of the 5 papers existed as cited. My wife was confused, spent a lot of time trying to figure out what was wrong and why she couldn't find any of the papers. A Google Scholar search turned up 2 of the papers which were close enough to the citation to be the ones with some logical thinking, but the other 3 were not even matchable. In the end, the time spent trying to sort out valid vs invalid citations, and find valid replacements, was significantly greater than just doing the search and looking through the abstracts. PS: LLMs are fine for information that can be "fuzzy": suggest places to go on vacation in September, plan a birthday party, etc. But I wouldn't consider that to be a "revolutionary" advance.
tveita 1y ago
> If a calculator works great 99% of the time you could not use that calculator to build a bridge. We know for certain that certified lawyers have committed malpractice by using ChatGPT, in part because the made-up citations are relatively easy to spot. Malpractice by engineers might take a little more time to discover.
- bagacrap 1y ago
  Engineers' work is also externally verifiable, e.g. by unit tests for software, but I'm assuming by other sorts of automated protocols for civil engineering. I would hope a bridge is not built without triple checking the various outcomes.
  - yencabulator 1y ago
    If that argument were to save anyone, it would have saved the lawyers too.
  - orwin 1y ago
    Well, most of the LLM-generated code i serve are unit tests (and scripts), so hopefully, those are good enough to catch my mistakes :)
vinni2 1y ago
> If a calculator works great 99% of the time you could not use that calculator to build a bridge. But if the alternative is doing calculations by hand (writing code manually) there is a higher chance of making mistakes. Just like calculations are double checked while building bridges unit tests and code reviews should catch bugs introduced by LLM written code.
- banannaise 1y ago
  Code review is your last (and worst) line of defense. Humans are not good at needle-in-a-haystack tasks.
the_snooze 1y ago
>If a calculator works great 99% of the time you could not use that calculator to build a bridge. That's happened before with far higher correctness rate than 99%, and it cost Intel $500M. Reliability and accuracy matter. https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug
worldsayshi 1y ago
This doesn't seem like the first time engineers try to work with something useful that is only partially reliable. The solution is to play at its strengths and reinforce it with other mediums. You don't build structures with pure concrete. You add rebar. You don't build ships out of only sail and you don't build rail with just iron. You compose materials in a way that makes sense. LLMs are most useful when the output is immediately verifiable. So let's build frameworks that take that to core. Build everything around verification. And use LLMs for its strengths.
oconnor663 1y ago
A pedantic but maybe-not-entirely-pedantic point: It depends on what you mean by 99%. If the calculator has a little gremlin in it that rolls a random 100-sided die, and gives you the wrong answer every time it rolls a 1, then you certainly can use it to build a bridge. You just need to do each calculation say 10 or 20 times and take the majority answer :) If the gremlin is clever, it might remember the wrong answers it gave you, and then it might give them to you again if you ask about the same numbers. In that case you might need to buy 10 or 20 calculators that all have different gremlins in them, but otherwise the process is the same. Of course if all your gremlins consistently lie for certain inputs, you might need to do a lot of work to sample all over your input space and see exactly what sorts of numbers they don't like. Then you can breed a new generation of gremlins that...
- franktankbank 1y ago
  Yea I know, can't really understand why people have such a problem with this. Just ignore the wrong answers and be thankful when it gives you a right answer. Picky bastards.
tom_m 1y ago
I believe it absolutely will. I think eventually we'll get to a point where people will be measured on now well they can get the AI to behave and how good they are at keeping cost down. My boss built an AI workflow that cost over $600 that does the same thing I already gave him that cost less than $30. He just wanted to use tools he found and did it his way. Now, this had some value, it got more people in the company exposed to AI and he learned from the experience. It's his prerogative as he's the owner of the company. Though he also isn't concerned about the cost and will continue to pay much more. For now. I think as time goes on this will be more scrutinized.

by brentm 1y ago

andrewmutz 1y ago

What we are seeing with our customers is that LLM errors are a very manageable problem. End users adapt pretty quickly to the idea that AI systems aren't perfect. In many cases AI products are doing tasks that used to be done by humans and these humans were making mistakes too, so the end user is used to the idea that the task will get accomplished with some non-zero error rate. You just need to build your products in a manner where the user has the ability to easily double check the results whenever they like. Then they can audit as they see fit, in order to get used to the accuracy level and to apply additional scrutiny to cases that are very important to their business.

brentm 1y ago
Yea I just think the true unlock in productivity will come from not requiring a human in the loop.
insane_dreamer 1y ago
> the user has the ability to easily double check the results whenever they like if the user is able to so easily verify that the results are accurate, that means that they are able to generate accurate results through other means, which means they don't need the LLM in the first place
- tom_m 1y ago
  True, but people are so enamored by what they can do that they rarely seem to think about that. We will over spend on AI purely because we think it's cool.
- Ukv 1y ago
  I don't think that's necessarily true - many tasks are difficult to solve but easy to verify. If I ask "place names that end with um", or "good ideas for a birthday party" I can pretty much verify the answer just by reading it. In other cases, clicking through to check that a linked source supports a claim is easier than researching to find and summarize the source in the first place would be.
  - insane_dreamer 1y ago
    > I can pretty much verify the answer just by reading it Only if you have domain knowledge. In both of your examples, you have to 1) know geography to determine whether "Técolum" and "Tolum" are indeed city names or just made up; and 2) know what might be acceptable ("good idea") or not at a birthday party. Yes, it'll probably save you some time, but it's not orders of magnitude. > In other cases, clicking through to check that a linked source supports a claim this supposes that the AI provides a link for every fact. Google search + Gemini does, but most LLM interfaces don't. secondly, if I have to click through every link and read through the source to determine whether details of a "summary" are correct or not, that really does not save me much time from conducting a search and looking through the linked sources myself Anecdote from a couple of weeks ago. My wife's professor sent her 5 citations and summaries related to a medical research project. She didn't say they were LLM generated, but it was obvious (to me, not my wife) they were, by the formatting alone. None of the 5 papers existed as cited. My wife was confused, spent a lot of time trying to figure out what was wrong and why she couldn't find any of the papers. A Google Scholar search turned up 2 of the papers which were close enough to the citation to be the ones with some logical thinking, but the other 3 were not even matchable. In the end, the time spent trying to sort out valid vs invalid citations, and find valid replacements, was significantly greater than just doing the search and looking through the abstracts. PS: LLMs are fine for information that can be "fuzzy": suggest places to go on vacation in September, plan a birthday party, etc. But I wouldn't consider that to be a "revolutionary" advance.

tveita 1y ago

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. We know for certain that certified lawyers have committed malpractice by using ChatGPT, in part because the made-up citations are relatively easy to spot. Malpractice by engineers might take a little more time to discover.

bagacrap 1y ago
Engineers' work is also externally verifiable, e.g. by unit tests for software, but I'm assuming by other sorts of automated protocols for civil engineering. I would hope a bridge is not built without triple checking the various outcomes.
- yencabulator 1y ago
  If that argument were to save anyone, it would have saved the lawyers too.
- orwin 1y ago
  Well, most of the LLM-generated code i serve are unit tests (and scripts), so hopefully, those are good enough to catch my mistakes :)

vinni2 1y ago

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. But if the alternative is doing calculations by hand (writing code manually) there is a higher chance of making mistakes. Just like calculations are double checked while building bridges unit tests and code reviews should catch bugs introduced by LLM written code.

banannaise 1y ago
Code review is your last (and worst) line of defense. Humans are not good at needle-in-a-haystack tasks.

the_snooze 1y ago

>If a calculator works great 99% of the time you could not use that calculator to build a bridge. That's happened before with far higher correctness rate than 99%, and it cost Intel $500M. Reliability and accuracy matter. https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug

worldsayshi 1y ago

This doesn't seem like the first time engineers try to work with something useful that is only partially reliable. The solution is to play at its strengths and reinforce it with other mediums. You don't build structures with pure concrete. You add rebar. You don't build ships out of only sail and you don't build rail with just iron. You compose materials in a way that makes sense. LLMs are most useful when the output is immediately verifiable. So let's build frameworks that take that to core. Build everything around verification. And use LLMs for its strengths.

oconnor663 1y ago

A pedantic but maybe-not-entirely-pedantic point: It depends on what you mean by 99%. If the calculator has a little gremlin in it that rolls a random 100-sided die, and gives you the wrong answer every time it rolls a 1, then you certainly can use it to build a bridge. You just need to do each calculation say 10 or 20 times and take the majority answer :) If the gremlin is clever, it might remember the wrong answers it gave you, and then it might give them to you again if you ask about the same numbers. In that case you might need to buy 10 or 20 calculators that all have different gremlins in them, but otherwise the process is the same. Of course if all your gremlins consistently lie for certain inputs, you might need to do a lot of work to sample all over your input space and see exactly what sorts of numbers they don't like. Then you can breed a new generation of gremlins that...

franktankbank 1y ago
Yea I know, can't really understand why people have such a problem with this. Just ignore the wrong answers and be thankful when it gives you a right answer. Picky bastards.

tom_m 1y ago

I believe it absolutely will. I think eventually we'll get to a point where people will be measured on now well they can get the AI to behave and how good they are at keeping cost down. My boss built an AI workflow that cost over $600 that does the same thing I already gave him that cost less than $30. He just wanted to use tools he found and did it his way. Now, this had some value, it got more people in the company exposed to AI and he learned from the experience. It's his prerogative as he's the owner of the company. Though he also isn't concerned about the cost and will continue to pay much more. For now. I think as time goes on this will be more scrutinized.