4 ms·

Yes. Numbers / math is pretty much instant hallucination. But. Try this approach instead: have it generate python code, with print statements before every bit

by LASR 2y ago

Yes. Numbers / math is pretty much instant hallucination. But. Try this approach instead: have it generate python code, with print statements before every bit of math it performs. It will write pretty good code, which you then execute to generate the actual answer. Simpler example: paste in a paragraph of text, ask it to count the number of words. The answer will be incorrect most of the time. Instead, ask it to out each word in the text in a numbered list and then output the word count. It will be correct almost always. My anecdotal learning from this: LLMs are pretty human-like in their mental abilities. I wouldn't be able to simply look at some text and give you an accurate word count. I would point my finger / cursor to every word and count up. The solutions above are basically giving LLMs some additional techniques or tools, very similar to how a human may use a calculator, or count words. In the products we've built, there is an AI feature that generates aggregations of spreadsheet data. We have a dual unittest & aggregator loop to generate correct values. The first step is to generate some unittests. And in order to generate correct numerical data for unittests, we ask it to write some code with math expressions first. We interpret the expressions, and paste it back into the unittest generator - which then writes the unittests with the correct inputs / outputs. Then the aggregation generator then generates code until the generated unittests pass completely. Then we have the code for the aggregator function that we can run against the spreadsheet. Takes a couple of minutes, but pretty bulletproof and also generalizable to other complex math calculations.

ddfs123 2y ago
>I wouldn't be able to simply look at some text and give you an accurate word count. I actually think you would, lots of time I've been surprised by how accurate people can eye-ball stuffs.
AdieuToLogic 2y ago
> Yes. Numbers / math is pretty much instant hallucination. Programming is codified applied mathematics and involves numbers in all but the most trivial programs. > LLMs are pretty human-like in their mental abilities. LLM's are algorithms. Algorithms do not have "mental abilities", people do. Anthropomorphizing algorithms only serves to impair objective analysis of when they are, or are not, applicable. > In the products we've built, there is an AI feature that generates aggregations of spreadsheet data. We have a dual unittest & aggregator loop to generate correct values. > The first step is to generate some unittests. And in order to generate correct numerical data for unittests, we ask it to write some code with math expressions first. We interpret the expressions, and paste it back into the unittest generator - which then writes the unittests with the correct inputs / outputs. > Then the aggregation generator then generates code until the generated unittests pass completely. Then we have the code for the aggregator function that we can run against the spreadsheet. How is this not a classic definition of overfitting[0]? Or is the generated code intentionally specific to, and only applicable for, a single spreadsheet? 0 - https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting
- unoti 2y ago
  > Anthropomorphizing algorithms only serves to impair objective analysis of when they are, or are not, applicable. Actually, in this case, comparing how we as humans think to how LLM's work is in fact useful. It's hard for us to eyeball a word and say how many consonants are in it, we need to count. I wouldn't ask a human to eyeball a tax return and tell me what the totals are reliably without giving them the tools to add things up. LLM's are the same way. It's true that anthropomorphizing in general can be a trap, but when working with LLM's it can be a useful guide in pointing the way towards workable solutions.
  - AdieuToLogic 2y ago
    > Actually, in this case, comparing how we as humans think to how LLM's work is in fact useful. Agreed. Contemplating the difference between what people and LLM's are is very useful IMHO. Understanding this is key to making informed decisions as to when LLM's can provide real value. The assertion originally proffered, however, is quite different than your nuanced perspective: > > LLMs are pretty human-like in their mental abilities. It is this to which I object.
    - Ironchefpython 2y ago
      I think it would be more accurate to say that LLMs are pretty human-like in their limitations.
- Mashimo 2y ago
  > Programming is codified applied mathematics and involves numbers in all but the most trivial programs. I don't understand that. Can you expand on it? When does the AI or the programmer need to do math calculations?
  - hsjdhdvsk 2y ago
    Everything a computer does is math...
gloosx 2y ago
>I wouldn't be able to simply look at some text and give you an accurate word count Actually this ability is at very core of human brain. Yet we humans traded most of it for the ability to speak better. Nevertheless, brain can still read very fast as well as count objects/words really fast, but as you've never really trained, this part of the brain is optimised a lot to do other stuff. Try scanning the texts diagonally, and try to come up with a random word count and what this text is meaning. On the first tries you will be making a lot of errors, but eventually, just 100 hours of training (much less when you are kid) and you will be able to scan the texts in seconds and extract the meaning and the word count very accurately. This is a real technique people use for reading fast. The thing is – the best and most useful ability of human brain is to adapt, from the very moment it adapted to the harsh reality of being expelled from trees and living on the ground – that's how language was created in the first place, as a result of this adaptation. LLM's can't adapt today, nor doesn't need to adapt, so their mental abilities will never come close to the brain which was adapting for millions of years. Something which is not adapting will become obsolete and come to it's end, this is a fundamental law.

by LASR 2y ago

ddfs123 2y ago

>I wouldn't be able to simply look at some text and give you an accurate word count. I actually think you would, lots of time I've been surprised by how accurate people can eye-ball stuffs.

AdieuToLogic 2y ago

> Yes. Numbers / math is pretty much instant hallucination. Programming is codified applied mathematics and involves numbers in all but the most trivial programs. > LLMs are pretty human-like in their mental abilities. LLM's are algorithms. Algorithms do not have "mental abilities", people do. Anthropomorphizing algorithms only serves to impair objective analysis of when they are, or are not, applicable. > In the products we've built, there is an AI feature that generates aggregations of spreadsheet data. We have a dual unittest & aggregator loop to generate correct values. > The first step is to generate some unittests. And in order to generate correct numerical data for unittests, we ask it to write some code with math expressions first. We interpret the expressions, and paste it back into the unittest generator - which then writes the unittests with the correct inputs / outputs. > Then the aggregation generator then generates code until the generated unittests pass completely. Then we have the code for the aggregator function that we can run against the spreadsheet. How is this not a classic definition of overfitting[0]? Or is the generated code intentionally specific to, and only applicable for, a single spreadsheet? 0 - https://en.wikipedia.org/wiki/Overfitting https://en.wikipedia.org/wiki/Overfitting

unoti 2y ago
> Anthropomorphizing algorithms only serves to impair objective analysis of when they are, or are not, applicable. Actually, in this case, comparing how we as humans think to how LLM's work is in fact useful. It's hard for us to eyeball a word and say how many consonants are in it, we need to count. I wouldn't ask a human to eyeball a tax return and tell me what the totals are reliably without giving them the tools to add things up. LLM's are the same way. It's true that anthropomorphizing in general can be a trap, but when working with LLM's it can be a useful guide in pointing the way towards workable solutions.
- AdieuToLogic 2y ago
  > Actually, in this case, comparing how we as humans think to how LLM's work is in fact useful. Agreed. Contemplating the difference between what people and LLM's are is very useful IMHO. Understanding this is key to making informed decisions as to when LLM's can provide real value. The assertion originally proffered, however, is quite different than your nuanced perspective: > > LLMs are pretty human-like in their mental abilities. It is this to which I object.
  - Ironchefpython 2y ago
    I think it would be more accurate to say that LLMs are pretty human-like in their limitations.
Mashimo 2y ago
> Programming is codified applied mathematics and involves numbers in all but the most trivial programs. I don't understand that. Can you expand on it? When does the AI or the programmer need to do math calculations?
- hsjdhdvsk 2y ago
  Everything a computer does is math...

gloosx 2y ago

>I wouldn't be able to simply look at some text and give you an accurate word count Actually this ability is at very core of human brain. Yet we humans traded most of it for the ability to speak better. Nevertheless, brain can still read very fast as well as count objects/words really fast, but as you've never really trained, this part of the brain is optimised a lot to do other stuff. Try scanning the texts diagonally, and try to come up with a random word count and what this text is meaning. On the first tries you will be making a lot of errors, but eventually, just 100 hours of training (much less when you are kid) and you will be able to scan the texts in seconds and extract the meaning and the word count very accurately. This is a real technique people use for reading fast. The thing is – the best and most useful ability of human brain is to adapt, from the very moment it adapted to the harsh reality of being expelled from trees and living on the ground – that's how language was created in the first place, as a result of this adaptation. LLM's can't adapt today, nor doesn't need to adapt, so their mental abilities will never come close to the brain which was adapting for millions of years. Something which is not adapting will become obsolete and come to it's end, this is a fundamental law.