4 ms·

They process the audio but they stumble enough with recall that you cannot really trust it. I had a problem where I used GPT-4o to help me with inventory manag

They process the audio but they stumble enough with recall that you cannot really trust it. I had a problem where I used GPT-4o to help me with inventory management, something a 5th grade kid could handle, and it kept screwing up values for a list of ~50 components. I ended up spending more time trying to get it to properly parse the input audio (I read off the counts as I moved through inventory bins) then if I had just done it manually. On the other hand, I have had good success with having it write simple programs and apps. So YMMV quite a lot more than with a regular person.

anon291 2y ago
> They process the audio but they stumble enough with recall that you cannot really trust it. I will wave my arms wildly at the last eight years if the claim is that humans do not struggle with recall.
- th0ma5 2y ago
  So are they human like and therefore not anything special or are they super human magic? I never get the equivocation when people complain how there is no way to objectively tell what out is right or wrong people either say they are getting better, or they work for me, or that people are just as bad. No they aren't! Not in the same way these things are bad.
  - anon291 2y ago
    Most people will confidently recount whatever narrative matches their current actions. This is called rationalization, and most people engage in it daily.
- vlovich123 2y ago
  I will wave my arms wildly if the claim is that LLM struggle with recall is similar to human-like struggle with recall. And since that's how we decide on truth, I win?
  - anon291 2y ago
    what we call hallucination in LLMs is called 'rationalization' for humans. The psychology shows that most peoples do things out of habit and only after they've done it will explain why the did it. This is most obviously seen in split brain patients where the visual fields are then separated. If you throw a ball towards the left side of the person, the right brain will catch the ball. if you then ask the person why they caught the ball the left brain will make up a completely ridiculous narrative as to why the hand moved (because it didn't know there is a ball. This is a contrived example, but it shows that human recollection of intent is often very very wrong. There are studies that show this even in people with whole brains.
    - vlovich123 2y ago
      You're unfortunately completely missing the point. I didn't say that human recall is perfect or that they don't rationalize. And of course you can have extreme denial of what's happening in front of you even in healthy individuals. In fact, you see this in this thread where either you or the huge number of people trying to dissillusion you from the maximal position you've staked out on LLMs is wrong and one of us is incorrectly rationalizing our position. The point is that the ways in which it fails is completely different from LLMs and it's different between people whereas the failure modes for LLMs are all fairly identical regardless of the model. Go ask an LLM to draw you a wine glass filled to the brim and it'll keep insisting it does even though it keeps drawing one half-filled and agree that the one it drew doesn't have the characteristics it says such a drawing would need and still output the exact same drawing. Most people would not fail at the task in that way.
      - anon291 2y ago
        > In fact, you see this in this thread where either you or the huge number of people trying to dissillusion you from the maximal position you've staked out on LLMs is wrong and one of us is incorrectly rationalizing our position. I by no means have a 'maximal' position. I have said that they exceed the intelligence and ability of the vast majority of the human populace when it comes to their singular sense and action (ingesting language and outputting language). I fully stand by that, because it's true. I've not claimed that they exceed everyone's intelligence in every area. However, their ability to synthesize wildly different fields is well beyond most human's ability. Yes, I do believe we've crossed the tipping point. As it is, these things are not noticeable except in retrospect. > The point is that the ways in which it fails is completely different from LLMs and it's different between people whereas the failure modes for LLMs are all fairly identical I disagree with the idea that human failure modes are different between people. I think this is the result of not thinking at a high enough level. Human failure modes are often very similar. Drama authors make a living off exploring human failure modes, and there's a reason why they say there are no new stories. I agree that Human and LLM failure modes are different, but that's to be expected. > regardless of the model As far as I'm aware, all LLMs in common use today use a variant of the transformer. Transformers have much different pitfalls compared to RNNs (RNNs are parlticularly bad at recall for example). > Go ask an LLM to draw you a wine glass filled to the brim and it'll keep insisting it does even though it keeps drawing one half-filled and agree that the one it drew doesn't have the characteristics it says such a drawing would need and still output the exact same drawing. Most people would not fail at the task in that way. Most people can't draw very well anyway, so this is just proving my point.
wahnfrieden 2y ago
You must use it to make transcripts and then write code to process the values in the transcripts
XenophileJKO 2y ago
Likely the issue is how you are asking the model to process things. The primary limitation is the amount of information (or really attention) they can keep in flight at any given moment. This generally means for a task like you are doing, you need to have sign posts in the data like minute markers or something that it can process serially. This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently.
- anon291 2y ago
  > This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently. Ranking / sorting is O(n log n) no matter what. Given that a transformer runs in constant time before we 'force' it to output an answer, there must be an M such that beyond that length it cannot reliably sort a list. This MUST be the case and can only be solved by running the model some indeterminate number of times, but I don't believe we currently have any architecture to do that. Note that humans have the same limitation. If you give humans a time limit, there is a maximum number of things they will be able to sort reliably in that time.
  - christianqchung 2y ago
    Transformers absolutely do not run in constant time by any reasonable definition, no matter what your point is.
    - anon291 2y ago
      They absolutely do given a sequence size. All models have max context lengths. Thus bounded by a constant

anon291 2y ago

> They process the audio but they stumble enough with recall that you cannot really trust it. I will wave my arms wildly at the last eight years if the claim is that humans do not struggle with recall.

th0ma5 2y ago
So are they human like and therefore not anything special or are they super human magic? I never get the equivocation when people complain how there is no way to objectively tell what out is right or wrong people either say they are getting better, or they work for me, or that people are just as bad. No they aren't! Not in the same way these things are bad.
- anon291 2y ago
  Most people will confidently recount whatever narrative matches their current actions. This is called rationalization, and most people engage in it daily.
vlovich123 2y ago
I will wave my arms wildly if the claim is that LLM struggle with recall is similar to human-like struggle with recall. And since that's how we decide on truth, I win?
- anon291 2y ago
  what we call hallucination in LLMs is called 'rationalization' for humans. The psychology shows that most peoples do things out of habit and only after they've done it will explain why the did it. This is most obviously seen in split brain patients where the visual fields are then separated. If you throw a ball towards the left side of the person, the right brain will catch the ball. if you then ask the person why they caught the ball the left brain will make up a completely ridiculous narrative as to why the hand moved (because it didn't know there is a ball. This is a contrived example, but it shows that human recollection of intent is often very very wrong. There are studies that show this even in people with whole brains.
  - vlovich123 2y ago
    You're unfortunately completely missing the point. I didn't say that human recall is perfect or that they don't rationalize. And of course you can have extreme denial of what's happening in front of you even in healthy individuals. In fact, you see this in this thread where either you or the huge number of people trying to dissillusion you from the maximal position you've staked out on LLMs is wrong and one of us is incorrectly rationalizing our position. The point is that the ways in which it fails is completely different from LLMs and it's different between people whereas the failure modes for LLMs are all fairly identical regardless of the model. Go ask an LLM to draw you a wine glass filled to the brim and it'll keep insisting it does even though it keeps drawing one half-filled and agree that the one it drew doesn't have the characteristics it says such a drawing would need and still output the exact same drawing. Most people would not fail at the task in that way.
    - anon291 2y ago
      > In fact, you see this in this thread where either you or the huge number of people trying to dissillusion you from the maximal position you've staked out on LLMs is wrong and one of us is incorrectly rationalizing our position. I by no means have a 'maximal' position. I have said that they exceed the intelligence and ability of the vast majority of the human populace when it comes to their singular sense and action (ingesting language and outputting language). I fully stand by that, because it's true. I've not claimed that they exceed everyone's intelligence in every area. However, their ability to synthesize wildly different fields is well beyond most human's ability. Yes, I do believe we've crossed the tipping point. As it is, these things are not noticeable except in retrospect. > The point is that the ways in which it fails is completely different from LLMs and it's different between people whereas the failure modes for LLMs are all fairly identical I disagree with the idea that human failure modes are different between people. I think this is the result of not thinking at a high enough level. Human failure modes are often very similar. Drama authors make a living off exploring human failure modes, and there's a reason why they say there are no new stories. I agree that Human and LLM failure modes are different, but that's to be expected. > regardless of the model As far as I'm aware, all LLMs in common use today use a variant of the transformer. Transformers have much different pitfalls compared to RNNs (RNNs are parlticularly bad at recall for example). > Go ask an LLM to draw you a wine glass filled to the brim and it'll keep insisting it does even though it keeps drawing one half-filled and agree that the one it drew doesn't have the characteristics it says such a drawing would need and still output the exact same drawing. Most people would not fail at the task in that way. Most people can't draw very well anyway, so this is just proving my point.

wahnfrieden 2y ago

You must use it to make transcripts and then write code to process the values in the transcripts

XenophileJKO 2y ago

Likely the issue is how you are asking the model to process things. The primary limitation is the amount of information (or really attention) they can keep in flight at any given moment. This generally means for a task like you are doing, you need to have sign posts in the data like minute markers or something that it can process serially. This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently.

anon291 2y ago
> This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently. Ranking / sorting is O(n log n) no matter what. Given that a transformer runs in constant time before we 'force' it to output an answer, there must be an M such that beyond that length it cannot reliably sort a list. This MUST be the case and can only be solved by running the model some indeterminate number of times, but I don't believe we currently have any architecture to do that. Note that humans have the same limitation. If you give humans a time limit, there is a maximum number of things they will be able to sort reliably in that time.
- christianqchung 2y ago
  Transformers absolutely do not run in constant time by any reasonable definition, no matter what your point is.
  - anon291 2y ago
    They absolutely do given a sequence size. All models have max context lengths. Thus bounded by a constant