5 ms·

I've been thinking about this lately and it seems to me that what these models are very good at is generating text that has the right structure, but of all the

I've been thinking about this lately and it seems to me that what these models are very good at is generating text that has the right structure, but of all the permutations with the right structure only a few actually contain useful and correct information and it only hits on those by chance. And, since the real value in communication is the information contained, that puts a fairly low ceiling on the value of their output. If it can't be trusted without careful review by someone that really understands the subject and can flag mistakes then it can never truly replace people in any role where correctness matters and that's most of the roles with a lot of economic value.

int_19h 3y ago
If that were the case, outputs would be consistently nonsense - the number of possible variations of text like "colorless green ideas sleep furiously" is so much larger than the meaningful subset, the probability of hitting the latter by chance would be zero for all practical purposes.
- cageface 3y ago
  Only if the words were chosen simply at random in sequence and of course they're not this simplistic. They're constrained by the attention models so they do much better than this but they're still random. You can control the degree of randomness with the temperature knob.
  - int_19h 3y ago
    This part about "constrained by the attention model" is doing a lot of implicit work here to dodge the question why GPT-4 can verifiably reason about things in text.
    - cageface 3y ago
      It also demonstrably is either flat out wrong about a lot of things or completely invents things that don't exist. It's a random process that sometimes generates content with actual informational value but the randomness is inherent in the algorithm.
sfink 3y ago
> And, since the real value in communication is the information contained, that puts a fairly low ceiling on the value of their output. ...then it can never truly replace people in any role where correctness matters and that's most of the roles with a lot of economic value. I think the thrust of your argument is correct: tasks where correctness matters are inherently less suited to AI automation. But I think that's more a matter of trying to use an LLM for a job that it is the wrong tool for. I think there are many economically valuable roles that are outside of that limited zone, and there will be a lot of people using AI for what AI is good at while the rest of us complain about the limitations when trying to use it for what it isn't good at. (I do a lot of that too.) Which is probably a waste of time and energy that could be better spent learning how to effectively use an LLM rather than trying to push it in directions that it is incapable of going. I haven't played much with LLMs yet, so I personally don't have a great sense for what it is good at, and I haven't come across anyone else with a good rundown of the space either. But some things are becoming clear. LLMs are good at the "blank page" problem, where you know what you want to do but are having a hard time getting started with it. An LLM-generated starting point need not be correct to be useful, and in fact being incorrect can be an advantage since the point is what it triggers in the human's brain. LLMs are good at many parts of programming that humans are weak at. Humans tend to need to have a certain level of familiarity and comfort with a framework or tool in order to even begin to be productive in it, and we won't use more advanced features or suitable idioms until we get into it enough. An LLM's training data encompasses both the basic starting points as well as more sophisticated uses. So it can suggest idiomatic solutions to problems up front, and since the human is deciding whether and how to incorporate them, correctness is only moderately important. An incorrect but idiomatic use of a framework is close to a correct idiomatic use, while a human-generated correct but awkward use can be very far away from a correct idiomatic use. Image generation seems similar. My impression is that Midjourney produces good looking output but is fairly useless when you need to steer it to something that is "correct" with respect to a goal. It's great until you actually need to use it, then you have to throw it out. Stable diffusion produces lower quality output but is much more steerable towards "correctness", which requires human artistic intervention. So there seems to be a common theme. Something like: LLMs are highly useful but require a human to steer and provide "correctness", whatever that might mean in a particular domain.

int_19h 3y ago

If that were the case, outputs would be consistently nonsense - the number of possible variations of text like "colorless green ideas sleep furiously" is so much larger than the meaningful subset, the probability of hitting the latter by chance would be zero for all practical purposes.

cageface 3y ago
Only if the words were chosen simply at random in sequence and of course they're not this simplistic. They're constrained by the attention models so they do much better than this but they're still random. You can control the degree of randomness with the temperature knob.
- int_19h 3y ago
  This part about "constrained by the attention model" is doing a lot of implicit work here to dodge the question why GPT-4 can verifiably reason about things in text.
  - cageface 3y ago
    It also demonstrably is either flat out wrong about a lot of things or completely invents things that don't exist. It's a random process that sometimes generates content with actual informational value but the randomness is inherent in the algorithm.

sfink 3y ago

> And, since the real value in communication is the information contained, that puts a fairly low ceiling on the value of their output. ...then it can never truly replace people in any role where correctness matters and that's most of the roles with a lot of economic value. I think the thrust of your argument is correct: tasks where correctness matters are inherently less suited to AI automation. But I think that's more a matter of trying to use an LLM for a job that it is the wrong tool for. I think there are many economically valuable roles that are outside of that limited zone, and there will be a lot of people using AI for what AI is good at while the rest of us complain about the limitations when trying to use it for what it isn't good at. (I do a lot of that too.) Which is probably a waste of time and energy that could be better spent learning how to effectively use an LLM rather than trying to push it in directions that it is incapable of going. I haven't played much with LLMs yet, so I personally don't have a great sense for what it is good at, and I haven't come across anyone else with a good rundown of the space either. But some things are becoming clear. LLMs are good at the "blank page" problem, where you know what you want to do but are having a hard time getting started with it. An LLM-generated starting point need not be correct to be useful, and in fact being incorrect can be an advantage since the point is what it triggers in the human's brain. LLMs are good at many parts of programming that humans are weak at. Humans tend to need to have a certain level of familiarity and comfort with a framework or tool in order to even begin to be productive in it, and we won't use more advanced features or suitable idioms until we get into it enough. An LLM's training data encompasses both the basic starting points as well as more sophisticated uses. So it can suggest idiomatic solutions to problems up front, and since the human is deciding whether and how to incorporate them, correctness is only moderately important. An incorrect but idiomatic use of a framework is close to a correct idiomatic use, while a human-generated correct but awkward use can be very far away from a correct idiomatic use. Image generation seems similar. My impression is that Midjourney produces good looking output but is fairly useless when you need to steer it to something that is "correct" with respect to a goal. It's great until you actually need to use it, then you have to throw it out. Stable diffusion produces lower quality output but is much more steerable towards "correctness", which requires human artistic intervention. So there seems to be a common theme. Something like: LLMs are highly useful but require a human to steer and provide "correctness", whatever that might mean in a particular domain.