6 ms·

This take strikes me as a little off. Programming languages are language. Unlike natural languages they are also based on context-free grammar. So an understand

by alpos 4y ago

This take strikes me as a little off. Programming languages are language. Unlike natural languages they are also based on context-free grammar. So an understanding of programming languages should actually be easier for even a general language model to incorporate than natural languages. We can expect a bot like this to not really get context clues in natural language, although they seem to be getting better at that, but context is not necessary to have a true and functional understanding of a programming language. That was the point of creating such languages. Using an API that doesn't exist but logically should once the use cases are demonstrated is not an example of lacking understanding, it is an example of advanced insight. A human might have invented the necessary functions inline with the rest of the project but if they are expressing functionality that is commonly applicable, then a common API for those functions is what the humans would eventually converge upon to clean up the code from the initial inline implementation, making it more consistent and readable.

layer8 4y ago
The failure to realize that the API doesn’t exist and therefore the code won’t work in practice, however, is a major lack of insight and understanding.
- alpos 4y ago
  Agreed. That does seem to be an example of a language model failing to understand the context in which the question was asked. The user was implicitly expecting code that would function when executed immediately and as written with no additional supporting libraries included. This is different from code that would function correctly when executed after having downloaded relevant existing packages. Which is different from code that would function if executed along side additional supporting code from private libraries the user might not have access to. Etc... Yet any of those answers fit for the same prompt, "create an R script that does such and such". The bot's lack of insight is on the likely intention behind the prompt rather than on the requested language. I'd say if it produces any code that fits the syntax and grammatical structure of the requested language, that's enough to say it understands the language.
gattilorenz 4y ago
My answer was mildly tongue in cheek, and I see where you’re going. On the other hand, one of the other posters asked “to generate a parallax effect in Qt/QML. It simply used a QML Elemened with the name Parallax”. Is this an insight, or is this answering “yes, I could” to “could you pass me the salt?”. Maybe the line between the two is a fine one, and I didn’t realize that yet. In general, it feels like copying part of the question (“write parallax code”) in the answer is the easy part of the task…
- alpos 4y ago
  Saw that, and yea, that's totally a fall-back cop-out type answer. I was pretty much just questioning everyone dismissing the whole category of answers like the R script one. To me there does seem to be some nuance to here that's worth noticing. Some examples of this type of response are indeed too cheap and can be chalked up to lack of training data or something. But in other cases it's actually not immediately obvious whether the answer the user got was their fault for not specifying that they are expecting code that works without additional supporting libraries. A language model can't reasonably be expected to understand an expectation of usability or fitness for purpose in a context the user didn't specify.
  - gattilorenz 4y ago
    Yes, indeed there are some extra nuances that should not be automatically dismissed. > A language model can't reasonably be expected to understand an expectation of usability or fitness for purpose in a context the user didn't specify. I agree, but I think we’re at the same time expecting the LM to “understand” a lot more

by alpos 4y ago

layer8 4y ago

The failure to realize that the API doesn’t exist and therefore the code won’t work in practice, however, is a major lack of insight and understanding.

alpos 4y ago
Agreed. That does seem to be an example of a language model failing to understand the context in which the question was asked. The user was implicitly expecting code that would function when executed immediately and as written with no additional supporting libraries included. This is different from code that would function correctly when executed after having downloaded relevant existing packages. Which is different from code that would function if executed along side additional supporting code from private libraries the user might not have access to. Etc... Yet any of those answers fit for the same prompt, "create an R script that does such and such". The bot's lack of insight is on the likely intention behind the prompt rather than on the requested language. I'd say if it produces any code that fits the syntax and grammatical structure of the requested language, that's enough to say it understands the language.

gattilorenz 4y ago

My answer was mildly tongue in cheek, and I see where you’re going. On the other hand, one of the other posters asked “to generate a parallax effect in Qt/QML. It simply used a QML Elemened with the name Parallax”. Is this an insight, or is this answering “yes, I could” to “could you pass me the salt?”. Maybe the line between the two is a fine one, and I didn’t realize that yet. In general, it feels like copying part of the question (“write parallax code”) in the answer is the easy part of the task…

alpos 4y ago
Saw that, and yea, that's totally a fall-back cop-out type answer. I was pretty much just questioning everyone dismissing the whole category of answers like the R script one. To me there does seem to be some nuance to here that's worth noticing. Some examples of this type of response are indeed too cheap and can be chalked up to lack of training data or something. But in other cases it's actually not immediately obvious whether the answer the user got was their fault for not specifying that they are expecting code that works without additional supporting libraries. A language model can't reasonably be expected to understand an expectation of usability or fitness for purpose in a context the user didn't specify.
- gattilorenz 4y ago
  Yes, indeed there are some extra nuances that should not be automatically dismissed. > A language model can't reasonably be expected to understand an expectation of usability or fitness for purpose in a context the user didn't specify. I agree, but I think we’re at the same time expecting the LM to “understand” a lot more