4 ms·

Given it's not grammatically or well-formed, I don't feel it's reasonable to assume a given reading as there are multiple possible readings that all requires as

by vidarh 3y ago

Given it's not grammatically or well-formed, I don't feel it's reasonable to assume a given reading as there are multiple possible readings that all requires assuming a certain set of corrections to punctuations or words. Frankly, the only "correct" response to this is to not offer a solution, but to suggest one or more possible readings and asking for corrections, as it's not solvable without knowing how to resolve the grammatical errors.

washadjeffmad 3y ago
In the absence of any prior knowledge, perhaps. But if you're aware that "missionaries and cannibals" [1] is a common toy problem for LLMs, this variation becomes an apparent anti-logic puzzle. Also, because we know that punctuation, capitalization, and special characters are removed prior to vectorization, they can be disregarded by us, as well. We do not need more perfect grammar or syntax. Thus, there is sufficient probability of intent in the information encoded that we can reject any attempts to solve it that use cannibalism as a constraint. That is the test. [1] https://en.m.wikipedia.org/wiki/Missionaries_and_cannibals_problem https://en.m.wikipedia.org/wiki/Missionaries_and_cannibals_p...
- vidarh 3y ago
  It might well be an "anti logic" puzzle but that doesn't change the severely broken grammar which makes it unclear how exactly the rules are changed. And the fact several of us were left here wondering about the meaning is clear evidence that it's unreasonable to expect it to be comprehensible to an LLM, and even more so to treat it as a good test, because it's severely compromised by the broken description. I stand by my assessment, which is that an LLM should be rated as failing this unless it responds by either setting out a plausible interpretation and asking if it is right or asking for a clarification. And so should people. The illogical response here is to assume toknow (rather than is guessing) what was intended. The expected response would be to be confused, as you can see from the replies.

by vidarh 3y ago

washadjeffmad 3y ago

In the absence of any prior knowledge, perhaps. But if you're aware that "missionaries and cannibals" [1] is a common toy problem for LLMs, this variation becomes an apparent anti-logic puzzle. Also, because we know that punctuation, capitalization, and special characters are removed prior to vectorization, they can be disregarded by us, as well. We do not need more perfect grammar or syntax. Thus, there is sufficient probability of intent in the information encoded that we can reject any attempts to solve it that use cannibalism as a constraint. That is the test. [1] https://en.m.wikipedia.org/wiki/Missionaries_and_cannibals_problem https://en.m.wikipedia.org/wiki/Missionaries_and_cannibals_p...

vidarh 3y ago
It might well be an "anti logic" puzzle but that doesn't change the severely broken grammar which makes it unclear how exactly the rules are changed. And the fact several of us were left here wondering about the meaning is clear evidence that it's unreasonable to expect it to be comprehensible to an LLM, and even more so to treat it as a good test, because it's severely compromised by the broken description. I stand by my assessment, which is that an LLM should be rated as failing this unless it responds by either setting out a plausible interpretation and asking if it is right or asking for a clarification. And so should people. The illogical response here is to assume toknow (rather than is guessing) what was intended. The expected response would be to be confused, as you can see from the replies.