4 ms·

Do you think the performance can be improved if the representation of the level is different? I've seen AI struggle with ASCII, but when presented as other dat

Do you think the performance can be improved if the representation of the level is different? I've seen AI struggle with ASCII, but when presented as other data structures, it performs better. edit: e.g. JSON with structured coordinates, graph based JSON, or a semantic representation with the coordinates

hajile 1y ago
If it struggles with the representation, that makes it an even better test of the AI's thinking potential.
- eru 1y ago
  I'm not sure. Adding superficial difficulties to an IQ test for humans doesn't (necessarily) improve it as an IQ test.
RainyDayTmrw 1y ago
In the limit case, to an actual general intelligence, representation is superfluous, because it can figure out how to convert freely. To the extent that the current generation of AI isn't general, yeah, papering over some of its weaknesses may allow you to expose other parts of it, both strengths and other weaknesses.
- kadoban 1y ago
  A human can easily struggle at solving a poorly communicated puzzle, especially if paper/pencil or something isn't available to convert to a better format. LLMs can look back at what they wrote, but it seems kind of like a poor format for working out a better representation to me.
  - kinduff 1y ago
    I found some papers [n] about this. And I think the answer is yes, the format matters asnd hence the representation. I wonder if the author would be willing to try with another representation. [1]: Does Prompt Formatting Have Any Impact on LLM Performance? https://arxiv.org/html/2411.10541v1 https://arxiv.org/html/2411.10541v1 [2]: Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey https://arxiv.org/html/2402.17944v2 https://arxiv.org/html/2402.17944v2
QuadmasterXLII 1y ago
These models can “code,” but they can’t code yet. We’ll know that they can actually code once their performance on these tasks becomes invariant to input representation, because they can just whip up a script to convert representations.

hajile 1y ago

If it struggles with the representation, that makes it an even better test of the AI's thinking potential.

eru 1y ago
I'm not sure. Adding superficial difficulties to an IQ test for humans doesn't (necessarily) improve it as an IQ test.

RainyDayTmrw 1y ago

In the limit case, to an actual general intelligence, representation is superfluous, because it can figure out how to convert freely. To the extent that the current generation of AI isn't general, yeah, papering over some of its weaknesses may allow you to expose other parts of it, both strengths and other weaknesses.

kadoban 1y ago
A human can easily struggle at solving a poorly communicated puzzle, especially if paper/pencil or something isn't available to convert to a better format. LLMs can look back at what they wrote, but it seems kind of like a poor format for working out a better representation to me.
- kinduff 1y ago
  I found some papers [n] about this. And I think the answer is yes, the format matters asnd hence the representation. I wonder if the author would be willing to try with another representation. [1]: Does Prompt Formatting Have Any Impact on LLM Performance? https://arxiv.org/html/2411.10541v1 https://arxiv.org/html/2411.10541v1 [2]: Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey https://arxiv.org/html/2402.17944v2 https://arxiv.org/html/2402.17944v2

QuadmasterXLII 1y ago

These models can “code,” but they can’t code yet. We’ll know that they can actually code once their performance on these tasks becomes invariant to input representation, because they can just whip up a script to convert representations.