6 ms·

Yeah the issue tends to be as the length increases the performance drops off. Your approach reminds me of Hattie's work: https://arxiv.org/pdf/2211.09066.pdf h

Yeah the issue tends to be as the length increases the performance drops off. Your approach reminds me of Hattie's work: https://arxiv.org/pdf/2211.09066.pdf https://arxiv.org/pdf/2211.09066.pdf I do wonder if the observable drop-off in performance on these formal languages / arithmetic tasks is something more fundamental, and related to the attention mechanism; namely that it can't attend further back as the input gets longer (possibly due to the way attention scores are computed).

DougBTX 3y ago
Interesting, this is a harder problem than I'd guess. Prompts like this fail around line 16 with ChatGPT using GPT-3.5: Show the full output of this Python script in a single code block, with no other commentary: ``` lst = [1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0] x = 0 for i, bit in enumerate(lst): x ^= bit print(x) ``` After a bit of "debugging", it looks like the errors are introduced because it fails to read the correct values from the input. Explicitly prompting it to label the input values works reliably up to 60, then it hits the ChatGPT output limit and cuts off the response: Show the full output of this Python script in a single code block, with no other commentary: ``` lst = [1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0] for i, bit in enumerate(lst): print(f"{i}:{bit}") x = 0 for i, bit in enumerate(lst): x ^= bit print(f"{i}:{bit}:{x}") ```

DougBTX 3y ago

Interesting, this is a harder problem than I'd guess. Prompts like this fail around line 16 with ChatGPT using GPT-3.5: Show the full output of this Python script in a single code block, with no other commentary: ``` lst = [1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0] x = 0 for i, bit in enumerate(lst): x ^= bit print(x) ``` After a bit of "debugging", it looks like the errors are introduced because it fails to read the correct values from the input. Explicitly prompting it to label the input values works reliably up to 60, then it hits the ChatGPT output limit and cuts off the response: Show the full output of this Python script in a single code block, with no other commentary: ``` lst = [1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0] for i, bit in enumerate(lst): print(f"{i}:{bit}") x = 0 for i, bit in enumerate(lst): x ^= bit print(f"{i}:{bit}:{x}") ```