3 ms·

That’s an established technique with papers written on the topic and everything. Anecdotally I tested this by having GPT4 translate Acadian cuneiform — which i

That’s an established technique with papers written on the topic and everything. Anecdotally I tested this by having GPT4 translate Acadian cuneiform — which it can just barely do. I had it do this four times and it returned four gibberish answers. I then prompted it with the source plus the four attempts and asked for a merged result. It did it better than the human archeologists did! More readable and consistent. I compared it with the human version and it matched the meaning. Expensive now… soon to be standard?

dontupvoteme 3y ago
Makes sense, it's too obvious to not have already been studied :) Do you know what I might search for to find info on it? >Expensive now… soon to be standard? With translation, better safe than sorry? It's a very important field and preserves human history so, why not?
- jiggawatts 3y ago
  It's trivial to test, just use ChatGPT yourself and ask it to solve the same problem several times in new sessions. Then paste in all attempts and ask for a combined result. The main issue is context length: if you use 4 attempts you have to have to fit in the original question, four temporary answers, and the final answer. So that's 6 roughly equal sized chunks of text. With GPT4's 8K limit that's just 1300 tokens per chunk, or about 900 words. That's not a lot! The LLMs with longer context windows are not as intelligent, and tend to miss details or they don't follow instructions as accurately. Right now this is just a gimmick that demonstrates that more intelligence can be squeezed out of even existing LLMs...
intended 3y ago
I don’t know if I will sound rude, but your example itself illustrates the crux of problem. The only way you could know that the output was wrong was because you could verify it in the first place. You can’t verify answers for questions in unfamiliar domains - or even for novel questions in your own domain. Hah, it feels like a weird version of P!=NP.

dontupvoteme 3y ago

Makes sense, it's too obvious to not have already been studied :) Do you know what I might search for to find info on it? >Expensive now… soon to be standard? With translation, better safe than sorry? It's a very important field and preserves human history so, why not?

jiggawatts 3y ago
It's trivial to test, just use ChatGPT yourself and ask it to solve the same problem several times in new sessions. Then paste in all attempts and ask for a combined result. The main issue is context length: if you use 4 attempts you have to have to fit in the original question, four temporary answers, and the final answer. So that's 6 roughly equal sized chunks of text. With GPT4's 8K limit that's just 1300 tokens per chunk, or about 900 words. That's not a lot! The LLMs with longer context windows are not as intelligent, and tend to miss details or they don't follow instructions as accurately. Right now this is just a gimmick that demonstrates that more intelligence can be squeezed out of even existing LLMs...

intended 3y ago

I don’t know if I will sound rude, but your example itself illustrates the crux of problem. The only way you could know that the output was wrong was because you could verify it in the first place. You can’t verify answers for questions in unfamiliar domains - or even for novel questions in your own domain. Hah, it feels like a weird version of P!=NP.