3 ms·

Coincidentally, I was just experimenting with this paper on Claude. It seems pretty good. It's basically just a matter of demonstrating an in-context example of

Coincidentally, I was just experimenting with this paper on Claude. It seems pretty good. It's basically just a matter of demonstrating an in-context example of the reasoning you want it to use. You can probably get even better results by providing more than one example that are dissimilar. I'm surprised nobody has published a method just like this already. Another in-context method I use is to give the LLM like 3 examples of inputs with ideal outputs and ask it to return me an instruction (or directive) that, if included in the inputs would help it generate those outputs. I used it to test LSAT logical reasoning questions and, despite the instruction just seeming like very basic instructions, it no doubt produced more accurate results when the instruction was included.

catlover76 3y ago
> Another in-context method I use is to give the LLM like 3 examples of inputs with ideal outputs and ask it to return me an instruction (or directive) that, if included in the inputs would help it generate those outputs. I used it to test LSAT logical reasoning questions and, despite the instruction just seeming like very basic instructions, it no doubt produced more accurate results when the instruction was included. Sounds useful and quick to try. Curious as to what made you choose LSAT logical reasoning questions? I used to be an LSAT tutor lol
- jakderrida 3y ago
  > Curious as to what made you choose LSAT logical reasoning questions? I used to be an LSAT tutor lol Full disclosure: They were actually GMAT critical reasoning from gmatclub. I was also a tutor that strayed from Stats and Math to help a particular person out for GMAT. Three reasons I always use them... First is that it's testing only the reasoning and the ability for the model to use reasoning itself and, for LSAT and GMAT, at levels exceeding an average person's capabilities. Second is that it supplies everything the model needs and can't be solved based on remembered training data. Even testing the model's grade school science knowledge is just a test of the ability for the model to retain and regurgitate what it was trained on, which is an ability I'm not interested in testing. Third is that I happen to be a natural grandmaster at the GMAT critical reasoning practice questions. For every GMAT prep program, they supply a crappy and convoluted step-by-step process or visualization. All I needed was my simple bridge analogy and every question was about finding, building, or destroying the bridge that holds the argument together.

catlover76 3y ago

> Another in-context method I use is to give the LLM like 3 examples of inputs with ideal outputs and ask it to return me an instruction (or directive) that, if included in the inputs would help it generate those outputs. I used it to test LSAT logical reasoning questions and, despite the instruction just seeming like very basic instructions, it no doubt produced more accurate results when the instruction was included. Sounds useful and quick to try. Curious as to what made you choose LSAT logical reasoning questions? I used to be an LSAT tutor lol

jakderrida 3y ago
> Curious as to what made you choose LSAT logical reasoning questions? I used to be an LSAT tutor lol Full disclosure: They were actually GMAT critical reasoning from gmatclub. I was also a tutor that strayed from Stats and Math to help a particular person out for GMAT. Three reasons I always use them... First is that it's testing only the reasoning and the ability for the model to use reasoning itself and, for LSAT and GMAT, at levels exceeding an average person's capabilities. Second is that it supplies everything the model needs and can't be solved based on remembered training data. Even testing the model's grade school science knowledge is just a test of the ability for the model to retain and regurgitate what it was trained on, which is an ability I'm not interested in testing. Third is that I happen to be a natural grandmaster at the GMAT critical reasoning practice questions. For every GMAT prep program, they supply a crappy and convoluted step-by-step process or visualization. All I needed was my simple bridge analogy and every question was about finding, building, or destroying the bridge that holds the argument together.