4 ms·

> Find the least odd prime factor of 2019^8+1 God that's absurd. The mathematical skills involved on that reasoning are very advanced; the whole process is a b

> Find the least odd prime factor of 2019^8+1 God that's absurd. The mathematical skills involved on that reasoning are very advanced; the whole process is a bit long but that's impressive for a model that can potentially be self-hosted.

pitpatagain 2y ago
Also probably in the training data: https://www.quora.com/What-is-the-least-odd-prime-factor-of-2019-8-1-1 https://www.quora.com/What-is-the-least-odd-prime-factor-of-... It's a public AIME problem from 2019.
- dartos 2y ago
  People have to realize that many problems that are hard for humans are in a dataset somewhere.
  - zamadatix 2y ago
    In a twofold way: 1) Don't bother testing it with reasoning problems with an example you pulled from a public data set 2) Search the problem you think is novel and see if you already get an answered match in seconds instead of waiting up to minutes for an LLM to attempt to reproduce it. There is an in-between measure of usefulness which is to take a problem you know is in the dataset and modify it to values not in the dataset on measure how often it is able to accurately adapt to the right values in its response directly. This is less a test of reasoning strength and more a test of whether or not a given model is more useful than searching its data set.
    - dartos 2y ago
      I’d be really interested in that kind of study. My intuition has slowly been building towards thinking of LLMs (and potentially all generative models) are essentially search algorithms
gowld 2y ago
The process is only long because it babbled several useless ideas (direct factoring, direct exponentiating, Sophie Germain) before (and in the middle of) the short correct process.
- Vetch 2y ago
  I think it's exploring in-context. Bringing up related ideas and not getting confused by them is pivotal to these models eventually being able to contribute as productive reasoners. These traces will be immediately helpful in a real world iterative loop where you don't already know the answers or how to correctly phrase the questions.
  - int_19h 2y ago
    This model seems to be really good at this. It's decently smart for an LM this size, but more importantly, it can reliably catch its own bullshit and course-correct. And it keeps hammering at the problem until it actually has a working solution even if it takes many tries. It's like a not particularly bright but very persistent intern. Which, honestly, is probably what we want these models to be.
sebastiennight 2y ago
Wait I didn't read the source of that quote, but is the answer "2"? Since (2019^8 (an odd number) plus one) is an even number, it only has one prime factor which isn't odd : 2.

pitpatagain 2y ago

Also probably in the training data: https://www.quora.com/What-is-the-least-odd-prime-factor-of-2019-8-1-1 https://www.quora.com/What-is-the-least-odd-prime-factor-of-... It's a public AIME problem from 2019.

dartos 2y ago
People have to realize that many problems that are hard for humans are in a dataset somewhere.
- zamadatix 2y ago
  In a twofold way: 1) Don't bother testing it with reasoning problems with an example you pulled from a public data set 2) Search the problem you think is novel and see if you already get an answered match in seconds instead of waiting up to minutes for an LLM to attempt to reproduce it. There is an in-between measure of usefulness which is to take a problem you know is in the dataset and modify it to values not in the dataset on measure how often it is able to accurately adapt to the right values in its response directly. This is less a test of reasoning strength and more a test of whether or not a given model is more useful than searching its data set.
  - dartos 2y ago
    I’d be really interested in that kind of study. My intuition has slowly been building towards thinking of LLMs (and potentially all generative models) are essentially search algorithms

gowld 2y ago

The process is only long because it babbled several useless ideas (direct factoring, direct exponentiating, Sophie Germain) before (and in the middle of) the short correct process.

Vetch 2y ago
I think it's exploring in-context. Bringing up related ideas and not getting confused by them is pivotal to these models eventually being able to contribute as productive reasoners. These traces will be immediately helpful in a real world iterative loop where you don't already know the answers or how to correctly phrase the questions.
- int_19h 2y ago
  This model seems to be really good at this. It's decently smart for an LM this size, but more importantly, it can reliably catch its own bullshit and course-correct. And it keeps hammering at the problem until it actually has a working solution even if it takes many tries. It's like a not particularly bright but very persistent intern. Which, honestly, is probably what we want these models to be.

sebastiennight 2y ago

Wait I didn't read the source of that quote, but is the answer "2"? Since (2019^8 (an odd number) plus one) is an even number, it only has one prime factor which isn't odd : 2.