3 ms·

This is impressive. I wonder how quickly and how well this could be adapted to other domains in science and engineering. It could be a really useful tool for en

by jkic47 5y ago

This is impressive. I wonder how quickly and how well this could be adapted to other domains in science and engineering. It could be a really useful tool for engineering teams

tsimionescu 5y ago
Have you looked at the actual questions or prompts? It's doing only trivial translations of the actual input question (which is often arbitrarily more complex than the original textbook question) into Python code. A lot of the time, they don't even attempt exact solutions, they directly ask for iterative approaches: Original textbook question (number theory course): > The product of the positive integer divisors of a positive integer n is 729. Find n. Actual model input/prompt: > Write a function which finds the divisors (including 1 and itself) of the input and multiplies them together. Iterate over all possible integers until it finds an input where the product is 729. Model output program: def divisors(n): divisors = [] for i in range(1, n+1): if n % i == 0: divisors.append(i) return divisors def product(n): return reduce(lambda x, y: x*y, n) def find_n(n): for i in range(1, n): if product(divisors(i)) == 729: return i print find_n(1000000) Not only is the prompt explaining exactly what to do, it's using an iterative approach instead of an analytical approach, as might be expected from a math class. Note: any errors in the program are my own transcribing error.
- YeGoblynQueenne 5y ago
  As noted in another comment, that is legit program synthesis from a complete specification in natural language. The workflow described in the paper and abstracted in your comment can be very useful, as long as the user can have good confidence in the correctness of the results (or the ability to eyball and correct them as needed). The problem is that this approach relies on a large language model trained on a copy of the entire web (GPT-3, trained on Common Crawl plus network extras), fine-tuned on github (Codex) and then fine-tuned again on the kinds of problems that it is supposed to solve. That is an insane amount of resources to spend to solve simple programming problems with known solutions that can be implemented by hand at much lower cost and effort. In that sense it's a little bit disappointing: so much data, so much compute, and all you can do is tell computer how to code python?
  - Isinlor 5y ago
    > and then fine-tuned again on the kinds of problems that it is supposed to solve Could you point me to where they claim to have fine-tuned Codex? From what I can see they claim: > We use OpenAI’s davinci-codex engine for all of our generations. We fix all of Codex’s hyperparameters to be the same for all experiments: top-p which is the portion p of the token probability mass a language model samples from at each step is set to 1, sampling temperature is set to 0 (i.e. argmax), and response length is set to 200 tokens. BTW - OpenAI Davinci costs $0.06 per 1000 tokens. Codex is currently free in closed beta, but I guess cost will be the same. I would be happy to pay even 100x ($6) for correct solutions to advanced mathematical problems. The issue is that this paper has ridiculous evaluation and Codex does not work anywhere near as good as they claim. It is pure hype by people who do not appear to be affiliated with OpenAI.
    - YeGoblynQueenne 5y ago
      >> Could you point me to where they claim to have fine-tuned Codex? I can't find that in the paper so I must have imagined it. >> Codex is currently free in closed beta, but I guess cost will be the same. In my previous comment, I'm referring to the "amount of resources" needed to train GPT-3 and then fine-tune Codex, of course.
  - tsimionescu 5y ago
    This is legit program synthesis for an iterative brute force solution to the problem. But it is in no sense "solving university level mathematical problems". It's not even figuring out that it can brute force it based on the original problem - a human looked at the original problem and told the model how to brute force it, and it did. It's cool that it did, but this achievement has nothing to do with the title and abstract of the paper.
    - YeGoblynQueenne 5y ago
      I agree. The paper exaggerates. I was commenting on the workflow you described.

by jkic47 5y ago

This is impressive. I wonder how quickly and how well this could be adapted to other domains in science and engineering. It could be a really useful tool for engineering teams

tsimionescu 5y ago

Have you looked at the actual questions or prompts? It's doing only trivial translations of the actual input question (which is often arbitrarily more complex than the original textbook question) into Python code. A lot of the time, they don't even attempt exact solutions, they directly ask for iterative approaches: Original textbook question (number theory course): > The product of the positive integer divisors of a positive integer n is 729. Find n. Actual model input/prompt: > Write a function which finds the divisors (including 1 and itself) of the input and multiplies them together. Iterate over all possible integers until it finds an input where the product is 729. Model output program: def divisors(n): divisors = [] for i in range(1, n+1): if n % i == 0: divisors.append(i) return divisors def product(n): return reduce(lambda x, y: x*y, n) def find_n(n): for i in range(1, n): if product(divisors(i)) == 729: return i print find_n(1000000) Not only is the prompt explaining exactly what to do, it's using an iterative approach instead of an analytical approach, as might be expected from a math class. Note: any errors in the program are my own transcribing error.

YeGoblynQueenne 5y ago
As noted in another comment, that is legit program synthesis from a complete specification in natural language. The workflow described in the paper and abstracted in your comment can be very useful, as long as the user can have good confidence in the correctness of the results (or the ability to eyball and correct them as needed). The problem is that this approach relies on a large language model trained on a copy of the entire web (GPT-3, trained on Common Crawl plus network extras), fine-tuned on github (Codex) and then fine-tuned again on the kinds of problems that it is supposed to solve. That is an insane amount of resources to spend to solve simple programming problems with known solutions that can be implemented by hand at much lower cost and effort. In that sense it's a little bit disappointing: so much data, so much compute, and all you can do is tell computer how to code python?
- Isinlor 5y ago
  > and then fine-tuned again on the kinds of problems that it is supposed to solve Could you point me to where they claim to have fine-tuned Codex? From what I can see they claim: > We use OpenAI’s davinci-codex engine for all of our generations. We fix all of Codex’s hyperparameters to be the same for all experiments: top-p which is the portion p of the token probability mass a language model samples from at each step is set to 1, sampling temperature is set to 0 (i.e. argmax), and response length is set to 200 tokens. BTW - OpenAI Davinci costs $0.06 per 1000 tokens. Codex is currently free in closed beta, but I guess cost will be the same. I would be happy to pay even 100x ($6) for correct solutions to advanced mathematical problems. The issue is that this paper has ridiculous evaluation and Codex does not work anywhere near as good as they claim. It is pure hype by people who do not appear to be affiliated with OpenAI.
  - YeGoblynQueenne 5y ago
    >> Could you point me to where they claim to have fine-tuned Codex? I can't find that in the paper so I must have imagined it. >> Codex is currently free in closed beta, but I guess cost will be the same. In my previous comment, I'm referring to the "amount of resources" needed to train GPT-3 and then fine-tune Codex, of course.
- tsimionescu 5y ago
  This is legit program synthesis for an iterative brute force solution to the problem. But it is in no sense "solving university level mathematical problems". It's not even figuring out that it can brute force it based on the original problem - a human looked at the original problem and told the model how to brute force it, and it did. It's cool that it did, but this achievement has nothing to do with the title and abstract of the paper.
  - YeGoblynQueenne 5y ago
    I agree. The paper exaggerates. I was commenting on the workflow you described.