4 ms·

Pretty insane. I thought AI had a ways to go before it could solve problems like these. https://minerva-demo.github.io/#category=Algebra&index=1 https://minerva

Pretty insane. I thought AI had a ways to go before it could solve problems like these. https://minerva-demo.github.io/#category=Algebra&index=1 https://minerva-demo.github.io/#category=Algebra&index=1

riku_iki 4y ago
So, I look at the internet for some questions, for example: https://minerva-demo.github.io/#category=Algebra&index=3 https://minerva-demo.github.io/#category=Algebra&index=3 And here you go: https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=PA126&dq=1296%3D64%3D2434&source=bl&ots=HCvmdpH8JZ&sig=ACfU3U2s9UaZUJo1B1sB13zbVZHPgxiBhw&hl=en&sa=X&ved=2ahUKEwjcmr2hsNb4AhUYD0QIHWFqAQ4Q6AF6BAgCEAM#v=onepage&q=1296%3D64%3D2434&f=false https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=... So, they may have data leak in training data. I am wondering what would be model answer if they slightly change numbers.
- ramasesh 4y ago
  They have this plot in the paper. See Figure 5. Average accuracy (averaged over problems) does not change.
  - riku_iki 4y ago
    The question listed in paper (they modified 20 out of thousands in MATH dataset) as example is very far in calculating complexity from what I quoted in my comment above. Their question for your convenience: At each basketball practice last week, Jenny made twice as many free throws as she made at the previous practice. At her fifth practice she made 48 free throws. How many free throws did she make at the first practice?
  - riku_iki 4y ago
    > accuracy (averaged over problems My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated average accuracy of this answers), which is not target metric of benchmark. Additionally, accuracy degradation is clearly visible on graphs, first graph has 4 problems out of 20 bellow the line, last graph has 14 problems bellow the line.
    - lewkowycz 4y ago
      The model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie we use majority voting or self-consistency.
      - riku_iki 4y ago
        Target benchmark accuracy: accuracy of "most common answer". Accuracy in Figure 5: average accuracy over 64 answers. This is mathematically different metrics.
- deleted 4y ago
  [deleted]

riku_iki 4y ago

So, I look at the internet for some questions, for example: https://minerva-demo.github.io/#category=Algebra&index=3 https://minerva-demo.github.io/#category=Algebra&index=3 And here you go: https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=PA126&dq=1296%3D64%3D2434&source=bl&ots=HCvmdpH8JZ&sig=ACfU3U2s9UaZUJo1B1sB13zbVZHPgxiBhw&hl=en&sa=X&ved=2ahUKEwjcmr2hsNb4AhUYD0QIHWFqAQ4Q6AF6BAgCEAM#v=onepage&q=1296%3D64%3D2434&f=false https://books.google.com/books?id=QutS5unZDLYC&pg=PA126&lpg=... So, they may have data leak in training data. I am wondering what would be model answer if they slightly change numbers.

ramasesh 4y ago
They have this plot in the paper. See Figure 5. Average accuracy (averaged over problems) does not change.
- riku_iki 4y ago
  The question listed in paper (they modified 20 out of thousands in MATH dataset) as example is very far in calculating complexity from what I quoted in my comment above. Their question for your convenience: At each basketball practice last week, Jenny made twice as many free throws as she made at the previous practice. At her fifth practice she made 48 free throws. How many free throws did she make at the first practice?
- riku_iki 4y ago
  > accuracy (averaged over problems My understanding is that it is average over 64 solution attempts (model generated 64 possible answers, and they calculated average accuracy of this answers), which is not target metric of benchmark. Additionally, accuracy degradation is clearly visible on graphs, first graph has 4 problems out of 20 bellow the line, last graph has 14 problems bellow the line.
  - lewkowycz 4y ago
    The model gets a question and outputs one answer, so this is the target benchmark. to do this, 64 samples are generated and we pick the most common answer, ie we use majority voting or self-consistency.
    - riku_iki 4y ago
      Target benchmark accuracy: accuracy of "most common answer". Accuracy in Figure 5: average accuracy over 64 answers. This is mathematically different metrics.
deleted 4y ago
[deleted]